Robots.txt

[netsavy006]

How to I make a Robots.txt file that allows all search engines to index the site?

 

[Nathan]

http://www.robotstxt.org/ :great:

 

[Doctor Tech]

For webmasters who have done this:

1. Is there a change in the amount of traffic?
2. What is a rough estimate of how much your traffic increased by?

I just want to get a general idea of how effective this method is.

 

[Joel]

Haha, i dont even know what robot txts are.

 

[Nathan]

For webmasters who have done this:

1. Is there a change in the amount of traffic?
2. What is a rough estimate of how much your traffic increased by?

I just want to get a general idea of how effective this method is.

All this does is allow you to control whether or not bots index your site/forum or not, it's always good to allow bots such as googlebot to index your site/forum etc.

 

[Doctor Tech]

I understand, but don't search engine bots automatically index your site even without the Robots.txt?

 

[Nathan]

Yeah most of the time, although sometimes they don't (it's best to submit your URL yourself) however as I said having a robots.txt allows you to control whether or not they index your site, so you can stop bots from indexing your site which is useful for the spamming bots/foreign bots.

 

[Hitakashi]

It's also good if you want to block a bot that is just going around your site to take up bandwidth

 

[Wrighty]

Bots will find your site eventually and index it eventually. Use the robots.txt file if there is an area that you don't want indexed. 🙂

Ie.a member only area.