Using robots.txt - Great for SEO

[Kaiser.]

Do you use a robots.txt file on your website?

Basically what it does is it disallows bots from visiting certain places that you put in the robots.txt file, if you use Google's Webmaster tools you can see the places where you get crawl errors, and you can put those places in your robots.txt to prevent Google from crawling it again. Its great for better SEO results.

Here is AdminBB's robots.txt http://adminbb.org/robots.txt

 

[el canadiano]

Here's mine.

http://www.mk3dsforum.com/robots.txt

I have a modified version from the XenForo community.

 

[Kaiser.]

Here's mine.

http://www.mk3dsforum.com/robots.txt

I have a modified version from the XenForo community.

Yea I am using the one posted in that thread over at XenForo with some edits.

 

[el canadiano]

I still don't get why /posts/ is disallowed, though. As far as I know, posts/XXXX/ redirects to the corresponding thread.

 

[Kaiser.]

I still don't get why /posts/ is disallowed, though. As far as I know, posts/XXXX/ redirects to the corresponding thread.

I was thinking that myself, but I have alot of crawl errors with /posts/ so I just disallowed it..