Forum robots.txt File

[RedmondG]

Hey FP,

We've all be there when there are so many guests on your forum you start to question what's going on. Of course, most of them are bots and some harmful - mainly just looking for email addresses to fire spam at.

Came across this beast of a robots.txt file and thought I would share. It's not made by me, all credit goes to mitchellkrogza.

The robots.txt file can be found at https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/robots.txt/robots.txt

 

[Carlos X]

It's a non-issue. All it does is blacklists or whitelists pages for google to slurp, and search.

 

[Deleted member 4320]

It can be useful to hide certain sections you don’t or do want crawled. It’s a handy thing to configure. Plus it’s not just Google that uses Robots.txt. Other search engines are available.