Robot.txt files

[Edd Bundy]

Does anyone know if you can edit a robot.txt file for a free forum as such as invision or forumotion? (and if so how?)
As i'm setting up a new forum (to test things on) which in turn could really help.
Anyway thanks for the replies to anyone that can help me.

 

[Null-11]

For free forum's I don't think you can edit them. I used Forumotion for a while and I never was able to edit it.

 

[FWhite]

Personally, don't bother with it. Most major search engines disregard the robots.txt file and don't follow the rules of it, and it can present a major security risk to hackers.

 

[CyberFreak]

Not with a free host no... Search engines do follow it (some better then others though) although bad bots don't.

Also if you do have one, be careful because as dWhite said it can cause security issues as if you store db backups for example in /abc directory for example, don't add that to the disallow bit of the robots.txt file. People will see that and think what is this directory for and then look at i and find all your backups or any other files you have stored there. Only use it to block bots to files and directories that users and bots would know about not hidden ones that only you know about otherwise you are telling the world about it.

 

[FWhite]

Not with a free host no... Search engines do follow it (some better then others though) although bad bots don't.

My experience with robots.txt has issued me favor of not using it anymore. Google and Yahoo both disobeyed it and still indexed a portion of my site that I had under Disallow rules.

 

[CyberFreak]

Google i find follows it really well... Yahoo does but it seems to take them months to start to follow it though.

 

[Watch Me Sink]

I always store my backups on my local HD rather than the server.
Its good if the server goes down....forever -🙁

 

[CyberFreak]

I only keep file backups on the server for a short while when making upgrades but i never keep any db backups on the server just incase they get found.

 

[Watch Me Sink]

I only keep file backups on the server for a short while when making upgrades but i never keep any db backups on the server just incase they get found.

If people found a DB backup of here, im sure it would be on DigitalPoint for a few hundred bucks.

 

[CyberFreak]

Exactly... file backups don't really contain anything bad apart from then db config file which could be just as serious.

 

[HB.]

Personally, don't bother with it. Most major search engines disregard the robots.txt file and don't follow the rules of it, and it can present a major security risk to hackers.

I don't know how you concluded your findings but I totally disagree from my own personal experience plus the people I host. Ever since I adjusted my robots.txt, all the major bots followed it. This in return equaled less bandwidth being consumed, and useless pages being indexed. Also, I would highly recommend not to store a backup in your allocated webspace. Either purchase a backup server if your website is really active or backup your information to your local hard drive.

To answer the original question, as already mentioned in the topic, the answer is NO.

 

[Raymond]

Yes, I bought a only tower computer to keep all of my website info.