What is robots.txt?

[twister]

What is robots.txt?<br /><br />-- 14 Apr 2014, 16:27 --<br /><br />The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable

 

[Amy Avery]

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

 

[RH-Calvin]

Robots.txt is a file which defines webpages of your website which you want to hide from search engine bots. The pages listed in robots.txt are never crawled by search engine spiders, but are visible to your website visitors.

 

[twister]

thanx guys, giving your views regarding robot.txt.