Robot.txt

[Amy Avery]

What is Robot.txt? How to use.

 

[Sam Fischer]

With this attribute, you're telling the search engines what to do with your pages:

index/noindex - This tells the engines whether to show your page in search results, or not.
follow/nofollow - This tells the engines what to do with links on your pages: whether they should trust and "follow" your links to the next page or not.

 

[RH-Calvin]

Robots.txt helps to denote pages on your website which you do not want search engines to crawl. These pages are visible to your website readers, but they will never be crawled and indexed by search engines.