robots.txt?

[thalada]

What does your robots.txt looks like?

I personally don't have one. I know it's helpful to keep away the bad bots & saving the bw, i think it's unnecessary for new forums.

 

[Tucker933]

It actually doesn't block bad bots, or anything at all for that matter. It's just a note to crawlers for what items or directories you don't want indexed.

 

[Page]

All mine says to the link of my Sitemap

 

[RH-Calvin]

Robots.txt is a good file to have on your website. It helps to prevent bots to crawl certain webpages which you want to restrict from search engines.

 

[thalada]

It actually doesn't block bad bots, or anything at all for that matter. It's just a note to crawlers for what items or directories you don't want indexed.

Yes, yes you are right.

I meant to say the main purpose of having the robots.txt is to keep away the bad bots & save resources.

 

[Tucker933]

It actually doesn't block bad bots, or anything at all for that matter. It's just a note to crawlers for what items or directories you don't want indexed.

I meant to say the main purpose of having the robots.txt is to keep away the bad bots & save resources.

Lol what I'm saying though is it will never block bad bots. The robots.txt file only works on bots that are specifically instructed by their creators to follow them, which is almost exclusively only crawlers that search engines use, with the few exceptions only being other "good" bots.

 

[thalada]

It actually doesn't block bad bots, or anything at all for that matter. It's just a note to crawlers for what items or directories you don't want indexed.

I meant to say the main purpose of having the robots.txt is to keep away the bad bots & save resources.

Lol what I'm saying though is it will never block bad bots. The robots.txt file only works on bots that are specifically instructed by their creators to follow them, which is almost exclusively only crawlers that search engines use, with the few exceptions only being other "good" bots.

Right, even the good bots can be resource intensive.

 

[Page]

Helpful tip add

# Allow crawling of all content
User-agent: *
Disallow:

 

[Tucker933]

Right, even the good bots can be resource intensive.

That's not at all what I was addressing, if you'll read the post you quoted.

 

[david hong]

Robots.txt file is used to guide the search engine automatically to the page you want it to search for and then index the page. Most sites also have the directories and files do not need to search engine robots to visit

 

[daica85]

All you guys can visit here for more detail: https://moz.com/learn/seo/robotstxt

Good luck to you,

 

[Page]

I know what Robots does, Just impressed that there are allot of websites that don't have that file

 

[Kanak Trades]

Robots.txt is using for security purpose. If you cannot index any file in search engine so using robot.txt

 

[StevenTr]

Robots.txt is used to when you want to prevent Googlebot from going to some area of the website, it saves resources