Robots.txt help

[Yehya]

Is it a good idea to exclude background images?

Edit: is it a good idea to block .htaccess using robots.txt?

 

[legalize]

htaccess can't be accessed, so disallowing it from being crawled wouldn't make a difference.

I would disallow directories containing design files, no need for them showing up on Google Image search 😛

 

[Yehya]

Thank you.

 

[Yehya]

Different help now.
I want to see which links and directories it is blocking. I want to literally see which links.
I know i can edit the file and look at the blocks, but i want to check if i have any typos, etc.

 

[legalize]

Make sure there are no typo-s yourself 😉

 

[Yehya]

That's not the biggest issue. I want to see if it is working correctly. I have my robots.txt in a subfolder, in my root not a checker or google noticed it for days.

I copy/pasted it to my subfolder and it picked it up instantly. So i want to know whether i should add my subfolder directory or not. Since it is already in it.

But i guess i have to have the sub-directory name written. True?



Even if it is buried 12 directories long, for example: /directory1/files/php/nonsense/stupid/robots.txt
I would have to include /directory1/files/php/nonsense/stupid/ before blocking any file in that directory right?

 

[legalize]

the robots.txt should be in your root directory, the spiders check www.yoursite.com/robots.txt to see if one exists.

As for sensitive directories, I wonder if it's possible to restrict access via permission rather than storing it in the robots.txt file, haven't tried that but it's an idea.

 

[Yehya]

Thanks for your help.