Whitehouse.gov
- Started
- Last post
- 7 Responses
- monoboy
Not sure if this should be viewable...
http://www.whitehouse.gov/robots…
- mikeim0
Is there a way to hide "robots.txt" files? It does seems pointless to have an easily accessible file telling people where they shouldn't look (oh, and search engines).
- jamble0
I don't think you can hide robots.txt as it needs to be called that so search engines can read it and know what they're supposed to not index although I have my doubts as to how many respect it anyway.
If you've got folders on your server you don't want people/bots to see, you're probably better off ditching robots.txt anyway and just protecting them with a .htaccess file/password.
There's plenty of info via google about setting it up and this is a good starter.
- emecks0
robots.txt is basically to tell the spiders which pages they can or cannot go to. It is used to stop the spiders trying to access areas that you don't want indexed OR parts of the site that may result in the spider "looping" which they apparently don't much like :)
- menos0
hmmm this is very interesting... so if you don't want files or pages indexed by search engines you just use a robots.txt file? but surely anyone can try and find these txt files and work their way around, no?
- It's not really the most foolproof way of hiding content you don't want indexing by search engines
jamble
- It's not really the most foolproof way of hiding content you don't want indexing by search engines
- harlequino0
On the converse side, does using a robot.txt file in any aid in getting content indexed better in addition to metatags and keywords in your copy? Or are they unrelated?
- slinky0
i used to work at WhiteHouse.gov (not .com of course)