Whitehouse.gov
Whitehouse.gov
Out of context: Reply #4
- Started
- Last post
- 7 Responses
- emecks0
robots.txt is basically to tell the spiders which pages they can or cannot go to. It is used to stop the spiders trying to access areas that you don't want indexed OR parts of the site that may result in the spider "looping" which they apparently don't much like :)