New whitehouse.gov
Out of context: Reply #46
- Started
- Last post
- 50 Responses
- TheBIueOne0
via:
http://www.kottke.org/09/01/the-…"Here's the robots.txt file from whitehouse.gov yesterday:
User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/textAnd it goes on like that for almost 2400 lines! Here's the new Obamafied robots.txt file:
User-agent: *
Disallow: /includes/That's it! BTW, the robots.txt file tells search engines what to include and not include in their indexes. "