Robot Txt files
Robot Txt files
- Started
- Last post
- 2 Responses
- Weyland0
decide what you want google to crawl and block all the rest?
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/etc
- Weyland0
PS: if you have subdomains like mobile.yoururl.com or rss.yoururl.com that don't resolve to the same web root put a robots.txt file in each subdomain's web root folder aswell!
- i do have subdomains... i'll upload those files there as well. super helpful Weyland.ohhhhhsnap