Robot Txt files
Robot Txt files
Out of context: Reply #1
- Started
- Last post
- 2 Responses
- Weyland0
decide what you want google to crawl and block all the rest?
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/etc
decide what you want google to crawl and block all the rest?
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
etc