Blocking Search Engines
- Started
- Last post
- 7 Responses
- paulsmith
OK,
Here is my problem. A client wants part of their site blocking from search engines such as google etc but wants other area's available to search engines.
I have only used ROBOT.txt files in a very simple way until now. Does anybody know the line to exclude specific directories ? Thanks in Advance.
- jox0
the meta tag "robots=none"
or was it "no"..? Not quite sure.
- 4cY0
- paulsmith0
Thanks chaps.
- Bertrand0
You can put the instruction in your robots.txt file:
User-agent: *
Disallow: /foldername
- unknown0
is the part he wants blocking called 'personal pics' or 'last weeks key party'????
- JazX0
robots suck
- paulsmith0
LOL. No it is a staff area for a school website.
It is password protected but to avoid confusion, he doesn't want google or other robots to pick it up and link to it.
I think most Robots suck, some are ok but most just mess up your server logs with junk. ;-(