I'm not an expert on robots.txt either, but I was under the impression that you needed a User-agent line for it to work, so try this
Quote:
User-agent: *
Disallow: /post-*.html
Disallow: /updates-topic.html*
Disallow: /stop-updates-topic.html*
Disallow: /ptopic*.html
Disallow: /ntopic*.html
|
Also, not all robots understand the wildcards yet, and some completely ignor the robots.txt, so if you are having trouble with a particular crawler, you may need to add differant sections for each one, or even ban there IP blocks (usually required for the email harvestors etc).
Coop
