Computer Geeks

Computer Geeks

Geek Shop

Geek News

Geek Stuff

Science Geek

Computer Gaming

Linux Chat

Building Websites

Computer Forums

Computer Help Forum

Computer Hardware Forum

Computer Software Programs


Go Back   Computer Forums > Geek Stuff
FAQ Community Calendar Today's Posts Search

Geek Stuff General forum for discussing anything geek. If your topic doesn't fit in any of the other forums then post it here.

Computer Geeks
» Active Discussions
Computer Geeks
No Threads to Display.
» Other Websites
- Software Publishing

- Server Hardening
 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #4  
Old 04-21-2006, 06:53 AM
Coop Coop is offline
Member
GB Beginner
 
Join Date: Apr 2006
Posts: 34
Default

I'm not an expert on robots.txt either, but I was under the impression that you needed a User-agent line for it to work, so try this

Quote:
User-agent: *

Disallow: /post-*.html
Disallow: /updates-topic.html*
Disallow: /stop-updates-topic.html*
Disallow: /ptopic*.html
Disallow: /ntopic*.html
Also, not all robots understand the wildcards yet, and some completely ignor the robots.txt, so if you are having trouble with a particular crawler, you may need to add differant sections for each one, or even ban there IP blocks (usually required for the email harvestors etc).

Coop
__________________
Coop
Reply With Quote
 


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Powered by vBadvanced CMPS v3.2.3

All times are GMT -5. The time now is 07:50 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
HTML Help provided by HTML Help Central.