Webmaster Forum - View Single Post

I, Brian · 12-15-2003, 02:16 PM

I wonder who could have been behind it?

There's actually a good list of bots to avoid in the robots.txt file of the site linked top of the page - lots of unpleasant ones singled out. May be worth copy/pasting something of that into my own robots.txt files.

12-15-2003, 02:16 PM	#6 (permalink)
I, Brian Inactive Join Date: 10-26-03 Posts: 2,466 iTrader: 0 / 0% Latest Blog: None	I wonder who could have been behind it? There's actually a good list of bots to avoid in the robots.txt file of the site linked top of the page - lots of unpleasant ones singled out. May be worth copy/pasting something of that into my own robots.txt files.