Become.com bots

greg · 03-11-2005, 10:08 AM

Has anyone else had a problem with become.com bots hammering their site? I took a look at their website and it is aparently some sort of product search engine like foogle. They have been nailing me lately though, browsing every page on my site at least 4-5 times a day. I am thinking about blocking them. I was just wondering if anyone else has gone through this.

gorillaboy · 03-11-2005, 10:23 AM

They seem to behave nicely on my site. They visit each day but not multiple times.

Personally, I never block bots that might end up sending me good traffic.

greg · 03-11-2005, 10:30 AM

well, i have a rule about not blocking bots either, unless i find out they are spam bots, which i need to go through again and block them since i changed host because i am getting those again.

But with become.com it is a product website and clearly i am not selling any products on there so i will likely return few results if any.

gorillaboy · 03-11-2005, 10:38 AM

I'd block it then!

Let me see your list of blocked bots.

Here's mine (I know I'm missing a bunch):

CherryPicker
CopyRightCheck
dumbot
EmailCollector
EmailSiphon
EmailWolf
ExtractorPro
JoeDog
Missigua Locator 1.9
NICErsPRO
Ocelli
SiteSnagger
SurveyBot
Teleport
Uptimebot
WebBandit
Website Downloader http:
WebStripper

greg · 03-11-2005, 10:50 AM

i don't remember them anymore honestly. When i changed host i just redid the whole site so I need to start over. The old one i had used b2 blog and spam bots would hit the comments like crazy. i would get over 300 comments sometimes in a day. So i blocked all them. Now they still keep hitting the site.

hatchet · 03-11-2005, 11:28 PM

Yeah become and grub-client has been hitting me pretty hard - mostly grub-client though.. and apparently they don't know what robots.txt is..

greg · 03-13-2005, 11:40 AM

most of the non-mainstream bots ignore robots.txt, you just usually have to IP ban them

gorillaboy · 03-13-2005, 11:41 AM

Add "Ocelli" to my list of banned bots. WHAT A HOG!

DOS · 03-13-2005, 03:16 PM

I am getting a ton of become.com visits myself. Kinda sucks because it doesn't seem like it will be of any benefit to me.

pcarlow · 03-13-2005, 05:00 PM

Unless I'm paying for extra bandwidth or something I could care less what bots come to my sites. They usually don't use enought bandwidth to make it worth my time to fool with.