How Many Of These Spiders Are Crawling Your Site?

[peter_d]

Some are worthwhile. Many are just bandwidth suckers.

Hat-Tip: http://www.skrenta.com/2007/08/the_1...lly_crawl.html


1612960 Yahoo! Slurp help.yahoo.com
365308 msnbot search.msn.com/msnbot.htm
148090 Googlebot www.google.com/bot.html
140120 VoilaBot www.voila.com
68829 Ask Jeeves/Teoma about.ask.com
62005 psbot www.picsearch.com/bot.html
39193 BecomeBot www.become.com/site_owners.html
30006 WebVac www.WebVac.org edu
29778 ShopWiki www.shopwiki.com/wiki/Help:Bot shopping
22124 noxtrumbot www.noxtrum.com
20963 Twiceler www.cuill.com/twiceler/robot.html startup
17113 MJ12bot majestic12.co.uk/bot.php
15650 Gigabot www.gigablast.com/spider.html
10404 ia_archiver www.archive.org
9337 Seekbot www.seekbot.net/bot.html
9152 genieBot www.genieknows.com
7246 FAST MetaWeb www.fastsearch.com enterprise
7243 worio bot worio.com
6868 CazoodleBot www.cazoodle.com
6608 ConveraCrawler www.authoritativeweb.com/crawl enterprise
6293 IRLbot irl.cs.tamu.edu/crawler
5487 Exabot www.exabot.com/go/robot
4215 ilial www.ilial.com/crawler
3991 SBIder www.sitesell.com/sbider.html
3673 boitho-dcbot www.boitho.com/dcbot.html enterprise
3601 accelobot www.accelobot.com memetracker
2878 Accoona-AI-Agent www.accoona.com
2521 Factbot www.factbites.com
2054 heritrix i.stanford.edu
2003 Findexa www.findexa.no
1760 appie www.walhello.com
1678 envolk www.envolk.com
1464 ichiro help.goo.ne.jp/door/crawler.html
1165 IDBot www.id-search.org/bot.html
1161 Sogou www.sogou.com/docs/help
1029 Speedy Spider www.entireweb.com

[mac1234]

well described will search for who all have visited my site