This is the report for our website, for the 5 days of January so far.
Last visit
9 different robots
Yahoo Slurp
229
Googlebot
92
Unknown robot (identified by 'bot/' or 'bot-')
7
Ask
7
MSNBot
14
Yahoo! Slurp China
1
MSNBot-media
187
Unknown robot (identified by 'crawl')
14
Unknown robot (identified by 'spider')
1
Yahoo absolutely loves our site, as you can see, and we rank very highly for a good number of competitive key phrases.
We also do reasonably well on Google too.
I would say the following things are crucial to attracting regular crawls.
- Build a Google Site Map [ and ensure you get it verified by google]
Have plenty of relevent, quality content. [ if you havent, why expect anyone to be interested in your site? ]
Add new content reasonably regularly
Add new pages
ensure your page file names are relevent