Google Crawl How Often?

[dsarathy]

Initially the crawlers visit your site through quality external links pointing to your site. Then the subsequent visit is dependent on the new content addition to your site periodically. The crawler schedules your site to be crawled after some time (say x number of days). The crawler visits your site after x days and find whether there is any change in the content. If there is a change, it schedules the next crawling to half of the initial time (i.e. x/2 days). If a site is successfully updating content effectively everyday, then they have all chances to make their site crawlable everyday. Blogging is one of the activities, which aids in achieving this goal.

D Sarathy.
RankQuest

[Thomas Schulz]

Originally Posted by TheBrain

My situation is very weird.
I have several sites and most of them are visited once in a while by Googlebot, but there is one, which I don't want indexed, where Google is there all day and night, ignoring robots.txt, metatags and all the unimaginable tricks I'm using to kick it off.

Are you sure it is GoogleBot? Have you checked the IP?
User agent IDs are very easy to fake.

[chillfire]

Originally Posted by mazter

You can see actual 'crawl rate' from Google Webmaster tools.
https://www.google.com/webmasters/tools/

According to that tool, at least 150 pages are crawled daily.

I think google sees a 'crawl' differnetly to I think it is, my definition (oh I am sure I will get flamed for this);

one crawl is a single session of spidering from a single spider/bot

In the sitemap tools, the crawl rate is really small (5 or so) but in my tracking system (reads sessions, IP etc ) It is really high, still upwards of 300 hits (approx 60-100 crawls each with different sessions, sometimes differnet IPs?)

So I think any numbers they have to be taken with a grain of google salt.

Just signed up for the microsoft.co.uk officelive beta (giving away free .co.uk domains eith the system) and they have launched their own site reporting, it includes spider activity for all spiders.... one thing missing from the google tools. Seems like it might just be good as well.

the admin tools are fairly simplistic, but good for none designers/developers, see www.chillfireltd.co.uk for a laugh, would be interested to see the traffic reports spike ha ha.

[Thomas Schulz]

Originally Posted by TE75

Some of my sites get crawled at least once a day, others may get crawled every 3 - 4 days. I have noticed that once I submit a sitemap, my time between crawls usually increases after a week or so.

This will be a pure guess from my side. But if you use LastMod in your <forum rules> file, I guess Google may decide not to crawl as much...? Although Google has AFAIK stated that sitemaps do not influcence their normal crawling.

[agile]

last time google crawled my website was 1 month ago(my home page) but it crawl regularly my innerpages

[potchie]

about my site google crawled it everyday but about msn and yahoo i dont know, maybe they change the algo.

[coolguy27]

Originally Posted by intruth

It seems like my site gets crawled by google every 3-4 days but yahoo and msn crawl it daily...

How often does Google crawl your site?

Usually 3 to 4 days.... And it depends on how you optimized your page often.

[sirkakashi08]

i think twice a week!

[hawashp]

how could i know that

[1more]

Out of curiosity, what are you doing to see how many times google has crawled your site?

Googlebot 15741 601.62 MB 31 May 2007 - 06:31
MSNBot 502 13.68 MB 31 May 2007 - 06:10
Yahoo Slurp 497 14.09 MB 31 May 2007 - 00:39

Those are the stats I got as of earlier today using Awstats via my cpanel.

How accurate are those numbers? If they are indeed what you guys are talking about, those three listed above run my site daily.

[viswablr]

I think any site that has forum, RSS feeds or anything else that google believes are indicators are periodical content changes, such sites are crawled quite often, than a purely static website.

[temyong]

It depends on how you update your site sometimes 3-5 days or 7 Google crawls