How do I know when the Google bot crawls my site?

davelev · Share | #1 (**permalink**) 11-22-2011, 01:40 AM

Hello,

I was wondering how it would be possible to detect when Google periodically craws my site thoroughly... I mean when the 'Google dance' comes, there are some intensive crawls, as far as I know from my colleagues.

Is there a script, program that shows us when it happens?

AWSTATS and Google Analytics don't show it.

Steve_gts · #2 (**permalink**) 11-22-2011, 02:56 AM

Probably the easiest way is to use the seobook toolbar, just install it, then click the little arrow next to PR and it will show you when a page was cached, then click on it and it will show you the cached page. It has a couple of other handy tools on there too.

yokowasis · #3 (**permalink**) 11-22-2011, 03:02 AM

I don't really know about Google Analytics. But I am sure anything crawl your website will be recorded by the website who provide live web stats tracking. And it is marked as "bot". I know because I have seen it on my website.

towelk · #4 (**permalink**) 11-23-2011, 07:10 AM

The easiest way is to regularly visit your Google Webmasters Tool account, if you don't have one, yet....create and set it up.

There, you will find as to when Google bog last crawled your site. You can even schedule it as to how many times you want your website to be crawled. But, for me...the recommended setting works fine.

C.Rebecca · #5 (**permalink**) 11-25-2011, 12:57 AM

Two ways to check bots visit on your website,
- Check the last caching date of your website
- Check the bot's impression on your robots.txt files

icecube_media · #6 (**permalink**) 11-25-2011, 01:59 AM

there is one tool name selfseo one can try that.

brijesh · #7 (**permalink**) 11-25-2011, 09:32 AM

that time is not fixed by any of toolbar but as far as my concern about that when you change something in your web page then google crawl can crawl your site and you can know it by google pagerank cheker tool which can install in our web browser and move cursor over their and see cache page option it will show that when your page has been crawled.

amjith66 · #8 (**permalink**) 11-29-2011, 03:16 AM

Just check your site cache date it will shows whether Google crawls or not..

~kev~ · #9 (**permalink**) 11-29-2011, 06:05 AM

Check your google webmaster tools, I think there is a place in there that says the last time google did a crawl.

A couple of things though:

New sites - google might crawl your site every 2 weeks or so.

Older mature sites that are over 1 year old - google crawls the site on a full time basis.

Site that is several years old - google stays on the site all the time looking for new content.

KRex · #10 (**permalink**) 11-30-2011, 06:53 PM

I use Google WebMaster Tools for this, it shows last 90 days activity of GoogleBots. You will get the following Statistics for 90 days.
1. Pages Crawled per day.
2. Kilobytes downloaded per day.
3. Time spent downloading a page.
With High Low and Average values.

Crawls error shows when GoogleBot unable to crawl your website or when they find some errors.

You can also use Fetch as GoogleBot option to see how your pages appears to Google. There you also have an option to submit it to index, do it only when your pages are new or you recently did some changes.

davelev · #11 (**permalink**) 12-07-2011, 12:36 PM

Thanks for tips everyone, I will check all these and get back a.s.a.p.

webnewbie · #12 (**permalink**) 12-08-2011, 08:39 AM

Same here! But this shows the snapshot of my site 5 hours ago.Is it the time delay for Google to index my site?

jeresteem99 · #13 (**permalink**) 12-09-2011, 12:32 AM

use of lot of FREE tools to check it.

Web master tool is one of them.

Ayansarei · #14 (**permalink**) 12-13-2011, 02:29 AM

you can check by google manually through:
cache:domain write in google search box.

Alexander P. · #15 (**permalink**) 12-13-2011, 03:23 AM

Other than using seo tools, I do search my site pages or whatever site I want to check and then check the cache on the google search results and it will appear on the upper right of the page when your website or page has been last crawled by google bots

archidesigngroup · #16 (**permalink**) 12-15-2011, 11:39 PM

just check the Google's cache date to get the last visiting details for google bot..

bbcsite · #17 (**permalink**) 12-16-2011, 12:00 AM

are you using PHP based cms? if yes the this tutorial might be helpful for you to know when googlebot access your site: googlebot access

Dirky47 · #18 (**permalink**) 12-16-2011, 12:07 AM

Quote:

Originally Posted by Steve_gts

Probably the easiest way is to use the seobook toolbar, just install it, then click the little arrow next to PR and it will show you when a page was cached, then click on it and it will show you the cached page. It has a couple of other handy tools on there too.

I will do what you said. May I know what's the SEO tool?

Jashlee · #19 (**permalink**) 12-16-2011, 12:17 AM

This is very simple. You can download Google toolbar to know about cache and crawling and another way is Google webmaster tool.

davidfrankk · #20 (**permalink**) 12-16-2011, 02:19 AM

Download Google's Toolbar. It show's when (with time) your page / site was crawled.