How often does Google crawls your website? - Page 2

stacy131 · 01-02-2008, 07:12 AM

My site gets about 25-30 Googlebot crawls per day. Usually there is a plus some number denotation after it in my statistics section, i.e. 25+2. The +2, as I have noticed whenever I wake up the next day and check, usually turns out to be the number of newly indexed pages on that particular day.

Whether that's true, I don't know, but for a newbie "webmaster" such as myself, it's been a pretty consistent theory.

sagefool · 01-02-2008, 08:04 AM

I get crawled about once every 3-4 days. By this schedule, it should have crawled me on the first. The bot seems to have taken off for the New Year.

Nikhil · 01-02-2008, 08:07 AM

Quote:

Originally Posted by stacy131

My site gets about 25-30 Googlebot crawls per day. Usually there is a plus some number denotation after it in my statistics section, i.e. 25+2. The +2, as I have noticed whenever I wake up the next day and check, usually turns out to be the number of newly indexed pages on that particular day.

Whether that's true, I don't know, but for a newbie "webmaster" such as myself, it's been a pretty consistent theory.

I guess googlebot stays on your site all the time and whenver they get time off your site they come to webmaster's like us.............can you tell me what are you doing so special that Google crawls your site 25+ times a day? I am eager to see your answer?

gemispence · 01-02-2008, 09:17 PM

Hi All - Interesting replies and I have a question of my own. My site is ranked within the first 10 results on about 5 of the most popular search terms so I'm happy about that.

As I was about to create a site map tonight, I looked at Google's diagnostics of my site and it looks like Googlebot last crawled my site on August 3rd, 3 months ago, which is odd considering I add am constantly adding fresh content and I still have really good rankings. Anyone have any ideas other than what was mentioned?
My site is [link removed]. -

Thanks in advance.

StarLab · 01-03-2008, 12:43 AM

Now that Dec Stats are complete, I thought I would post an update.

Keep in mind my blog is now a little over 5 weeks old, 4 weeks of that existed in Dec.

975 hits which only accounted for 2.32% of my total hits to the site. That averages out to ~31 hits per day.

What did I do to get that?

Submitted my URL to Google
Created and implemented a Google sitemap
Updated content daily (missed a couple days, however)
Running WordPress (not sure if that has to do with anything, but thought it important to list anyway)

Overall, I'm fairly happy with my site as it has seen some decent traffic for being the blog equivalent to Seinfeld... a "blog about nothing"... lol

gemispence - That is odd that August was the last time Google crawled your site. Especially if you are updating regularly. Hopefully someone will have suggestions for you... I'm at a loss on that, however...

woodcs82 · 01-03-2008, 01:35 AM

Google bot and yahoo bot can crawl Daily sometimes!

stacy131 · 01-03-2008, 10:25 PM

Well, they always say content is king. Whether that's true is debateable, but it my case, it IS!

I run an e-commerce site that offers unique products that people can't just stumble upon, they search for them by name because of their uniqueness and rarity. For that reason, in my case, content IS king.

People type in all sorts of combinations of keywords for my site. When dealing with selling specific products, those keywords are very descriptive.

You can't be very descriptive with only one or two keywords, so my, let's not forget to mention, new and by no means professionally made or SEO-optimized site has seen great success from those odd "tail" keywords that come from many combinations of keywords put together in one search.

Now it's only a matter of time before I start showing up for the shorter, more frequently searched for keywords.

I write my content not just to describe my product to someone who is on the site at that moment but also, for someone who needs to find my site first. And that, folks, is done by using Google as your translator.

1websitedesign · 01-03-2008, 10:39 PM

i think in 1 month in 2 months in 3 days in 2 weeks......

Bogart · 01-04-2008, 03:20 PM

My static sites get crawled 2 or 3 times a week. If you have adsense or update frequently, Google crawls more.

shakir · 01-05-2008, 01:01 AM

almost once in 2 weeks crawl my site

businessservicesuk · 01-05-2008, 04:21 AM

My site gets crawled most days 1650 hits in Dec.

New articles are generally indexed within a few hours

deezines · 01-05-2008, 03:31 PM

This is the report for our website, for the 5 days of January so far.
Last visit
9 different robots

Yahoo Slurp 229
Googlebot 92
Unknown robot (identified by 'bot/' or 'bot-') 7
Ask 7
MSNBot 14
Yahoo! Slurp China 1
MSNBot-media 187
Unknown robot (identified by 'crawl') 14
Unknown robot (identified by 'spider') 1

Yahoo absolutely loves our site, as you can see, and we rank very highly for a good number of competitive key phrases.
We also do reasonably well on Google too.

I would say the following things are crucial to attracting regular crawls.

Build a Google Site Map [ and ensure you get it verified by google]
Have plenty of relevent, quality content. [ if you havent, why expect anyone to be interested in your site? ]
Add new content reasonably regularly
Add new pages
ensure your page file names are relevent

stacy131 · 01-06-2008, 09:56 AM

Deezines, do you mind sharing what you did to become so popular on Yahoo? I have submitted by site URL and sitemap there and have almost as many pages indexed by them as I do in Google, but your popularity in Yahoo, compared to Google is unsurpassed? What tricks did you use with Yahoo?

deezines · 01-06-2008, 04:25 PM

Quote:

Originally Posted by stacy131

Deezines, do you mind sharing what you did to become so popular on Yahoo? I have submitted by site URL and sitemap there and have almost as many pages indexed by them as I do in Google, but your popularity in Yahoo, compared to Google is unsurpassed? What tricks did you use with Yahoo?

I dont think we use any 'tricks' as such.

What I will say is this, and it i'm likely to get flamed...the headlong rush to try and garner loads of mildly themed incoming links is in my opinion overated.
We concentrate on good content, refeshing it, and use sensible semantics to target our keywords/phrases.
Yahoo really likes keywords...it also really likes Meta title and Description...so pay a LOT of attention to getting these right; but remember, your Title & Description semantics need to be placed properly then throughout your content copy.

I dont think I am allowed to identify our site, otherwise you would be welcome to go take a good look around! lol.

We come page one on Google [ and I think position 1 ] for about 12 - 15 keyphrases with approx 1.5 million upwards competition.
On Yahoo, the situation is even better.

4 years ago we used to send out around 400 - 600 brochures a month by post to our target market [ SME's ]
Now they are gathering dust in the office. All our new enquiries come from the web now.

SEO is a fascinating and also frustrating thing.....
I do believe in trial and error.....try a change and see if the SE's like it, or not.
Key is, dont make loads of changes all at once because you'll never know which were positive or vice versa.
Pay close attention to the Stats package you use to monitor traffic and visitor habits....it will tell you all sorts of invaluable information that you can use to tweak your Metas and/or Copy content. Quite often it will throw up a Keyphrase that you are not using but realise you should be.

I check ours daily.

Hope some of this helps, and i havent bored everyone to death!

D.