Is Google bot having troubles to crawl sites?

[Natural Elements]

Last week Google bot stopped to crawl sites for 2 days, and many webmasters observed the same issue on many forums. Then since last week the Google bot seems to crawl extremly slow and that's very unusual.

I have a dynamic statistic program integrated to my site so I know when something goes wrong pretty fast.

I am wondering if you guys noticed any of that.

[james70]

Thanks for your thread about it. I was thinking any problem happened on my sites. But, now seems most of the websites got affected.

Some of my new webpages really slow to get indexed.

[WebshoppeSolutions]

um .. I think that it's been Yahoo that's slowed, in that their crawl rate has always been well ahead of Google by at least 1,000 .. this month though, so far, Yahoo is just ahead of Google in their crawl rate by a slim margin of only 100.

GoogleBot has been a benchmark crawl for so long, that I've used it to compare the crawl rates of some of the others .. 900 to 1,200 per month is the norm for me (from Google), and right now, GoogleBot is snuggled right in there with a 1,097 and the month isn't quite over yet. (my main site isn't as large as some others).

The two Bots that have me scratching my head lately has been Twiceler (Cuil), and MSNBot (Bing) ..

Cuil has absolutely hammered my site with crawls this month, and I've logged over 15 different individual IP addresses for it's Twiceler Bot .. Bing, on the other hand, has fallen off of the face of the earth, in that after a consistant crawl rate of about 30 visits per month, I'm showing nothing for the Bing (MSNBot) so far in June.

as an aside, I've noticed more of my main site's pages are listed in the index this month, and all of the pages on my secondary sites are listed, whereas before only about half of the pages on my secondary sites were listed in months previous to June.

My crawl numbers, listed above, doesn't include Google crawls for wap/mobi, images, or adsense.

[Natural Elements]

Thanks WebshoppeSolutions.

I did not see any problem with Yahoo and Bing so far. Google bot was fine until June 15th, the last week is a mess.

The Google bot is not acting normally for a week now and is not crawling or indexing the new pages.

[WebshoppeSolutions]

Originally Posted by Natural Elements

Thanks WebshoppeSolutions.

I did not see any problem with Yahoo and Bing so far. Google bot was fine until June 15th, the last week is a mess.

The Google bot is not acting normally for a week now and is not crawling or indexing the new pages.

I was wondering if it all might have something to do with eCommerce sites, but when I look at a few of my client sites, I'm seeing that this might not be the case .. still, the internet is a big place, and maybe Google hasn't gotten around to throwing out it's snarky crawls to them.

Over on Webmasterworld there has been some discussion on this issue too .. most of those are eCommerce site owners as well .. and after reading around on the net about this for a few days, I'm coming to the very light conclusion that the DC (data center) might be playing a role in this.

I'm not quite completely sold on the bad neighborhood angle/concept yet, but thoughts are leaning that direction with the more that I read.

Things have changed, are changing, and have yet to change for the many, and unless one of my sites, or client sites see this marked change, it will be hard for me to put my finger on exactly what might be going on.

Google, over the years, has been a constant .. sites come and go, algorithms get tweaked, and life goes on .. but aside from all of that and the so-called normalcy of the internet, this is something that leaves me in a bit of a quandry.

[~kev~]

Originally Posted by Natural Elements

I am wondering if you guys noticed any of that.

Nope - I have not noticed anything like that. To be perfectly honest, I do not have time to pay attention to details like that. Once or Twice a month I "might" log into my google analytics account and check the number of indexed pages, check my backlinks, check the sitemaps,,,,, and that is about it.

Over the past couple of years, I got tired of asking "why did google delist 200,000 of my pages", why did google do that, why did google do this,,,. So I just roll with it and try not to worry.

[Natural Elements]

Originally Posted by ~kev~

Over the past couple of years, I got tired of asking "why did google delist 200,000 of my pages", why did google do that, why did google do this,,,. So I just roll with it and try not to worry.

I am getting tired of this too, business is not good no more on internet for small businesses.

[WebshoppeSolutions]

Originally Posted by Natural Elements

I am getting tired of this too, business is not good no more on internet for small businesses.

Maybe it isn't, but then again, maybe it still is.

Business can still be good for small businesses online if they spent more of an effort targeting local/regional sales, as opposed to national ones. With the economy such as it's been lately, more people are shopping a little closer to home these days .. the direct human factor comes into play a lot easier when your client is just down the road, or in the next town down the road.

Use "Local" as a selling point, and chances are pretty good that you'll win the day over anyone else that sells from clear across the country.

[Natural Elements]

Originally Posted by WebshoppeSolutions

Maybe it isn't, but then again, maybe it still is.

Business can still be good for small businesses online if they spent more of an effort targeting local/regional sales, as opposed to national ones. With the economy such as it's been lately, more people are shopping a little closer to home these days .. the direct human factor comes into play a lot easier when your client is just down the road, or in the next town down the road.

Use "Local" as a selling point, and chances are pretty good that you'll win the day over anyone else that sells from clear across the country.

We spent a lot of effort but like I said business is not good anymore. Thanks for your suggestion, but locally brick and mortar businesses close their doors everyday in our state, so selling online is the only way to make it.

Our e-commerce represent for us much more than just a simple business.

Anyway, the Google bot issue just add one more problem to us, because the traffic from Google is not the same, orders are going down and technically new pages are not indexed like it should be.

[Ding2005]

The main Google bot stopped working properly already from the beginning of June. Many sites I monitor get no update of "cache:" for three weeks already. Can be related to Caffeine introduction.

The GoogleNews bot is still working perfectly fast - within 10 minutes the news are in index.

[Natural Elements]

Thanks James70 and Ding2005, I guess we have to keep an eye on the GoogleBot

[Groovystar]

Probably it is issues relating to their new search engine style, Caffeine. It's understandable that during its first days there would be a few bugs.

[eimroda]

could this be the reason why some of my sites were thrown to the 100th page from page 1 this June? Some have already "recovered" but some are not yet back in their previous ranks.

[jhon321]

i dont think so it is having problem to crawl sites.

[MarcoFromParis]

Hi, do you still have trouble with Google crawling your sites ?
We have several sites in the travel industry ( 1 Million page each, half indexed).
The crawling rate before the 15th of June ( day of caffeine ?) was around 50K pages per day and per site.
The 15th the crawl rate dropped to 5K pages a day.... 10 times less than the average before.
So, it's now 15 days that the crawl rate is really low and we notice since few days that traffic starts decreasing as well

Webmaster tool is ok ( not 404, time downloading a page is 400ms...)

Every comments or help here is more than welcome.
Thanks a lot.

[Natural Elements]

Bonjour et bienvenue Marco de Paris, es tu Francais?

All I can tell you is that Google bot doesn't work like before, and perhaps the Google team is still working on it.

[joeimo]

I noticed Googbot slowing down on my pages too.. Yahoo still going strong though

[jennifer852]

i have not noticed it yet may be i am not looking crawling rate this much seriously but it seems that all are discussing on that . i believe we need to check on google blog about it..

[sarahberra]

How do you know when the google bot crawls your site???

[Natural Elements]

It's already been said:

I have a dynamic statistic program integrated to my site so I know when something goes wrong pretty fast.