How deep does google crawl on its daily visit?

crazyhorse · 08-23-2004, 12:37 AM

By checking my awstats i have noticed that google isnt crawling my site deeper then directoryname/directoryname/directoryname. Im thinking of changing the directory and getting it closer to the root. Any suggestions?

nextGoal100kVisitors · 08-23-2004, 01:05 AM

Doesn't this depend on number of inbound links you have? i.e., if you have only one inbound link, then Google only scratches surface. If you have 10, 20 inbound links, Google follows these links and crawl 10, 20 times. As it crawls over and over, it crawls deeper and deeper. I thought that's how crawlers crawl ...

So, to answer question, I think you need many more inbound links, BUT if I'm wrong, I'm sure SEO experts in this forum will correct me.

crazyhorse · 08-23-2004, 01:51 AM

Well it does crawl the site but to a cerain extension , not deeper then three levels but i ll wait for the end of the month and see what the results will bring up

Dominic · 08-23-2004, 06:30 AM

In my stats there is a file status 'no change since last visit' which the spiders may / would pay attention to. But really asside from that I find the sites that have the most frequently updated / pages added are spidered most often.

Johan007 · 08-23-2004, 06:48 AM

I don’t find directory structure matters.

I find pages linked from highest PR page (home) are spidered quicker. Speed is a real issue with my type of news media website and something I am always looking to improve on. – So any other tips please let me know too.

Currently it takes 4 days to include a page linking of my homepage... and thats not fast enough!