|
Actually, the Alltheweb and Google links do differ.
A particular problem for many big vBulletin 3's that I know of is that Google simply fails to index the dynamic content of them - even though they are supposed to have session IDs for a range of bots disabled. It's almost as if Google decides that there are too many links on such sites, and simply covers the static content (I'm talking about huge sites such as vBulletin.com, and sitepointforums.com).
Yet smaller vBulletin's, such as my own, don't seem to have triggered this block.
Alltheweb has no problem, though.
You can see this yourself by checking backlinks for vB.com or SPF in both.
Ironically, Google has indexed more pages on my forums than vb and SPF combined - even though my sites combined have less than 15,000 posts, compared to their combined post count of 1.5 million.
Does anyone have prior evidence of Google having a feature that refuses to index too large a set of dynamic content from any single site?
|