Quote:
|
Originally Posted by spidersam
Disallow: /forums/sutra*.html$
Disallow: /forums/ptopic*.html$
Disallow: /forums/ntopic*.html$
Disallow: /forums/ftopic*asc*.html$
From my understanding, this was supposed to not allow Google from crawling duplicate content...not stop it from crawling 'any' content.....but I am a supreme amatuer at this...so I was just following directions
|
Where are you bothered about google crawling duplicate content, after all, with twice the content crawled, cached, and listed, you have twice the probability of you pages appearing in SERPS