Thread: What is in your robots.txt file?
View Single Post
Old 05-04-2008, 05:19 PM   #3 (permalink)
m42
v7n Mentor
 
Join Date: 03-23-08
Posts: 1,390
iTrader: 0 / 0%
Latest Blog:
None

m42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest order
Send a message via Yahoo to m42
Quote:
Originally Posted by Steven Myers View Post
Mine just includes the directories allowed and not allowed, specific spiders to crawl and that is it. Very low in text.
How do you determine which spiders to allow?
__________________
Diet Fads
m42 is offline   Reply With Quote