Webmaster Forum


Go Back   Webmaster Forum > Web Development > Web Design Lobby
What is in your robots.txt file? What is in your robots.txt file?
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

Web Design Lobby Forum for general web design issues not specific to scripting or graphics.

   

Reply
 
LinkBack Thread Tools Display Modes
Old 05-04-2008, 06:11 PM   #1 (permalink)
m42
v7n Mentor
 
Join Date: 03-24-08
Posts: 1,733
iTrader: 0 / 0%
Latest Blog:
None

m42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest order
What is in your robots.txt file?
Bandwidth is cheap, but for small business owners attempting to squeeze the most value out of every penny spent, crawling bots can become a nuisance. Robots.txt files provide a quick barrier between those tedious bots and your content.

What are some of your reasons for maintaining a robots.txt file?
What do you include in your robots.txt file?

__________________
Individualism | Diet Fads
m42 is offline  
Add Post to del.icio.us
Reply With Quote
Old 05-04-2008, 06:13 PM   #2 (permalink)
Contributing Member
 
TechSup's Avatar
 
Join Date: 12-27-07
Location: North Carolina
Posts: 655
iTrader: 1 / 100%
TechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really niceTechSup is just really nice
Mine just includes the directories allowed and not allowed, specific spiders to crawl and that is it. Very low in text.
__________________
Best Regards, TechSup
Google your question(s) before you ask.
TechSup is offline  
Add Post to del.icio.us
Reply With Quote
Old 05-04-2008, 06:19 PM   #3 (permalink)
m42
v7n Mentor
 
Join Date: 03-24-08
Posts: 1,733
iTrader: 0 / 0%
Latest Blog:
None

m42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest orderm42 is a web professional of the highest order
Quote:
Originally Posted by Steven Myers View Post
Mine just includes the directories allowed and not allowed, specific spiders to crawl and that is it. Very low in text.
How do you determine which spiders to allow?
__________________
Individualism | Diet Fads
m42 is offline  
Add Post to del.icio.us
Reply With Quote
Old 05-05-2008, 02:46 PM   #4 (permalink)
Contributing Member
 
Join Date: 02-07-07
Posts: 189
iTrader: 0 / 0%
Latest Blog:
None

3DProf4online is liked by many
I am not sure, but as far as I know the Disallow: directory is used for this purpose. Please correct me if I am not right.
__________________
Search engine optimization and submission techniques - promote a website effectively!
Basic web services - update your site easily!
3DProf4online is offline  
Add Post to del.icio.us
Reply With Quote
Old 05-05-2008, 02:49 PM   #5 (permalink)
Contributing Member
 
Join Date: 02-07-07
Posts: 189
iTrader: 0 / 0%
Latest Blog:
None

3DProf4online is liked by many
for example if you write in robots.txt:

User-agent: Aport
Disallow: /eng
Disallow: /news

this means to forbid Aport robot to index links which begin from /news
__________________
Search engine optimization and submission techniques - promote a website effectively!
Basic web services - update your site easily!
3DProf4online is offline  
Add Post to del.icio.us
Reply With Quote
Go Back   Webmaster Forum > Web Development > Web Design Lobby

Reply

« Previous Thread | Next Thread »


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Q about tweaking my robots.txt file RawDepth SEO Forum 2 04-13-2008 07:01 PM
Google and Robots.txt file noob_0001 Google Forum 0 04-01-2005 09:54 PM
Working with the robots.txt file Pannu Marketing Forum 0 05-24-2004 09:10 AM


Sponsor Links
Get exposure! Get exposure! Find Scripts Web Hosting Directory Get exposure! SEO Blog


Site Navigation: v7n Home .::. Graphics .::. Scripts .::. V7N Web Directory .::. V7N Technology Blog .::. V7N Affiliate Program .::. Advertise

Partners: Search Engine Optimisation .::. Webmaster Talk .::. CPA Affiliates

All times are GMT -7. The time now is 08:25 AM.
© Copyright 2008 V7 Inc

Contact Us - Internet Marketing - Top