Webmaster Forum

Go Back   Webmaster Forum > Marketing Forums > SEO Forum
What is robot.txt What is robot.txt
Register Rules/FAQ Members List RoundTable Chat Search Today's Posts Mark Forums Read

SEO Forum Search engine optimization discussions.


Reply
 
LinkBack Thread Tools Display Modes
Old 08-31-2009, 04:37 AM   #1 (permalink)
v7n Mentor
 
Join Date: 07-24-06
Posts: 691
iTrader: 1 / 100%
Latest Blog:
None

nasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nicenasty.web is just really nice
Quote:
Originally Posted by bermuda View Post
Many websites do not have the robots files on their servers and in these situations, the search engines would consider crawling all parts of the websites including the images folders. It is still recommended using the files even if you are not going to block any folders.
Some crawlers usually check if such file exists. Having empty one means cleaner error log.
nasty.web is offline  
Add Post to del.icio.us
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > SEO Forum

Reply

« Previous Thread | Next Thread »

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes
Hybrid Mode Hybrid Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
robot spider polymath1980 Web Design Lobby 7 01-22-2009 04:43 AM
Would You Have Sex With a Robot? earthbound soul Forum Lobby 53 07-28-2008 05:56 AM
iTunes Robot? GeXus SEO Forum 1 05-01-2006 02:20 PM
Robot tag globalmr Coding Forum 2 02-04-2005 05:19 PM
Making a robot.txt file edbarton Coding Forum 5 10-13-2003 10:55 AM


Sponsor Links
Get exposure! Contextual Links V7N SEO Blog V7N Directory


Site Navigation: v7n Home .::. Graphics .::. Scripts .::. V7N Web Directory .::. V7N Technology Blog .::. V7N Affiliate Program .::. Advertise

Partners: Search Engine Optimisation .::. Dedicated Server Hosting .::. Webmaster Talk .::. CPA Affiliates .::. Dedicated Server Hosting

Site Sponsored by a Hivelocity Dedicated Server.

All times are GMT -7. The time now is 10:29 PM.
© Copyright 2008 V7 Inc
Powered by vBulletin
Copyright © 2000-2009 Jelsoft Enterprises Limited.

Contact Us - Internet Marketing - Top

Search Engine Optimization by vBSEO 3.3.0 ©2009, Crawlability, Inc.