Webmaster Forum

Go Back   Webmaster Forum > Web Development > Coding Forum

Coding Forum Problems with your code? Discuss coding issues, including JavaScript, PHP & MySQL, HTML & CSS, Flash & ActionScript, and more.

Thread Tools Display Modes
Share |
Old 02-04-2005, 11:35 AM
globalmr globalmr is offline
Junior Member
Join Date: 10-26-04
Posts: 2
iTrader: 0 / 0%
Question Robot tag

On all my websites I have this meta tag and I am told it is correct.(<meta name="Robots" CONTENT="index,follow">) My logs show that last month on 1 site I had 19 failed request for /robots.txt, another had 47, I do not know what this means. I did some research and it seems I am missing the robot.txt folder on my server. I was not aware I had to upload and create this new folder on server. I know the tag is not that neccessary for engines to index whole site.
My question is:
A. do I create this folder and name it robot.txt on the server?
B. What do I put in the folder?
c. What does that exactly mean. failed request for /robot.txt

Advise appreciated. Thanks
Reply With Quote



Old 02-04-2005, 02:06 PM
imaginemn's Avatar
imaginemn imaginemn is offline
v7n Mentor
Join Date: 02-18-04
Location: Brownsville, Texas
Posts: 1,354
iTrader: 0 / 0%
There could be some bots looking specifically for that file. You could just create a blank robots.txt and save it in the root if you are concerned about your stats. One of the main reasons to have this file is so you don't have to code the meta tags. NOT ALL spiders read META Tags. The index,follow tag is pretty much useless because of this. The spiders will automatically index the page unless you don't want it to and that is where the robots.txt file comes into action.

Here is an example of what could be in your robots.txt file if you don't want your cgi-bin and images folders indexed by spiders.

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

Reply With Quote
Old 02-04-2005, 05:19 PM
globalmr globalmr is offline
Junior Member
Join Date: 10-26-04
Posts: 2
iTrader: 0 / 0%
Very helpful and direct information.
Thank you.
Reply With Quote
Go Back   Webmaster Forum > Web Development > Coding Forum


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
iTunes Robot? GeXus SEO Forum 1 05-01-2006 01:20 PM
Making a robot.txt file edbarton Coding Forum 5 10-13-2003 09:55 AM

V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory

All times are GMT -7. The time now is 07:43 PM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC