Robot tag

[globalmr]

On all my websites I have this meta tag and I am told it is correct.(<meta name="Robots" CONTENT="index,follow">) My logs show that last month on 1 site I had 19 failed request for /robots.txt, another had 47, I do not know what this means. I did some research and it seems I am missing the robot.txt folder on my server. I was not aware I had to upload and create this new folder on server. I know the tag is not that neccessary for engines to index whole site.
My question is:
A. do I create this folder and name it robot.txt on the server?
B. What do I put in the folder?
c. What does that exactly mean. failed request for /robot.txt

Advise appreciated. Thanks

[imaginemn]

There could be some bots looking specifically for that file. You could just create a blank robots.txt and save it in the root if you are concerned about your stats. One of the main reasons to have this file is so you don't have to code the meta tags. NOT ALL spiders read META Tags. The index,follow tag is pretty much useless because of this. The spiders will automatically index the page unless you don't want it to and that is where the robots.txt file comes into action.

Here is an example of what could be in your robots.txt file if you don't want your cgi-bin and images folders indexed by spiders.

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

imaginemn

[globalmr]

Very helpful and direct information.
Thank you.
Frederick