Robots.txt?

[leeppp222]

I have just checked my Matrix Stats for broken URLs and noticed that every day I get the following showing:

reffering URL: (a website design company who I have never heard of)
missing page: /robots.txt
average hits per day: 4

This may be a silly quesion but why is it showing this?

[winelight]

Do you have a robots.txt file?

http://www.robotstxt.org/wc/robots.html

[leeppp222]

No I haven't - should I have one?

[winelight]

It depends. If you read the FAQ at the link I gave it should make it clear whether or not you need one. No harm is likely to result if you don't create one but everyone's situation is different.

[scisoft]

I do not have robots.txt as well but I am really getting annoyed from the robot.txt not found hits in my log.

[winelight]

If you don't actually need one in order to restrict access to parts of your site, you can still create one and just leave it blank!

[scisoft]

Just done it:
User-agent: *
Disallow: /temp/

[winelight]

Now, if only I knew how to get rid of all those attempts to access owssvr.dll etc.!

[Immo]

You should be able to use .htaccess to block requests for the file.

[winelight]

Ah yes, good idea, thanks.

[later]

It took me a while to figure out the best way. Now I have:

Redirect permanent /_vti_bin/owssvr.dll http://domain.com/404.php

in .htaccess, and 404.php sends a 404 header. This way I get no entries in the error log and the client gets a 404 which is correct.

[clasione]

You know -- Altavista's paid inclusion wasn't indexing one of my sites cause it had no robotos.txt - I had to add one to get crawled.... That was about a year ago......

food for thought

[christper2000]

I beleive that you odn't need robots.txt anymore, just get at least one good link that is crawled and has a good page rank to get it index