Robots.txt

m2k · 08-12-2004, 07:32 AM

Hi. I redesigned what used to be a one-page site and launched about 1.5 months ago. Same url for the home page. None of the new pages have been indexed by Google, Yahoo, etc. yet.

I was just looking at the site's web stats and under August 2004 "Code 404 Not Found" Requests, it read:

Hits: 11 URL: /robots.txt

Is this creating an indexing problem? Thanks. Sorry, still very new at this...

Immo · 08-12-2004, 07:37 AM

No its just the search engines or other spiders checking if you have any instructions, it doesnt stop them indexing the site.

If in doubt add a robots.txt file and simply add the file and leave the contents blank to get rid of the error.

I, Brian · 08-12-2004, 03:07 PM

Just got to be patient with Google indexing.

sem4u · 08-13-2004, 04:23 AM

As ncsuk says, just create a blank robots.txt file. It will get rid of all of the robots.txt errors in your logs.

JayM · 08-13-2004, 04:31 PM

User-agent: *
Disallow:

Even better than a blank document - add the above lines - this says that all robots are allowed to visit all files... unless of course you do want them to stop visiting somewhere on your site.