Making a robot.txt file

edbarton · 10-13-2003, 09:28 AM

How do you do this, I have never done this before

I want a robot file that stops it from indexing two pages and also that makes the spiders revisit every week

**Cricket** · 10-13-2003, 09:31 AM

I wrote an article on creating a simple robots.txt file.

http://www.mississippi-photo-gallery...e_articles.htm

Perhaps that will help you! If not I will be happy to point you in the right direction.

Mel · 10-13-2003, 09:38 AM

HI Ed:
You can use a robots.txt file to stop one or all robots from indexing your pages, but you cannot make them visit you on your terms.

To make a robots.txt file just create a plain vanilla text file (notepad or any other text editor will work fine but not Wordpad) then add these lines:
User-agent: *
Disallow: names of pages or directories you do not want spiders to visit one page or directory on a line.

Then save this file to the ROOT directory of you site.

You can do lots more with a robots.txt file, too. There is full information on the protocol at W3C.org

edbarton · 10-13-2003, 09:42 AM

According to something I read you can do something like

useragent:*
revisit:7 days

Mel · 10-13-2003, 09:47 AM

Yes you can do that if you want to but it will have no effect on when the spiders visit you.

You are probably thinking about the Meta robots tag that uses that kind of syntax and which is generally misunderstood. Such a request in that tag asks the spiders NOT to visit you for at least 7 days.

**Cricket** · 10-13-2003, 09:55 AM

There's only one sure way to get the robots to visit you on a regular basis and thats through backlinks.

Robots follow links... the more sites that are linked to you, the more often they visit...

We have been very lucky (knock on wood) because the crawlers we care most about (Google Etc) visit nearly daily.