Best tutorial for using a robot.txt file?

Johnathan · 11-09-2004, 01:47 PM

I've recently just become familiar that google can use a robot file to track all the links of your site?

Anyway.. I'd like to make one... since alot of my pages arent getting indexed..

I was wondering what the best site is that shows how to make a robot.txt file?

.. that tells me, what exactly i do.

Thanks

baggiho · 11-09-2004, 02:17 PM

I learn to make robot.txt through this page http://www.seorank.com/robots-tutorial.htm.

I like it and let me knot alot

awall19 · 11-09-2004, 02:41 PM

robots TXT files are for excluding robots from files...they will not help you get pages indexed.

Johnathan · 11-11-2004, 07:15 PM

oh. ok thank you awall19

after reading the above think i figured that out as well. is there any kind of file that search engines look for... that gives the engine a list of all the links that you WANT the engine to crawl?

thanks

awall19 · 11-12-2004, 10:21 AM

Quote:

Originally Posted by Johnathan

is there any kind of file that search engines look for... that gives the engine a list of all the links that you WANT the engine to crawl?

point links at your pages. make sure that your pages link to your other pages...your site and the links on its pages and links pointing into your site are what search engines use

Papadoc · 11-12-2004, 02:23 PM

Use the robots.txt file to reduce bandwidth on indexing such things as image folders, if you have production folders that you do not want indexed, or if there are sections of your site that it makes no sense for people to enter your site.

If you have a large site, it can help the SE find your pages if you have a table of contents page or a site map that explodes into links. However, you do have to link to this page or it will never be seen either.