|
Use the robots.txt file to reduce bandwidth on indexing such things as image folders, if you have production folders that you do not want indexed, or if there are sections of your site that it makes no sense for people to enter your site.
If you have a large site, it can help the SE find your pages if you have a table of contents page or a site map that explodes into links. However, you do have to link to this page or it will never be seen either.
|