Robots.txt

andrewp · Share | #1 (**permalink**) 10-11-2006, 04:10 PM

Do I need a robots.txt page for my websites? I have some 'page not found' in my website stats and they are for robots.txt because I haven't created one.

How would I go about creating this page?

Bernard · #2 (**permalink**) 10-11-2006, 04:40 PM

Use notepad:

http://www.robotstxt.org/wc/exclusion-admin.html

webzenstar · #3 (**permalink**) 10-11-2006, 06:29 PM

The robots.txt file is easy to set for a smallish site, for instance, a typical 4 page small company web site since there's nothing to "restrict" bots from seeing, such as:

User-agent: *
Disallow:

These two lines of code, placed in a robots.txt file (within your site's root directory), mean that any bot crawler can explore, well... any directory!

With this in mind, please consider that if there exists information within a web site structure that shouldn't be seen by bots, it probably shouldn't be seen by anyone other than the intended users so -typically- the "secret" directory will be protected within a server controled "realm" (requiring a username and password to gain access to).

You should also note that even if your robots.txt file indicates a directory is off-limits to bots, such as:

User-agent: *
Disallow: /sensitive-folder-that-should-not-be-scanned

To this day, nobody knows -for sure- if all the bots out there actually respect this directive... so be careful (better safe than sorry)!

Well, in any case, it's better to have a robots.txt file than not

andrewp · #4 (**permalink**) 10-12-2006, 03:22 PM

I have now created a robots.txt file and it now allows robots to search all pages.

Wouldn't all pages be crawled automatically if no robots.txt file exists?

Cheers for the help

Bernard · #5 (**permalink**) 10-12-2006, 03:49 PM

Generally, yes, the major search engines will crawl your entire site if no robots.txt is present, but I don't like seeing those robots.txt 404 errors in my web stats, so I always include one.

I think it may affect how aggresively the majors crawl your site (ie. not having one will cause crawling/indexing to be slower), but I have never tested this hypothesis to be sure.

parusa619 · #6 (**permalink**) 10-19-2006, 12:52 PM

Quote:

Do I need a robots.txt page for my websites? I have some 'page not found' in my website stats and they are for robots.txt because I haven't created one.

Of course, but it depends on the page you want to configure, most probably dynamic pages.

Blazed · #7 (**permalink**) 10-19-2006, 07:18 PM

check this out what google has to say about robots.txt
http://www.google.com/support/webmas....py?topic=8475

Manish Pandey · #8 (**permalink**) 10-27-2006, 01:24 AM

You could also make use of the rel="nofollow" tag...

DebbieV · #9 (**permalink**) 10-27-2006, 02:20 AM

Quote:

Originally Posted by Manish Pandey

You could also make use of the rel="nofollow" tag...

I have my doubts about how effective this tag actually is.

Manish Pandey · #10 (**permalink**) 10-27-2006, 03:50 AM

why...???...

Bernard · #11 (**permalink**) 10-27-2006, 04:54 AM

rel="nofollow" offers no guarantee that the target page will not be crawled and indexed (it could be linked from somewhere else without rel="nofollow"

robots.txt can ensure a page is not cached, but no guarantee that it wont appear in SERPs. Google will list pages without the official title and no description if it finds enough links to it to figure out what it's about.

meta robots="noindex,nofollow" (without any robots.txt restrictions) should get an already indexed page to be removed from Google's index (no cache, title, description as above).

vistadivine.com · #12 (**permalink**) 10-27-2006, 05:45 AM

Well it is effective.

Akash

DebbieV · #13 (**permalink**) 10-27-2006, 10:49 AM

Quote:

Originally Posted by vistadivine.com

Well it is effective.

Akash

What are you referring to and what do you base this on?

Manish,

I've seen indexed sites with what we believe to have only nofollow links. I realize there is the possibility that there might be other links, but it's not likely.

I believe the major search engines do follow these links and index the content but do not provide the target url with any benefit (anchor text, PR, etc) for the link in the search results.

vistadivine.com · #14 (**permalink**) 10-27-2006, 04:33 PM

Well because the no-follow tags do work, I have tested it out on 1 of my pages.

Akash

Manish Pandey · #15 (**permalink**) 10-27-2006, 04:46 PM

Quote:

Originally Posted by DebbieV

What are you referring to and what do you base this on?

Manish,

I've seen indexed sites with what we believe to have only nofollow links. I realize there is the possibility that there might be other links, but it's not likely.

I believe the major search engines do follow these links and index the content but do not provide the target url with any benefit (anchor text, PR, etc) for the link in the search results.

Well Deb then what is the use of this tag if they get indexed... well i want a certain link of my page not to be followed.. and according to you its been followed moreover indexed... then are they kept on another index... called the "nofollow index"... i guess its not logical...

Well i believe that its similar to the "noodp" tag where the bots know that they dont have to follow/use the description from the odp... isn't it..??..

vistadivine.com · #16 (**permalink**) 10-27-2006, 04:50 PM

Yes Manish you are right and noodp tag does work isn't it and the no-follow tag too isn't it.

Akash

DebbieV · #17 (**permalink**) 10-27-2006, 06:30 PM

Quote:

Originally Posted by Manish Pandey

Well Deb then what is the use of this tag if they get indexed... well i want a certain link of my page not to be followed.. and according to you its been followed moreover indexed... then are they kept on another index... called the "nofollow index"... i guess its not logical...

The whole premise behind this tag was to stop comment spam from skewing the search results. While the links are not given any weight towards ranking, link juice, PR, etc., I still believe they are crawled and the pages are indexed. Just because a page is indexed does not mean it has any chance to rank in the search results.

Manish Pandey · #18 (**permalink**) 10-27-2006, 09:04 PM

Oops... are you forgetting the thing called "on-page" optimization... well if there is a very small competition then only "on-page" optimization could take you to the top 10... isn't it..??..

vistadivine.com · #19 (**permalink**) 10-27-2006, 10:13 PM

Yes but still on-page optimization is neccessary along with high quality links so that you can give your best.

Akash

Manish Pandey · #20 (**permalink**) 10-27-2006, 10:42 PM

Ahem!

We are not talking about on-page optimization and links... hehe... what we're discussing is that if a page is indexed then it may show up in SERPs for the less competitive terms...