Tell about robot.txt

[chinku]

Can any help me to know about robot.txt, and how it helps while crawling?

[jannah is awesome]

A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines.

The file itself is a simple text file, which can be created in Notepad. It need to be saved to the root directory of your site, that is the directory where your home page or index page is. All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site.

Read further more factoids about robots.txt

[Puripong]

Originally Posted by jannah is awesome

A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines.

The file itself is a simple text file, which can be created in Notepad. It need to be saved to the root directory of your site, that is the directory where your home page or index page is. All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site.

Read further more factoids about robots.txt

Great reply as usual Green rep added. I would recommend you reading this for further reference

Robots.txt Tips and Tricks |SEO Updates, News

10 Things you can do with Robots.txt for SEO

[nsontaylo]

robots.txt bis file placed on your website is just a simple text file that contains commands that tell web crawlers what they should or should not indexing in search engines. It has special meaning to most of the honorable robots on the web. By establishing a few rules in this text file, you can tell robots not to crawl and index files of certain directories on your site, or at least.

[m112]

google webmaster tools has got tool to help setup proper robots.txt

[shaaam2007]

robots.txt is a simple text file to tell a search engine that which pages are not indexable/crawl able. Which page are transfered and which are permanentlys deleted.

User-agent: *

Disallow: /downloadthankyoupage.php
Disallow: /error_log
Disallow: /page2.html

this is simplest robots.text file