importance of Robot.txt file

Humak · 03-15-2008, 11:02 PM

Can anyone tell me importance of Robot.txt file ?

kind regards

idisplayforless · 03-16-2008, 12:40 AM

its not that important...unless you have some pages that you want to block google from crawling

--bk · 03-17-2008, 11:15 AM

In two words ... "Very Important".

A while back, within a year or so, Google and Yahoo! both arose and said they would now look to the robots.txt file to find the link to your sitemap.xml file.

The sitemap.xml file is important because it lists all the pages to your site in a nice robot friendly way.

At the same time they asked if webmasters could please use the robots.txt file to tell their bots where NOT to be ... the original use for this file.

Their request is serious. They use up a lot of bandwidth searching for things that they can't or won't index. Images, javascript, dead files, notes and a huge array of different things people keep on their servers.

So, if you will kindly prepare a list of places to NOT have the robots visit, they will be happy.

A happy robot is a wonderful thing!

Plus ... they'll find your sitemap.xml file every single time they visit your site.

--BK

mrichards1984 · 03-18-2008, 12:59 AM

It's all important when you don't want your download pages to be indexed!

tomfrederick · 03-18-2008, 02:10 AM

check this link http://www.seoresource.net/robots-metatags.htm

jabo · 03-18-2008, 02:46 AM

robots.txt is important if you want to restrict access to some pages of your website...

netrepreneur · 03-18-2008, 12:08 PM

Very Very Important!

By disallowing the search engines to read all your code - css, javascript etc you can feed them only your Sites Content ie:keyword optimized copy!

This can have a dramatic effect on your se rankings. Best practices always...