Webmaster Forum

Go Back   Webmaster Forum > Marketing Forums > Google Forum

Google Forum Discuss Google related issues.


Closed Thread
 
Thread Tools Display Modes
Share |
  #1  
Old 05-31-2010, 03:37 AM
chinku chinku is offline
Banned
 
Join Date: 04-21-10
Posts: 10
iTrader: 0 / 0%
Tell about robot.txt

Can any help me to know about robot.txt, and how it helps while crawling?
 

Advertisement

Advertisement

  #2  
Old 05-31-2010, 04:06 AM
jannah is awesome jannah is awesome is offline
Banned
 
Join Date: 03-27-10
Location: from the 3rd most corrupt
Posts: 410
iTrader: 0 / 0%
A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines.

The file itself is a simple text file, which can be created in Notepad. It need to be saved to the root directory of your site, that is the directory where your home page or index page is. All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site.

Read further more factoids about robots.txt
 
  #3  
Old 05-31-2010, 12:48 PM
Puripong's Avatar
Puripong Puripong is offline
Contributing Member
 
Join Date: 03-21-10
Location: Somewhere in your mind
Posts: 695
iTrader: 0 / 0%
Quote:
Originally Posted by jannah is awesome View Post
A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines.

The file itself is a simple text file, which can be created in Notepad. It need to be saved to the root directory of your site, that is the directory where your home page or index page is. All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site.

Read further more factoids about robots.txt
Great reply as usual Green rep added. I would recommend you reading this for further reference

Robots.txt Tips and Tricks |SEO Updates, News

10 Things you can do with Robots.txt for SEO
 
  #4  
Old 06-17-2010, 02:37 AM
nsontaylo nsontaylo is offline
Junior Member
 
Join Date: 06-17-10
Posts: 4
iTrader: 0 / 0%
robots.txt bis file placed on your website is just a simple text file that contains commands that tell web crawlers what they should or should not indexing in search engines. It has special meaning to most of the honorable robots on the web. By establishing a few rules in this text file, you can tell robots not to crawl and index files of certain directories on your site, or at least.
 
  #5  
Old 06-17-2010, 03:02 AM
m112 m112 is offline
Member
 
Join Date: 06-15-10
Posts: 42
iTrader: 0 / 0%
google webmaster tools has got tool to help setup proper robots.txt
 
  #6  
Old 06-17-2010, 03:36 AM
shaaam2007 shaaam2007 is offline
Banned
 
Join Date: 02-17-10
Posts: 128
iTrader: 0 / 0%
robots.txt is a simple text file to tell a search engine that which pages are not indexable/crawl able. Which page are transfered and which are permanentlys deleted.

User-agent: *

Disallow: /downloadthankyoupage.php
Disallow: /error_log
Disallow: /page2.html

this is simplest robots.text file
 
Go Back   Webmaster Forum > Marketing Forums > Google Forum

Closed Thread


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is robot.txt Manikkavalli SEO Forum 31 10-24-2009 03:22 AM
robot spider polymath1980 Web Design Lobby 7 01-22-2009 04:43 AM
Robot tag globalmr Coding Forum 2 02-04-2005 05:19 PM


V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory


All times are GMT -7. The time now is 11:04 AM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC