Webmaster Forum

Go Back   Webmaster Forum > Marketing Forums > Marketing Forum

Marketing Forum Marketing, branding and advertising discussions. Online and off-line marketing discussions.

Thread Tools Display Modes
Share |
Old 09-16-2004, 06:33 AM
Alan30 Alan30 is offline
Contributing Member
Join Date: 03-28-04
Posts: 73
iTrader: 0 / 0%
Robot text file.

What is the Robot Text File?

The robot text file is used to disallow specific or all search engine spider’s access to folders or pages that you don't want indexed.
Why would you want to do this?
You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.

How do I use it?

You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow.

What does it look like?

It's really nothing more than a "Notepad" type .txt file named "robots.txt"
The basic syntax is

User-agent: spiders name here
Disallow:/ filename here

If you use
User-agent: *
The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.

To disallow an entire directory use
To disallow an individual file use
You have to use a separate line for each disallow. You cannot you for example use
You should use

For a list of spider names visit
Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.robotstxt.org/wc/active/html/index.html
For help on creating robot text files there is a program call robogen.
There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/

Alan Murray is an Internet Webmaster Professional and runs his own web design company. http://www.designprofessional.co.uk
Reply With Quote



Old 09-17-2004, 11:27 AM
Ferre's Avatar
Ferre Ferre is offline
No Longer Active
Join Date: 10-15-03
Location: Amsterdam, Netherlands
Posts: 6,897
iTrader: 0 / 0%
I have a question. Does:

Disallow: Forum/admin/

disallow the forum as well?
Reply With Quote
Old 09-17-2004, 11:48 AM
TGR TGR is offline
Senior Member
Join Date: 09-02-04
Location: Chicago, IL
Posts: 323
iTrader: 0 / 0%
Reply With Quote
Old 09-17-2004, 03:13 PM
Ferre's Avatar
Ferre Ferre is offline
No Longer Active
Join Date: 10-15-03
Location: Amsterdam, Netherlands
Posts: 6,897
iTrader: 0 / 0%
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > Marketing Forum


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
importance of Robot.txt file Humak SEO Forum 6 03-18-2008 12:08 PM
The proper usage of the robot.txt file vipenterprises Marketing Forum 0 02-13-2005 07:33 PM
Best tutorial for using a robot.txt file? Johnathan Google Forum 5 11-12-2004 02:23 PM
Using ROBOT Meta Text Buskerdoo SEO Forum 3 02-29-2004 08:39 PM
Making a robot.txt file edbarton Coding Forum 5 10-13-2003 09:55 AM

V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory

All times are GMT -7. The time now is 03:06 AM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC