Webmaster Forum

Go Back   Webmaster Forum > Web Development > Coding Forum

Coding Forum Problems with your code? Discuss coding issues, including JavaScript, PHP & MySQL, HTML & CSS, Flash & ActionScript, and more.


Reply
 
Thread Tools Display Modes
Share |
  #1  
Old 10-13-2003, 09:28 AM
edbarton's Avatar
edbarton edbarton is offline
Senior Member
 
Join Date: 10-13-03
Location: England
Posts: 1,757
iTrader: 0 / 0%
Making a robot.txt file

How do you do this, I have never done this before

I want a robot file that stops it from indexing two pages and also that makes the spiders revisit every week
 
Reply With Quote

Advertisement

Advertisement

  #2  
Old 10-13-2003, 09:31 AM
Cricket's Avatar
Cricket Cricket is offline
No Longer Active
 
Join Date: 10-13-03
Location: Texas
Posts: 42,181
iTrader: 0 / 0%
I wrote an article on creating a simple robots.txt file.

http://www.mississippi-photo-gallery...e_articles.htm

Perhaps that will help you! If not I will be happy to point you in the right direction.
 
Reply With Quote
  #3  
Old 10-13-2003, 09:38 AM
Mel's Avatar
Mel Mel is offline
Contributing Member
 
Join Date: 10-13-03
Location: Malaysia
Posts: 2,029
iTrader: 0 / 0%
HI Ed:
You can use a robots.txt file to stop one or all robots from indexing your pages, but you cannot make them visit you on your terms.

To make a robots.txt file just create a plain vanilla text file (notepad or any other text editor will work fine but not Wordpad) then add these lines:
User-agent: *
Disallow: names of pages or directories you do not want spiders to visit one page or directory on a line.

Then save this file to the ROOT directory of you site.

You can do lots more with a robots.txt file, too. There is full information on the protocol at W3C.org
 
Reply With Quote
  #4  
Old 10-13-2003, 09:42 AM
edbarton's Avatar
edbarton edbarton is offline
Senior Member
 
Join Date: 10-13-03
Location: England
Posts: 1,757
iTrader: 0 / 0%
According to something I read you can do something like

useragent:*
revisit:7 days
 
Reply With Quote
  #5  
Old 10-13-2003, 09:47 AM
Mel's Avatar
Mel Mel is offline
Contributing Member
 
Join Date: 10-13-03
Location: Malaysia
Posts: 2,029
iTrader: 0 / 0%
Yes you can do that if you want to but it will have no effect on when the spiders visit you.

You are probably thinking about the Meta robots tag that uses that kind of syntax and which is generally misunderstood. Such a request in that tag asks the spiders NOT to visit you for at least 7 days.
 
Reply With Quote
  #6  
Old 10-13-2003, 09:55 AM
Cricket's Avatar
Cricket Cricket is offline
No Longer Active
 
Join Date: 10-13-03
Location: Texas
Posts: 42,181
iTrader: 0 / 0%
There's only one sure way to get the robots to visit you on a regular basis and thats through backlinks.

Robots follow links... the more sites that are linked to you, the more often they visit...

We have been very lucky (knock on wood) because the crawlers we care most about (Google Etc) visit nearly daily.
 
Reply With Quote
Go Back   Webmaster Forum > Web Development > Coding Forum

Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
importance of Robot.txt file Humak SEO Forum 6 03-18-2008 12:08 PM
Making a PDF File Fill-Inable theChronic Coding Forum 6 08-30-2006 05:28 AM
The proper usage of the robot.txt file vipenterprises Marketing Forum 0 02-13-2005 07:33 PM
Best tutorial for using a robot.txt file? Johnathan Google Forum 5 11-12-2004 02:23 PM
Robot text file. Alan30 Marketing Forum 3 09-17-2004 03:13 PM


V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory


All times are GMT -7. The time now is 10:50 PM.
Powered by vBulletin
Copyright © 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC