Webmaster Forum


Go Back   Webmaster Forum > Marketing Forums > SEO Forum
Robots.txt Robots.txt
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read

SEO Forum Search engine optimization discussions.

   

Reply
 
LinkBack Thread Tools Display Modes
Old 08-25-2006, 02:03 PM   #1 (permalink)
Inactive
 
Join Date: 07-10-06
Posts: 22
iTrader: 0 / 0%
Latest Blog:
None

timtom is liked by somebodytimtom is liked by somebodytimtom is liked by somebody
Send a message via AIM to timtom
Robots.txt
If i want google to crawl/index all the pages throughout my site, do i need a robots.txt file?
timtom is offline  
Add Post to del.icio.us
Reply With Quote
Old 08-25-2006, 02:11 PM   #2 (permalink)
Contributing Member
 
bratwebdesign's Avatar
 
Join Date: 04-05-06
Location: Nebraska
Posts: 560
iTrader: 0 / 0%
Latest Blog:
None

bratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to beholdbratwebdesign is a splendid one to behold
Send a message via MSN to bratwebdesign
No you do not really need one, but some crawlers will look for a robots.txt file once they get to your site. I would just create a robots.txt with the following and upload it to your server.

Quote:
User-agent: *
This way the crawlers who do look for a robots.txt will find it.
__________________
Omaha Web Design | Cube Directory
bratwebdesign is offline  
Add Post to del.icio.us
Reply With Quote
Old 08-25-2006, 05:52 PM   #3 (permalink)
Inactive
 
BinkyMelnik's Avatar
 
Join Date: 06-06-06
Location: New York, NY
Posts: 586
iTrader: 0 / 0%
Latest Blog:
None

BinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to beholdBinkyMelnik is a splendid one to behold
timtom:

Quote:
If i want google to crawl/index all the pages throughout my site, do i need a robots.txt file?
What you need it for is if you have places you *don't* want spidered, like a directory of scripts, or images, that kinda stuff.

You're gonna be spidered otherwise, like it or not.
BinkyMelnik is offline  
Add Post to del.icio.us
Reply With Quote
Old 08-25-2006, 10:29 PM   #4 (permalink)
Contributing Member
 
KC Tan's Avatar
 
Join Date: 07-31-06
Posts: 145
iTrader: 0 / 0%
KC Tan is liked by many
Hi timtom,

The following is a great resources for robots.txt

http://www.robotstxt.org/wc/robots.html
__________________
Free Web Analytics- Essential Information to study and optimize your site based on traffic behavior.
Make money on the internet
KC Tan is offline  
Add Post to del.icio.us
Reply With Quote
Old 02-19-2007, 06:56 PM   #5 (permalink)
Inactive
 
Join Date: 01-30-07
Location: cairo
Posts: 58
iTrader: 0 / 0%
Latest Blog:
None

MaZaGAngeE is liked by many
Send a message via ICQ to MaZaGAngeE Send a message via MSN to MaZaGAngeE Send a message via Yahoo to MaZaGAngeE Send a message via Skype™ to MaZaGAngeE
i did creat a robots.txt file and added this "User-agent: * "
do i need to add anything else ?
MaZaGAngeE is offline  
Add Post to del.icio.us
Reply With Quote
Go Back   Webmaster Forum > Marketing Forums > SEO Forum

Reply

« Previous Thread | Next Thread »


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Need help with robots.txt mitra SEO Forum 10 05-28-2007 04:28 PM
Robots realestate V7N Directory Issues 4 08-15-2004 10:57 PM
Robots.txt m2k SEO Forum 4 08-13-2004 04:31 PM
robots.txt John Scott Web Design Lobby 6 10-15-2003 08:14 AM


Sponsor Links
Get exposure! Get exposure! Find Scripts Web Hosting Directory Get exposure! SEO Blog


Site Navigation: v7n Home .::. Graphics .::. Scripts .::. V7N Web Directory .::. V7N Technology Blog .::. V7N Affiliate Program .::. Advertise

Partners: Search Engine Optimisation .::. Webmaster Talk .::. CPA Affiliates

All times are GMT -7. The time now is 03:57 AM.
© Copyright 2008 V7 Inc

Contact Us - Internet Marketing - Top