Webmaster Forum

Go Back   Webmaster Forum > Web Development > Web Design Lobby

Web Design Lobby Forum for general web design issues not specific to scripting or graphics.


Reply
 
Thread Tools Display Modes
Share |
  #1  
Old 05-05-2008, 12:24 PM
3DProf4online 3DProf4online is offline
Contributing Member
 
Join Date: 02-07-07
Posts: 224
iTrader: 0 / 0%
example of robots.txt file

could you please provide me with the real example of robots.txt file, I would like to know how it looks like?
 
Reply With Quote

Advertisement

Advertisement

  #2  
Old 05-05-2008, 07:40 PM
StarLab's Avatar
StarLab StarLab is offline
v7n Mentor
 
Join Date: 10-29-07
Location: Ontario, Canada
Posts: 3,444
iTrader: 0 / 0%
Rather than a file which won't really explain anything...

http://www.freefind.com/library/howto/robots/

... some explanations to go along with it.

The default robots.txt that ships with Joomla 1.5 looks like this:
Code:
User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /xmlrpc/
HTH
 
Reply With Quote
  #3  
Old 05-05-2008, 11:25 PM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Administrator
 
Join Date: 10-29-07
Location: Canada
Posts: 26,710
iTrader: 5 / 100%
www.robotstxt.org/ would be a good place to learn about the robots.txt file.
 
Reply With Quote
  #4  
Old 05-06-2008, 10:53 AM
TravisT TravisT is offline
Contributing Member
 
Join Date: 09-19-07
Posts: 77
iTrader: 0 / 0%
I thought that wikipedia had a very good source too: http://en.wikipedia.org/wiki/Robots.txt
 
Reply With Quote
  #5  
Old 05-07-2008, 09:36 PM
3DProf4online 3DProf4online is offline
Contributing Member
 
Join Date: 02-07-07
Posts: 224
iTrader: 0 / 0%
where is it possible to find robots.txt file in hosting Cpanel?
 
Reply With Quote
  #6  
Old 05-07-2008, 09:54 PM
HTMLBasicTutor's Avatar
HTMLBasicTutor HTMLBasicTutor is offline
Administrator
 
Join Date: 10-29-07
Location: Canada
Posts: 26,710
iTrader: 5 / 100%
File Manager.
http://www.cpanel.net/docs/cpanel/File_Manager.htm

It should be in the root of your website, if you have one already.
 
Reply With Quote
  #7  
Old 05-07-2008, 10:10 PM
Sean@WMS's Avatar
Sean@WMS Sean@WMS is offline
Contributing Member
 
Join Date: 04-11-08
Location: Eureka, CA USA
Posts: 137
iTrader: 0 / 0%
Quote:
Originally Posted by 3DProf4online View Post
where is it possible to find robots.txt file in hosting Cpanel?
I don't know about CPanel, but one's robots.txt file has nothing to do with control panel, editor, etc. Simply upload a robots.txt file to your web root.

This should get you started in the right direction:
Code:
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
This is typically very simple stuff.

We recommend that you at least have a robots.txt file in you web root with at least the following content:
Code:
User-agent: *
You can then apply other rules.

NB: We haven't found that search engines, even Google, really respect the fules set in this file, so if you REALLY need to protect content, don't depend upon this; use a login of some sort.
 
Reply With Quote
Go Back   Webmaster Forum > Web Development > Web Design Lobby

Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is in your robots.txt file? m42 Web Design Lobby 4 05-05-2008 01:49 PM
Q about tweaking my robots.txt file RawDepth SEO Forum 2 04-13-2008 06:01 PM
Google and Robots.txt file noob_0001 Google Forum 0 04-01-2005 09:54 PM
Working with the robots.txt file Pannu Marketing Forum 0 05-24-2004 08:10 AM


V7N Network
Get exposure! V7N I Love Photography V7N SEO Blog V7N Directory


All times are GMT -7. The time now is 07:44 AM.
Powered by vBulletin
Copyright 2000-2014 Jelsoft Enterprises Limited.
Copyright © 2003 - 2018 VIX-WomensForum LLC