example of robots.txt file

[3DProf4online]

could you please provide me with the real example of robots.txt file, I would like to know how it looks like?

[StarLab]

Rather than a file which won't really explain anything...

http://www.freefind.com/library/howto/robots/

... some explanations to go along with it.

The default robots.txt that ships with Joomla 1.5 looks like this:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/

HTH

[htmlbasictutor]

www.robotstxt.org/ would be a good place to learn about the robots.txt file.

[TravisT]

I thought that wikipedia had a very good source too: http://en.wikipedia.org/wiki/Robots.txt

[3DProf4online]

where is it possible to find robots.txt file in hosting Cpanel?

[htmlbasictutor]

File Manager.
http://www.cpanel.net/docs/cpanel/File_Manager.htm

It should be in the root of your website, if you have one already.

[Sean@WMS]

Originally Posted by 3DProf4online

where is it possible to find robots.txt file in hosting Cpanel?

I don't know about CPanel, but one's robots.txt file has nothing to do with control panel, editor, etc. Simply upload a robots.txt file to your web root.

This should get you started in the right direction:

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

This is typically very simple stuff.

We recommend that you at least have a robots.txt file in you web root with at least the following content:

User-agent: *

You can then apply other rules.

NB: We haven't found that search engines, even Google, really respect the fules set in this file, so if you REALLY need to protect content, don't depend upon this; use a login of some sort.