Robots.txt question

Thanol · 06-04-2005, 09:01 AM

It's been a while since I've had to write one, but if I put Disallow: ?o= it disallows everything that starts with ?o= correct?

WhatiFind · 06-04-2005, 03:49 PM

Don't know for sure, but I believe it goes something like this: Disallow: *?o=
check this page for more info: http://www.robotstxt.org/wc/exclusion-admin.html

John Scott · 05-30-2006, 05:43 AM

I have never done a robots.txt for a subdomain. Am I correct is assuming that the subdomain gets its own robots.txt?

Say I wanted to exclude http://directory.v7n.com/cgi-bin/

How would I write that up?

Louis · 05-30-2006, 06:03 AM

What's exactly the purpose of a robots.txt? I have an idea of what it do but I'm not grasping the bigger picture.

What would/could it bring me on my domain for example?

John Scott · 05-30-2006, 06:15 AM

The robotx.txt file is the first file a spider visits, and you can tell the spider where to not go, etc.

Louis · 05-30-2006, 06:20 AM

Yes, but what can it bring me?

John Scott · 05-30-2006, 06:21 AM

Fame and riches?

Louis · 05-30-2006, 06:25 AM

Hmmm, Is that how you did it John?

John Scott · 05-30-2006, 06:28 AM

LOL.

I wish it were that easy.

Kalina · 05-30-2006, 06:34 AM

I think John just finds a rich girlfriend.

Sabeur · 05-30-2006, 07:06 AM

Quote:

I think John just finds a rich girlfriend.

where? / Who? / What? / When?
--
Is there away to know if robots.txt works? because i use for most of my websites.

John Scott · 05-30-2006, 07:33 AM

Quote:

Originally Posted by Colleen

I think John just finds a rich girlfriend.

LOL. I wish. *Dreaming*

lordspace · 05-30-2006, 07:52 PM

Hi all,

JohnScott, I think that you have to have a robots.txt file in the document root of virtual domain in this case: directory.

e.g.:
in /home/v7n/www/directory/robots.txt

User-agent: *
Disallow: /cgi-bin/

Svet

Buskerdoo · 05-30-2006, 08:01 PM

I'd also like to block a se and it's been awhile since I've set one up.

How would I block google from a subdirectory?

COBOLdinosaur · 05-30-2006, 09:25 PM

Quote:

How would I block google from a subdirectory?

the Google spider uses the user agent id of Googlebot. It will comply to instructions in teh robots.txt if you follow the stadards for robot exclusion:

User-agent: Googlebot
Disallow: /your directory

Detailed control instruction for Google spiders are here:

http://www.google.ca/support/webmast...y?answer=35303

The Google spiders are well behaved if the site does not have a lot of crap and script generated links. AFAIK Google bot is the only one that will obey rel="nofollow" on a link and that gives you very detailed control of what they index.

Cd&

John Scott · 05-31-2006, 12:37 AM

Quote:

Originally Posted by lordspace

Hi all,

JohnScott, I think that you have to have a robots.txt file in the document root of virtual domain in this case: directory.

e.g.:
in /home/v7n/www/directory/robots.txt

User-agent: *
Disallow: /cgi-bin/

Svet

Thanks Svet

WagerX · 05-31-2006, 12:51 AM

Even though googlebots behave most of the time.. I've had a page listed even though I have excluded it in the robots.txt file.(There since website inception). Here's the case:
Exclusion: >>http://www.paidx.com/robots.txt<<
Webpage listed: >>http://www.paidx.com/affiliate/index.asp<<

It even has a PR2 ??

I have only 1 inbound link to that page from a PR3 page.

So does that mean that inbound links force robots to ignore exclusions in robots.txt??

lordspace · 05-31-2006, 07:15 AM

Hi WagerX,

Quote:

So does that mean that inbound links force robots to ignore exclusions in robots.txt??

It seems is possible to have you PR but this page is not shown in results.

if you search for your domain in google only main page is shown, so r