What directory does robots.txt need to be placed in?

spidersam · 03-31-2004, 10:41 AM

What directory does robots.txt need to be placed in? In regards to allowing forum crawling....

Andy · 03-31-2004, 11:06 AM

Your robots.txt only needs to be placed in the root directory, any references to lower level directories are made from there.

However robots.txt. is an exclusion standard so in terms of "Allowing Crawling" you shouldnt need to do anything except install your forums search engine hack and link to the forum index.

spidersam · 03-31-2004, 02:02 PM

so, in the www/ folder?

I added this as a hack, but am not even sure what it does

Disallow: /forums/sutra*.html$
Disallow: /forums/ptopic*.html$
Disallow: /forums/ntopic*.html$
Disallow: /forums/ftopic*asc*.html$

MadHatter · 03-31-2004, 02:07 PM

This stops bots crawling any of the topics!
(I think, confirmation please...)

spidersam · 03-31-2004, 02:30 PM

Yes! Confirmation Please!

MadHatter · 03-31-2004, 02:34 PM

I am pretty sure that if you specify them rules for 'media partners google' that google cannot crawl any of those pages (the *'s mean anything, like any post number)

spidersam · 03-31-2004, 02:48 PM

Can someone confirm MadHatter's suspicion? I'm totally clueless...

compar · 03-31-2004, 03:46 PM

The question is what was MadHatter trying to achieve with that entry?

spidersam · 03-31-2004, 03:49 PM

Disallow: /forums/sutra*.html$
Disallow: /forums/ptopic*.html$
Disallow: /forums/ntopic*.html$
Disallow: /forums/ftopic*asc*.html$

From my understanding, this was supposed to not allow Google from crawling duplicate content...not stop it from crawling 'any' content.....but I am a supreme amatuer at this...so I was just following directions

MadHatter · 03-31-2004, 03:50 PM

Quote:

The question is what was MadHatter trying to achieve with that entry?

The guy said he added the hack but wasnt sure what it did, I was telling him what i thought it did, then I explained about the rule being specified for media partners google to be in effect for googlebot, and then explained what the *'s mean.
Thus answering the question of: Not sure what it does

MadHatter · 03-31-2004, 03:52 PM

Quote:

Originally Posted by spidersam

Disallow: /forums/sutra*.html$
Disallow: /forums/ptopic*.html$
Disallow: /forums/ntopic*.html$
Disallow: /forums/ftopic*asc*.html$

From my understanding, this was supposed to not allow Google from crawling duplicate content...not stop it from crawling 'any' content.....but I am a supreme amatuer at this...so I was just following directions

Where are you bothered about google crawling duplicate content, after all, with twice the content crawled, cached, and listed, you have twice the probability of you pages appearing in SERPS

spidersam · 03-31-2004, 03:57 PM

I don't know....I was literally following step-by-step directions.....that was one of the steps.....and that's what it said that step did.....

If I remove that, will it matter in terms of getting my forum pages indexed? That's all I'm looking to get done....

MadHatter · 04-01-2004, 02:39 AM

A Disallow: rule
Stops a bot from crawling that page...
so if you remove them, but allow google to crawl all pages then your more likely to get ranked.
I only put disallow rules in for my images dir... otherwise your images end up showing up on google image search.

compar · 04-01-2004, 06:54 AM

Quote:

Originally Posted by MadHatter

I only put disallow rules in for my images dir... otherwise your images end up showing up on google image search.

Do you think that's where Google gets images from? What stops them from lifting right off the web pages?

Fruit & Veg · 04-01-2004, 11:53 AM

eh? The images on a webpage are referenced by their url - eg. www.domain.com/images/image1.gif - how else can you get an image by not using this url? There's no backdoor.

Google obeys robots.txt so you're kosher MadHatter.

spidersam - I recommend you start again with your robots.txt file, you could be screwing things up and not knowing. Check out - http://www.searchengineworld.com/cgi-bin/robotcheck.cgi and http://www.robotstxt.org/ to learn more.

Where did you copy that bit of code from? Doesn't that page explain what you are actually doing?

compar · 04-01-2004, 12:17 PM

Quote:

Originally Posted by Fruit & Veg

eh? The images on a webpage are referenced by their url - eg. www.domain.com/images/image1.gif - how else can you get an image by not using this url? There's no backdoor.

But anybody can lift an image right off a web site. Right click and save image. So do you think Google needs direct access to the image folder to get the file?

MadHatter · 04-01-2004, 12:52 PM

Yes it does!
You see, if i set a dissallow rule saying that google cannot use anything in the images dir, google simply indexes the page without being allowed to get the images on the page served... it is like it is getting the little red x.

Quote:

But anybody can lift an image right off a web site. Right click and save image.

Yes... but that actually downloads the image... from /images/logo.jpg
it is like right click and save target as for a zip file, you are still requesting the zip file from the downloads directory!

spidersam · 04-01-2004, 01:10 PM

This topic has gotten a little off of my original intent....which is fine.......

but let simplify my question..........Do I have to have a robots.txt file in order to get all of my forum pages indexed?

Fruit & Veg · 04-01-2004, 01:55 PM

No, but your forum pages have to be able to be indexed by a search engine.

Robots.txt is mainly used for blocking rather than allowing.

spidersam · 04-01-2004, 02:01 PM

I did install a hack....by looking at my forum, could you tell if i should be good?

www.lordsrendezvous.com/forums/