Password Protection

Aisne · 08-10-2006, 12:18 AM

I would like to know, if I have a section of my website that is private and I don't want it crawled by any search engine, if I password protect that folder would it still be able to be crawled?

spyrochan023 · 08-10-2006, 11:06 AM

Just put robot.txt to your pages so that the bots cant crawl that pages

Thanol · 08-10-2006, 11:14 AM

Quote:

Originally Posted by Aisne

I would like to know, if I have a section of my website that is private and I don't want it crawled by any search engine, if I password protect that folder would it still be able to be crawled?

You can password protect it fairly easily with htaccess, if you're host is using cPanel.

sixty6 · 08-10-2006, 12:47 PM

Just create robots.txt in your root and here is an example:

Quote:

User-agent: *
Disallow: /directory/

This will prevent all robots from accessing it.

JuggoPop · 08-10-2006, 12:53 PM

the question is

Quote:

if I password protect that folder would it still be able to be crawled?

not how do I or should I use a robot.txt page...

sixty6 · 08-10-2006, 12:57 PM

Yeah I think that if you do password protect directories using cPanel or .htaccess, it wouldn't be crawled by Google (if it did, it would have to be a blank page in the cache)

If they could actually access the content, it would be a major security problem!

vangogh · 08-10-2006, 04:33 PM

I think password protection will keep spiders out. Like sixty6 said it would be a real security issue if they could still get through.

As for robots.txt it doesn't prevent robots or keep them out. It's just a file telling robots what you don't want to have indexed. It's up to the specific robot in question whether or not to honor it. Supposedly the major engines will all honor it, but given that pages I have disallowed in my robots.txt did make it into the index I question this a little.

sixty6 · 08-10-2006, 04:44 PM

Quote:

Originally Posted by vangogh

I think password protection will keep spiders out. Like sixty6 said it would be a real security issue if they could still get through.

As for robots.txt it doesn't prevent robots or keep them out. It's just a file telling robots what you don't want to have indexed. It's up to the specific robot in question whether or not to honor it. Supposedly the major engines will all honor it, but given that pages I have disallowed in my robots.txt did make it into the index I question this a little.

Yeah, it is up to them, but most robots do follow the rules!

Paul_F · 08-11-2006, 04:03 PM

Definitely don't use robots.txt.

Search engines definitely won't be able to access the content if you password protect with .htaccess. They'll receive a 403(?) forbidden header and i would imagine no search engine would bother indexing such a page. If you protect with a dynamic form script then the page may be indexed but just with "Enter userid" text or similar depending on how you set up the password script.

sixty6 · 08-11-2006, 05:09 PM

It depends on how serious it is, if you simply just don't want it indexed, use robots.txt

But if it's some big secret, I guess you would have to use cPanel...

cimmeron · 08-11-2006, 05:23 PM

my question is, if you want the content password protected, why would you want it spidered in the first place?

sixty6 · 08-11-2006, 05:28 PM

The point is with .htaccess you wouldn't have to type a pass and username EVERY time you visit it or have a visitor visit it...

with robots.txt you can keep a good amount of robots away from it and save the hassle

If you're fine with typing in a pass and having your visitor do the same, go ahead!