Quote:
|
Originally Posted by odevimcom
I have a homework-help site with ten thousands of doc and pdf files. There's a monthly fee for membership and i protect my files from visitors via nt authentication. Anonymous access to homework files is forbidden . I open an nt account for every paid-member of the site, allowing them to download those files.
There's a link to every document on the site, so i would like to learn if Google could index this content even though they're "secured" with nt authentication. Or does this kind of operation protect the robots from crawling the documents on the server.
|
It will prevent robots from indexing it... as far as your web server is concerned, the spider is just another user.
You can of course be sneaky and look for robots by IP or user agent and setup your server to not authenticate them. But it's highly frowned upon by Google... and a very good way to get you penalized. Google wants the user to see the same thing they see. And doing that could be considered a form of cloaking to some degree...
- Shawn