Robots.txt Question

Calisonder · 05-04-2007, 05:33 PM

i understand that the robots.txt file must be placed in the root directory,
i.e. www.yourdomain.com/robots.txt

and won't work if u place it in a sub directory,
i.e. www.yourdomain.com/subdirectory/robots.txt

------------------------------------------------

But I want my site to be indexed, i just have a sub directory that i use to host verious unrelated files that i do not want indexed.

How can I accomplish this?

thanks

jumpenjuhosaphat · 05-04-2007, 05:36 PM

User-agent: *
Disallow: /folder

replace "folder" with your directory name.

Calisonder · 05-04-2007, 05:49 PM

thanks, i just figured it out about a min. ago.

aram535 · 05-04-2007, 07:55 PM

If at all possible make sure you also lock the directory with an empty index.html in the least and .htaccess file at the most.

Posting a directory in robot.txt leads "people" to look inside that directory to see what's there.

If you have no link to a directory and the directory that "this" directory is under is locked by a index.html or .htaccess file the robots will never find it unless you tell them where it is.

Aram