Well, Mitra, I really don't know an easy way to say it but I'll try to explain it as better as I can.
So,
Let's say you have this web site called "
MyWebSite" and the url will be, of course "
www.MyWebSite.com". Now, in this web site you create two subdomains: one called "
MySubdomainOne" and other called "
MySubdomainTwo"
.
Your web site structure should look like this:
MyWebSite (
this is the root folder)
> index.php
> page1.html
> page2.html
> robots.txt
>
MySubdomainOne (this is a folder, but also the root folder for the subdomain 1 and contains the following files)
-- index.php
-- page1.html
-- page2.html
>
MySubdomainTwo (this is another folder, but also the root folder for the subdomain 2 and contains the following files)
-- index.php
-- page1.html
-- page2.html
Now, as you said, you don't want the SE's robots to access your subdomain's files, so your robots.txt file should look like this:
# This will allow any spider to access all the files within your domain:
User-Agent: *
Disallow:
# This will block all spiders to access any file within this subdomain
User-Agent: *
Disallow: /MySubdomainOne/
# This will block all spiders to access any file within this subdomain
User-Agent: *
Disallow: /MySubdomainTwo/
because you're blocking the access to your subdomains, the spiders will be able to access only these files:
> index.php
> page1.html
> page2.html
> robots.txt located on the root folder (See above)
Note:
You have to specify the "
noindex,nofollow" attributes in each
head section of each existent page within your blocked subdomain:
<head>
<
meta name="robots" content="
noindex,nofollow" />
.....
</head>
This way, you don't need an individual robots.txt file for each subdomain.
I hope this helps you.
Linear Mode
