Quote:
Originally Posted by Natural Elements
I disagree with this tag, because it means you must leave the pages live. Pages live mean the page could be considered as duplicated, for instance the menu. If you are using a dwt template, you cannot remove the menu code for the page not used anymore, but leaving a page for this tag, no thanks.
The best way to go is robot.txt or webmaster tools.
|
I also use robots.txt to disallow some pages from spider visiting. But it seems to me there is a difference between those two options for disallowing. As far as I noticed, robots.txt file stops the robot from visiting listed URLs, but it's still continue listing within SERP for a little while. Another words, not wanted pages will remain in cache for a while. So, the tag "unavailable_after" looks like get rid of this cache at the day you specify. Correct me if I'm wrong.
So it's probably better to have both options. That's my opinion, but I could be wrong.
Have agreat day,
Peter