|
Q about tweaking my robots.txt file
Since my site is dynamic I am trying to fine tune my robots.txt file for best indexing results.
If I disallow a folder such as "includes," will this stop robots from accessing needed files for other pages? In other words, if a page from my root directory calls for a file in the "includes" folder, will the robots still retrieve it even though it has been disallowed?
|