Crawl-Delay – What,s the meaning of this?
I hope, you all must listen about Crawl delay. Lots of webmaster use “crawl-delay” word in the robots.txt file. I searched about this, on Google and got lots answer. But, I did not get any answer, which satisfied to me. So, I made topics on this and posted here.
Sometimes, you have seen this Line in robots.txt file.
Can someone explain? What is the meaning above line?