Cache creator & Crawler to detect dead links on website.

[Ins]

Hi,

I have about 25000 external links on my community website resource database; how easy it is to create a crawler to automatically detect the dead links and report them back to admin for manual removal, and,

How to create and display cache for all these external links, in case the external website is down. Just like Google has.

If you can refer me to some free/cheap software or literature available, it would be great.

Thanks in advance!

Ins

[lordspace]

here is the process with a meta code

1. while has some links
2. retrieve a link from the db
3. make a head request (via, fsockopen, cURL, wget, or perl's HEAD variable)
4. if not success update this record as a dead link
5. go to 1

[Ins]

Thanks, lordspace!

[alainkun]

You can use

www.google.com/webmasters/sitemaps/ Its will check all your pages and tell you wich one is dead.

http://www.xml-sitemaps.com/ It will do a list of all your internal pages and told you witch one is dead.