indexing non-public webpages

[Nealrm]

I get a report sent to me daily that has about 50-70 javascript links to real estate information. This is not information that is publicly available, so it is not indexed by anything like Google. I need to be able to index and search the linked pages so that I can find specific information quickly. I have tried saving the web pages (Didn't work), PDF files (logistically not feasible), using chrome history search (only partially successful).

Does any one know of a program that will index webpages as viewed??

[carwell]

Use firefox history box?

[techielog]

I think if it is not indexed on Google, it means the website owners might have set limitations on crawling their pages....Any respectable search engine would respect the website owner instructions and wont crawl... may be you need to go beyond...

[snakeair]

Originally Posted by techielog

I think if it is not indexed on Google, it means the website owners might have set limitations on crawling their pages....Any respectable search engine would respect the website owner instructions and wont crawl... may be you need to go beyond...

This is a report that is sent to the thread creator so obviously it's not indexed in the search engines.

Do you know of any program in which the thread creator could put the report on so he can search through it easily instead of scrolling through every single page and every single line?

[Jively]

I don;t think there's an off the shelf solution.

But with a little skill (and maybe Linux) I'd suggest whip up (or get someone else to write) a script that takes the URL's and runs them through curl (or some other command line web downloader like wget) that can save the HTML to a folder that you can then search using Google Desktop.

You could semi automate this, and if you've got control over your email setup, could pipe the output of the email directly into the script for full automation (so long as it's a unique email address).