Pulling content from search engine results...

JoeTMuse · 02-14-2008, 11:53 AM

Hello all! New to the forum, and I could use some help... I need to compile a list from search engine results. Is there any way to do this? Thanks for the help...

@

pinkfluffybunny · 02-14-2008, 01:26 PM

You mean like your current positions, gain loss etc.

Webposition pro

JoeTMuse · 02-14-2008, 01:46 PM

No, more along the lines of pulling actual search results and compiling a list of those results in an easy on the eyes form. Example: searching for Bars/Night Clubs in Michigan on Yahoo Yellow Pages, compiled into a list of these without all the other clutter. I don't know if these is even possible... Thanks for any help though...

@

nasty.web · 02-14-2008, 02:19 PM

Use Seo4FireFox extension by Aaron Wall. It has ability to save search results to CSV file.

JoeTMuse · 02-14-2008, 02:36 PM

Thanks nasty. Is there anyway to make it into an automated system? Instead of saving them for database purposes, I need them to be "live." In essence, I choose the search engine, tell it what to search for, and it fetches the results only including the info I want from the link... including all the following pages of said results. Whew... That was a mouthful. I know it's gotta be a pretty complicated process, and I can't quit scratching my head over it. Thanks again.

@

nasty.web · 02-14-2008, 02:40 PM

Scraping is against Googles TOS as I know. You (your IP) can get banned (I know this from my own experience

). Anyway, it's pretty build a scraper with some programming skills. If you know PHP, look at curl functions.

JoeTMuse · 02-14-2008, 02:48 PM

Thanks nasty, I'll take a look at curl functions. As far as scrapping, what if it's for private use? Nothing published, or being redistributed across the web...

@

nasty.web · 02-14-2008, 02:53 PM

From: http://www.google.com/support/webmas...n&answer=35769

Google does not recommend the use of products <...> that send automatic or programmatic queries to Google.

JoeTMuse · 02-14-2008, 03:01 PM

Well, that sucks! I guess, unless I wish to be banned from google, my little project is over. Although, after reading the link you posted, I think what I'm trying to do is different. Maybe I should just ask google for their point of view? lol Thanks for the help either way.

@

nasty.web · 02-14-2008, 03:05 PM

You can try to imitate human behavior. Don't scrape to much at a time (think about user with few browser tabs open) and make a reasonable breaks between requests.

JoeTMuse · 02-14-2008, 03:11 PM

What about an applet that would run from my end,as apposed to server side, pulling and placing the data in a spreadsheet? Would that be considered scraping, or would that essentially be what Seo4FireFox does?

nasty.web · 02-14-2008, 03:15 PM

What's a difference between your computer and so called server? IP address? Google has many visitors form different addresses and it bans for a suspicious behavior not for technology used.

JoeTMuse · 02-14-2008, 03:22 PM

Ahhh... I see says the blind man. That makes a lot of sense. Thanks for all the help, you've saved me months of headache now that I know it's not allowed.

@

nasty.web · 02-14-2008, 03:28 PM

As I said before, you can try to simulate human behavior. If you need small amounts of data - it will work.

JoeTMuse · 02-14-2008, 03:33 PM

I could do it in small batches. I'll give that a try. So you think cURL's the way to go?

@

nasty.web · 02-14-2008, 03:40 PM

Quote:

Originally Posted by JoeTMuse

I could do it in small batches. I'll give that a try. So you think cURL's the way to go?

@

yes, see: http://php.net/manual/en/ref.curl.php

JoeTMuse · 02-14-2008, 03:48 PM

Sweet! Thanks nasty, 'preciate it!

nasty.web · 02-14-2008, 03:53 PM

Quote:

Originally Posted by JoeTMuse

Sweet! Thanks nasty, 'preciate it!

My pleasure

DaGrip · 02-14-2008, 04:51 PM

I think there's a bit of confusion here about banning and what that word means in this context.

If you have a script on your website which is doing the scraping, there's a risk of the site being deindexed or 'banned'. Of course there are legitimate ways of doing it. Look into the Google API - they may not want you scraping their results but they do provide you with ways of using their data.

If you have a script on a separate domain which is doing the scraping and eg saving the results to a file or db for the main site to access. The domain doing the scraping may be deindexed but since that domain is just being used for the scraping and running other scripts who cares. Since there's no connection from the SEs viewpoint between the scraping domain and your site, there's no danger to your site.

If a search engine receives too many requests in a given period from the same IP address it is usual for the SE to place an automated temporary (usually 24 hours or less) ban on that IP address from making requests and receiving data - this has nothing to do with search engine indexing it's just a limit on the number of requests an IP can make.

If you are running a tool from your desktop it is your own IP that may be banned. If you're running a script on a server then it's the IP address of that domain which may be banned. Either way using proxies gets around it.

It takes a very high volume of requests to get an IP ban. Simply putting a few seconds delay between requests is usually enough to avoid it.

JoeTMuse · 02-14-2008, 05:21 PM

Thanks for the info. That's a definite help.

@