Puzzled by Website's Visitor Behavior

[billybw]

My website stats includes a Latest Visitors log, and I keep seeing something on it that puzzles me. At least once or twice a day, a visitor will appear to move through all the pages on the site at a very rapid rate. By this I mean that there is only one or two seconds spent on each page, before going to the next one.

By doing some testing, I've determined that It is PHYSICALLY IMPOSSIBLE for anyone to click through the pages on the site that quickly.

Also, these are real visitors, not robots or crawlers. I know this because they come from Google and I can see the search terms that they used to find the site. Also, sometimes this type of visitor will spend a few minutes looking at two or three pages, before suddenly starting to rapidly go through the rest of the site as I described above.

Does anyone have an explanation for what I'm seeing?

Thank you

[Nicole]

Maybe they are using one of those programs that download your site for viewing offline ?

[Weizheng]

Does your site rank high in search engine?

Could be one of those keyword analysis tools. When a keyword is entered, it does a search in Google then crawls the top N sites to look for keywords in titles and meta tags.

[billybw]

Thanks for the replies.

Weizheng-- This is a non-commercial site with no ads and nothing to sell, so I doubt that anyone would be analyzing the type of keywords that send visitors to the site. Also, sometimes a visitor will spend a few minutes looking at two or three pages before the rapid movement through the rest of the site begins.

Nicole -- You're idea that people may be using a program to download the entire site makes sense. Does anyone know how common it is for people to do this on the sites they visit? Also, what is this kind of program called?

Thank you

[waffles]

Fasterfox is an extension that can do it if they're using Firefox.

[StupidScript]

You're definitely seeing bot/crawler activity. As you noted, pages requests are too rapid for it to be a human who would need to wait for a page to load before clicking on the next link.

With regards to the Google query string, it's not unusual for a bot to post a query through Google and then spider the results, including any attached codes. I've seen hundreds of visits from obvious bots that include identical Google query strings ... it's not too complicated to accomplish.

It's also not difficult to program a bot to mimic human behavior by slowly going through the first couple of pages and then speeding up the rest of the crawl.

There are any number of reasons why a rogue bot might be crawling your site, from email harvesting to anti-competitive behavior to searching for unsecure forms to exploit to gathering your pages in order to build a duplicate site that potentially holds malware to ... ? Maybe your site is just one of hundreds of thousands on a list being used by bots for who-knows-what purpose.

On the benign side, it may very well be users who liked what they saw on the first couple of pages and then use a "site-grabber" to download every page for offline viewing, as noted above.

If you are concerned about the behavior, you should check more deeply into those visits and see if there exists some unique identifier you could use to block them, like their IP address, although I'm pretty sure you'll find that the variety of IPs and User-agents will be extensive. Also, there are scripts/programs out there that can "throttle" request activity. Using the software to exclude valid bots, like Googlebot, you could restrict every visitor to, say 1 request per 10 seconds. Most rogue bot programs won't wait around, that long. They'll keep making their speedy requests, and if most of them are denied, due to the throttling, they'll finish their run and leave without all they came for.

[Taltos]

Heck, IE7 and I think IE6, all you have to do is File >> Save As and it will save the whole site in a folder. It creates a .mht file, I think.

[billybw]

I have done some more checking, and am convinced that these are real visitors, for several reasons:

1. Not all of them are sent by Google. Some come from Yahoo, and some come from a link on another site.

2. The search terms vary, and the entry pages vary.

3. In one case I noticed that the visitor waited about six hours, probably doing something else, then came back, looked at two more pages, than began the rapid movement through the site.

4. This is an environmental site about rare tree species, so there is no reason for automated crawlers to be repeatedly collecting the same information over and over every day.

[Taltos]

Sounds to me like somebody grabbing the whole site for school. Either a teacher to use it in class or a student for a paper. Just a guess but that my theory and I'm stickin' to it

Any chance you can tell the general map location by IP address? Are they from the same area? From the same school?

[billybw]

There is no way all of these visitors are coming from the same class or school. I first noticed this happening at least six months months ago, and it has happened hundreds of times since then. Also, as I said, these visitors find the site in different ways such as Google, Yahoo, an external link, etc. I just noticed today that one of them was sent by the Google Image search originally, looking for a photo of a particular tree blossom, then after about three minutes going rapidly through all the other pages as I've described earlier.

Also, in looking through some logs I've saved, a few of the visitors that do this were sent by google.co.uk and google.ca, so they must live in England and Canada.

But I am wondering why so many visitors to my site seem to be doing this. Does anyone know how common it is for people to save an entire site on their computer?

Thank you

[E.Scape]

take a good sentence from different pages of your site and quote text them in search with google, see if your content is being copied.

let us know the results and if you find copying do a whois on the domain being used for your records.

This may not be the answer but might help eliminate the possiblities.

Good luck

[billybw]

E.Scape -- I did some searches as you suggested and didn't find any other sites with any of the sentences I tested. It doesn't make sense anyway, that hundreds of other webmasters would be copying content from my site to their sites.

The only thing that seems to fit is Nicole's idea that some visitors are downloading the site for viewing off-line. After studying the data some more, I estimate that an average of about two visitors per day may be doing this, which is about 1.5% of all visitors to the site. Does anyone have any estimates for their sites that I can compare that with?

Thank you

[E.Scape]

I suggested this because it does happen and with so many "harvesting" signs It's at least good to eliminate the obvious.

I have alot of custom articles written for me and I have to check regularily for people copying my content. And as much as it doesnt make sence that people would do this for the obvious lack of attention to their sites, this is what most MFA and PPC sites run off. Even duplicated content gets indexed and as a result brings SE traffic to an ad splash page.

Good luck and let us know if you find the problem, this can help others here.

Jack

[Taltos]

Yes, I have had a couple sites that ranked very well in a particular industry that was still somewhat young on the internet. Those site got ripped off regularly and duplicated almost completely in a couple of cases. That made it pretty easy to find the stolen sites and send DMCA nasty-grams. With the exception of one Canadian site on a Canadian host that didn't both with DMCA, the rest quickly were taken down volentarily and they went on to do their own work. Which resulted in much worse SERPs than they had with my stolen site

But those were commercial sites. Yours is not so it is less likely to get copied like that. It definitely happens more than you would like to think.

[billybw]

Quote:
Good luck and let us know if you find the problem, this can help others here.

What problem are you talking about? I'm not aware of any problem. I alreadfy told you that no one is copying my site, so there's no problem there. So what are you talking about?

[Taltos]

I think he's just asking that if you figure out this weird behavior, let us know what you discover. I know I am curious now. It is kind of strange but so far appears harmless. Good luck on the hunt

[billybw]

Taltos --Nicole already gave the explanation in the second post on this thread. I don't know why it's so hard for some people here to understand it.

[Taltos]

Sorry, in later posts you didn't sound convinced that was the answer. Problems solved.

[Tanya]

OFF TOPIC


Pokes Taltos

[Taltos]

DOH! Hey, its been long enough that I don't have the chair and rope right here but their not far away! You come back here!