11 Year Old Site on a Google Rollercoaster Looking for Answers

[bruceeisner]

This question is regard to one of my websites which is 11 years old.The site is called Mind Media at htp://www.mindmedia.com

Part of the website is a directory of websites created by a program called Hyperseek. The program creates can create either static or dynamic pages but I have chosen to create static pages for the past several years.

The program also creates the directory in two possible ways

1) All the pages of one directory or 2) separate directories for each category and subcategory. I had the site built both ways.

About two years ago, i began to notice that Google started giving good SERP to the directory. It always indexed the pages in the single directory and mostly ignored the duplicate pages in the multiple directory. On the other hand, Yahoo liked the multiple directories.
Occasionally I would switch back and forth between the two modes so that the site would appear to be build one way or the other.

About 10 months ago, i started creating a sitemap with Gsitecrawler, a program that creates a sitemap and uploads it to your site. I have submitted updated sitemaps since I started using the Google webmaster program.

My SERP improved gradually until last November 15th or so. I had a couple of server outaages and so when the results dropped, I attributed the drop to the server downtime.

But then I was on a roller-coaster. After about a month, my SERP shot up to the heights they had been at before the drop. Then they plunged again a week or so later.

I read about duplicate content. The first thing I thought of was the two directories. So I deleted the multiple directories and wrote permanent redirects from those pages to the corresponding pages in the single directory.

That 2343over a 1000 redirects. Its hard to tell what the results were because fluctuations between almost all time highs and lows kept happening.

I Begin to analyze some of the reports on the Google Webmaster area for answers. i noticed that it was showing an increasing number of pages which ended with his m/cgi-bin/search/hyperseek.cgi?ID=xxxxxxxx where xxxxxxx are numbers.

I did not understand why the Gsitecrawler was restricting some of these and went to their help forum. Then I started looking at these pages in my site:domain.com results and noticed that these pages were actually other websites. The hyperseek had disguised redirects by creating these pages for each link.

So I thought that maybe these pages are considered duplicate content. I wrote a new robot.txt that restricted Google for the cgi-bin/search/ directory and made sitemaps that matched.

I also put all the local pages back that I removed and got rid of the redirects. I figured that it was better that if the engines already knew about those earlier pages and asked for them, it was better to put it back then have all those redirects going.

Its been a couple of weeks since my last big drop and things have not improved. A lot of what i have done is guesswork so what I want to ask is if anyone knows whether each of the two types of pages that I thought might be duplicate content are actually considered duplicate content and frowned on by Google?

The most important question to me is what to do about the
m/cgi-bin/search/hyperseek.cgi?ID=xxxxxxxx pages. They how up in a site:mindmedia.com. But they never show up in search results. Is it better to restrict Google from seeing them or not.

One perspective is that something on my site was working and kept working better and then stopped. It is the content that Googlle thought were on those redirected pages or was it something else?

As you can see, I need some advice.