Engineering a SE Algorithm

[awall19]

I think the point that everybody is really missing is not that

it is hard to make a quality large scale hypertextual search engine.

Google results are not some amazing miracle of relevancy above those provided by Yahoo!. the hardest part about search is branding and marketing it...thus my suggestion is to get help from those deeply entrenched into the web design and development community.

[jeanboucher]

So better if you could have a search engine that does a meta-search of Google, Yahoo, MSN, and Teoma - if you could set up something to process those results in a specific way then you would be able to choose the best from the best - Google linkpop analysis, Yahoo's on-page elements, and Teoma's clustering technology - roll in into one and you could have a real power of an engine.

http://www.copernic.com/en/products/meta/index.html

[John Scott]

Originally Posted by I, Brian

Perhaps a better summary would be - don't try to stand against giants - just climb on their shoulders.

If you see Google's SERPs, you'd see that they are still filled with pages spewed by WebPositionGold and other automated software. The whole idea is to create a better search engine, not just another search engine.

[JayM]

Thats exactly what brian said - you take the best bits of each search engine and role them into one....

In turn, creating a better search engine.

[John Scott]

Originally Posted by OptWizard

>>>Link pop over on-page definitely an important concern, as links are ultimately harder to manipulate<<<

I have to disagree with that links are easy to manipulate especially if you own tons of domains. Also make it easier for the competition to "google bomb" for a lack of a better term.

Say you own a hosting company you could just put your link at the bottom of all your clients pages, for example if i owned a 'concrete' co and owned a hosting co then i could put that at the bottom of all my hosted pages no?!

Opt, which is easier to manipulate - links or on-page factors? Keyword density is a free, links aren't. Keyword density takes half a milisecond, links take a lot of time.

According to your algo, my page on Google's branding strategy ( http://www.v7n.com/google-branding-strategy.php ) should outrank www.google.com for the search term "Google", because it has a higher keyword density for that keyword. Pretty dumb, eh?

(No go back to IHY and whine about the big bad JohnScott, like you always do.)

[JayM]

My personal opinion would be if you can find a way of effectively dealing with spam - you will already be heads and shoulders above the rest.

It's been mentioned in this thread already - but theres only so many aspects to a search algorithm. Maybe you shouldn't be asking what makes a good search engine - but what makes a bad search engine........

Working backward is probably, sometimes the best way - you can develop an amazing search algoithm, but whats the first thing people are going to do? Try and spam it - so put a stop to it before they even have chance to do it.

I'd love to see a search engine which didn't take any cr*p - if there spamming, simply ban the domain.

I think more on page factors need to play a part than googles current algo - I think they actually lose relevance because they have gone so far away from it.... Some of the serps are laughable.

Then as Aaron said - the biggest part isn't writing it, but as you know - it's promoting it. As mentioned - what is the selling point for the search engine, what makes you so much better than everyone else?

I think - if you could turn around and say - well we have actively dealt with spam - our results contain the best information and the most relevant- it would be an excellent selling point.

[thebassman]

I may get totally shot down here, but I think ask.com was on to something when you type in a question... maybe it's better to have different "types" of searches... whether you're looking for information, shopping, photos, etc... like you can almost "pre-filter" the SERPs to say what type of sites you actually want to come up... :idea:

I'm not suggesting that sevenseek has a question format, but maybe some optional "filters" that visitors can use to help narrow their search critera down.... :

[John Scott]

http://www.v7n.com/basic-concepts.php


I think Bob Massa explained one thing very well in that article. Mathematical, automated algos can always be beaten by humans. That's why I think human review is important. An aggressive anti-spam stance should be taken, and any sites which serve up or use spewed pages should be dropped forever from the index. Something Google isn't willing to do.

[thebassman]

Yeah, I think that would play a big part in my above suggestion... humans can review and categorize sites that would fit in the "filters"

[JayM]

Originally Posted by JohnScott

http://www.v7n.com/basic-concepts.php


An aggressive anti-spam stance should be taken, and any sites which serve up or use spewed pages should be dropped forever from the index. Something Google isn't willing to do.

My type of search engine - once a spammer - always a spammer - they'll never change so lets not even pretend they will by giving second and third and fourth chances.......

[awall19]

Originally Posted by JohnScott

(No go back to IHY and whine about the big bad JohnScott, like you always do.)

11

[John Scott]

Originally Posted by awall19

11

Just annoying how he always goes over to IHY and whines about me.....

[awall19]

Originally Posted by JohnScott

Just annoying how he always goes over to IHY and whines about me.....

well I do not want to complain about him complaining, but I would probably find a critic of mine who opperated likeso to be rather annoying.

[Johan007]

As for link popularity you may want to limit the number of links one site can give another. So if a site has 10 pages the most it can give is 1 link. 100 page then 2 links, 200 pages can give 3 links.

I made an advanced SE with a coder and created the user interface. It was aimed at the education industry and we had our own spider. Link popularity was not used because it was not needed. After all the only sites in the directory are non-commercial research and education sites. We chose to ditch the effort because we could not afford to spend time on it though people did use it for 2 years.

The idea was you create “research topics” such as you favourite Football team, your are of works, your favourite music by clicking in a few keywords to that group and every day you can go and check a topic or create a new one. Works well if you spider sites like CNN and BBC.

[JayM]

1 link max from each domain would be a challenge ... kinda sort the men from the boys.

[Johan007]

Yeah that would be good too cos it would also simplify and speed up the coding making links domain specific rather than page specific. With the downside being that 1 page may never be associated with a another page under the same domain but this is very rare.

[I, Brian]

Originally Posted by JohnScott

If you see Google's SERPs, you'd see that they are still filled with pages spewed by WebPositionGold and other automated software. The whole idea is to create a better search engine, not just another search engine.

But...it depends on whether you'd want to develop an algo simply as an investment, or something to actually develop into a new search engine in the first place.

Trying to start a SE from scratch in an industry already dominated by billion-dollar giants would be an absolutely huge financial risk - especially when there are a lot of engines out there that have failed to make their way in the market. Everyone's after relvancy - so why are so many failing?

Having something on the lines of a meta-search engine - taking its results from the big players and adding your own filtering elements to the processing - would be a way to get the best from the giants, without having to make a comparable investment in resources as they've had to.

That way you can use their methods - links, on-page, clustering, etc, technologies, and try and get the best from them - make the results more relevant - manual penalties on specific domains, etc.

I actually think the future belongs to meta engines, because they're less focussed on any one ranking technology, and can take the best from whomever they please.

The webpositiongold pages - I'm not sure you'll ever see such pages gone. SEO is primarily built on accessibility - my understanding of WPG is that it simply exploits the accessibility elements to most effect.

2c, anyway.

[88sahara]

any domain with more than 2 hypens = spam!! Try that, just kidding, I know some of you out there have them that are legit...but think about it, a domain name with 3 hypens may be legit, but it is really a prime web destination? Same does for really long domains.

It is a stretch, and would unfortunately penalize some worthwhile players, but it is better to kill a few legit sites to get rid of the 1+ million spam sites like that.

I also like the once a spammer always a spammer idea.

On being human reviewed: this would be a MAJOR undertaking to ramping up fast, maybe you could include everything and have a team of 100+ volunteers that rate sites on a scale, not anyone could rate the sites, but the volunteers would be able to when they came to the site vote, you probably could scroung up 100 quality, long time posters on here alone.

[JayM]

I notice that sevenseeks directory/sign up pages/etc all duplicate that of bluefinds.

I'm guessing the idea is - internet.com for a PR8 - sell directory links once again...... pretty much duplicate what worked with bluefind. I'd hardly call that a financial risk.

What gets me - this isn't a search engine - it says bang at the top of the page - sevenseek web directory.

[John Scott]

It will be a portal type deal. Classifieds, job boards, email, etc. The search engine will take a while to develop. For right now it's just a web directory.