More new search engine development

(August 11 2003) Just as we thought the world risked ending up with two major search engines, Google and the forthcoming Yahoo!/Inktomi/AltaVista/AlltheWeb search engine, the Web seems full of new search engine projects.

MSN is developing a new competitor to Google. Pandia has also written about the new wotbot and NetNose search engines.

The Kaltix search engine

Today CNET reports that a spin-off company from Stanford (the university that gave us Google) is now developing a new technology that may lead to the birth of a new search engine.

The fact that Kaltix is formed by three members of Stanford's PageRank team says a lot about the seriousness of the endeavor. It was this research environment that gave birth to Google.

According to CNET the idea is to personalize the Pagerank system, apparently so that the search engine learns from searcher behavior.

IBM's Web Fountain search technology

Gary Price reports that IBM is developing a new search engine technology called "Web Fountain". It is unclear whether the company actually will develop a brand new Web search engine or if it would like to sell the technology to others.

According to Forbes the new engine reads and understands text, and uses natural language to make correlations between words. Unlike traditional search, Web Fountain searches everything on the Web, including chat rooms.

IBM says Web Fountain applies several technologies to the data, including machine learning, probability theory and pattern recognition, "processing tens of thousands of documents per second."

The primary focus is on "identifying patterns, trends and relationships in unstructured text data stores such as: news feeds, a full crawl of the worldwide web, industry-specific data sources and company documents."

One goal is clearly to develop a tool that can be used in marketing, enabling marketers "to use the Web as a surrogate for public opinion."

At the IBM Almaden Research Center Web site we are told that Web Fountain is to "transform passive, reactive organizations and processes into more proactive and agile businesses that can sense and respond to real-time internal and external events, issues, and marketplace changes."

This probably means that companies will be able to use this technology in order to go through vast amounts of current data on the Internet, so that they get an impression of how people or markets act right now.

The Nutch open source search engine

Gary has also found an article in Business 2.0 regarding a new open source search engine called Nutch.

According to the Nutch.org Web site, Nutch is to provide "a transparent alternative to commercial web search engines".

Nutch argues that only a search engine that makes its own search algorithms public can be fully trusted to be without bias (read: Google cannot be fully trusted).

How Nutch intends to beat spammers when the algorithm is made public is not explained.

Moreover, Nutch is not willing to go for any form of paid inclusion.

How Nutch is going to finance this endeavor is unclear.

As the Nutch organization explains it: "Currently, our primary costs are creating a large demo, with billions of pages indexed, that can handle substantial traffic. This takes a lot of hardware and bandwidth, which is not cheap." Because of this the founders ask for donations and help from developers.

It is interesting to note that Overture Research has donated hardware and helped to fund development.

Why Overture is helping to fund the development of a new independent competitor to its own search engines is unclear. Our guess is that they would like to use any useful innovation as an input to its own search engine technology.

One thing is for sure: No search engine will survive without any form for revenue. The dot.com collapse demonstrated how hard it can be to survive as a search portal. Banners alone will not be enough.

The importance of new search engines

It is important to remember that Google once was an academic outsider facing search engine giants like AltaVista and Excite. Few could have guessed that Google finally would end up as the most important search engine in the world.

Most of these new upstarts will fail, or they will be bought by others in the way Teoma ended up as the search engine powering Ask Jeeves and Wisenut became a part of the LookSmart enterprise.

For the searchers this is all very well, as long as this technology development leads to (a) new, high quality search engines, or (b) the technology contributes to an improvement in already existing search engines.

It was competition that pushed the present search engine technology into the forefront. Google understood that the key to success was high quality search results that brought visitors back for more.

If any of these new innovators make a better search engine than Google, they deserve to win. They have a loooong way to go, however. The present Google and AlltheWeb technologies are the results of many years with research and experimentation.

Indeed, these new researchers may "stand on the shoulders of giants" (i.e. their predecessors), but knowledge does not come for free. It takes time to learn the ins and outs of modern search methods and theory, and Google and Overture will not give their know-how away for free.

Still, we wish them the best of luck!

More search engine news...