The Pandia Post Newsletter No. 11

Another one bites the dust

The dot.com era may be over, but the excitement certainly does not die in search engine circles. 2001 has been a frenzy of death and innovation.

Infospace has now bought the Excite.com portal from the bankrupt Excite@Home company. However, it has not bought the Excite search engine. Instead Infospace will use its own Dogpile metasearch engine and the Overture pay-per-click search engine to provide results. This means that Excite will end up as Go and NBCi, a shell intended to overwhelm faithful users with pay-per-click results until they discover what has happened and start using Google instead.

Please note, though, that we have nothing against metasearch engines. We have one ourselves. Nor do we in principle argue against paid results. If you are looking for something to buy, the yellow pages is a sensible place to start. In the case of Dogpile, however, a large majority of the result listing are paid text ads. Unless you are looking for something to buy, it will not provide you with a decent search experience.

We may be wrong, but it seems to us that Pandia Metasearch and Vivisimo are the only metasearch engines these days that do not include paid results on purpose.

Pandia Metasearch: http://www.pandia.com/metasearch/
Vivisimo Metasearch: http://vivisimo.com/
Dogpile: http://www.dogpile.com/
Excite: http://www.excite.com/

Search engine innovation

Although 2001 has seen the end of many loved old timers (Snap/NBCi, Infoseek/Go and Excite), this has also been the year of intense search engine innovation.

In an interview with Pandia Fast executive vice president and general manger Rob Rubin revealed that the company will launch a new and much bigger search engine index early next year containing some 2 billion documents. Fast also launched several new features at its AllTheWeb search site, including fresh news results.

Google obviously took notice of this. Like Fast, Google now searches top news websites throughout the world in order to find news stories relevant to users' queries. News results are presented to users in a "News" section at the top of search results pages. In addition, Google refreshes millions of web pages every day (Pandia included) to ensure that Google users have access to the most current information.

For a long time search experts have complained about the slow updates of search engine indexes (Excite and Webcrawler have been notorious in this respect). Google and Fast have listened.

Google also noticed Rubin's announcement of a future Fast 2 billion document index, and has now launched a search engine database containing no less than 3 billion documents. However, careful readers should note that we are no longer counting webpages only here.

2 billion are webpages (25 percent of which are non-English language web pages), 700 million are Usenet posts (i.e. newsgroup discussion messages), and 330 million are images. Out of the 2 billion webpages, some 0.5 billion are not fully indexed, i.e. they are only registered as links from other webpages. In this perspective Fast's objective of indexing the text of 2 billion "true" documents remains impressive.

Unlike the Fast search index, however, Google Web Search also offers users the ability to search for numerous non-HTML files such as PDF, Microsoft Office and Corel documents. We won't be surprised if Fast makes a similar move soon -- the company certainly has the technology needed to do so -- but at the moment Google is the only regular search engine to offer Adobe Acrobat PDF-files in its search results. That is important, not at least because a lot of government agencies and research institutions distribute information using this format.

Does the inclusion of 3 billion documents improve the quality of search results? Probably not. There is a psychological element that should not be underestimated, though. The search engines need to show users and investors that they are innovative and in front. Numbers like these are more impressive than long texts about new and improved algorithms.

On the other hand, Google has already shown that the search engine is able to produce relevant results, and the inclusion of more documents will make it easier to find more obscure pieces of information.

Why is it that Google and Fast succeed, while Excite dies? One possible reason is that they continue to invest in clever engineers and passionate PhD's Yes, companies like these need good economists that understand the need for revenue, but unless you can present a high quality service, it doesn't matter how many pay-per-click ads you include in the results.

Changes at Pandia:

Pandia Awards 2002

In March next year Pandia is going to present the second round of Pandia Awards to sites that excel in Internet searching, Web search guidance and search engine optimization information.

The following awards will be presented (2001 winners in brackets):

Best all round search site (Google)
Best professional search site (AltaVista)
Best metasearch site (first year)
Best site on searching (Search Engine Watch)
Best site on search engine optimization (first year)
Best search engine discussion forum (Webmaster World)
Best publication on searching (Alan M. Schlein)
Best publication on search engine optimization (first year)

The awards for this year are presented at http://www.pandia.com/post/007.html#award

Pandia needs your help in choosing the best search site for the 2002 awards. Please give your vote using the form to the right, then use the "Back" button to get back to this page.

The poll can also be found in the left hand margin of the Pandia home page: http://www.pandia.com/

If you feel strongly about who should win these awards, do not hesitate to send us a nomination (see contact page at http://www.pandia.com/info/contact.html). Please tell us why you think your nominee should win.

AltaVista adds search trends page

AltaVista has added a new column to its press section. AltaVista Search trends highlights interesting contrarian trends and presents various search and query statistics.

In this months issue AltaVista notes that the terrorist attacks has caused the number of searches for anthrax to outstrip those for the two greatest health hazards, heart disease and cancer.

On the bright side, AltaVista observes that Harry Potter is now more "popular" than Osama bin Laden. There is hope for the world yet.

AltaVista search trends http://www.altavista.com/sites/about/trends
Google has a similar service at http://www.google.com/press/zeitgeist.html

Inktomi adds new international functions

Most search engines will offer you the choice of restricting searches to languages, but what if you want to limit your search to a certain region? The fact that many sites world wide use generic domains like .com and .org means that it is not sufficient to sort by web address country codes.

By examining the link structure of the Web, Inktomi, which provides search results to sites like MSN and HotBot, is using contextual clues to identify content of interest to users in a particular region. In this way the search engine may distinguish between content from countries that share language but are culturally different.

Inktomi uses one database to serve all languages, including Japanese and Chinese.

By the way, search engine expert Gary Price reports that Terra Lycos is preparing a new European version of the HotBot search portal. That is interesting, because the lack of traffic from HotBot.com gives the impression that the brand is on its deathbed.

Inktomi press release: http://www.inktomi.com/new/press/2001/websearch.html

Google launches Google Catalogs

Google is beta testing a search engine for searching the content of more than 600 catalogs from American mail order retailers. Google scans printed copies of the catalogs and automatically converts the text portion to a format that can be searched. Google then uses the same algorithms that power Web search engines to enable search the catalog content and ensure that the most relevant catalog pages are presented first in your results.

Google notes that the scanning and character recognition processes are not perfect, so you may notice occasional errors in page numbers or encounter pages that are hard to read.

The catalogs are included for free, but Google is clearly looking for some additional ad revenue here.

Google Catalogs http://catalogs.google.com/

Go to the next page: On searching the Usenet
and the new Gripe.com search engine >>>

Sign up for our free newsletter today!