AllTheWeb adds updated news searching and topic oriented grouping of search results

We have said it before and we say it again: The Fast AllTheWeb search site is becoming one of the very best on the Web. On Monday November 12 Fast will add fresh news searching and other useful additions to its search engine.

November 10 2001.

This last year has probably been the worst in search engine history, with portals, engines and directories dying all over the Web. All is not gloom, however. Google is thriving, and the Norwegian search engine company Fast has become a treasure trove of innovation.

On Monday Fast is launching yet another new version of its showcase, the AllTheWeb search site. Executive vice president and general manger Rob Rubin took Pandia along for a ride, and we are glad to say that there are still search engine companies out there that use their Ph.D.'s to improve our searcher experience.

So, what's new? Well, a first glance does not reveal much. Fast has kept the lay out from the previous update, which is a good idea in our opinion. The interface is clean and easy to understand. There is nearly no distracting clutter, and they go easy on the graphics, which means that pages load fast.

Current news listings

One of the major complaints regarding Web search engine has been that the search results have been old and stale. AltaVista is struggling at the moment with an index that has portions that may be more than three months old, and the Excite database is a bit rusty, too. Google goes for a monthly update of its search engine index, but does spider (i.e. gather information from) news sites more frequently.

Fast is taking this problem very seriously, and does clearly intend to be the winner in this field. They normally spider the Web for seven days, and use another three days to digest the information, meaning that the index should be reborn every 11 days -- give or take a few.

The September 11 disaster proved, however, that this is not enough if you are searching for current news. AllTheWeb tries to solve this problem by adding three news listings at the top of the first page of search results. These listings are fetched from approximately 3000 news sites that are spidered 12 times a day, and most of these search results will be less than one hour old.

The news listings appear in a separate box near the top of the first page of results and it gives you the number of minutes since the last time the Fast spider visited the site.

This is an excellent example on how Fast is reusing technology developed for its Fast Data Search product, a real time search solution for larger corporations. This technology is used by companies like eBay and Reuters and on the scientific information search engine Scirus. This Web crawler can index 800 news articles per minute, according to Rubin.

Fast Topics result navigation

AllTheWeb has added a new feature called "Fast Topics", which groups search listings in several folders near the top of the first page of results. It's a bit similar to the topic features introduced by the new Teoma and WiseNut search engines, and the Excite zoom function.

In essence this feature helps you target your search more precisely. The category title may lead you to a group of sites that fit your needs more accurately than some of the first standard results.

Fast is basing its category tree on categories from the Open Directory Project, although some of the category descriptions are shortened a bit. The site listings found under each category, however, is fetched from Fast's own search engine index. If the search engine cannot find relevant categories in the ODP hierarchy -- for instance if your query is too specific -- it will try to group the top 200 results into new categories produced on the fly.

Query rewriting

AllTheWeb will do its best to interpret search queries in a meaningful way, i.e. it will try to rewrite your search into an optimal search, taking the search engine query language into consideration. Unlike other search engines, the new AllTheWeb will show you what it does in a separate text box in the right margin of the result page.

Rubin's favorite example is the query "Who is the Who?" This query contains nothing but stop words, i.e. small everyday words that many search engines skip when indexing pages.

Google brings up no results to this query unless you force it to take the stop words into consideration by entering a + sign in front of each and every word ("+who +is +the +who"), and even that query does not bring you information on the legendary rock group on the first couple of pages.

The Fast search engine consults a set of dictionaries when presented with a query like this, and strips off unnecessary words. This particular search query is interpreted as "the who", and due to linguistic analysis the search engine does produce search listings relevant to this topic. Among the Fast Topics one find relevant categories like "Music ->Who, the" and "Music, Townshend, Pete".

(And fans of British pop/rock will probably like to know that AllTheWeb also handles a query like "what is The The" quite well. They are very good at inventing absurd band names, the Brits. Unbeknown to Rubin, we tested names like "frankie goes to hollywood", "a flock of seagulls" and "johnny hates jazz" as well, and Fast immediately understood that we were thinking of pop groups.)

Advanced customization

Now, what if you want to turn off these features? There are probably search engine purists out there that would like to have full control of their queries. You may click on the "Repeat without rewrite" option in the "About your query" box to keep all the words in your query and to stop AllTheWeb from turning your query into exact phrases.

You may, however, also use the advanced customization feature. If you click on the "Customize" link on any page, you are brought to the AllTheWeb customization menu. There is nothing new in that, AllTheWeb has had this feature for some while now.

It has been extended, however, and made more easy to use. The various features are now divided on several pages. You may move from page to page by clicking on the "next" button near the end of the page or by selecting one of the tabs near the top.

As soon as you have finished filling out this "form", AllTheWeb will save the results in a cookie on your computer, ensuring that the same options are selected automatically the next time you visit this search engine.

Among the options available are:

Advanced Boolean searching

Alas, advanced Boolean searching (i.e. search term like AND, AND NOT, OR etc.) is still not supported. Rubin states, however, that Fast is very much aware of the needs of professional searchers and researchers and their need to build more complex search queries.

We got the impression that Fast's researchers and engineers already have developed such a search option. It will be implemented in the next AllTheWeb update. We certainly look forward to that.

The 2 billion page index

In August representatives of the Norwegian research team of Fast told the press that Fast would have ready a new 1.8 billion page search engine index by October, making Fast the biggest search engine in the world.

Actually, the new Fast spiders have started building this new database. However, it took Fast three months to consolidate all data in the company's Sacramento unit according to Rubin. They are now working on a new compression technology. "The challenge is now to get to the 2 million object goal within three months from now," he says to Pandia.

Rubin underlines that by "objects" he means regular files, not "link objects", i.e. links that are registered by the search engine, but not indexed by the spider. The reason is, of course, that Google counts such links when calculating the size of its database.

Fast has now some 625 million webpages and other files in its index. Google has 1.6 billion, out of which 1 billion are fully indexed.

Fast now uses three different types of spiders to fill its database:

A worthy contender to the search engine throne

By introducing these improvements the Fast team has clearly demonstrated that they are the masters of one of the very best search engines on the Web. The quality and relevance of search results is continously improving, partly because of the introduction of new anti-spam technology. When Internet searchers realize this, Google will have to face some stiff competition.

READ ON>>> More about the future of Fast and AllTheWeb!

Fast AllTheWeb
Fast Acquires Elexir GMBH To Improve Search Relevance through Deployment of Sophisticated Linguistic Techniques (press release)
AlltheWeb.com is Twice as Fresh as Google and has Six Times the News (press release)
Fast's New AlltheWeb Search Service Garners Industry Acclaim (Excite News)

More search engine news...

The Pandia Post is our free bimonthly newsletter on Internet searching, search engines and directories.

We will give you the latest news from the world of Internet exploring, useful tips on how to improve your searching skills or search engine ranking, as well as information on the development of the Pandia Search Central.

We will never give your address to any other company or organization.