Fast develops new and larger search engine index

August 27 2001

The Technology Director in the Norwegian search engine company Fast Search & Transfer ASA, John M. Lervik, has told the Norwegian Net site digi.no, that Fast will launch a new and much larger version of its search engine index this fall.

As the name of their own search site indicates, the goal is indeed to map "All the Web". The number of listed pages will be tripled to 1.8 billion webpages. Fast is aware that experts believe the Web has more pages than this. However, Fast will throw out duplicates and "junk" (probably doorway pages and pages with little or no text).

In order to achieve the new goal, Fast researchers in Trondheim, Norway, has developed a new spider that might even crawl dynamically produced pages (i.e. webpages produced by programs or scripts). Traditionally search engine spiders have avoided this kind of pages, out of fear of becoming trapped in an endless loop of new webpages.

However, Lervik adds that this does not mean that Fast will index all possible pages produced by sites like the online store Amazon.com. "But we will let the users find Internet-pages that only demands a click or two in order to bring you to your goal," he says.

The new crawler will start working in October.

"The number of pages is a very important factor," Lervik says. "Many users are searching for exact information. Moreover - in lack of any other quantifiable measures, the number of searchable pages becomes an important factor when users choose what search engine to use."

He is probably right; the psychological dimension to the number of pages race should not be underestimated. We would not be surprised if Google and Inktomi try to reach Fast's goal before Christmas.

Fast is also working on improving the quality and relevance of results, and would like to present the freshest search index available. Now new and updated webpages are registered every tenth day in average.

The Fast search engine is now used on practically all of the Lycos search sites around the world.

Fast has also had some success on the search engine technology market recently, as the famous online auction house, eBay, has decided to power its search facilities with Fast technology.

Fast skal dekke hele nettet til jul (Digi.no)
Fast company
All-the-web search engine

More search engine news...

The Pandia Post is our free bimonthly newsletter on Internet searching, search engines and directories.

We will give you the latest news from the world of Internet exploring, useful tips on how to improve your searching skills or search engine ranking, as well as information on the development of the Pandia Search Central.

We will never give your address to any other company or organization.