If you are interested in the history of Fast Search & Transfer, the Norwegian search company that is now to be acquired by Microsoft, you should take a look at the blog of Bjørn Borud of Google’s unit in Trondheim, Norway.

He has now written an article series on the history of Fast, a series which opens with the following wonderful paragraph:

“About 12 years ago, me and three other friends founded a company. We were young, single and what we lacked in experience and business sense, we made up for with pure hubris and a firm conviction that all problems are solvable.”

Admittedly, this was not Fast, but another company that would become a part of the Fast adventure, but the argument holds for Fast as well. You have to believe in yourself in order to succeed as an IT pioneer. These people were the Norwegian equivalents to people like Bill Gates, Steve Jobs, Larry Page and Sergey Brin.

Creativity and innovation

Bjørn discusses the role of nontraditional students and the use of UNIX at what was then the Technical College in Trondheim.

He underlines the role of an open culture that encourages discussions and interactive learning: “The most important resource you need in order to innovate and create, is good people.”

Most of all he stresses the importance of a “can-do” attitude where people — if necessary — rip a system apart and rebuild it if it is misbehaving.

Guardian Networks

His original company was called Guardian Networks AS, and was to produce “a hardened Linux for use in firewalls”. That didn’t work, but the company survived as a UNIX consultancy.

In 1998 the Guardian boys (they are all boys in such histories) were approached by Fast Search and Transfer, a company that had bought the rights to the university based FTPSearch (a search engine used to search for files on Internet servers).

The World’s Biggest Search Engine

Fast asked Bjørn & Co to build a crawler for their new Web search engine, which, according to Bjørn, was another type of technology altogether: “Web search and FTPSearch were in fact two completely separate and un-related code-bases.”

The first version of what was called TheWorldsBiggestSearchEngine (later known as AlltheWeb) were based on the master thesis — a research prototype — of Knut Magne Risvik:

“A ten page memo on the feasibility of building Alltheweb was hammered out by Knut Magne and sent off to Espen Brodin, who then got Robert Keith to cut us a check in order to make it so.”

Wooops!

The following story says a lot about the process of search engine innovation:

Finn Arne [Gangstad] did most of the design and implementation on the first version of the crawler. The basic design and implementations were simple, tidy and quite sound. I liked it a lot. The thing was that Finn Arne had had very little exposure to web software. While he was good at writing very efficient networking and IO software, he had never bothered with HTML. So for instance, when he asked me which elements would contain URLs I told him. All of them. Including URLs in FORM elements. It didn’t occur to me why he was asking (he was writing the link extraction code).

Hilarity ensued when we unleashed the beast onto the web and people started getting empty submissions to their guestbooks. The crawler was following the ACTION URLs in the FORM tags. You only needed to type in “braindamage” and “fastcrawler” into Dejanews to survey the damage.

The size of it all

Developing a web crawler turned out to be a much harder problem than they had thought. In Bjørn’s words: ” Not only does it have to be capable of handling vast amounts of information fast, but it traverses the web — a universe filled with the craziness and laziness of a million people who create all sorts of complexities.”

Bjørn points to the fact that Web-scale systems are much more difficult to handle than Enterprise-scale solutions. The amount of traffic and data is so much larger, and you have to develop a system that can handle exponential growth.

Open source

Using open source tools is part of the solution:

“Open source gives you more control,” Bjørn says. “When problems arise, you can address them. This is far more valuable than any support contract you can get for a closed source system.”

The Lycos portal

The first Fast web search customer was Lycos: “Our search results were presented, dressed up in drag, on their portal pages… The density of STUFF on these pages was obscene. Every square millimeter was full of blinky stuff, links, advertising and even more links.”

Ah, those were the days, before Google came and ruined it all with simplicity.

Image search

The collaboration with Lycos also gave birth to Fast’s first image search engine, and Bjørn tells the story of how you use simple hacks to solve complex problems.

(See Pandia’s January 2000 article Fast gets bigger - again.)

You can read it all here:

Ancient history, Part 1: early years
Ancient history, Part 2: using open source
Ancient history, Part 3: more on the origins of FAST Search