Behind the scenes at MSN desktop search

Pandia takes a look at two online video interviews with the teams developing the Microsoft Toolbar Suite for desktop searching.

(December 27 2004) The search engine companies devlop complex relationships with their users and customres.

At the one hand they want to be open and friendly, building goodwill among researchers, librarians, search engine experts and search engine marketers.

At the other hand they must keep much of their work secret, partly because of their competitors and partly because of some of the more aggressive search engine optimizers -- people that will use any scrap of information in order find new ways of getting to the top of search results.

Information and communication channels

In general the search engines have opted for the friendly approach, being open when possible.

Hence you will find search engine employees like GoogleGuy and MSNDude taking part in discussions at search engine forums.

This year all the search engines have established their own weblogs where they present products and ideas. They are also present at relevant search engine conferences.

Google even has a public "lab" section, where people may test works in progress.

Microsoft and software developers

Microsoft is, of course, much more than a search engine company. It is first and foremost a software company, and as such it needs to get software programmers to develop new programs for its Windows operating system and related platforms.

This means a need for relevant online resources and support, but also for online communities where software developers can discuss problems and opportunities. Microsoft also needs bug reports and other types of feedback that can be used to improve products and services.

One such community is Channel 9 of the Microsoft Developers Network (MSDN), named after the cockpit communication channel.

Channel 9 is interesting also from a search engine perspective, as Microsoft recently has been using the site to present the development of their Toolbar Suite, i.e. their desktop search tools -- software for searching your own computer (see Pandia's article on MSN's toolbar suite).

Interviews with the Toolbar Suite developer teams

Microsoft actually has two Toolbar Suite teams, one in Redmond, Washington, and one Mountain View, California, and there are video interviews with both of them at the Channel 9 site.

The Redmond video presents David Dawson, Chris McConnell, the brothers Dave and Paul Steckler and Tom Laird-McConnell. In the Mountain View video you meet Bubba Murarka and Marek Gorecki.

Those of you who picture Microsoft buildings as evil deathstars inhabited by soulless storm troopers ruled by the evil Darth Gates, should definitely take a look at these videos, as they give you a rare glimpse into what's happening behind the scenes at Microsoft and what kind of people are actually working there.

These are all men in their twenties and thirties. (Yepp, no women!) They are computer geeks, for sure, but a friendly and enthusiastic bunch of people, and we wouldn't hesitate having a beer with them anytime.

They are clearly proud of the work they have done on the Microsoft desktop tools, and do not hesitate to talk a lot about the process of the developing the software and of contemporary and future features.

The needs of the advanced expert user

For instance, in one video the MSN Toolbar Suite research team underlines that they already are testing a more advanced version of the Microsoft desktop tool inhouse.

One of the things they are focusing on is the needs of the advanced expert user -- i.e. the information professional that has a lot of data, both locally on his or her computer, and on the net.

Microsoft will make it easier to restrict queries to --let's say -- emails from this or that person.

Desktop vs. web search

The Microsoft developers underline the difference between desktop search and web search, pointing to the fact that the searcher has a clearer idea about what to search for when searching his or her own files and emails.

That being said, Microsoft is clearly considering a closer integration of web and desktop search results, although they will not go in the same direction as Google.

Google is now presenting desktop search results in the same way web search results. Microsoft will stick to column view, making it easier to sort search results according to various fields, like date, type of document, sender, headline etc.

The use of computer resources

Desktop search tools do not actually search all your files every time you search your computer for files and emails. Instead they search a database containing information on your files. Obviously, the desktop search tool will have to build this database.

One of the problems with desktop search is that the software needs to make use of CPU resources to index the hard disk and then to keep the database up to date.

The Microsoft programmers admit that they continue to work on new ways to ease the burden on the computer, so that the desktop tool does not overwhelm the system too much, interrupting other processes.

For instance: Should an email be indexed right away, or within 30 seconds? The goal is clarly to get an "extremely up to date" database, meaning that new emails should be indexed as soon as possible.

Dog food users

The developers also tells a lot about how they develop and test the software, for instance by using what they call dog food users, i.e. Microsoft employees that are willing to test preliminary ("interim build" or "pre-beta") versions and that gives the developers feedback and advice.

All of Microsoft's 50.000 employees can take part in the dog food process, although not all of them take part.

Still, the developers get one mail from every five dog food user every day. Even Bill Gates might send the team an email saying: "Why does it do this?" or "I would like to have it to do that!"

"Secret" toolbar features

The Toolbar Suite teams take their time presenting various not so well know Toolbar Suite features.

One such useful feature is "search as you type". Macintosh users know this already from their hard drive search (and iTunes). Google is experimenting with a similar online web search feature.

Search as you type means that the desktop software starts showing results as soon as you have entered a few letters in the search forms. What it does is to guess what you are going to write based on the letters entered and show results accordingly.

As a matter of fact, the Toolbar Suite has a feature like this already. As soon as you start entering a word in the toolbar search form, a small window will pop up, giving a list of programmes and files that match the letters entered. Hence if you put in the letter W, the software will display programmes like Word, media files with this letter in the file name etc.

You can also use the toolbar form to enter URLs (to go to a certain web pages), or enter a command (start with an equal sign and add a command as you would in your Start Run box, i.e. =cmd to get a command prompt). or c:\ to launch an Windows Explorer windows with the C drive files and folders.

You may also make your own shortcuts, by typing @ in the search form. This will give you a shortcut tutorial. You may for instance make a shortcut called p that automatically opens the Pandia homepage in Internet Explorer.

Features that have not been included

The team decided not to index browser cache files based on input from test users. First of all cache files (i.e. local copies of web pages visited by the user) added too much noise to the results.

Second the cache files may contain passwords, private emails and other security related material, and third some users found it disturbing that other users could use the desktop search tools to find out what web sites they had been visiting.

The MSN developers have so far not included an easy way to index and search a large number of internal network servers at the same time. But they will look at the search multiple machines in the future.

The developers in Redmond and in Silicon Valley have also made a statistics feature for the desktop search tool, in essence giving you the possibility to map what search terms you use the most. This feature has so far not been implemented in the public version of the program.

One of the videos are close to one hour long, so get yourself a cup of coffee before viewing them.

List of all videos
Interview with MSN Desktop Search Silicon Valley Team (Windows Media Video File, 53 minutes)
Interview with MSN Toolbar Suite Redmond Team (Windows Media Video File, 23 minutes)
Channel 9 Wiki MSN Search Feedback
Google's own blog
Yahoo's own blog
MSN's own blog
Other desktop search tools

More search engine news...