Google continue to use human input for the evaluation of search results, Peter Norvig of Google says.

Technology Review has an interesting interview in its January/February 2008 issue. They have interviewed Peter Norvig, director of research at Google.

Norvig mentions following factors when asked about how they make Google’s accuracy better:

• They track what users are clicking on. If they click on number one and do not come back to Google, that is taken as a quality sign: the searchers have found what they are looking for.
• Google hires people to review the quality of search results for specific queries. The myth about Google’s search results being fully automatic is clearly not true.
• Google does user experience tests in laboratories and in peoples’ homes, observing searchers in order to pinpoint difficulties.
  
• Google is not interested in natural language search, to the extent that searchers should be able to enter regular queries into the search field — e.g. “What is the capital of France?”. However, Google is working on understanding the meaning of concepts, like for instance the difference between “York” and “New York”.
• Google is working on search personalization, tailoring search results to the interests of the individual (Universal Search). At the moment they are focusing on news stories, as there is a limited number of them. Doing the same on the scale of the Web will take time.
• They continue to work on the integration of different types of search results (e.g including news and blog search listings in regular web search results).
• Google is getting into speech recognition.
• Google is working on interfaces on phones and other mobile Web devices: “Where you have a tiny screen and awkward keyboard.”

Although all of these factors have been known for a long time, it is useful to get a prominent Google employee to sum them up in this way.

Using human controllers

The most interesting point for us is Norvig’s presentation of the use of human editors or controllers:

“Another way we do it is to randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment. We train them on how to identify spam and other bad sites, and then we record their judgments and track against that.”

The question is how Google makes use of this information. They may take one of two approaches:

1. They can use their input to adjust search results manually. To give an example: Several reviewers find that Site X is able to get users relevant information in an efficient way, and the site is therefore given a boost in the rankings. This is a very risky procedure, as Site X may loose quality over time. The more sites you give special treatment, the more complex the follow up procedures become.
2. On the basis of the human recommendations Google can do an analysis, trying to identify why the automatic algorithms do not reward the sites the human searchers find to be the best. Google can then adjust the algorithm accordingly.

Danny Sullivan over at Search Engine Land argues that there is nothing wrong in Google using human experts in the development of the search engine algorithm (i.e. option 2 above):

Over time, Google’s algorithms (as those with the other major search engines) have continued to change and be refined. But the one unifying element to them all is that they’re trying to do what humans would do. Would humans like pages that rank well because they are stuffed full of nonsensical keywords? No. Enter a filter for that. Would humans like fresh news content getting to the top of the results for search queries that spike in reaction to a news event? Yes — and enter a change for that.

We think Danny is right. As long as Google uses human input to improve the overall algorithm, making the search results more useful for humans, this is the obvious way to go.

As regards reward sites on an individual basis — well, that will definitely be a risky strategy. Google is using the automatic nature of the process to defend itself against disappointed webmasters in court, and admitting that it is rewarding sites on a personal basis would undermine that positions.

There is one exception to this rule, though, and that is the ability to remove sites from the index altogether. Google is clearly using human input to identify and remove spam sites and other sites that give no useful information whatsoever. We are fine with that.