The Search Engine Scene In 2015

Pandia is currently celebrating its 8 anniversary; our site is a very old one in Internet terms. However, we are not going to mark this occasion by writing long articles about what has happened since December 1998. Instead we are going to tell you what is going to happen during the next eight years.

Due to a freak accident, Pandia's newsletter mailing service opened a digital wormhole in cyberspace in January 2015, sending us a very interesting issue of the Pandia Post. We are going to reprint several of the newsletter articles included during the next two weeks. An here's the first one:

Search engine trends from 2007 to 2015

Pandia would like to wish all its readers, listeners and viewers a happy new year!

2014 was definitely an exciting year for searching, and 2015 seems to become even more important. In this issue of the Pandia Post we will discuss the major developments in the online search sphere for the last eight years, and take a look at possible future developments. Let us get right down to the basics:

Google still going strong, Yahoo and Microsoft tie the knot

We have seen an impressive development of search technologies during the last eight years, and we will come back to the major innovations later in this newsletter. Still, it should be noted that the old fashioned text search we learned to love in 1990′s is still around. Google will still feed you blue links on a white web pages, HTML style, and people continue to click on text ads.

A searcher visiting from the year of 2007 will still find most of its his or her favorites. Google, Yahoo! and Ask are still there, and the only major newcomers on the European and American scenes are Chinese Baidu and Scandinavian

However, MSN Search aka Windows Live Search  is no more. It didn't have to go that way. When Microsoft bought Yahoo! in 2008, they did actually discuss making Yahoo! a subsite of Windows Live and abandon the Yahoo! search engine technology.

Fortunately, experts convinced them otherwise. The experts pointed out that this would be the same mistake as Yahoo! had done when taking over AlltheWeb and AltaVista in 2003. At that time Yahoo! abandoned the best technology (AlltheWeb) and kept the weakest one (Inktomi).   Microsoft decided to keep  best search engine (Yahoo!) and discontinue Windows Live Search. The Microsoft team of search engineers became part of the Yahoo! Search team in 2009.

The fact that Yahoo! was kept as a separate identity turned out to be a great success. The new engineers and researchers from Microsoft found a new home where there was more room for creativity than in the Microsoft mastodon, and together the two teams managed to face some of the major problems search engines where facing those days: the size of the Web and the horrible amount of search engine spam.

Personalized search

Like Google, Yahoo! started focusing on personalized in earnest around 2007. Any user willing to sign up for an account  soon found that the search engine started to learn from his or her search behavior, identifying themes, topics and web site neighborhoods of particular interest. Relevant pages were given a boost in his or her search engine results.

At the same time the search engines started to weed out sites and pages he or she did not find interesting (for instance by measuring how much time the person spent on that page).The information gathered from registered users was also used to influence the results for non-registered users. Needless to say, scraper sites and pages of low quality found it hard to compete under such circumstances.

This development led to a renewed interest in so-called white hat, organic search engine marketing. It made no sense to aim for at "top 10 ranking" anyway, for the site that ranked as No. 1 for Betty, would rank as No. 42 in Kim's search results. Instead the goal became to generate traffic and sales - which, of course, should have been the main objective all along.

Social search

The search engines also found other ways of identifying high quality pages that went beyond the "PageRank" methodology developed by Google in the 1990′s (i.e. where Google calculated the quality and relevance of a page based on the number and quality of inbound links).

Around 2006-2007 Yahoo! started in earnest to make use of information gathered from its two bookmarking services, MyWeb and (they were merged into Yahoo! in 2009). Sites that were favored by users of these services began to see a slight improvement in rankings.

Webmasters realized this, of course, and the black hat camp started spamming these services. However, by 2009 Yahoo! was getting so good at identifying "unnatural bookmarking," that it managed to reduce the problem.

Google bought Magnolia and Simpy in 2007 (both were immediately merged with Google Bookmarks), and Ask acquired Bluedot in 2008. Now all the big three had efficient bookmarking services that could deliver input to search algorithms.

This did not mean that they abandoned the old ways of deciding page rank. On-page factors like keyword density and use of tags remained important, as did the link structure of the web. However, the search algorithms had now become so complicated that it became impossible for search engine marketers to "reverse engineer" them with any kind of accuracy.

Yahoo! managed to get through its 2006-2007 crisis, mostly due to its expertise in the social arena. Yahoo! had for a long time been better than Google in community building and Google's acquisition of YouTube in 2006 did not change that. Yahoo! (now a part of Microsoft) could use this expertise in developing and buying even more social services, making its own visitors create even more searchable content.

Moreover, Google's acquisition of YouTube did not become the success it had hoped for. YouTube experienced a tremendous growth in 2006 and 2007, mainly thanks to teenagers viewing "cool" video clips. Unfortunately, the attention span of a modern teenagers is shorter than an MTV video (now on average lasting 1.50 minutes), and they soon moved over to new and more fashionable places to hang out (triple X in 2009, monsterSUSHI in 2011 and Pluto the Dog in 2012).

Now, of course, teenagers have abandoned the Web all together, having moved over to virtual realities like 3DLife and Otherland. YouTube has evolved into a site for news-clips, 2010′s comedies and live family snap shots.

Office online

Google had a tremendous success with its online office strategy, however. Google was not the first in this arena. As early as in the 1990′s Microsoft and Yahoo! had presented online mail services (Hotmail and Yahoo! Mail) whereby users could read their mail from any computer in the world.

Google followed up on this paradigm with a large number of online services: mail (GMail), word processor (Google docs), spreadsheet (ditto), web site designer (Web Creator), web analytics, personal discussion forums (Orkut and Groups) and more. Most of these were available by 2006, but they were improved significantly in the years to come.

Moreover, as broadband became "broader" and more ubiquitous services were added. In 2007 Google added Gdrive (free online storage), in 2008 Google Presentations (a Powerpoint killer letting you use your browser for presentations), Google Podcaster in 2009 (for recording and distributing sound files) and Google Image and Video in 2010 (a photo and video editor).

All these services were funded by advertising. For a fee users could also upgrade to better and more advanced services and get rid of the ads.

(This development was not without controversy. On the 2010 New Dehli Webmaster World conference a Google representative let slip that they were using the content of personal mails and files to influence the search results of Google subscribers. This was another factor in the ongoing struggle to provide more accurate and spam-free SERPs. Some bloggers presented this as a violation of people's privacy: "Google is reading your mail!" It took some time for Google to explain that no, they were not reading anyones mail, but yes, they did use automatic software to spider the content of mailboxes and file directories in order to map the interest of the user.)

Given that all of these Google office services could be used through any browser in any operating system, Microsoft realised it was in big trouble. Not only could you now perform nearly any regular office task without the Windows operating system. You did not have to buy Microsoft Office either.

By 2009 several computer manufacturers had started selling cheap office LINUX PCs that hid the complexity of LINUX for the user, booting directly up in a free version of the Firefox or Opera browsers, with all the relevant Google services presented in separate tabs. By 2011 you didn't even need a PC. Who needed Vista?

Microsoft tried to meet this challenge with Windows Live, but they botched it, due to a confusing use of  brands, and the fact that they had given Google such a head start. By 2011 45 percent of American small- and medium sized companies had switched to Google Tools (as the package was called now), some 8 percent were using Yahoo! Toolbox and 5 percent other competing online office services.

Search 2015: When media equals the Internet

In part two of our foresight exercise we look at the role of new media in 2015. In a few years time web search expanded into these new media territories.

The merger of Internet and media

Microsoft lost the control of the office tool market to Google, who moved everything from wordprocessors to graphics software to the Web. This development did not mean the end of Microsoft's Internet presence, however.

In 2007 Microsoft did the right thing: It turned the Xbox gaming computer into an advanced media station. Microsoft had seen that the Internet, radio, TV and film distribution was on the point of merging, and decided that the Xbox was the right tool for delivering such content to the home. They were right.

By 2012 TV and radio as we had known them were dead. Only pensioners and neo-luddites were willing to have their lives scheduled by  TV companies. They wanted to see their TV programs and movies when it suited them, and were even willing to pay to get rid of the advertising, if needed.

As broadband became a way of life, distribution via FM or regular wireless TV distribution (analogue or digital) became meaningless, and by 2012 95 percent of sound and video media content was distributed via the Internet, through IP delivery.

All the major media corporations went bankrupt. In the US NBC, ABC and CBS all died in the period of of 2011 to 2013. Fox survived until 2014, mainly due to its conservative user base. BBC survived as an online content provider in the UK, but not as a regular broadcaster.

In essence this meant that the TV set and the radio receiver had been replaced by Internet computers. The Xbox was by 2009 the most popular media content computer in the world. Sony came in second, with its Bravia line of TV screens (the computer being part of the TV screen itself) and Playstation 4. Apple TV came in third and Indian Krishna fourth.

The Internet is everywhere

The Xbox of today is not only a gaming console. It is able to transfer pictures and sound to any receiver in your home, including the living room cinema, the home office work screen, the door of your fridge, your kitchen window, i.e. any flatscreen that is able to present digital information.

The Internet is everywhere, as most of us will know. The average Chinese bathroom mirror will now identify its user and present relevant pop-up widgets or windows with daily schedules, weather reports, the latest news, web feeds from blogs and podcasts - and chances are that the mirror is powered by an Xbox.

However, while Microsoft conquered the home, Apple conquered your hand bag and pocket. While the Windows operating system is close to irrelevant, Apples MacOS is till living strong. But not because of the MacBook and the iMac. Although these machines continued to sell even after most Windows machines were replaced by LINUX/Firefox constellations  and later with any flat-screen, digital paper or digital surface - after all, the Macs were so much more sexy than a Dell or an HP - the MacOS survived mainly due to the iPod.

The 2007 iPod (called iPhone for a short time) gave the world a truly functional portable computer that combined the music and video player with an Internet computer and a voice communication device. The big touch-screen became the paradigm for portable computing for several years, not to be replaced until 2011, and then by yet another Apple concept.

Internet media and search

By the end of the first decade of the 21st century it became clear that Internet searching was more important than ever. Not only was there a need for search engines for searching web pages. Now the Internet contained:

  • Nearly all TV programs from all over the world. Given the TV companies' need to generate revenue, you could also download programs from their back catalogue at an affordable price.
  • Ditto for movies. You could now download any movie you want to your living room cinema display. Now there were millions of movies to choose from, from Lang's Metropolis from 1927 to Coppola's Five Carrots and a Coyote from 2014.
  • Billions of radio programs and podcasts
  • Nearly all phonograms (songs and albums) from 1970 onwards, and a large number of those published before that time as well.

Many TV programs and movies contain hidden text (close captioning) in the local language or subtitles in foreign languages, and the search engines may make use of this text when indexing this content. However, many older TV shows lack such text, as do most radio programs and podcasts. By 2006 many companies had launched services that searched the content contained in the soundtrack of media files. By 2011 most of these companies were bough by the major search engines, and by that time all of them included information from such files in their regular search engine results.

Book and magazine search

Google and Yahoo's book search services also expanded, and by 2012 all new books were automatically included in their book search databases. By 2013 most magazines and newspapers were also added, and the indexing of old literature continued. Now Google claims that it has scanned and indexed nearly all English Language books published in the USA, Canada, Ireland, the Kingdom of England and Wales, and the republic of Scotland since 1850.

What made this possible was probably the success of Apples iTunes media store. Media producers and publisher finally realised that downloading did not constitute at threat to their livelihood, but rather presented them with the possibility to distribute content for a very low price to a world wide audience.

The long tail

The existence of "the long tail", ie. that fact that even the most obscure topics will generate enough sales when you add up all the enthusiasts existing in the word, suddenly turned the back catalogs of record companies and book publishers into gold mines. The enormous increase in sales also led to significant drop in prizes, restricting illegal downloads to the rooms of teenagers.

For the search engines this opened up a new source of revenue in addition to the traditional pay-per-click advertising. Google had for several years had a special music search feature in its regular search results, linking album entries to various online music stores.

It launched its own online media store in 2008, negotiating agreements with all the major record companies and book publishers. Microsoft/Yahoo! did the same, while Ask merged with in 2009, offering not only media files, but also the possibility of linking book search results with the Amazon bookstore. If you have an Ask/Amazon account, one click only is enough to bring a book into your physical mail box.

And yes, paper based books are as popular as ever. Until someone makes a color printer that can bind books for you, the bookstore will never die. CDs, DVDs, BlueRays and HDDVDs, however, are now history. Who needs a physical copy when you can download a high definition, wide screen,  Dolby Surround movie in minutes?

Note: This is an archived article published By Per and Susanne Koch, Published on 15 January 2007