(Feb 25 2002) Lycos.com is today launching its new InSite paid inclusion program. It is a variant of the Fast AllTheWeb PartnerSite program, which is finally out of the testing stage. Pandia takes a look at the concept of paid inclusion.

The search for search engine revenue

It is a fact of life, search engine must earn money in order to provide their services, and that has not been easy since the dot.com crash and since Web surfers discovered that they didn't have to click on ad banners in order to feel content.

So what do you do, if you are a search engine company? Some, like Google and Fast, sell their search engine technology to other companies, so that they may use it to power their own internal databases and intranets.

Some add so-called "pay-per-click" search results, listings that are in fact paid text ads. All the major search portals -- with the exception of Google -- uses Overture as a pay for placement provider.

And then there is paid inclusion, which must not be confused with pay-per-click text ads. Paid inclusion is a tricky area. The idea is that Webmasters and companies are to pay the search engine to get included in the regular search engine index, i.e. in the standard search results that follow the pay-per-click results on most search portals.

Relevance vs. paid inclusion

It is problematic, because the regular search engine databases are to include the most relevant pages on the Web. You are to get the most relevant results when you search for a particular query. The rise of Google proves that searchers will go to search sites that delivers high quality results if they get the choice.

The problem is that if you let sites buy their way into the database, you add a new factor to the algorithms that decide what pages are to be included, and that factor is money. Money does not necessarily equal quality, whatever they say.

The search engines are very much aware of the problem, and as far as we know, none of the major paid inclusion programs let program membership deliberately influence the ranking of the pages.

Hence paying to get included in the Inktomi database will not necessarily bring you a higher ranking in the search results on search portals using the Inktomi database. The same rules applies to all pages in the database, paying and non-paying.

More cynical souls may suspect that the search engines will start dropping non-paying pages from their index in order to "encourage" webmasters to pay for their inclusion. And, indeed, the chances of non-paying. pages being deleted is much larger than for pages taking part in these programs.

Still, so far we have found no proof of search engines deliberately deleting pages in order to force sites to pay.

When pages are dropped, this may be for various reasons. Sometimes the search engines flush out less popular pages in order to make room for others.

Inktomi may transfer them to less fashionable parts of their databases, meaning that the chances are that they will not get a high ranking on more general search queries. Then there are technical glitches. From time to time tens of thousands of pages just disappear for no particular reason.

Fortunately, search engines like Google and Fast have found that it is good PR to be know as "the largest search engine in the world", meaning that they cannot afford to leave out too many relevant pages.

Fast is aiming for a database of 2 billion "objects" (i.e. Web pages, images and other file types) by the end of March 2002, and can hardly afford to start dropping high quality pages in order to lure site owner into their program.

So provided that you deliver content rich Web pages, and you do not spam the search engines (i.e. do not try to influence your ranking in a devious and unethical manner), and that there are other sites linking to your site (indicating quality and giving the search engine spiders a way to find your site) you will actually get included.

So what's the point in paying?

Why should you pay to be included?

Well, there are a large number of commercial sites out there whose livelihood depends on a continuous presence in the search engine databases. They are actually willing to pay to be 100% sure that their site does not drop out at any time, even if it is included again the next time the search engine robot come their way.

Moreover, normally search engine spiders will visit you no more than once every fortnight or even less often, meaning that your listings will seldom be up to date. If you refresh your site on a daily basis, adding for instance new products or news items, you would naturally prefer the search engine robots to visit you more often. The Inktomi and Fast programs guarantee that they will come back at least once every 48 hours. AltaVista will revisit your site once a week.

Regular spidering gives the webmaster an added bonus. He or she may make small adjustment to the webpage coding and copy in order to see if it is possible to improve the search engine ranking. This is admittedly not as easy as it used to be, but the fact that you may measure the effect one or two days later (instead of one or two months) makes it much easier.

Some may argue that the search engines in this way actually encourages webmasters to manipulate their rankings, as long as they pay for it, and to a certain extent this is true. As we shall see, Fast and Lycos have actually added search engine optimization tips and information to their new services, in effect telling paying customers how to improve their rankings.

Other sites, not taking part in these programs, will not get access to this information, and as a result their rankings may suffer, meaning that it is not necessarily the "best" or the most relevant site that gets the top position.

On the other hand, webmasters studying search engine optimization techniques, will always get the upper hand, and you do not have to subscribe to these services to get this information. If you are a regular reader of Pandia and related sites, you will probably gain access to this information anyway.

Moreover, no search engine will encourage you to spam the search engine. Paying sites will be punished just a severely for cheating as non-paying. sites.

What the search engines will do is to give you common sense optimization advice like adding keywords to your title-field, headlines and regular text, having many content rich pages and so on. Fair enough.

Go on to part II, on the new Lycos InSite paid inclusion program >>>