Google adds Sitemap feeds for URL submissions

Google lets webmasters submit lists of new webpages for Google to crawl. This will help Google getting accurate information about your web pages, updates and site structure.

(June 4 2004) Yahoo! is now the only search engine that offers paid inclusion, letting webmasters pay to ensure that the search engine crawls and includes new web pages added to the site.

This is a very useful feature for large shopping sites and other sites that add a large number of new or updated pages on a regular basis and that want to ensure that the search engine result listings stay fresh and up to date.

Webmasters taking part in the Yahoo paid inclusion program normally inform Yahoo! about new pages by the use of a special XML text file containing information about new pages, their URLs, content and so on.

Google argues that no one should pay to be included in the regular search results, and is apparently not planning to implement a similar fee based service.

However, Google does see the advantage of having access to data that makes it easier for the search engine crawlers to find new files.

Because of this the company is now testing a new service called Google Sitemaps.

What is Google Sitemaps?

Google Sitemaps is a free tool for webmasters that helps improve a site's coverage in the Google index. It is a system that lets you tell Google about changes made at your site.

Google argues that by using Sitemaps to inform and direct their crawlers, they will expand their coverage of the web and improve the time to inclusion in their index.

Generating XML files

All you need to do is place a Sitemap-formatted file on your web server that enables Google's crawlers to find out what pages are present and which have changed recently. You must then inform Google that you have added or updated this file.

Like Yahoo! Google asks webmasters to deliver the information in specially formatted XML text files.

You can generate a XML sitemap file by using Google's Sitemap Generator.

The Sitemap Generator is quite a complicated tool, though. You obviously need to have knowledge of uploading files to your web server and connecting to your web server. In addition, you must know how to install and run server scripts, and Python version 2.2 must be installed on your server.

If the Sitemap generator is a little too much for you, you can still submit a Sitemap to the Google Sitemaps program in simple text format (i.e. a text file containing a list of URLs).

We would guess that third party developers will eventually deliver software that can generate Sitemap XML files by alternative routes.

Who will benefit?

Google says that the Sitemap feature is intended for all web site owners, and that in most cases, webmasters will benefit from Sitemap submission, and in no case will they be penalized for it.

The service is perfect for those that run large database driven sites with a large number of pages -- especially if you have a script driven system that generates URLs with a large number of parameters (including & and ? signs).

However, if you run one or more small sites that are updated infrequently, this is probably not for you.

Moreover, if you are the type that responds to terms like "Apache", "UNIX" and "XML" and "Python" with "huh?", "eh?", "huh?" and "HUH!???", you should probably go out in the sun and enjoy a cup of latte instead.

If you find this feature to be too intimidating, you can always let Google know about your pages the old fashioned way, e.g. by adding a regular site map, i.e. a ordinary web page containing links to your new webpages.

Google will not punish sites without Sitemap feeds, and will continue to crawl sites as it did before.

Including links to new webpages on the home page or some of your most visited pages on the site will for instance ensure that Google find these pages quickly.

However, Google may eventually expand its current reporting system, giving users access to data showing popular search terms and click-through rates. If this happens, the added value may make this system valuable even for very small sites.

No guarantees

SitemapsAdvisor, a Google employee taking part in Search Engine Watch discussions, says that the program is a complement to, not a replacement of, the regular crawl:

"The benefit of Sitemaps is two fold: For links we already know about thru our regular spidering, we plan to use the metadata you supply (e.g., lastmod date, changefreq, etc.) to improve how we crawl your site. For the links we don't know about, we plan to use the additional links you supply, to increase our crawl coverage."

Note that using Sitemaps does not guarantee that Google will crawl all of your URLs, nor will they necessarily get crawled any faster -- at least not for the time being.

However, Google will use the data in your Sitemap to learn about your site's structure.

This will allow them to improve their crawler schedule to better crawl your site.

Google Sitemaps Home Page

More search engine news...