Google is including web site content found by using site search forms.

Search engines like Google are very good at following regular web links. If they find a properly coded link at a web page, their spider will go to that page and index it if it finds it worth while.

However, many sites have hidden their content behind a search form. I.e. the only way you can access the information is by searching their database using their search form.

Search engine spiders have, until now, ignored such search forms. This is why people talk about “the hidden web” or “the invisible web”.

The Webmaster Central Blog now reports that Google is letting their spiders do test searches when finding forms:

Specifically, when we encounter a form element on a high-quality site, we might choose to do a small number of queries using the form. For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML. Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made.

The spider will not index pages found in directories banned by the local robots.txt file and will in general try not to overwhelm a site with such requests. So, if you want to stop Google form indexing a database, edit your robots.tx file.

Moreover, Google will not let pages found via search forms replace web pages found through regular links.

Note also that Google is now parsing some types of Javascipt and Flash links. This does not mean that you should abandon the general rule of always including a regular HREF URL-link in addition to Flash and Javascript links like, for instance, the ones found in various menus.

See also: Matt Cutts: Solved: another common site review problem
Beyond Search: Google Forms: A Data Snout for a Bigger Creature