Yahoo introduces the “class=robots-nocontent” attribute.

It is a general problem for search engines that a lot of web page content is repetitive: navigation menus, headings, ads etc. (”boilerplate content”).

The search engines do obviously have algorithms that help them identify and disregard such content, but that hasn’t stopped Yahoo! from asking webmasters for help.

The Yahoo! Search Blog has announced the birth of a new tag that webmasters may include in their templates: the robots-nocontent tag.

The tag will, as Yahoo! puts it, “indicate to our crawler what parts of a page are unrelated to the main content and are only useful for visitors.”

Note that this is not a new metatag. Instead you use CSS class attribute and add it to a div, span or p-tag. These are the examples given by Yahoo!:

<div class="robots-nocontent"> This is the navigational menu of the site and is common on all pages. It contains many terms and keywords not related to this site</div>

<span class="robots-nocontent"> This is the site header that is present on all pages of the site and is not related to any particular page</span>

<p class="robots-nocontent"> This is a boilerplate legal disclaimer required on each page of the site</p>

<div class="robots-nocontent"> This is a section where ads are displayed on the page. Words that show up in ads may be entirely unrelated to the page contents</div>

Google has had a related tag for Adsense sites (to help the Adsense text ad system identify the core content of a page), but that tag has — as far as we know — not been used by the regular search engine spiders of Google.

Matt Cutts of Google has drawn attention to one open question regarding the Yahoo! tag: Will Yahoo! follow links that are part of sections tagged as “robots-nocontent”? We don’t know.

Should you start using the tags? Hm, we are in doubt.

If you have templates generating a lot of boilerplate content the tags may give a boost to keyword phrases that are present in the core article text of a page.

On the other hand, we would argue that Yahoo! has failed if it is unable to identify the core text without the help of webmasters. After all, most sites are unlikely to make use of the tags, and it seems unreasonable to us that these sites are to be “punished” ranking wise for not following Yahoo’s recommendations.

Still, if it turns out that the tags will have an effect on rankings, SEO-savvy webmasters will have no choice but to make use of it. Stay tuned!

Yahoo Supports New Robots-Nocontent Tag To Block Indexing Within A Page Search Engine Land
How do I mark web page content that is extraneous to the main unique content on the page? Yahoo! Help
Yahoo! Supports Robots-Nocontent: Enabling Organic Search Page Section Targeting SE Roundtable