Help, My Site Has Been Banned By Search Engines!
It is the ultimate search engine marketing nightmare: Your site has
been removed from any search engine’s index and your traffic is falling.
By all means panic. Then draw your breath and read our Banned by search engine Survival Guide.
When your pages disappear
There are those that wake up one morning and find that all of their web pages has disappeared from the search index. Others find that any search engine has removed one page at the time, finally ending up with the home page, and then: none at all.
A search engine ban can also be more subtle. The pages remain in the their database, but they end up at page 722 for all relevant search queries.
If they had good rankings before, this is more likely than not caused by the wrath of the world's mightiest search engine.
For the sake of argument we will believe you when you say that you don't know why Google (or Yahoo!, Ask.com, MSN Search or any other search engine) has decided to ban your site and your web pages.
Moreover, we also take it that this is not the result of some mysterious flux in the search engine databases -- i.e. you have really been banned for some weeks now.
What do you do?
First check whether the disappearance of your pages have a natural cause. For instance: See if someone has changed the robots.txt file and inadvertently stopped the search engine spiders from entering your site.
It could also be that someone has included meta tags that exclude the robots (NOINDEX).
If your web server has too much downtime, that might also hurt. If that is the case, get a new web host or new and stable web server.
It could also be that the ban is caused by someone else. For instance: It could be that someone has copied your site, and that search engine, finding identical pages, has decided to give your competitor the benefit of doubt.
If this is the case, you must search for a text string that is unique for your web pages and see if you can find the thief. Then demand that the site owner (or -- if that does not work -- his or her web host) remove the offending site, while at the same time informing search engines about the matter.
What is spam?
However, more likely than not you have done something that is in violation with search engine’s written or un-written anti-spam regulations. That is: There is something about your site that leads them to believe that you are trying to unduly influence your search engine rankings.
Now this is indeed a tricky "judicial" area. Given that search engines accepts that web site owners optimize their sites for better search engine listings, it is somewhat hard to decide what is kosher and what is not.
Still, search engine’s own guidelines gives you clear indications of what they consider to be big "no-no's":
Search engines always expect that you should make pages for users, not for search engines: "Don't deceive your users, or present different content to search engines than you display to users," they say, and goes on to list the following banned advice:
- Avoid hidden text or hidden links.
- Don't employ cloaking or sneaky redirects.
- Don't send automated queries.
- Don't load pages with irrelevant words.
- Don't create multiple pages, subdomains, or domains with substantially duplicate content.
- Avoid "doorway" pages created just for search engines, or other "cookie cutter" approaches such as affiliate programs with little or no original content.
Google also says:
"Avoid tricks intended to improve search engine rankings. A good rule of thumb is whether you'd feel comfortable explaining what you've done to a website that competes with you. Another useful test is to ask, 'Does this help my users? Would I do this if search engines didn't exist?'"
Actually, this advice does not make as much sense as they would like it to.
The fact is that search engine will not punish you for so-called "organic search engine optimization", as long as the page make perfect sense for any human reader.
Hence, search engines will not ban your for sprinkling relevant keyword phrases thought the text, as long as the copy reads as normal text.
We will take a look at Google's list of advice, one by one.
And please remember: Even if you know that you haven't broken these rules, you could still considered a search engine spammer. Check whether anyone else have been involved in the design of the site.
If you have inherited the responsibility for the site from someone else, he or she can have broken the rules. If a search engine marketing company has been involved, they can have stepped beyond the boundaries of proper search engine optimization practices.
Moreover, are you sure your local computer expert may not have tweaked the site a bit, just to be helpful?
What search engines considers unacceptable spam
Use historical data to track
As mentioned earlier on a Pandia’s article, search engines keeps copies of its own search index. If it grows suspicious of your search engine optimization techniques, it may go back into its archives and see what your site (or even you as a registered web site owner) did years ago. If your combined practices of yesterday and today clearly identifies you as a spammer, you are dead.
The sophistication level of search technology will grow exponentially.
Search engines may also have developed intuitive and memory skills that will make them look almost super-human:
Avoid hidden text
Hidden text is an old timer in the history of search engine optimization. In the early jurassic period -- i.e. in the mid 1990's -- webmasters discovered that search engines rewarded web pages with a lot of occurrences of relevant keyword phrases.
Hence they would repeat the keywords again and again, hiding the text by giving the text the same color as the web page background. The search engine spiders would read the text, the human visitors would not.
Needless to say, you should avoid such tricks. The search engine spiders will compare the font color with the background color if you use old fashioned HTML.
To find hidden text you could either look at the HTML code itself (View Source) or mark the whole web page with your mouse. This will make hidden text stand out.
CSS-savvy coders may try to get around this by including the colors in their style sheets. We have no proof that the search engine check the style sheets for color codes, but be sure that your competitors will do so -- and they will report you if they find out.
The same applies if you use one CSS layer to cover another.
If you use regular HTML, be careful when you enter colored text in colored table cells. Search engines will compare the text color with the regular web page color, not the color of the table cell. Hence white text in a black table cell on a white web page will look like white on white to a search engine -- and they may consider it spam.
Avoid hidden links
Many webmasters turn out so called "doorway pages" designed specifically for the search engines (we'll say more about them later on).
These webmasters would prefer that regular visitors who come to the site via the main entrance -- the home page -- do not find these pages, as they do not reflect well upon the quality of the site as a whole.
However, the web master must include links for the search engine to follow in order for these pages to be indexed. Hence the need for hidden links. Hidden links can also be used to interlink web sites in order to enhance link popularity.
There are several ways of coding hidden links. One way is to put in a transparent 1 x 1 pixel picture file and then turn this image into a link using the A HREF syntax.
Here's an example:
<a href="http://mysite.com/doorwaypage.html"><img src="../graphics/blackdot.gif" width="1" height="1" hspace="0" vspace="0" border="0" alt="Relevant keywords"></a>
Another way to do this is to make a single dot (.) into a link, including the relevant keyword phrases in the ALT parameter.
If you find such links on your website, remove them immediately.
Don't employ cloaking.
Cloaking is a very popular technique among search engine optimization experts who have a more than a little knowledge about servers and obscure programming languages.
It is possible to distinguish a search engine spider from a regular human visitors. Hence these experts develop scripts that sniff out the search engine spiders in order to deliver them highly optimized web pages.
Regular visitors get different unoptimized web pages.
In the best case scenario there is a relationship between the page the spider reads and the one you are served. In the worst case scenario, the cloaked page tells the search engine about children playing, while the real one is about adults "playing".
If you have never heard of cloaking, ask your computer expert or search engine optimization company if they have.
Avoid sneaky redirects
There are more primitive -- and inefficient -- ways of hiding a regular web pages from the search engines. One is redirects.
The following code embedded in the web page will tell the web browser to open a new page (hiddenpage.html) in the browser window within 0 (zero) seconds:
<meta http-equiv="Refresh" content="0; URL=http://mysite.com/hiddenpage.html">
The page with the redirect code will be optimized for the search engines, the hidden page is probably not.
Search engines find such code highly suspicious. If you need to use a redirect meta tag, include a NONINDEX tag as well. By using that tag you tell the search engine not to include the page in the index, thus avoiding suspicion:
<meta name="ROBOTS" content="NOINDEX">
By the way: the best way of doing redirects -- for instance if you have moved a page permanently -- is by doing a 301 redirect through changing the server's .htaccess file.
As Shari Thurow of Grantastic Designs has pointed out, 301s are also the best way of redirecting traffic from one domain to another, i.e. if you have several domain names pointing to the same site.
However, you will see if this technique has been used if you put your mouse cursor outside the web page itself when the page is loading. You will then see the "real" page that is hidden when the mouse cursor is within the browser window.
Don't send automated queries to any search engine
"Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our terms of service. Google does not recommend the use of products such as WebPosition Gold that send automatic or programmatic queries to Google."
Well, we have never heard of anyone who have been banned by Search engines for using such tools with common sense. But if you keep checking all your pages for a large number of keywords every day, they will probably notice.
Hence: if you do use such programs (and that makes sense if you have a few sites) do not go over board! Check your search engine rankings no more than once a week, or even less often for your not that important pages.
Search engines will seldom visit a page more than once a month unless it is updated regularly.
Don't load pages with irrelevant words.
If your page is about lawn movers, do not repeat the name of Britney Spears a dozen times throughout the text. The search engines won't like it, your visitors won't like it and Britney Spears won't like it.
(And no, they will probably not punish us for including the name Britney Spears three times on this page. Hmmm, maybe her fans are interested in search engine optimization?)
But do include relevant words, in the regular text, in the TITLE-tag and elsewhere as described in the Pandia Search Engine Marketing 101 tutorial.
Don't create multiple pages, subdomains, or domains with substantially duplicate content.
This is where some search engine optimization companies gets to excited, as these are techniques that tend to give results fast, i.e. before any search engine discovers what's taking place and decide to ban the whole site.
At the "Cleaning Up The Mess" session of the latest Search Engine Strategies conference Anne Kennedy of Beyond Ink mentioned quite a few practices that can get you banned, including.
- Multiple domains that are established to bring traffic to your site.
- Pages on other domains that are established to redirect traffic to your site.
- Mirror sites, i.e. exact copies of the same site on different domains.
- Multiple search results (i.e. too many pages in the search engine results pointing to your site or to pages that redirect searchers to your site).
If you have been using a search engine marketing company, make sure that they haven't established such domains and doorway pages pointing to your site.
Search engines also frown upon links from so-called free for all directories, i.e. sites that are established only to deliver inbound links to sites for a certain price or for a link back.
Avoid "doorway" pages created just for search engines
Webmasters as well as search engines disagree about what a doorway page is. Google is clearly thinking about pages that are only made in order to attract search engine spiders and that lead human visitors to more informative pages on the site.
Hence these are not pages that are normally found in the navigation menu of a site.
Such doorway pages come in two flavors:
- Nonsense pages that repeat the same keyword all over again or that has text that is so filled with keyword that the sentences become meaningless.
- Pages with a few paragraphs of understandable text that is relevant to the main keywords for which the page is optimized, but that is clearly designed to lead the visitor elsewhere.
If there are a large number of similar pages that only differ as regards the main keywords phrases, you can be quite sure that someone is trying to spam the search engine.
However, Google, Yahoo!, Teoma or any other decent search engine will not punish you if you write a large number of distinct, relevant and informative articles that are optimized for relevant keyword phrases.
If your readers will find these articles interesting and useful, the search engines will also like them.
Avoid "cookie cutter" approaches such as affiliate programs with little or no original content
Google says that you should "avoid 'cookie cutter' approaches such as affiliate programs with little or no original content." Now, that is a rather cryptic remark!
We interpret this to mean that you should avoid establishing separate sites or doorway pages with no or little content of use to visitors and that are exclusively made to promote a product or service sold by someone else.
Search engine will not ban you for including links to affiliate programs if you provide high quality content, though. If that was the case, Pandia would have been banned years ago.
Google also warns you about bad neighborhoods:
"Don't participate in link schemes designed to increase your site's ranking or PageRank. In particular, avoid links to web spammers or "bad neighborhoods" on the web as your own ranking may be affected adversely by those links."
Oh yes, this is important. First of all: Check to see if your site is listed on any free-for-all link exchanges. If your site is listed, kindly ask the web site owner to remove you. (Or -- at least -- do not pay them for a renewal!)
Next, search Google and Yahoo! for inbound links and see if any of these come from dubious sites. What a bad neighbor is, is not totally clear, but you should probably look out for links from sites like these:
- Sites are run by active spammers
- Free-for-all link farms or directories
- Adult sites (unless you run one yourself)
- Gambling sites
Active spammers are hard to identify, but you will often find that they run networks of a large number of sites that are interlinked. If they use any of the spamming techniques described in this article, you should also be very careful.
Obviously, if you cannot find the site in any of the search engine, that is a warning sign. A Page Rank of 0 or grey could also indicate that the site has been punished by Google (but it could also be that it is a new page).
For a discussion of Page Rank or PR, see the Pandia Search Engine Marketing 101 tutorial.
As regards the other kinds of bad neighborhoods, we would not loose a night sleep if we found a few links from such sites. Pandia has a couple of admirers in the more seedy part of the Web, and none of the major search engine has not pulled the plug yet.
How to get back into the search engine index
At the Search Engine Strategies conference in Stockholm, Shari Thurow of Grantastic Designs gave some sound advice on how to get relisted by any serious search engine
- First, clean up your act.
- Then send them a very polite email where you acknowledge and describe what you (or your search engine marketing firm) has done.
- Tell that you have read their guidelines and that you will never do anything like this again.
- If you hired an search engine marking firm that spammed on your behalf, state the firm's name and contact information.
- Include full contact information.
This is what Danny Sullivan and others call the "Kiss Up Letter". Indeed, do apologize and make no excuses.
Search engine will often reply to such polite requests and you may find some of your pages back in the index within a couple of weeks. However, it can take months before the search engine has spidered your whole site again.
How to get search engines to tell you when you have crossed the line
It is actually possible to get search engines to inform you when they believe you have broken the rules.
Webmasters using webmaster tool feature of any search engine, may (we repeat: may) get a mail from their account section telling them when they have crosse the line.
Search engine query syntax
To check for search engine listings in any search engine do a search for: site:mydomain.com mydomain.com. The relevant syntax for Yahoo! is site:mydomain.com or domain:mydomain.com
To check for links from other sites, search for link:mydomain.com. Note, however, that this search may not give you the full list of inbound links. At Yahoo!, search for linkdomain:mydomain.com
Note: This is archived article and not updated since longer. Some of the info on article may be outdated.
Authors: Per and Susanna Koch