Demystifying Your Competitor’s Ranking Success

By Ross Dunn - July 21, 2006

We have all been there, “how the heck do they always get #1?” It is a constant frustration for many a client and, well, even myself occasionally. The fact is that much of the time there are a few solid reasons behind the search engine success of any website and it is important to learn what these reasons are before trying to compete. How is this done? Therein lies the subject of this article; how do you determine what your competitor has done to win the search engine war?

Demystifying your competitor’s success requires you to put on your detective garb because you are going to have to investigate all aspects of their website; even the deepest darkest corners. In the following instructional I will lead you through a hypothetical investigation of a competitor who is ranking for the phrase “voip services”. In each step I will choose the more popular result that I find when I do similar competitor analyses professionally. So please take note, the sample is only the most popular result; occasionally there are truly baffling cases of competitor success which have required heftier investigation leading to differing conclusions of what they did to succeed.

STEP 1: Start with the Basics: Is The Site Optimized?

Visit your competitor’s website and look at the content on the page. Does the keyphrase they are ranking under appear often throughout the body text? Does the phrase appear in the headings, Meta Tags, Title and in their menu? If this is so it generally means they have focused completely on attaining the keyphrase they are ranking highly for. In other words it is quite possible, and even likely, that they have sacrificed other keyword goals in order to achieve this single goal.

Example Results of Step 1:

“ The competitor’s page has utilized the ‘voip services’ often but not in a spammy manner (not obvious overuse) throughout the body text on their home page. The page topic is clearly focused on this phrase because the Title tag states ‘VOIP Services – by VOIP Business Systems’ and the headings and menu items are clearly related. Conclusion, I have no doubt that this site has focused on this phrase and that a search engine spider will have no problem logically identifying the subject of this page as ‘voip services’.”

STEP 2: Is Site Structure Playing A Role In Their Success?

Site structure is a fundamental reason for search engine success. I cannot stress enough just how important it is to have a website that can be freely and simply indexed by search engines. The fact is that the more legitimate and useful content that a search engine has access to on a website the better the environment for top rankings. To put this inquiry simply: “how well is the website indexed?” Each search engine may react slightly different to the technology used within a website so let’s tackle the 2 largest and most revealing engines:

Test Google and Yahoo: On each search engine use the following syntax replacing “competitor” with your competitor’s site: site:www.competitor.com . This search will reveal how many pages that each major engine has indexed within the competitor’s site. Now that you have a list of the pages indexed, look and see if each indexed page has its own title and description. The answer will determine whether each page within their site has been individually optimized.

It is also crucial to look at the URL’s for each link. Does each link appear to have many symbols such as “%&”? Often a website with a lot of extraneous code in their URL’s will not be 100% indexed because the search engines tend to shy away from pages that are created dynamically (by databases – which these characters state). If there are no extraneous characters it may be quite revealing if you know the site is far too big to be anything but driven by a database. In this situation it is apparent that the competitor’s website has been tweaked by a programmer to allow the search engines to index everything. This tweak is extremely common and highly recommended for anyone who has a database driven website with unfriendly URLs (contact me for more info).

Google Sitemap

Another useful and revealing tactic is to determine whether your competitor is using Google Sitemap to improve the indexing of its website. To find this out with reasonable chances of success, try entering the following URL into your browser’s address bar: www.competitor.com/sitemap.xml. This is the default file name for Google sitemaps so if your competitor has one this will often provide you with a resulting page full of xml data. Here is an example of what you would see if your search were successful (from StepForth’s website). If you do find a sitemap then it is reasonable to assume that your competitor is relatively well informed on the latest search engine tactics and that Google Sitemaps may play a small role in their online success.

Comparing Actual Site Size

Finally you need to know exactly how many pages your competitor actually has online. This way you can determine how well their website has been indexed. To do this there are a variety of software options but one simple and free option is to use Xenu’s Link Sleuth. Xenu is a free site analysis tool available here; in the past I have found it very simple, small to install and unobtrusive. Install the program and select a new project where you just need to enter the address to spider and let it go to work. Ultimately you can ignore a lot of the information because you are not trying fix dead links. The key is the “Statistics for Managers” section located at the bottom of the report. The number of pages within the site can be found next to the “text/html” statistic.

Example Results of Step 2

“ I saw that Google had indexed about 9000 pages and Yahoo had indexed 138,000 pages. I am quite impressed with these indexing; obviously my competitor’s site is easily indexed by search engine spiders. To back this up I see that the URL’s do not have any extraneous characters yet they are also very long… this database-driven website must have been tweaked for search engines.

"Yahoo had more results than Google but that isn’t a surprise since Yahoo tends to be more forthcoming with its site-specific queries. A Google Sitemap does exist on the competitor’s site but because Yahoo has also indexed the site well I don’t suspect it has played a significant role. As for the ratio of actual pages to those indexed… it appears that Yahoo has actually indexed more pages than my software found. Overall I would say that this competitor has spent a lot of time making their site search engine friendly. I should compare these results with my own site… perhaps I need to follow suit!”

STEP 3: Where, Oh Where Do They Get Links?

These days another fundamental issue are backlinks (links pointing to their site); just how many have your competitors got that you don’t have OR just how good are they? For example you may find that your competitor has thousands of links but when you research them they appear to be less than relevant to their site topic. Only relevant links offer any advantage. In other words, if your site is about VOIP and you have a link from a hair salon you are not likely to get an iota of benefit from this link. That said, if you have a link from a telephone sales website then there is an obvious correlation in relevance and your perceived search engine popularity will increase.

In order to discover this information you have two options. One, perform a manual search of your competitor’s backlinks where you can randomly review the quality of their links (time consuming but ok for small sites). The other option is to purchase a program like OptiLink which is designed specifically to simplify the investigation process; especially for large competitor sites. For the purposes of this article I will stick to the manual option but I do recommend checking out OptiLink because it also provides other amazing analysis tools that will help you get better rankings.

Manual Research

Open two web browsers and then follow these instructions:

In one browser visit Google and type in the following: link:www.competitor.com

The resulting list will include a sampling of the backlinks that Google ‘sees’. Now I note “sampling” because Google is not very forthright with this information. For example I can state with little doubt that Google actually sees just as many backlinks that Yahoo does yet Google always shows fewer in their results.

In the other browser visit Yahoo and type in the following: linkdomain:www.competitor.com

The resulting list is likely a very realistic assessment of all the notable links your competitor has online.

Now that you have a list for each major search engine you can tackle randomly investigating the links within each.

Some Points to Look Out For:

Are the sites relevant to your competitor’s site? If they are not relevant you can consider them a low quality link with likely zero benefit.

Do the sites appear to be reputable? In other words do they have obvious Spam? If there is obvious Spam you can ignore this site.

Do the sites have decent PageRank? (use the Google Toolbar to deduce this)
Low PR sites (>4) are generally less helpful. High PageRank sites (5>) often have better impact because they are considered reputable so most outgoing links will be votes of confidence for the destination site. Please note that this is a topic thick with opinions but these are fair general statements.

Do they appear to be link farms? If they are, this link can generally be discounted.

Are the links paid? If you notice that the links shown are paid links take note of the site and consider contacting them for their rates. Even if you decide not to proceed with their service it can be a revealing piece of information.

Example Results for Step 3:

“ I decided to review 50 different back links on each search engine and I discovered something very interesting. First of all, of the 100 links that I reviewed I noticed that my competitor actually owns the sites! I figured this out by reviewing the Whois of each site because they appeared to be so promotional. Each site is informational, well designed and obviously created to build traffic to his main site. This is not a tactic that I want to follow due to the time it takes and from an ethical standpoint so I chose to focus on the other links I found. Unfortunately it appears that the other links were not too revealing; none of them appeared to be of especially high quality and most appeared to be from an old link building campaign because they were link farms (an old and dangerous tactic).

"I did have one lucky hit! I found about 10 websites where my competitor is paying for links. These sites appear to be of very good quality and the number of outgoing links is very low. I am going to find out if advertising on them is affordable.”

STEP 4: Your Site – Compare and Contrast

Now run all of the same analysis you just did for your competitor on your own site and record your discoveries.

STEP 4: The Resulting Conclusions

Compare and contrast the information that you have compiled from your competitor and your own website and create a game plan.

Example Conclusions: The Game Plan
In point form this is what I believe I need to review on my own site in order to compete:

1. I must simplify the URL’s within my website. My research has shown that my site has several roadblocks because my site is only 20% indexed in both Google and Yahoo.

2. I need to optimize each major page within my website paying particular attention to the customization of each title tag. I noticed that each of my competitor’s top performing pages (despite being deep in their site) was well optimized.

3. It may not be the biggest help but I am going to add a Google Sitemap to my website once it has been optimized.

4. I need to attain more backlinks. I see how my competitor has gained backlinks but I have no interest in pursuing this angle. Instead I am going to hire a link building firm as well as start my own blog on VOIP emerging technology that I hope will produce enough interest to get some backlinks.

Other Forms of Competitor Analysis

The analysis that I outlined above is used when you are trying to beat out a competitor under a particular keyphrase target. But this is only one form of competitor analysis.

For example, perhaps you know a competitor down the street from your office is doing exceptionally well online. In order to find out how they were doing well you would approach an analysis differently. You would first need to discover what keyphrases the competitor is targeting and what exactly they are doing to obtain rankings under them. Additionally you would need to discover if this competitor is profitable for reasons other than search engine rankings. For example, they may be acquiring traffic from another website through a paid advertisement or a mutual partnership. There are many ways that a business can profit online and once in a while I find a completely new and occasionally cool concept when I research competitors for my clientele.

Just a reminder, the steps noted above represent just the tip of an iceberg for researching your competitor’s success. If you are finding that significant pieces of the puzzle still remain unanswered after following these recommendations please do not hesitate to give me a call (250-385-1190).

About The Author:

Ross Dunn is the CEO of StepForth Placement Inc., a search engine marketing company founded in 1997 and based in Victoria, British Columbia, Canada. StepForth provides cutting-edge search engine optimization services that provide highly successful, targeted results for its clientele. Ross Dunn is a Certified Internet Marketing and Business Strategist (CIMBS) with a background in web design and online marketing. His broad Internet experience in combination with a talented staff has made StepForth a name synonymous with top results.