Comparing Rank-Tracking Methods: Browser vs. Crawler vs. Webmaster Tools |
Comparing Rank-Tracking Methods: Browser vs. Crawler vs. Webmaster Tools Posted: 04 Sep 2013 04:05 PM PDT Posted by Dr-Pete Deep down, we all have the uncomfortable feeling that rank-tracking is unreliable at best, and possibly outright misleading. Then, we walk into our bossâs office, pick up the phone, or open our email, and hear the same question: âWhy aren't we #1 yet?!â Like it or not, rank-tracking is still a fact of life for most SEOs, and ranking will be a useful signal and diagnostic for when things go very wrong (or very right) for the foreseeable future. Unfortunately, there are many ways to run a search, and once you factor in localization, personalization, data centers, data removal (such as [not provided]), and transparency (or the lack thereof), itâs hard to know how any keyword really ranks. This post is an attempt to compare four common rank-tracking methods:
MethodologyWe started by collecting a set of 500 queries from Moz.comâs Google Webmaster Tools (GWT) data for the month of July 2013. We took the top 500 queries for that time period by impression count, which provided a decent range of rankings and click-through rates. We used GWT data because itâs the most constrained rank-tracking method on our list â" in other words, we needed keywords that were likely to pop up on GWT when we did our final data collection. On August 7th, we tracked these 500 queries using four methods: (1) Browser â" PersonalizedThis is the old-fashioned approach. I personally entered the queries on Google.com via the Chrome browser (v29) and logged into my own account. (2) Browser â" IncognitoAgain, using Google.com on Chrome, I ran the queries manually. This time, though, I was fully logged out and used Chromeâs incognito mode. While this method isnât perfect, it seems to remove many forms of personalization. (3) CrawlerWe modified part of the MozCast engine to crawl each of the 500 queries and parse the results. Crawls occurred across a range of IP addresses (and C-blocks), selected randomly. The crawler did not emulate cookies or any kind of login, and we added the personalization parameter (â&pws=0â) to remove other forms of personalization. The crawler also used the â&near=usâ option to remove some forms of localization. We crawled up to five pages of Google results, which produced data for all but 12 of the 500 queries (since these were queries for which we knew Moz.com had recently ranked). (4) Google Webmaster ToolsAfter Google made data available for August 7th, we exported average position data from GWT (via âSearch Trafficâ > âSearch Queriesâ) for that day, filtering to just âWebâ and âUnited Statesâ, since those were the parameters of the other methods. While the other methods represent a single data point, GWT âAvg. positionâ theoretically represents multiple data points. Unfortunately, there is very little transparency about precisely how this data is measured. Once the GWT data was exported and compared to the full list, there were 206 queries left with data from all four rank-tracking methods. All but a handful of the dropped keywords were due to missing data in GWTâs one-day report. Our analyses were conducted on this set of 206 queries with full data. Results: CorrelationsTo compare the four ranking methods, we started with the pair-wise Spearman rank-order correlations (hat tip to my colleague, Dr. Matt Peters, for his assistance on this and the following analysis). All correlations were significant at the p<0.01* level, and r-values are shown in the table below:
*Given that the ranking methods are analogous to a repeated analysis of the same data set, we applied the Bonferroni correction to all p-values. Interestingly, almost all of the methods showed very strong agreement, with Personalized vs. Incognito showing the most agreement (not surprisingly, as both are browser-based). Hereâs a scatterplot of that data, plotted on log-log axes (done only for visualizationâs sake, since the rankings were grouped pretty tightly at the upper spots):
Crawler vs. GWT had the lowest correlation, but itâs important to note that none of these differences were large enough to make a strong distinction between them. Hereâs the scatterplot of that correlation, which is still very high/positive by most reasonable standards:
Since the GWT âAverageâ data is precise to one decimal point, thereâs more variation in the Y-values, but the linear relationship remains very clear. Many of the keywords in this data set had #1 rankings in GWT, which certainly helped boost the correlations, but the differences in the methods appear to be surprisingly low. If you're new to correlation and r-values, check out my quick refresher: the correlation "mathographic". The statement "p<0.01" means that there is less than a 1% probability that these r-values were the result of random chance. In other words, we can be 99% sure that there was some correlation in play (and it wasn't zero). This doesn't tell us how meaningful the correlation is. In this particular case, we're just comparing sets of data to see how similar they are â" we're not making any statements about causation. Results: AgreementOne problem with the pair-wise correlations is that we can only compare any one method to another. In addition, thereâs a certain amount of dependence between the methods, so itâs hard to determine what a âstrongâ correlation is. During a smaller, pilot study, we decided that what weâre really interested in is how any given method compares to the totality of the other three methods. In other words, which method agrees or disagrees the most with the rest of the methods? With the help of Dr. Peters, I created a metric of agreement (or, more accurately, disagreement). Iâll save the full details for Appendix A at the end of this article, but hereâs a short version. Letâs say that the four methods return the following rankings (keeping in mind that GWT is an average):
This yielded a disagreement score for each of the four methods for each of the 206 queries. We then took the mean disagreement score for each method, and got the following results:
GWT showed the highest average disagreement from the other methods, with incognito rankings coming in on the low end. On the surface, this suggests that, across the entire set of methods, GWT disagreed with the other three methods the most often. Given that weâve invented this disagreement metric, though, itâs important to ask if this difference is statistically significant. This data proved not to be normally distributed (a chunk of disagreement=0 data points skewed it to one side), so we decided our best bet for comparison was the non-parametric Mann-Whitney U Test. Comparing the disagreement data for each pair of methods, the only difference that approached statistical significance was Incognito vs. GWT (p=0.022). Since I generally try to keep the bar high (p<0.01), I have to play by my own rules and say that the disagreement scores were too close to call. Our data cannot reliably tell the levels of disagreement apart at this point. Results: OutliersEven if the statistics told us that one method clearly disagreed more than the other methods, it still wouldnât answer one very important question â" which method is right? Is it possible, for example, that Google Webmaster Tools could disagree with all of the other methods, and still be the correct one? Yes, itâs within the realm of possibility. No statistic will tell us which method is correct if we fundamentally distrust all of the methods (and I do, at least to a point), so our next best bet is to dig into some of the specific cases of disagreement and try to sort out whatâs happening. Letâs look at a few cases of large-scale disagreement, trying not to bias toward any particular method. Case 1 â" Personalization BoostMany of the cases where personalization disagreed are what youâd expect â" Moz.com was boosted in my personalized results. For example, a search for âseo checklistâ had Moz.com at #3 in my logged-in results, but #7 for both incognito and crawled, and an average of 6.7 for GWT (which is consistent with the #7 ballpark). Even by just clicking personalization off, Moz.com dropped to #4, and in a logged out browser a few days after the original data collection, it was at #5. Whatâs fascinating to me is that personalization didnât disagree even more often. Consider that all of these queries were searches that generated traffic for Moz.com and Iâm on the site every day and very active in the SEO community. If personalization has the impact we seem to believe it has, I would theorize that personalized searches would disagree the most with other methods. Itâs interesting that that wasnât the case. While personalization can have a huge impact on some queries, the number of searches it affects still seems to be limited. Case 2 â" Personalization PenaltyIn some cases, personalization actually produced lower rankings. For example, a search for âwhat is an analystâ showed Moz.com at the #12 position for both personalized and incognito searches. Meanwhile, crawled rankings put us at #3, and GWTâs average ranking was #5. Checking back (semi-manually), I now see us at #10 on personalized search and up to #2 for crawled rankings. Why would this happen? Both searches (personalized vs. crawled) show a definition box for âanalystâ at the top, which could indicate some kind of re-ranking in play, but the top 10 after that box differ by quite a bit. One would naturally assume that Moz.com would get a boost in any of my personalized searches, but thatâs simply not the case. The situation is much more complex and real-time than we generally believe. Case 3 â" GWT (Ok, Google) Hates UsHereâs one where GWT seems to be out of whack. In our one-day data collection, a search for âseoâ showed Moz at #3 for personalized rankings and #4 for incognito and crawled. Meanwhile, GWT had us down in the #6 spot. Itâs not a massive difference, but for such an important head keyword, it definitely could lead to some soul-searching. As of this writing, I was showing Moz.com in the #4 spot, so I called in some help via social media. I asked people to do a logged-in (personalized) search for âseoâ and report back where they found Moz.com. I removed data from non-US participants, which left 63 rankings (36 from Twitter, and 27 from Facebook). The reported rankings ranged from #3 to #8, with an average of 4.11. These rankings were reported from across the US, and only two participants reported rankings at #6 or below. Hereâs the breakdown of the raw data:
You can see the clear bias toward the #4 position across the social data. You could argue that, since many of my friends are SEOs, we all have similarly biased rankings, but this quickly leads to speculation. Saying that GWT numbers donât match because of personalization is a bit like saying that the universe must be made of dark matter just because the numbers donât add up without it. In the end, that may be true, but we still need the evidence. Face ValidityUltimately, this is my concern â" when GWTâs numbers disagree, weâre left with an argument that basically boils down to âJust trust us.â This is difficult for many SEOs, given what feels like a concerted effort by Google to remove critical data from our view. On the one hand, we know that personalization, localization, etc. can skew our individual viewpoints (and that browser-based rankings are unreliable). On the other hand, if 56 out of 63 people (89%) all see my site at #3 or #4 for a critical head term and Google says the âaverageâ is #6, thatâs a hard pill to swallow with no transparency around where Googleâs number is coming from. In measurement, we call this âface validityâ. If something doesnât look right on the surface, we generally want more proof to sort out why, and thatâs usually a reasonable instinct. Ultimately, Googleâs numbers may be correct â" itâs hard to prove theyâre not. The problem is that we know almost nothing about how theyâre measured. How does Google count local and vertical results, for example? What/who are they averaging? Is this a sample, and if so, how big of a sample and how representative? Is data from [not provided] keywords included in the mix? Without these answers, we tend to trust what we can see, and while we may be wrong, itâs hard to argue that we shouldnât. Whatâs more, itâs nearly impossible to convince our clients and bosses to trust a number they canât see, right or wrong. ConclusionsThe âgoodâ news, if weâre being optimistic, is that the four methods we considered in this study (Personalized, Incognito, Crawler, and GWT) really didnât differ that much from each other. They all have their potential faults, but in most cases theyâll give you an answer thatâs in the ballpark of reality. If you focus on relative change over time and not absolute numbers, then all four methods have some value, as long as youâre consistent. Over time, this situation may change. Even now, none of these methods measure anything beyond core organic ranking. They donât incorporate local results, they donât indicate if there are prominent SERP features (like Answer Boxes or Knowledge Graph entries), they donât tell us anything about click-through or traffic, and they all suffer from the little white lie of assumed linearity. In other words, we draw #1 - #10, etc. on a straight line, even though we know that click-through and impact drop dramatically after the first couple of ranking positions. In the end, we need to broaden our view of rankings and visibility, regardless of which measurement method we use, and we need to keep our eyes open. In the meantime, the method itself probably isnât critically important for most keywords, as long as weâre consistent and transparent about the limitations. When in doubt, consider getting data from multiple sources, and donât put too much faith in any one number. Appendix A: Measuring DisagreementDuring a pilot study, we realized that, in addition to pair-wise comparisons of any two methods, what we really wanted to know was how any one method compared to the rest of the methods. In other words, which methods agreed (or disagreed) the most with the set of methods as a whole? We invented a fairly simple metric based on the sum of the differences between each of the methods. Let's take the example from the post â" here, the four methods returned the following rankings (for Keyword X):
That sounds a lot more complicated than it actually is. Let's calculate the disagreement score for method 1, which we'll call "M1" (likewise, we'll call the other methods M2, M3, and M4). I call it a "disagreement" score because larger values ended up representing lower agreement. For M1 for Keyword X, the disagreement score is calculated by: sqrt(abs(M1-M2)) + sqrt(abs(M1-M3)) + sqrt(abs(M1-M4))The absolute value is used because we don't care about the direction of the difference, and the square root is essentially a dampening function. I didn't want outliers to be overemphasized, or one bad data point for one method could potentially skew the results. For Method 1 (M1), then, the disagreement value is: sqrt(abs(2-1)) + sqrt(abs(2-1)) + sqrt(abs(2-2.8))...which works out to 2.89. Here are the values for all four methods:
Finally, for each method, we took the mean disagreement score across the 206 keywords with full ranking data. This yielded a disagreement measurement for each method. Again, these measurements turned out not to differ by a statistically significant margin, but I've presented the details here for transparency and, hopefully, for refinement and replication by other people down the road. Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read! |
Why Local Businesses Don't Need Big Budgets for Their Content Marketing Posted: 04 Sep 2013 03:07 AM PDT Posted by MatthewBarby This post was originally in YouMoz, and was promoted to the main blog because it provides great value and interest to our community. The author's views are entirely his or her own and may not reflect the views of Moz, Inc. Time and again we hear the same old argument that we shouldn't be building links, but that we should be focusing on developing exciting and unique content that encourages organic linking all by itself. I completely agree with this statement, but I see businesses misinterpret this far too often. If you run a local business and have a relatively small budget for your online marketing, is spending that money on a flashy infographic going to be the best use of your resources? More often than not, it isn't. Big brands vs. local businessesThe marketing goals for local business and multinational brands are often quite similar in basic principleâ"they want to establish their brand name as a leader within their field and the geographic areas that they serve. The big difference here is that a local business is looking for large levels of brand exposure to their customers on a micro-level (i.e. they want to reach their customers within the local area that they serve). Major brands also want maximum brand exposure to their customers but on a macro levelâ"they want to be recognised the world over, and to do this they need to spend big on reaching everyone. This is where the content of these two types of businesses becomes different. Content is EVERYTHINGWhen we think of "content," we typically think of articles, webpage text, and imagery. This is one of the greatest reasons for failure within content marketing campaigns. Having a very linear and restricted view of what content is will only restrict and inhibit results.
Let me elaborate a little on the points I've made above. Let's take the example of a local coffee shop (you can take a look at the discussion I had on inbound.org about this as well). Turning your staff into contentMost local businesses are heavily focused around delivering a high level of customer service to gain positive feedback for their business. Herein lies an amazing opportunity for content that will boost the awareness and reach of your brand.
Within a coffee shop business, the staff are just as important as the food/drink itself. In some cases they are more important. If you have built in strong customer service that sets you apart from your competitors then, believe it or not, people will talk about you. Considering the work that goes into building local citations and reviews for SEO, utilising your staff to encourage these makes sense. Getting creative with your staff
source: http://www.flickr.com/photos/hootersgirls/7735589086/in/set-72157630960120760
For years, major international brands have been looking at ways to differentiate themselves from their competitors. Just look at the likes of Hooters; everyone knows about them because of their staff and the outfits that they wear. One simple thing that has got them where they are today, and that has become one of the most iconic pieces of content within the restaurant industry is the Hooters uniform. I'm not saying that you should start making all your staff wear hot-pants and revealing topsâ"as fun as that might sound, but what I'm saying is that you can give your staff a unique edge that makes them a piece of creative content. For example, I visited a coffee shop a few weeks back and some of the staff had aprons on with a big QR code on the front that said "zap me for a 10% discount." When you scanned the QR code you had to like their Facebook page and the member of staff would give you a discount there and then... awesome! Turning your shop/office into contentLike with your staff, your business premises can become an awesome content example. With the example of a local coffee shop, it's easy to get creative on a small budget to gain attentionâ"create a unique style and do something different. Also, make it easier to tie in your offline presence with your online presence. For example, you could have your Twitter handle stenciled onto the wall. If you want to get really creative then you could have a chalkboard where you write out your recent tweets as they happen in chalkâ"this would certainly engage your customers within the shop and I can guarantee it would get people talking about you.
Design your space to be unique and it could become one of the best evergreen content assets that your business could have. The above photo is one that I took whilst visiting the Shakespeare and Company vintage bookstore in Paris. I read somewhere that this is one of the most photographed bookstores in the world - you will see why when you step inside!
Photo: by http://www.camenzindevolution.com at http://mz.cm/17CDwr4 This is just what many major international brands doâ"the above image is from the Google Tel Aviv office, and yes, that's a slide! I remember when I visited the Burberry offices in London a couple of years ago and they had a catwalk in the lobby area with models walking down it all dayâ"that certainly got me talking. These ideas are completely relevant and applicable to local businesses, and they don't necessarily need to involve huge slides or catwalks! Your products are contentThe products and services that you offer can both be used within other content or as creative content in their own right. You don't need to have the most unique product in the world, and sometimes this isn't actually possible within standardised product industries. Let's look at a coffee shop, for example. We might say that our coffee shop sells a range of different coffees, some sandwiches and a selection of cakes/pastries. The first thing you would think of to make them stand out from the competition is quality. The only problem here is that it's hard to show the quality of these products to someone who hasn't actually tasted them; plus, when all of your competitors claim to have the "best quality products," it can be hard to stand out. With this in mind, why not try something a little different...
source: http://www.flickr.com/photos/krobison/3346410954/ I know, pretty awesome isn't it? Apparently "latte art" is pretty popular, and to be honest, if that was served in front of me, the first thing that I'd do would be to take a picture and share it on Facebook/Twitter.
source: http://www.flickr.com/photos/jamieanne/3991004528/ Seasonal product themes can be another fantastic way to augment your current product offering and turn it into creative content. At the local coffee shop that I go to, they create crazy cupcake variations on Halloween and decorate the whole shop. That's just asking to be talked about on social media and can result in some tasty links as a byproduct. Your product/service collateral can be contentI'm guessing that you can see where I'm starting to go with this idea that anything can be content, no matter how boring it is. Let's take some food and drink menus as an example. In its purest form, a menu should deliver the goal of displaying what your business has to offer to your customers. Now, when we bear in mind that over 90% of the customers in a coffee shop are going to take a look at this menu, there is a huge opportunity to get creative. The above video shows how the Global Mundo Tapas restaurant in the North Sydney Rydges Hotel has replaced all of its menus in favour of actual iPads. The customer can view the whole menu, get more information on each of the different dishes and they can actually place their order live on the tablet. There could be huge potential for linking this in with social media and really connecting the link between online and offline interactions with customers. Another great example is from Duo restaurant in New York. Their menus actually light up when you open them, which makes them look pretty awesome. Again, this has resulted in the restaurant being mentioned on several niche-relevant blogs that will boost their rankings and give them a wider reach for their brand (it made me notice them, and I'm on the other side of the world!). The branding of your business is contentThe values, culture and vision of your company says a lot about your brand. Multinational brands use their slogans as one of their most valued content assets. If I mentioned "Just Do It" or "They're Gr-r-r-eat!" then you would know exactly which companies/products I'm talking aboutâ"this can be the same for local businesses, but on a different scale. One hilarious example of some amazing branding was from a local sofa company near the place where I grew up, called "Sofa King." They drove around in these big white vans every day that had "Our prices are Sofa King Low!" written on them. They got loads of press coverage from this because they were told they had to remove them and actually got a mention on the TV as well. This may not be the most transferable example but I love telling this story to people because it's just so funny! Another great way that a lot of cafes and restaurants can use their company values to build content is by partnering up with higher welfare suppliers and institutions. If we look at the likes of the Rainforest Alliance, an international charity focused around conserving biodiversity and improving the working conditions of third-world suppliers. They actually offer a way to certify your business as being Rainforest Alliance approved. This can then be used to build some positive PR around the business, especially within local publications. Your customers are contentYes, that's right... your customers are content too. If you run a local business then you're likely to be customer-facing a lot of the timeâ"why not take advantage of that? I worked on a recent campaign with a UK restaurant chain where we ran a competition to eat free for a year at the restaurant. All you had to do to enter was like the Facebook page, send a tweet (that we had pre-defined) and also give some feedback on their experience at the restaurant. We ended up with over 10,000 entries into the competition and the Twitter account was going crazy for weeks. We ended up driving back some good quality links to the website in the process by running an effective link prospecting campaign alongside this, but more importantly we grew the social following dramatically and had loads of people talking about the brand. Another tactic that I've used in the past, and a tactic that could be used within the coffee shop example, is to bring in a group of bloggers and let them eat for free in exchange for writing up a blog on their experience. It's important to keep things impartial here, but getting a write up from a respected blogger can do wonders for your brand and will be an awesome link back to your websiteâ"it doesn't cost a lot either. Content is EVERYTHINGAs you've probably now realised, content doesn't just mean blog articles or infographics. Content can come in the shape of all sorts of things, both online and offline. The important thing is to understand the end goal of your content marketing strategy and how this then ties in with your social media and SEO campaigns. When run on their own, content marketing, SEO and social media can be very expensive for local businesses, but when you integrate these together and take full advantage of the resources at your disposal, it will be both cost effective and bring in far greater results. This doesn't need to apply to just local businesses either. A perfect example of this is with the post that I wrote for Moz a short while ago that was a case study on the link building campaign that I carried out for my travel blog. Case studies are a fantastic way to squeeze out more links to your website and can be awesome pieces of content that can apply to blogs, major brands and small businesses alike. A few actionable content ideas for local businessesYour staff:
Your premises:
Your products:
Product/service collateral:
Your customers:
Some comments from our Inbound.org discussionVictor Pan: Content for content's sake doesn't work. âDoing' SEO without knowing what you're doing doesn't work. However, both methods work when executed correctly. It's not whether or not it is done, but rather how it is done. Time is scarce for small businesses. They don't have time to learn the âhow' and execute it correctly on their first try. All too often, they visit the wrong neighbourhood on the web and do what they've learned... or hire the wrong people. Bad blogging (scraping/plagiarism/panda) can get you penalized just as well as bad links (irrelevant neighbourhoods/fishy anchor text/penguin) - so I wouldn't be so quick to say one is better than the other, or that you need both. Have you run into a business that has gone through the hands of a rogue SEO who did low quality content spun from a competitor and a list of comment links from forums? I have. It's not pretty. If I had to choose, I'd say it's easier for small businesses to be smart about content generation. SEO? That's not part of their core business - it's inefficient for them to learn it beyond the absolute fundamentals. Martin Harris: Whilst the above analogies are great, i think the point here is how to get the best out the client's time. Ultimately it's what going to be getting them more traffic and if (like most of my clients) you pay an hourly rate; 3-4 hours a month of SEO specific time, content marketing, won't cut it. But here lies the problem; they should be treated as separate entities. Put it this way: would creating great promotional and traffic relevant content on social media drive more traffic or would ranking for a targeted niche relevant term? It's both. For small businesses, getting natural links from content marketing should be a by-product not a SEO strategy.
Slava Rybalka: 1) as for me, content marketing involves both, and first of all, it's like you said, being creative and notice what is going on in your daily business operations and what you can turn into content 2) I have seen the same effect, however, I tend to focus more on content rather than links, because: 1. Links tend to disappear over time, whereas your contents stays on your site, you don't have control over your links but you have control over your content 2.if you have content that resonates with you target audience, great links will come naturally and recently we have seen the cases when few links can make a difference in search results, since Google is focusing more on quality of links. There are other things that come to my mind but these are 2 main points. View the full discussion here: http://www.inbound.org/articles/view/content-marketing-the-ultimate-seo-office-discussion TL;DR
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read! |
You are subscribed to email updates from Moz Blog To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |
Niciun comentariu:
Trimiteți un comentariu