Illustrated Guide To Web Experiments

---

Illustrated Guide To Web Experiments Posted: 26 Mar 2012 02:16 PM PDT Posted by Thomas Høgenhaven Web experimentation is a great tool to increase engagement and conversion rates. The primary strength of experiments is the possibility to isolate variables, and thus examine causality between different metrics such as tagline and conversion rate.   Much of the literature on experimental design has its roots in statistics and can be quite intimidating. To make it more accessible, I introduce the illustrated guide to web experiments (with some help from my brother, Andreas Høgenhaven, who kindly made the illustrations).   Before getting started on the experiment, you need to get the basics right: Test metrics that align with your long term business goals. Test big changes, not small. And remember that the test winner is not the optimal performance, but only the best performing variation we have tested. It doesn’t mean that you have found the all time optimal performing variation. You can (almost) always do better in another test.   A/B or MVT One of the first things to consider is the experimental design. An A/B test design is usually preferred when one or two factors are tested, while a multivariate test (MVT) design is used when two or more independent factors are tested. However, it is worth noting that 2+ factors can be tested with A/B/n tests or with sequential A/B tests. The downside of using A/B test for several factors is that it does not capture interaction effects. MVT Face-off: Full Factorial vs Fractional Factorial So you want to go multivariate, huh? Wait a second. There are different kinds of multivariate tests. If you have ever visited Which MVT, you probably came across terms such as full factorial, fractional factorial, and modified taguchi. Before getting into these wicked words, let's get our multivariate test down to earth with an example. In this example we have 3 different factors, and each factor has two conditions.     In this case there are 3 factors each with 2 combinations, giving a total of 23 = 8 groups. In the full factorial design, all possible combinations are tested. This means 8 variations are created, and users are split between these. In the following table, +1 Indicates condition on while -1 indicates condition 2.     This design is not too bad when we have 3 factors with 2 conditions in each. But if we want to test 4 factors each comprising 4 conditions, we will have 44 = 256 groups. Or if we want to test 10 different factors with 2 conditions in each, we will end up with 210 = 1,024 groups. This will require a lot of subjects to detect any significant effect of the factors. This is not a problem if you are Google or Twitter, but it is if you are selling sausages in the wider Seattle area (You can calculate the test duration time with Google's Calculator and VisualWebsiteOptimizers Calculator. These calculators are, however, based on very imprecise data because the change in conversion rate is unknown. That is kinda the point of the test).   Enter fractional factorial design. The fractional factorial design was popularized by Genichi Taguchi and is sometimes called the Taguchi design. In a fractional factorial design, only a fraction of the total number of combinations are included in the experiment. Hence the name. Instead of testing all possible combinations, the fractional factorial design only tests enough combination to calculate the conversion rate of all possible combinations.   In this example, it is sufficient to run 4 different combinations, and use the interaction between included factors to calculate combination of factors not included in the experiment. The 4 groups included are ABC; A + (BC); B + (CA); C + (BA).   Instead of testing Factor A 3 times, it is only tested once while holding B and C constant. Similarly, Factor B is tested once while holding A and C constant, and Factor C tested once while holding A and B constant. I'll not deep too deeply into the statistics here, as the experimental software does the math for us anyway.   The fractional factorial test assumes that the factors are independent of one another. If there are interactions between factors (e.g. image and headline), it'd affect the validity of the test. One caveat of the fractional factorial design is that one factor (e.g. A) might be confounded with two-factor interactions (e.g. BC). This means that there is a risk that we end up not knowing if the variance is caused by A or by the interaction BC. Thus, if you have enough time and visitors, full factorial design is often preferable to fractional factorial design.   Testing The Test Environment With The A/A Test Most inbound marketers are quite familiar with A/B tests. But what is less known is the A/A test. The A/A test is useful as a test of the experimental environment, and is worth running before starting A/B or MVT tests. The A/A test shows if the users are split correctly, and if there are any potential misleading biases in the test environment.     In the A/A design, users are split up like they are in an A/B or MVT test, but all groups see the same variation. We want the test results to be non-significant, and thus see no difference between the groups. If the test is significant something is wrong with the test environment, and subsequent tests are likely to be flawed. But as discussed below, an A/A test is likely to be significant sometimes, due to random error / noise.   The A/A test is also a good way to show co-workers, bosses, and clients how data fluctuate, and that they should not get too excited when seeing an increase in conversion rate with 80% confidence. Especially in the early phases of experiments.   Statistical Significance In the ideal experiment, all variables are held constant except the independent variable (the thing we want to investigate, e.g. tagline, call to action, and images). But in the real world, many variables are not constant. For example, when conducting an A/B test, the users are split between two groups. As people are different, the two groups will never comprise similar individuals. This is not a problem as long as the other variables are randomized. It does, however, inflict noise in the data. This is why we use statistical tests.     We conclude that a result is statistically significant when there is only low probability that the difference between groups is caused by random error. In other words, the purpose of statistical tests is to examine the likelihood that the two samples of scores were drawn from populations with the same mean, meaning there is no "true" difference between the groups, and all variation is caused by noise.     In most experiments and experimental software, 95% confidence is used as the threshold of significance, although this number is somewhat arbitrary. If the difference between two group means is significant at 98% probability, we accept it as significant even though there is a 2% probability that the difference is caused by chance. Thus, statistical tests show us how confident we can be that difference in result are not caused by chance / random error. In Google Website Optimizer, this probability is called chance to beat original.   Pro Tip: Ramp Up Traffic To Experimental Conditions Gradually One last tip I really like is ramping up the percentage of traffic sent to experimental condition(s) slowly. If you start out sending 50% of the visitors to the control condition, and 50% to the experimental condition, you might have a problem if something in the experimental condition is broken. A better approach is to start sending only 5% of the users to the experimental condition(s). If everything is fine, go to 10%, then 25% and finally 50%. This will help you discover critical errors before too many users do it.   Ps. If you want to know more / share your knowledge on experiments and CRO tools, you might want to have a look at this CRO Tools Facebook Group. Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Unnatural Link Warnings and Blog Networks Posted: 26 Mar 2012 04:26 AM PDT Posted by Carson Ward Advice and Confessions from a Reformed Link Network Spammer There was only one time when link building was the easiest, least challenging part of my SEO work. I was a link network spammer, relying upon services with names like "blog networks" and "article networks." These services allow paid subscribers to post their content to a network of sites for the sole purpose of building links. Because the sites aren't meant to be read by people, many networks accept and actively encourage spun content. The resultant content quality is absolutely horrific. I pulled this example from an active blog network site. "Sodium Body of water Town" is spun garbagese for "Salt Lake City." Using blog networks, I had multiple sites penalized, re-included, and, once I had learned the ropes, I even had a few that gained rankings and escaped unharmed. I ultimately gave up my spam-content ways because it became clear that it was not an effective long-term strategy. The writing was on the wall - Google was getting smarter, and I was at risk of losing any time I invested. And the writing was, and still is, on the wall. Google rolled out Panda, which dealt a heavy blow to some blog and article networks that had paid almost no attention to users. Starting this year, blog networks, both private and public, starting dropping. In one of the highest-profile incidents, Google crippled BMR's blog network. Webmasters began receiving warnings in Google Webmaster Tools around the same time that now strike fear into the hearts of those using manipulative or questionable tactics to build links:  Google Webmaster Tools notice of detected unnatural links to http://example.com/ Dear site owner or webmaster of http://example.com/, We've detected that some of your site's pages may be using techniques that are outside Google's Webmaster Guidelines. Specifically, look for possibly artificial or unnatural links pointing to your site that could be intended to manipulate PageRank. Examples of unnatural linking could include buying links to pass PageRank or participating in link schemes. We encourage you to make changes to your site so that it meets our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results. If you find unnatural links to your site that you are unable to control or remove, please provide the details in your reconsideration request. If you have any questions about how to resolve this issue, please see our Webmaster Help Forum for support. Sincerely, Google Search Quality Team In short: you're caught - the game is up. Some sites received this warning without penalties, and some sites were penalized immediately. In either case, the links need to be taken down to retain or regain rankings. So what now? Identifying Posts from Blog Networks There are plenty of webmasters, site owners, and SEOs who have no idea what Google is talking about when they receive the warning. For example, I recently spoke with a friend from a reputable SEO agency who had the bad lack of taking over the same week his client was penalized for the previous SEO firm's work. In any case, the first step to fixing the problem is finding the links in question. We can use tools like Open Site Explorer and Google/Bing Webmaster tools to find bad links. Because BMR's sites have already been outed, penalized, and rendered useless for the foreseeable future, we can use an old BMR site as examples of what to look for: That's right - you can get misinformation about water shoes, Dubai shopping, and constipation remedies - all in one place! BMR posts actually had limited quality reviews, making them higher quality than most existing blog networks.  The identifying marks of a blog network post are as follows: Terrible content and a boring template; you find yourself asking, "Why would anyone ever subscribe to this blog?" Topics are jumped, unrelated to the site's theme, and categorization is poor. There is rarely an about section, author name, or means of contact. Lots and lots of exact-match anchor text seemingly pointing to sites at random. Posts tend to be 400-500 words with 2-3 links per post - generally all to the same site. If you have recently received an unnatural link warning, there's a pretty good chance that you have a lot of links like this in your link portfolio. You will also wish to look for footer/blogroll links, especially from irrelevant sites, and any other links that were clearly not intended to be viewed by visitors.  Fixing and Recovering From Unnatural Link Penalties There is a way back to Google's good graces, but it's not going to be fast, and it's unlikely that your traffic will reach the same heights it once did if you relied heavily on link networks or paid links to gain rankings. Once you're caught, you must sacrifice all paid or spammy links, submit a reconsideration request, and develop a legitimate backlink portfolio. Removing Penalization-Inducing Links The removal of links has to be thorough, or the reconsideration request will be denied. If you rent links on a monthly basis, the obvious step is to stop paying for the links and request their removal. If your links were built by an external company, contact them and see what they can do about taking any links from blog networks offline. For BMR users, I recommend clicking the following link, which BMR was professional enough to offer: Take them all down! Now leave the site without exporting those low-quality posts that you won't need, thanks to the fantastic new link-building strategy you're going to develop. Other link networks often provide simple solutions to taking down your links - speak with the person who put these links up if you don't have access. In the case of paid links not submitted through a blog network, you may have to contact the site directly and request removal of links. It's a tedious process, but a reasonable effort has to be made.  Submitting a Reconsideration Request If your site hasn't been penalized yet, but you received the warning, you can skip this step. Do not skip the step above, as you will eventually face into a penalty if you don't clean up your act and link portfolio. In a blog post on 6 Ways to Recover from Bad Links, Dr. Pete offers some advice for reconsideration requests: Be honest, specific and detailed. Show that you’ve made an effort. Act like you mean it (better yet: mean it). You have to explain that you have changed your views and your strategic focus. A good way to show effort is by including a link to an accessible Google Doc spreadsheet showing the bad links, which ones were removed, and which ones you made unsuccessful efforts to remove. Be specific, and touch on everything requested on the reinclusion request form: Tell us more about what happened: what actions might have led to any penalties, and what corrective actions have been taken. If you used a search engine optimization (SEO) company, please note that. Describing the SEO firm and their actions is a helpful indication of good faith that may assist in evaluation of reconsideration requests. If you recently acquired this domain and think it may have violated the guidelines before you owned it, let us know that below. In general, sites that directly profit from traffic (e.g. search engine optimizers, affiliate programs, etc.) may need to provide more evidence of good faith before a site will be reconsidered. Be open and specific about what you were doing, what you changed to comply with Google's guidelines, and what you will do going forward. Right now there is doubtless a long line of websites requesting reinclusion, so make sure you've done a good clean-up. The last thing you want is to have to go through the whole process again in a few months. You'll also have to be patient, especially if you're a smaller site or lesser-known brand. In the meantime, though, we will develop a legitimate link-building strategy. Build a Legitimate Link Portfolio Under most conditions, Google appears to assess link penalties algorithmically. Most of the sites that I have seen receiving warnings about unnatural links have serious problems with the over-optimization of anchor text and links from low-quality sites. Low quality sites, in this diagram, refers to sites (and pages) that have little or no relevance, few incoming links, unnatural link portfolios of their own, and few branding signals. Having too much exact-match anchor text from legitimate domains is a hard thing to do, but it has happened through things like widgets with unnatural anchor text. Building links from legitimate sources is hard, and in competitive and boring industries it take a lot of creativity and work. Some have taken this as a license to manipulate rankings and build spammy links. In competitive and boring industries; however, the online world looks very much like the real world. The secret to success in both is a unique selling point (USP): what makes you different or better than your competitors? Your USP can be customer experience, site interactivity, prices, or content resources. Online businesses will profit in much the same way that offline businesses did and continue to. It is time to start thinking of ways to build links and attract users in a way that is scalable, effective, and long-term. Building links manually is boring, difficult, and often unrewarding. There's a reason a lot of industry leaders have been talking about content marketing. New Questions and Concerns There's a lot of change in this industry, and we've become good at adapting and changing our roles. As with all significant changes, there have been a flood of questions about what to do. First, I do not think that this was related to Google's reported semantic search or the upcoming over-optimization penalty. The later was was pre-announced after large numbers of sites had already been pummeled my something else entirely.  My Blog Network is Running Strong. Should I Stop? Yes. Those who have not received warnings for using blog networks should recognize that blog networks are not a sustainable long-term strategy. You're spending time building bad links instead of relationships and branding. Additionally, Google has hit several blog networks, and it's likely to continue. Standing and waiting for the hammer to fall is strategy at its absolute worst. Couldn't I Hurt Competitors? The last defense for the link spammer is a fallacious line of reasoning: "search engines can't penalize me, because I could do it to my competitors." Yes, in theory, you could trigger a link spam penalty on a competitor site. You will find; however, that companies with strong branding signals who have built real editorial links - usually the companies that actually rank highly - are nearly impervious to link spam attacks.  Yes, you can do horrible things to other people - but why? Ethics aside, it just doesn't make sense in this industry to waste effort tearing others rather than building a site up. Restaurant owners don't go around attacking nearby restaurant owners' stores, because it takes a lot of effort and, even if you do temporarily close a restaurant, it doesn't really bring new customers. There are a lot of restaurants in the city - and even more sites on the internet. Note that there may be legal ramifications if you successfully harm a competitor with link spam. Once the subpoenas start falling, the invincible feeling of anonymity disappears quickly. Strategies for Agencies One line in the reconsideration request stood out to me: "Describing the SEO firm and their actions is a helpful indication of good faith" People rarely describe a company without mentioning its name. What actions might we expect from firms mentioned in multiple reconsideration requests? Direct action is unlikely, but companies who relied on networks to build links to clients' and their own sites may be wise to worry, devise a new strategy, and stop publishing crap. I have absolutely no interest in the obnoxious and mostly-imaginary "war" between white hat and black hat SEO. Whether you care about adding value to users - and at the world of link penalties and Panda, it's insane not to - it's time to drop tactics and schemes like blog networks. The links have been low value for a long time, but now they carry with them an unacceptable risk. Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

You are subscribed to email updates from SEOmoz Daily SEO Blog To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google Inc., 20 West Kinzie, Chicago IL USA 60610