SEOmoz Daily SEO Blog

---

Duplicate Content: Block, Redirect or Canonical Posted: 06 Jan 2011 03:43 AM PST Posted by benjarriola This post was originally in YOUmoz, and was promoted to the main blog because it provides great value and interest to our community. The author's views are entirely his or her own and may not reflect the views of SEOmoz, Inc. Duplicate content in SEO has been around for quite some time and even if Google has been saying they have been getting smarter and smarter in figuring out the best page to display in the SERPS from a list of duplicate content pages. They claim that it is something less to worry about today, than before. But knowing this issue exist, they give advice from various places, also in support threads, employee blogs, webmaster help videos, and many other places on how we should fix this issue. Some say simply block your duplicate content pages, some say redirect them. Maybe there is no 1 rule that best fits all situations, so I decided to enumerate the various ways to fix duplicate content issues, the differences so you can draw you own advantages and disadvantages to help you judge which method is the best to use for your specific situation. So let's go ahead and review each one. Blocking in Robots.txt Probably this is one of the most common suggestion used by many people, including several people from Google. This is also one of the oldest recommendations in the book and is probably outdated since there are many other things you can do today.   This would work in eliminating duplicate content. Search engine bots will see the robots.txt file and when it sees to exclude a URL of the hosted domain name, this URL is no longer crawled and indexed. Having said that, the only problem in using robots.txt in eliminating duplicate content is some people may be linking to the page that is excluded. That would prevent these links from contributing to your website's search engine ranking. Using the Meta Robots: NoIndex/Follow tag Another way to eliminate duplicate content, is to use the Meta Robots tag noindex/follow: <meta name="robots" content="noindex,follow" /> The rationale behind using this tag is the noindex value is telling search engines not to index the page, thus eliminating duplicate content. And the follow value is telling search engines to still follow the links found on this page, thus still passing around link juice. The problem is there are still some people that believes this does not work. Once it's noindex, most probably it is automatically nofollow as well, but then again, why was the value nofollow and follow invented for the robots meta tag if you are not given the power to separate this out from the index and noindex? Crawled or not, this has to be tested out. I believe Rand has taken Google's word for it  that this tag works. Upon searching around for people that tested this with anchor text using unique words, I found Scott M. Clay from UK doing some test. Well for me, for some reason, can never be satisfied by results and post by other people including Matt Cutts statements sometimes. And the only reason why I haven't tested this myself for a long time was there are just many other alternatives in fixing duplicate content that I didn't find the need to really know how search engines really treat this noindex/follow tag. But if any of the readers has done a good test on this, maybe you can publish your results here and also say how you did your test. The 301 Redirect A lot of people in the industry love the 301 redirect to fix duplicate content. Because so many people have tried it out and many know it works. It has also been abused in many shady ways too, but that's not my topic. So what really happens in a 301 redirect in treating duplicate content? The nice thing about this compared to the two methods above is we are really sure based on statements from the respective search engines, as well as testing by numerous people (which probably includes you, the reader of this blog), knows that a link going to a page that 301 redirects will be considered as a link of the destination page of the redirect. This seems like the ultimate fix to all duplicate content issues, but actually, there is also a good reason to use the next methods I will mention. This blog post though is not about how to do 301 redirects but if ever just in case that is what you were searching for, 301 redirects can be done on the webserver software (Apache, IIS, etc.) or through server-side programming (PHP, ASP/.net, ColdFusion, JSP, Perl, etc.). Probably a good starter guide for different 301 redirect implementations is the guide by WebConfs. The Canonical Link Tag The nice thing about the canonical link tag, search engines behave in the same way how it would look at a 301 redirect. It is not going to index the duplicate content page. Only the destination page will appear in the search engine index. All links going to the duplicate content pages will be counted as links of the main content page.   <link rel="canonical" href="http://(main content page)" /> If Google treats the canonical link tag in a very similar way how 301 redirects are treated, the main difference is what the user experience is. A 301 redirect, well... redirects. While the canonical link tag does not. So you can imagine when this might be better than a 301 redirect, when users may not want to be redirected. Let's say you are browsing a department store website. And a business traveler is looking for different traveling bags and also needed a laptop bag and arrived to a URL like this: http://www.example.com/travel/luggage/laptop-bags/targus/ While let's say there is some computer geek that wants a new laptop and  a bag to go along with it and ended up in a URL like this: http://www.example.com/electronics/computers/laptops/accessories/laptop-bags/targus/ Let's say these two pages are duplicate content pages on the same department store website, but doing a 301 redirect to fix the problem, messes up the user experience. If the buyer's train of thought in this example was to buy different bags, if they get 301 redirected to the computers section, makes them lost and would need to do some extra effort to go back to the luggage. Which the geek laptop buyer looking for different accessories would not want to be redirected to the luggage since he may be looking for more laptop accessories. Although a canonical link tag does not redirect, you still have to choose which one would be the main page search engines would display in search engine results. The Alternate Link Tag The alternate link tag, is very similar to the canonical link tag. Although this is used mainly for International or Multilingual SEO purposes. <link rel="alternate" hreflang="en" href="http://www.example.com/path" /> <link rel="alternate" hreflang="en" href="http://www.example.co.uk/path" /> <link rel="alternate" hreflang="en" href="http://www.example.com.au/path" /> The Canonical link tag will remove all other duplicate content, but for the Alternate link tag, all pages will still be index, but this helps guide Google choose the best result for the individual country versions of Google. And eliminates the problems Google may run into treating pages as duplicate content. To sum things up, here is a simply guide when to use which type of redirect in different cases of duplicate content: Alternate Link Tag International pages, multilingual pages, intended for different countries. Canonical Link Tag Multiple categories and subcategories with different category paths, but the same content. Example: http://www.example.com/products/laptops/sony/ http://www.example.com/products/sony/laptops/ Tracking codes, Session IDs mainly because redirection sometimes interferes with the functionality of the tracking codes and sessions. Example: http://www.example.com/path/file.php?SID=BG47JF448JD6I7TGF439LVFD476 http://www.example.com/path/file.php?utm_whatever=5uck3rs http://www.example.com/path/file.php Different variable orders due to how some CMS platforms are created. Example: http://www.example.com/path/file.php?var1=x&var2=y http://www.example.com/path/file.php?var2=y&var1=x 301 Redirect Cases where a redirection does not bother the user experience such as www and non-www, index files, trailing slashes, hosting IP address. Example: http://www.example.com/ http://example.com/ http://www.example.com/index.html http://www.example.com http://123.123.123.123/ Domain changes, and URL changes of pages that no longer exist. Example: http://www.example.com/old_folder/old_file 301 redirects to http://www.example.com/new-folder/new-file/ http://www.example.net/ 301 redirects to http://www.example.com/ Meta Robots NoIndex/Follow Probably the best place to use this is in a list of archived post, such as a blog. Where the main URL of the individual blog post or the permalink may have content that is posted as a duplicate somewhere in the archive view by date, the category view, the author view, tag topic views, or in the pagination of older blog post from the blog homepage. You cannot really do a 301 redirect, nor do a canonical link tag since these pages may have more than 1 blog post listed and you will have to finalized where the 301 redirect should go or where the canonical link tag should point to. Thus I would take my chances using the Meta Robots tag, NoIndex,Follow, and hopefully all the links still help. Robots.txt I no longer see a need to use robots.txt in duplicate content issues. The natural linking is something too precious to lose. Just use robots.txt to really block of content that does not need to be indexed at all, duplicate content or not.   Disclaimer: Although I have in my examples, PubCon and CSI Miami, both websites do not have duplicate content. The images are for example purposes only. As for SMX East, SMX Advanced London and SMX Australia, these pages also have no duplicate content. Photo of Brett Tabke, was by Andy Beal. CSI Miami photo of David Caruso by CBS Television/Alliance Atlantis. Photo of Danny Sullivan is a photo by SMX/3rd Door Media. All other brands used in this blog post are trademarks or registered trademarks of their respective owners. Do you like this post? Yes No

How to Leverage Year Over Year Data Successfully Posted: 05 Jan 2011 12:22 PM PST Posted by JoannaLord Oh the New Year. How we love you so. There are so many reasons to love you. Let us count the ways: Those "Best of 2010" blog posts finally come to an end. We all have a great reason to buy more stuff than we need because the entire world is on sale. Everyone's energy is sky high. It's incredibly contagious. YES! LET'S DO THIS! WORLD DOMINATION! Rarely does a New Year start without popping a bottle (or four) of champagne, which you all know is a favorite pastime of us Mozzers. We can all get excited as we start baby stepping toward the summer, which means sunshine in Seattle is officially less than 1/2 a year away. Not that we are counting down or anything. On top of all those excellent reasons I'd like to add my favorite part about the coming of a new year to the mix--it's time to explore your Year Over Year (YOY) data! Did you all just freak out like me? Yeah I thought so. What makes YOY data so valuable? YOY data is one of the few datasets that both offers a micro and macro view of your site's performance in one sitting. Throughout the year we work off mini-data segments (today compared to yesterday, week over week, month over month, etc.) and it is challenging to see outside influences. It is incredibly difficult to make valid conclusions off of data that exists in a vacuum. YOY analysis allows us the rare chance to compare large datasets side to side with similar outside factors already accounted for.  We can compare data and isolate out things like; year over year trends, year over year differences, anomalies that don't follow the grain, and so much more. In my opinion, all analysts should take a few hours in the next few weeks and just wander around their YOY analytics. Pull your performance reports, pull your traffic data, pull everything you can from 2009, and then pull it all for 2010. Put them side-by-side and then jump in there. For those of you wondering the easiest way to get YOY data, if you are in GA, note the below screenshot. You can set the "Compare to" dates to be 2010 and 2009, and don't forget you can change the data represented from visits to whatever you are interested in looking at. If you switched analytic packages in the past two years, you might need to pull the data into excel from your two sources and create graphs there. Here is an example of what our Operations team tracks in excel. We will definitely be using this data in the coming year for a lot of sign up YOY comparisons: When it comes to pulling YOY stats, it might take some time to get it formatted in a useful manner. Trust me though, it's worth the few hours you take playing in reports and excel. So what should you be looking at? 1. General health review This is YOY analysis at it's most basic--have any of your site stats absolutely plummeted? As you collect data over the year you may see slight dips, and they not be red flags at the time, but in your YOY analysis you can now see that you are averaging a 15% higher bounce rate across your site that the year before. Three words folks: bad news bears. This is what I mean by a general health review. You have internal goals for your site's performance, and YOY analysis lets you quickly identify any of the metrics that failed to meet your expectations. Specific metrics to keep an eye on are things like; analytic vitals (time on site, bounce rates, etc.) visitor engagement metrics (visitor recency, visitor frequency, etc.), and traffic drivers (branded term performance, your head term traffic, keyword queries that are historically conversion winners, etc.). Other performance metrics to keep front-of-mind are things like value per visit (VPV), and cost per visit (CPV). While it's great that this year you made twice as much more money as the year before, you should also know if you are paying 1/3 more to acquire those successes. What if you could tweak something on performance and get that spend back down while maintaining increased performance? Then you are 133% above last year. These caveats in ROI calculations can make a big difference as you expand channels, and grow programs. 2. Dive deep into significant fluctuations YOY data analysis is a great tool for gauging momentum gained or momentum lost on your site this past year. Do you see seasonal trends that show up year over year, and are they holding true? For some industries this may not be the case, but for quite a few industries you will see peaks and valleys around similar times of year. Pay special attention to the peaks. Believe it or not the valleys are easier to isolate out in smaller data segments, the real questions is did you see the seasonal jumps you usually see each year? If not, you may have an issue brewing. This is must easier to identify in YOY analysis. See below picture for an example of how you can isolate out fluctuations and then easily research if they are accounted for. ** The yellow highlights were peaks and valleys in 2010 that matched the norm for 2009, but the two pink highlights weren't represented in 2009. Further research revealed they were due to us launching the web app and one of Rand's most popular posts of 2010. These are good things to know.  3. Use YOY data to measure the success of last year's company goals I bet you all know if you hit your 2010 revenue goals, don't ya? How about traffic goals? What about your secondary metrics? Things like specific channel goals? Do you know if you are seeing that 10% jump in referral traffic you wanted to get by the end of the year? Do you know if you doubled the number of keywords driving traffic to your site like you wanted? I blame the fact that we are all in the Inbox weeds at the beginning of the year, but I am also guilty of forgetting to circle back on last year's goals. Your YOY data is especially valuable when you want to quickly identify if you accomplished all you set out to do. Plus you can show all of these pretty comparison charts to your boss when you ask for that 2011 bump in pay. 4. Use YOY data to guide this year's company goals We all know that accurate performance projection is an art form. In fact our VP of Marketing, Jamie Steven, is the in-house whiz on this, but I am realizing more and ore that good predictions involve a lot of data crunching. Last year's performance is one of those vital indicators not just on potential success, but on what times of year you may see drops you need to plan ahead for. Having YOY data to work from helps you be even more accurate in those predictions. You can see below what parts of the year we see a drop in one of our key metrics--signups by month. You can imagine that our acquisition goals for next year (set on a week by week basis) will be adjusted accordingly during those times of year. 5. Find the Wild Card of Data Awesomeness What the heck am I talking about? I'm talking about your low hanging fruit. When you do YOY research you often come across one or two metrics that literally have not seen much action or possibly seen recent minimal wins.  With a little effort these guys can be quick wins for the company. If you can isolate out one or two of these and then build marketing campaigns around them, you can see big successes early in the year. It's a great way to start the year, don't you think? A great examples would be to isolate out the long tail keyword wins (those words that ended up converting for you, or ones that now drive significant traffic) of the year before last, and take the ones of this past year, and see if you can combine them for a new category of long tail queries to go after. These would be less competitive to acquire traffic from, and you've already proven they will return. Below is an example from our December 2010 stats. You can see that phrases with "title tag" in them caught wind in Decemeber when compared to November. It would be worth it to see what title tag queries got traction in 2009, and start building content around the topic. People are looking for it, and it could be a quick 2011 content win for us. So there you have it. These are just some of the ways you can take your YOY stats and milk them for data goodies. I know the beginning of the year we are all looking forward, but before you do, I urge everyone to take a little time and look backwards. So cheers to a new year of data collection friends...may this year be full of beautiful charts, insightful finds, and the resulting ROI successes. Do you like this post? Yes No

You are subscribed to email updates from SEOmoz Daily SEO Blog To stop receiving these emails, you may unsubscribe now.	Email delivery powered by Google
Google Inc., 20 West Kinzie, Chicago IL USA 60610