| |||
| |||
SEO Finds in Your Server Logs, Part 2: Optimizing for Googlebot |
SEO Finds in Your Server Logs, Part 2: Optimizing for Googlebot Posted: 29 Jul 2013 07:50 PM PDT Posted by timresnik This is a follow-up to a post I wrote a few months ago that goes over some of the basics of why server log files are a critical part of your technical SEO toolkit. In this post, I provide more detail around formatting the data in Excel in order to find and analyze Googlebot crawl optimization opportunities. Before digging into the logs, itâs important to understand the basics of how Googlebot crawls your site. There are three basic factors that Googlebot considers. First is which pages should be crawled. This is determined by factors such as the number of backlinks that point to a page, the internal link structure of the site, the number and strength of the internal links that point to that page, and other internal signals like sitemaps. Next, Googlebot determines how many pages to crawl. This is commonly referred to as the "crawl budget." Factors that are most likely considered when allocating crawl budget are domain authority and trust, performance, load time, and clean crawl paths (Googlebot getting stuck in your endless faceted search loop costs them money). For much more detail on crawl budget, check out Ian Lurieâs post on the subject. Finally, the rate of the crawl â" how frequently Googlebot comes back â" is determined by how often the site is updated, the domain authority, and the freshness of citations, social mentions, and links. Now, let's take a look at how Googlebot is crawling Moz.com (NOTE: the data I am analyzing is from SEOmoz.org prior to our site migration to Moz.com. Several of the potential issues that I point out below are now solved. Wahoo!). The first step is getting the log data into a workable format. I explained in detail how to do this in my last server log post. However, this time make sure to include the parameters with the URLs so we can analyze funky crawl paths. Just make sure the box below is unchecked when importing your log file.
The first thing that we want to look at is where on the site Googlebot is spending its time and dedicating the most resources. Now that you have exported your log file to a .csv file, youâll need to do a bit of formatting and cleaning of the data. 1. Save the file with an Excel extension, for example .xlsx 2. Remove all the columns except for Page/File, Response Code and User Agent, it should look something like this (formatted as a table which can be done by selecting your data and ^L):
3. Isolate Googlebot from other spiders by creating a new column and writing a formula that searches for âGooglebotâ in the cells in the 3rd column.
4. Scrub the Page/File column for the top-level directory so we can later run a pivot table and see which sections Google is crawling the most
5. Since we left the parameter on the URL in order to check crawl paths, weâll want to remove it here so that data is included in the top level directory analysis that we do in the pivot table. The URL parameter always starts with "?," so that is what we want to search for in Excel. This is a little tricky because Excel uses the question mark character as a wildcard. In order to indicate to Excel that the question mark is literal, use a preceding tilde, like this: "~?"
6. The data can now be analyzed in a pivot table (data > pivot table). The number associated with the directory is the total number of times Googlebot requested a file in the timeframe of the log, in this case a day.
Is Google allocating crawl budget properly? We can dive deeper into several different pieces of data here:
In this example, the directory /webinars pops out as not getting enough Google attention. In fact, only the top directory is being crawled, while the actual Webinar content pages are being skipped. These are just a few examples of crawl resource issues that can be found in server logs. A few additional issues to look for include:
As a bonus, I have done a screencast of the above process for formatting and analyzing the Googlebot crawl. In my next post on analyzing log files, I will explain in more detail how to identify duplicate content and look for trends over time. Feel free to share your thoughts and questions in the comments below! Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read! |
You are subscribed to email updates from Moz Blog To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |
|
The easiest quick opportunity remains the same: Yuppie Information.
What information can you offer the connected and the curious that they don't already have? This group is not only the most eager group of early adopters around, but they are so digitally connected that reaching them is easier than ever before.
No, it's not going to change the world, not right away anyway. But yes, if you're hoping to quickly work your way up the adoption curve, offering timely information to connected, educated, urban youth is in fact a great place to start.
On the other hand, the green fields and real wins will come from connections, mesh businesses and leadership for groups that aren't as peripatetic or spoiled as the digital yuppies.
[You're getting this note because you subscribed to Seth Godin's blog.]
Don't want to get this email anymore? Click the link below to unsubscribe.
Your requested content delivery powered by FeedBlitz, LLC, 9 Thoreau Way, Sudbury, MA 01776, USA. +1.978.776.9498 |
Mish's Global Economic Trend Analysis |
Posted: 29 Jul 2013 12:39 PM PDT David R. Kotok, Cumberland Advisors Inc.'s chairman and chief investment officer believes Municipals Cheap After Detroit Filing Municipal bonds are an "outrageous bargain" in the wake of Detroit's bankruptcy filing, according to David R. Kotok, Cumberland Advisors Inc.'s chairman and chief investment officer. Kotok bases his arguments on a comparison on General Obligation Yields to US Treasuries. Muni vs. Treasury Yields More on Munis In a followup guest post on the Big Picture blog, Kotok offers More on Munis, Detroit, Bloomberg, Whitney & Wilson. In our recent commentary on municipal bonds and Detroit, we argued in favor of buying the highest-grade AAA tax-free municipal bond It currently yields more than the corresponding taxable US Treasury obligation.The Muni Market's Shifting Sands Writer, Mark J. Grant, author of Out of the Box expresses a different viewpoint in The Muni Market's Shifting Sands, a guest post on ZeroHedge. I began my career in this space. Fresh off my internship I was assigned to cover small banks in Missouri and Kansas and sell them Municipal Bonds. My fondest recollection was of a small bank in western Kansas that said he couldn't buy bonds now because, "There are green bugs in the milo." I had no idea what green bugs were then and wasn't too sure about milo.Expect More Fallout The fallout in Detroit is not over. There will be more Detroits for sure. However, we have not seen the final ruling from bankruptcy court. To what extent, if any, will courts rule "General Obligation bonds no longer have the first call on assets"? I don't know and no one else does either. Yet, I suspect that both bondholders and pensioners will take a decent-sized hit. A friend emailed a commonsense point of view earlier today. "Fairness of the pension level is irrelevant. It's what the debtor can afford. And Detroit can't afford much. In municipalities, as in private employment, the cost of getting a pension package your employer can't afford is ultimately that you don't have a pension package. This will be much tougher than private bankruptcies, though, because the PBGC does not cover municipal employee plans. So the estate of Detroit will have to pay something, because it is intolerable that old policemen and firemen suddenly resort to complete welfare. But it can't pay much. This will be brutal." Three Bad Assumptions Kotok points out that no AAA rated GO bond has ever defaulted. OK, but have we ever had a pension crisis before? Please consider three widely-held assumptions on GO bonds.
All three of those assumptions are false. And how many bonds are rated AAA because of those assumptions? Definition of "Cheap" The top chart shows that munis are cheap compared to treasuries (assuming no defaults). But does that make them cheap? This is not a position I endorse, but for the moment let's assume that Kotok is correct. That AAA rated bonds will not default. Are they cheap? Compared to what? It depends on the definition of cheap. What if treasuries are not "cheap"? After all, the same chart Kotok showed a few days ago would have shown the same thing at the beginning of May. 10-Year Treasury Yield Yield on the 10-year treasury rose over 100 basis points since the beginning of May. US treasuries were not cheap then. Thus, the notion of something is cheap compared to something else is a false dichotomy. At the beginning of May, neither treasuries nor munis were cheap. And what if neither is still cheap? What if the pension crisis shifted the sands? What if widely-held assumptions about AAA and defaults are invalid? As you can see, it's not as simple as Kotok's "GO's will not default so buy them because they are cheap compared to treasuries" point of view. Things changed, sands shifted, and right now, we really do not know by how much. So it's quite the stretch, for numerous reasons, to say munis are an "Outrageous Bargain". Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
Carl the Robot Bartender Mixes Drinks and Chats With Customers Posted: 29 Jul 2013 10:59 AM PDT Robots are taking on increasingly complex tasks. As a case in point, Carl the robot bartender serves customers at German bar. He's handy with a shot glass and customers travel from far and wide to admire him at work.Carl-like robots are not about to replace human bartenders en masse just yet, but technology is certainly advancing at breakneck speed. I find these kinds of articles fascinating. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
Former ECB Chief Economist Warns "ECB Will Soon Have to Support France with Bond Purchases" Posted: 28 Jul 2013 11:50 PM PDT Juergen Stark, former ECB chief economist (who resigned in 2011 over a dispute regarding bond purchases), says in an interview in Handelsblatt "The Euro crisis will worsen in late autumn" Via Google Translate A year ago, ECB chief Draghi announced plans to do anything to save the euro. The former ECB chief economist Juergen Stark considers this fatal. He fears that the ECB will soon have to support France with bond purchases.Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
You are subscribed to email updates from Mish's Global Economic Trend Analysis To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |
Damn Cool Pics |
The Powerful Effect of Bass [Video] Posted: 29 Jul 2013 06:39 PM PDT |
Posted: 29 Jul 2013 06:36 PM PDT |
Posted: 29 Jul 2013 05:46 PM PDT |
Tihana Nemcic, The Sexiest Football Coach Ever Posted: 29 Jul 2013 05:28 PM PDT |
You are subscribed to email updates from Damn Cool Pictures To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |
Facebook Twitter | More Ways to Engage