|
|
The rear view mirror is one of the most effective motivational tools ever created.
There's no doubt that many people speed up in the face of competition. We ask, "how'd the rest of the class do?" We listen for someone breathing down our necks. And we discover that competition sometimes brings out our best.
There's a downside, though. Years ago, during my last long-distance swim (across Long Island Sound... cold water, jellyfish, the whole nine yards), the competitiveness was pretty thick. On the boat to the starting line, there were hundreds of swimmers, stretching, bragging, prancing and working themselves up. By the time we hit the water, everyone was swimming someone else's race. The start was an explosion of ego and adrenaline. Twenty minutes later, half the field was exhausted, with three hours left to go.
If you're going to count on the competition to bring out your best work, you've surrendered control over your most important asset. Real achievement comes from racing ahead when no one else sees a path--and holding back when the rush isn't going where you want to go.
If you're dependent on competition then you're counting on the quality of those that show up to determine how well you'll do. Worse, you've signed up for a career of faux death matches as the only way to do your best work.
Self motivation is and always will be the most important form of motivation. Driving with your eyes on the rear view mirror is exhausting. It's easier than ever to measure your performance against others, but if it's not helping you with your mission, stop.
[You're getting this note because you subscribed to Seth Godin's blog.]
Don't want to get this email anymore? Click the link below to unsubscribe.
Your requested content delivery powered by FeedBlitz, LLC, 9 Thoreau Way, Sudbury, MA 01776, USA. +1.978.776.9498 |
Mish's Global Economic Trend Analysis |
IMF Needs Funding; My Dear Darling Posted: 25 Sep 2011 08:47 PM PDT Things are so dire the IMF is out of cash. Don't take that from me, take it from Christine Lagarde, head of the IMF. The Telegraph reports IMF may need billions in extra funding, says Lagarde Christine Lagarde said the money available to the organisation "pales in comparison to the potential financing needs of vulnerable countries".My Dear Darling My dear Darling, you are a complete fool (but no, I don't love you anyway). Here are the "Lessons of Lehman" Lessons of Lehman
Darling continues .... "The problem with the Greek crisis is that it has been allowed to run on and on and on."I happen to agree with that Darling sentence, unfortunately I completely disagree with the context. The smart thing to do would have been to let Greece default. It is now clear Greece is going to default anyway. History will show the world will not end. It will also show that bailouts to date have done nothing but harm. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
Desperate Times Lead to Desperate Lies; Europe Weighs the Weightless; Even Citigroup Sees the Lies Posted: 25 Sep 2011 05:35 PM PDT Are central bankers and politicians really as stupid as they sound or are they pathological liars who simply cannot help it? Check out these preposterous lies by Bank of France Governor Christian Noyer as quoted by Bloomberg in Noyer Sees 'Absolutely No Reason' to Use Bank Backstop Noyer Lies
All of those are blatantly preposterous. However, lie number 1 has to be one of the top lies of the year. "French banks have no toxic assets"?! For starters, what about Greek bonds about to take a 50% haircut or more in default? That lie is so ridiculous no one on the planet can possibly believe it. Greece vows to stay in the euro, never go bankrupt In case you need more laughs this Sunday, please consider Greece vows to stay in the euro, never go bankrupt Greek Finance Minister Evangelos Venizelos sought to reassure nervous markets and EU partners on Saturday by pledging his debt-ridden country would do whatever it takes to avoid default and stay in the euro zone.No Greek Paper Will Go Uncovered!? Excuse me for pointing out two simple facts.
Europe Weighs the Weightless, Comes Up with Wrong Answer Bear in mind the German supreme court has ruled out permanent bailout mechanisms. Moreover, the odds that Finland, Austria, Slovakia and the Netherlands would approve them is close to zero. Nonetheless, Europe Weighs Speedier Enactment of Permanent Rescue Fund to Stem Crisis European governments are exploring speeding the start of a permanent rescue fund for their cash- strapped economies amid fresh signs they may bolster efforts to halt the worsening sovereign debt crisis.Citigroup and JP Morgan Dismiss the Lies Lies from Europe are now so preposterous that even Citigroup can spot them. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
In Galling, Pathetic Whine, Deutsche Bank CEO Warns Against Opening "Pandora's Box" Posted: 25 Sep 2011 10:33 AM PDT In a galling, pathetic whine, Josef Ackermann, CEO Deutsche Bank warns of "Opening Pandora's Box" if banks have to take responsibility for their poor decisions. Yahoo! Finance reports Bank lobby rejects reopening of Greek rescue deal The international bank lobbying group that has been taking a leading role in negotiations on giving debt-ridden Greece easier terms for its bonds on Sunday rejected calls to impose larger losses on private investors.As noted moments ago in German Banking Group Warns of Contagion, Requests Taxpayers to Take Hit for Stupid Loans Made by Stupid Banks admission that losses on Greek loans will be far greater than 21% is starting to spread. Ackermann is a Liar There is no polite way to say it so I may as well be blunt: Josef Ackermann is a liar. Banks agreed to the 21% haircut in July for one reason and one reason only: It was nowhere near "fair" and banks knew they got off easy. The only "fair" proposal is for those who make stupid decisions to pay full price for those decisions. Ackermann wants taxpayer to bail him out. Screw Ackermann. And if Deutche Bank goes under as a result, Ackerman and the entire board should be fired. Indeed the entire banking system would be better off if failed bank executives get shown the door and their pensions set to zero when banks fail. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
Posted: 25 Sep 2011 10:14 AM PDT Slowly but surely (except from France), bankers are willing to admit things are more screwed up than they previously stated. For example, please consider German Banking Group Warns of further Greek Charges. Germany's private banks need to prepare for further fallout from the euro zone's sovereign debt crisis, the head of Germany's BdB banking association said on Sunday.Did you catch that? Schmitz is pleading for passage of the EFSF to dump losses made by stupid risk-seeking banks on the backs of overburdened taxpayers. The "private investors" he wants to bail out are the banks and the bondholders. Recovery will be delayed and moral-hazard policies rewarded once again unless banks and bondholders pay for the stupid risks they took. As I have noted before, banks are not lending anyway, so there is no risk of having responsible parties take responsibility for their actions. Indeed, the way to increase lending is to require bondholders (not taxpayers) to bring banks up to required capital levels. Thus, private investors should pay 100% of the price for their poor decisions. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
You are subscribed to email updates from Mish's Global Economic Trend Analysis To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |
The placebo effect isn't a lie. In fact, if you believe something is going to help you get better, it may very well do just that.
This very same effect works with stereo equipment, wine, politicians... just about everything where our belief intersects with reality.
You can believe that Ford is better than Chevy, that California reds are better than French ones and that your particular tribe is right (and that everyone else is wrong.)
Marketers love the placebo effect because it opens the door to stories and fables and word of mouth and varied perceptions. It gives marketers room to sell more than price and features. The first cultural byproduct this benefit creates is the notion that everyone is entitled to believe what they believe, and it's rude to question it.
The second, is a real problem, though. If you spend enough time experiencing your own take on reality, you come to believe that what works for you might actually be a universal truth. Marketing plus psychology might equal science, it seems.
For the placebo to work, you have to believe it, but sometimes believing requires suspension of your connection with verifiable fact.
When that happens, we might believe that we're entitled to believe things that conflict with demonstrable truth and an understanding of reality. With enough internal spin, you can believe that the moon walk was a fake, that levitation is possible and that the world is only 6,000 years old. You are welcome to believe that aqua metals will improve your sports performance and that z-rays will cure your arthritis, but only until it collides with things that are actually true. Placebos are a good thing, and everyone is entitled to their own beliefs, but they're not entitled to their own science.
We now have to deal with the fallout from personal science. We've so blurred the lines between stories we tell ourselves and our perception of the outside world that it's easy to be confused and easier still to confuse others if it advances your cause.
Consider the fact that the world is getting warmer. To be clear, everyone is entitled to have an opinion on what to do about global warming. The question I'm wondering about is whether we should solicit the opinions of the population as to whether or not it exists. We're asking people to bring their knowledge of statistics, earth science and atmospherics to bear on analyzing data... Of course, most people don't have that knowledge, or care that they don't. If all that matters is belief, why should they?
Dylan told us that you don't need a weatherman to know which way the wind blows... I'm not sure you need to take a poll either.
Before you send me an angry email, consider that the question of what we should do about the trend is a different discussion, one that should be had. The question of how (or if) we should take action is not what this post is about. The trend I'm concerned with is the notion that we're entitled to get upset when the truth doesn't match our point of view. Does the weather care what you think?
[You're getting this note because you subscribed to Seth Godin's blog.]
Don't want to get this email anymore? Click the link below to unsubscribe.
Your requested content delivery powered by FeedBlitz, LLC, 9 Thoreau Way, Sudbury, MA 01776, USA. +1.978.776.9498 |
Mish's Global Economic Trend Analysis |
Multi-Trillion Euro Bailout Plan Allegedly in the Works; Plan Has Failed Already Posted: 24 Sep 2011 08:17 PM PDT The rumor mills are flying this Saturday regarding a Multi-trillion plan to save the eurozone. Telegraph: European officials are working on a grand plan to restore confidence in the single currency area that would involve a massive bank recapitalisation, giving the bail-out fund several trillion euros of firepower, and a possible Greek default. German and French authorities have begun work on a three-pronged strategy behind the scenes amid escalating fears that the eurozone's sovereign debt crisis is spiralling out of control. Their aim is to build a "firebreak" around Greece, Portugal and Ireland to prevent the crisis spreading to Italy and Spain, countries considered "too big to bail". Mish: If that's the plan it, it has failed already. The crisis has already spread to Spain and Italy. In fact, one look at European bank stocks says it has spread to France and Germany as well. Telegraph: Sources said the plan would have to be released as a whole, as the elements would not work in isolation. Mish: Lovely. In a typical bicycle wheel if one spoke gets broken the wheel still works fine. In the proposed wheel, if a spoke breaks, the bicycle crashes. Telegraph: First, Europe's banks would have to be recapitalised with many tens of billions of euros to reassure markets that a Greek or Portuguese default would not precipitate a systemic financial crisis. The recapitalisation plan would go much further than the €2.5bn (£2.2bn) required by regulators following the European bank stress tests in July and crucially would include the under-pressure French lenders. Mish: Will French leaders and French banks go along? Just last week they were insistent that French banks were well capitalized. Telegraph: Officials are confident that some banks could raise the funds privately, but if they are unable they would either be recapitalised by the state or by the European Financial Stability Facility (EFSF) – the eurozone's €440bn bail-out scheme. Mish: Recapitalized "by the state" means taxpayers. Will Germany, Finland, Austria, and the Netherlands go along? Telegraph:The second leg of the plan is to bolster the EFSF. Economists have estimated it would need about Eu2 trillion of firepower to meet Italy and Spain's financing needs in the event that the two countries were shut out of the markets. Officials are working on a way to leverage the EFSF through the European Central Bank to reach the target. The complex deal would see the EFSF provide a loss-bearing "equity" tranche of any bail-out fund and the ECB the rest in protected "debt". If the EFSF bore the first 20pc of any loss, the fund's warchest would effectively be bolstered to Eu 2 trillion. If the EFSF bore the first 40pc of any loss, the fund would be able to deploy Eu1 trillion. Using leverage in this way would allow governments substantially to increase the resources available to the EFSF without having to go back to national parliaments for approval, which in a number of eurozone countries would prove highly problematic. Mish: This leveraged proposal with the ECB backing it up has already been rejected by the ECB. Moreover, such a proposal with the ECB taking leveraged risk would be in violation of the Maastricht Treaty. Telegraph: Gathering turmoil in financial markets has convinced Germany to begin work of some kind of variant of the US plan, despite having initially rejected the notion as unworkable as threatening to compromise ECB independence. The proposal would be hugely sensitive in Germany as its parliament has yet to ratify the July 21 agreement to allow the EFSF to inject capital into banks and buy the sovereign debt of countries not under a European Union and International Monetary Fund restructuring programme. The vote is due on September 29. Mish: The current EFSF proposal is sketchy enough already. It will likely pass. However, Merkel may go down in flames because of it. The Guardian notes "Merkel looks sure to win the Sept. 29 vote on the European Financial Stability Facility because opposition parties support the bill, designed to give the EFSF more powers after an agreement by EU leaders in July. However, her job could be on the line if she has to rely on the opposition and fails to persuade rebels from her conservative camp and the Free Democrats (FDP), her junior coalition partners. Opposition parties have said Merkel would be finished politically if that were the case and have threatened to call for fresh elections. If that happened, the ensuing uncertainty would send shockwaves through the euro zone as it tries to tackle its debt crisis." Bear in mind the above mess pertains to the existing proposal for 440 billion Euros. What would the vote be for a €2.5 billion proposal? Telegraph: As quid pro quo for an enhanced bail-out, the Germans are understood to be demanding a managed default by Greece but for the country to remain within the eurozone. Under the plan, private sector creditors would bear a loss of as much as 50pc – more than double the 21pc proposal currently on the table. A new bail-out programme would then be devised for Greece. Mish: Will the ECB, IMF, and France go along with that? What about the German parliament? Telegraph: Officials would hope the plan would stem the panic in the markets and stop bond vigilantes targeting Italy and Spain, which European and IMF figures believe should not be in any immediate distress but are in need of longer-term structural reform. Mish: So here we are, with a half-baked 2+ trillion Euro proposal, highly likely in violation of the Maastricht Treaty, that all 17 nations in the Eurozone would have to approve. Finland, Austria, the Netherlands, and Germany are already balking over various proposals and Finland in particular wants collateral. This multi-trillion idea is "in hope the plan would stem the panic in the markets and stop bond vigilantes targeting Italy and Spain". The plan is supposed to pass by November? Really? And the aim is to spend 2 trillion to stop something from happening that has already happened. Hope springs eternal. http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
EU Synopsis: No Method, No Strategy, No Calendar; Blatant Lies by French Finance Minister Posted: 24 Sep 2011 09:29 AM PDT The number of arrogant, unbelievable lies from the EU and ECB are flying so fast I do not have time to report on them all. Please check out these lies by French finance minister, François Baroin as reported by the New York Times in Amid Warning Signs, Hints That Europeans May Step Up Action Rising alarm that Europe is not addressing its economic problems with sufficient speed or force pervaded and soured the annual gatherings here on Friday of economists and policy makers from around the world.No Method, No Strategy, No Calendar François Baroin would not repeat the communiqué for the simple reason there was not a damn thing in it. I commented on the communiqué in Asia Pacific Equities Slump Following Ridiculous Pledge by G-20 Check out this statement by the G-20 following the recent multi-day plunge in global equities and commodities: "We commit to take all necessary actions to preserve the stability of banking systems and financial markets as required"OK François...
No Strategy, No Method There is no method, and there is no strategy. There is only talk of producing a strategy by November. Good luck with that given all the EU infighting. At this juncture I doubt the market will wait until November even if the EU could come up with a viable alternative to default (which it can't). Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
Why did Gold and Silver Plunge? No, It's Not CME Margin Hikes; What will the Fed do Next? Posted: 24 Sep 2011 01:17 AM PDT Many people have asked me to comment on the plunge in gold and silver. First let's take a look at the wrong answer: Case Closed: CME Hikes Gold, Silver, Copper Margins And there you have it: CME just hiked gold margins by 21%, silver by 16% and copper by 18%. Mystery solved.Sorry Tyler, wrong answer. Four Reasons for Metals Plunge
1. Fed Did Far Less than Expected The Fed did not do what everyone thought, which is to say something far more than "Operation Twist". As noted in advance, I explained why the Fed wouldn't do more than Operation Twist, in Six Things the Fed May Announce Tomorrow (But Likely Won't); Would Any of Them Matter? Gaming the Reaction. In short, the Fed did not print, or even threaten to print. Moreover the Fed committed to a strategy not through the end of this year, but all the way through June of 2012. Perhaps the Fed does more in the interim, perhaps not. For those expecting drama, the Fed's non-action was decidedly bearish for commodities in general, even gold. 2. Mutual Fund Redemptions Mutual fund cash levels are at or near record lows. In general, mutual funds were not prepared for the market selloff and sell orders came in. Rather than sell garbage like Bank of America at $6, mutual funds unloaded stuff like gold, taking profits. 3. Margin Calls at Hedge Funds Hedge funds unloaded gold and silver for the same reasons as mutual funds, but also because they mistimed the play and what Bernanke would do. Leverage works both ways. 4. China Growth Story Fading Commodities in general have been clobbered along with currencies of commodity producing countries because the global economy is slowing rapidly. As in 2008 there will be no decoupling. China is not a growth engine in any real sense of the word. Instead, China desperately needs demand from the US and Europe. Moreover, China is overheating and has a huge property bubble to boot, at precisely the wrong time. Commodities were set to plunge on the China story alone. Metals Volatility In the wake of increased volatility related to the above, the CME hiked margins. That likely added to the volatility but was not a fundamental "cause" of the plunge in precious metals. What will the Fed do Next? "Bay of Pigs" asks ... Mish, Any thoughts on what the FED will do next? I doubt they sit there and do nothing. Thanks Bay. That was a good question. This is why: 1. It is a single question, not five questions 2. It is a question on topic 3. It is a question I have not explained 10 times already 4. It is macro-based, not stock specific 5. It seeks an honest opinion rather than asking for something that may take hours of research Problems for Bernanke Market expectations were clearly for the Fed to do more. Goldman Sachs was shocked at the market reaction, I was not. See Goldman Surprised by Reaction to "Operation Twist" for details. The problem for Bernanke is every action he may take now has serious negative ramifications. Fort example, take Operation Twist: The flattening of the yield curve may (I doubt it) help mortgages by lowering mortgage rates. However, the flattening of the yield curve will without a doubt hurt banks struggling to make profits on spreads. The flattening of the yield curve also hurts those on fixed income as well as pension plans with 8.5% or so yield assumptions. The irony is pension plans might have gotten big returns had they been in treasuries, but they weren't because treasury yields were "too low". Instead, pension plans all plowed into foreign bonds, commodities, currencies, and global equities to make their 8.5% assumptions. So what is Bernanke to do? See Bernanke, a Complete Dunce, "Puzzled by Weak Consumer Spending" for more on how the self-proclaimed student of the great depression is clueless about the current depression. Bottom line: The Fed is more or less out of bullets. Moreover, Bernanke admitted he does not know why his policies are not working even though it is perfectly obvious. When backed in a corner, Bernanke may conceivably try nearly anything. However, Bernanke is just not that desperate yet. Right now, European banks are at far greater risk than US banks so Bernanke may easily bide his time. Silver Daily Chart Silver, once again, is acting more like a leveraged commodities plaything than a currency. I traded all my silver for gold on April 27, as noted in Taking Silver Profits - Swapping Silver for Gold. Gold is still higher than my swap point. At the time, I commented "I believe the price of silver is highly likely to revisit the low $20's at some point. Thus, I see no point in chasing silver higher here. Moreover, except for pure speculation, I see little reason to even hold silver in this spike." I really do not know if silver hits the low 20's or not, but I was not tempted by that previous decline to near $32. Had I bought it there, I made a mental sell at $40. Silver got all the way to $44 and to be honest I was wondering if it would take out my swap point. Silver breached $30 today. As I have commented many times, silver is a far riskier play than gold. I believe this volatility proves my point. CME margin hikes are not a cause of 40% collapse in silver from the top. No Hiding Places On September 19th I wrote No Hiding Spots Except Despised US Dollar: Equities Red, Metals Red, Energy Red, Grains Red No Hiding Spots Except Despised US DollarShort-term, I have been wrong about gold holding up. Then again, I really do not concern myself with short-term action. Moreover, gold is higher than it was the day I swapped it. The equity markets in general sure are not. Bernanke Will React It's a safe bet Bernanke will react, we just do not know when. Things may (or may not) get ugly for miners (especially silver) in the meantime. Those with cash, should be rooting for a selloff in gold and miners. In the meantime, hold a core position in gold. Take profits on big spikes and buy big dips. At some point that advice will stop working, I just do not think this is the time. Mike "Mish" Shedlock http://globaleconomicanalysis.blogspot.com Click Here To Scroll Thru My Recent Post List Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific. |
You are subscribed to email updates from Mish's Global Economic Trend Analysis To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |
Crawl Outage - An Update and What We're Doing |
Crawl Outage - An Update and What We're Doing Posted: 24 Sep 2011 03:56 AM PDT Posted by Bryce Howard Howdy folks! I wish I was writing you with better news, but in the spirit of TAGFEE, we want you to be as informed as possible about your PRO membership: Due to a major PRO web crawler service outage that occurred on Friday evening, crawler-related PRO features (link analysis and crawl diagnostics) are currently disabled. However, rankings, on-page optimization, and all tools except Crawl Test are functional. So what the *bleep* happened!? Amazon turned the lights out on us. Well, not exactly—I’ll explain. We host a number of our web applications from Amazon Web Services (AWS). For many of these hosts we pay a fixed rate per hour, however AWS offers an alternative billing model called spot instance pricing. Spot instance pricing is a method for purchasing excess computing power from AWS at a respectable discount. Everybody wins, we get a great price for the hundreds of computers we use daily while AWS is able to sell a resource that’s otherwise just sitting around idle. But the use of spot instance pricing comes at a risk: the computers hosting your services are only allocated to you as long as there is still excess capacity and that no one else is willing to bid more for those hosts than you are. If someone comes along offering to pay more, then AWS may revoke your hosts without any warning, leaving you to rebuild your services from scratch. This is not so bad if you can ensure you have enough computers left to still service requests… and therein lays the problem. Our Mistake The contract of spot instance pricing is quite clear: your servers may be arbitrarily taken from you, so you must be strategic about its usage. We unfortunately did not apply good strategy to our PRO web crawler configuration. Almost all of our service hosts were spot instances allocated with a dangerously low bid price (e.g. $2.00/hour), and they were all clustered within the same AWS availability zone (more on this later). So we put ourselves at risk with a low bid price, excessive use of spot instance pricing, and a poor distribution of hosts across AWS availability zones. We bet that there’d be little to no chance that AWS would reclaim our spot instance hosts but we bet very wrong. At approximately 6 PM PST, AWS terminated approximately 50% of our active spot instance hosts in the PRO crawler service cluster. Around this same time, the going spot instance price shot up to $2, our maximum bid price, which triggered this culling of our service hosts. Losing half our hosts wasn’t entirely catastrophic, however bad. In fact, it was a salvageable situation but then it got much worse. At 9PM PST we lost all of our service hosts that were spot instances (> 90%). The going spot instance price had jumped to $2.51/hour at this time, and given most of our hosts were bid at the price of $2/hour we effectively forfeited all rights to our previous claims. Our service wasn’t broken; it was just plain gone! This pretty graph from AWS accurately documents the spot instance pricing timeline for the day in question: Three practices that could have prevented this from occurring: 1) Use a spot instance price that is commensurate with the value of the service. If a host was very critical to service functionality, we should’ve bet a much higher price than the $2/hour. Using the spot instance pricing chart as a guide we should have at least used a bid of $3/hour or more to ensure better chance of avoiding host reclamation by AWS. Could we have predicted this optimal bid price? Likely not. Regardless, we should’ve bid what we thought the continued functioning of our service was worth. I think it’s easy to appreciate we now find that value much higher than the $2/hour we’d originally bid. 2) Distribute hosts across multiple availability zones. The initial increase in spot instance price occurred in one availability zone (us-east-1c), with the secondary increases in us-east-1c and us-east-1d. Had we spread our bets across multiple availability zones, we could have weathered this price volatility with at least half of our service hosts intact, even at the bid price of $2/hour. Although we were aware that prices could vary by availability zone, we did not use this to hedge our bets more effectively. 3) Use a mix of on-demand and spot instance pricing. On-demand priced hosts use a different pricing strategy where you agree to pay a fixed amount per hour to AWS but in return you get certain guarantees about your host claim, most notably it won’t be arbitrarily taken from you due to demand. Had we diversified our portfolio between on-demand and spot instance pricing we could’ve ensured at least minimal functionality of our service in the worst case while enjoying some good amount of cost savings in the best case. As with any critical investment you have to be strategic about minimizing your downside; we will do this moving forward. So where are things? To be frank, we are absolutely mortified that we’ve had to disable such an indispensable product feature as crawl diagnostics, especially when this service outage was otherwise avoidable. We are literally working day and night to re-enable the PRO app crawler service. Currently, we are rebuilding the API servers, the underlying NoSQL data store (Cassandra), and the various processing and crawling hosts. We are being very careful as we do this to avoid the previous mistakes, being strategic about diversifying pricing type (on-demand vs. spot-instance), distributing across availability zones and using a very competitive spot-instance bidding price. Most of the aforementioned service components are pretty easy to restore, but we have one unfortunate problem that will somewhat delay full restoration of the service: the terabytes of data generated by the hundreds of thousands of crawls we’ve executed over the last nine months. We must load this data from our backups (securely stored in AWS S3) into our NoSQL data store, something that by no means can be done quickly. Being perfectly transparent, this is an operation that could take the full duration of a week. We certainly don’t want to make anyone wait a full week just to see data that’s already a week out of date, so we plan to be a bit more clever with this service restoration, choosing the most optimal path to populate our database while also ensuring we preserve our weekly crawl cycle. Do we have all the solutions in place to achieve these goals? Not immediately, but we are making great progress and I’m very confident we will have more optimistic projections about service restoration in the next several days. Ok, so how exactly does this affect me again? As a PRO member you can still:
For the next week you won't be able to access:
Also as a reminder, none of the data is lost, we just need time to rebuild so we can access it. In the meantime, rest assured that we are doing everything we can to get your PRO functionality back up and running like it’s meant to be. We realize that many of you rely on this data to optimize your company’s and clients’ sites, and want to return service ASAP so you can continue to do what you do so well. Thank you for hanging in there with us as we learn from our mistakes.
|
You are subscribed to email updates from SEOmoz Daily SEO Blog To stop receiving these emails, you may unsubscribe now. | Email delivery powered by Google |
Google Inc., 20 West Kinzie, Chicago IL USA 60610 |