vineri, 19 august 2011

Mish's Global Economic Trend Analysis

Mish's Global Economic Trend Analysis

Former Moody's Senior Vice President Accuses Rating Agency of Fraud, Corruption, and Greed

Posted: 19 Aug 2011 01:59 PM PDT

In good times, no one wants to end a party and everyone is willing to turn a blind-eye to fraud, corruption, and excessive greed. Bear markets, however, expose the truth.

Massive fraud at Moody's now coming to light. Business Insider reports MOODY'S ANALYST BREAKS SILENCE: Says Ratings Agency Rotten To Core With Conflicts, Corruption, And Greed
A former senior analyst at Moody's has gone public with his story of how one of the country's most important rating agencies is corrupted to the core.

The analyst, William J. Harrington, worked for Moody's for 11 years, from 1999 until his resignation last year.

From 2006 to 2010, Harrington was a Senior Vice President in the derivative products group, which was responsible for producing many of the disastrous ratings Moody's issued during the housing bubble.

Harrington has made his story public in the form of a 78-page "comment" to the SEC's proposed rules about rating agency reform, which he submitted to the agency on August 8th. The comment is a scathing indictment of Moody's processes, conflicts of interests, and management, and it will likely make Harrington a star witness at any future litigation or hearings on this topic.

The primary conflict of interest at Moody's is well known: The company is paid by the same "issuers" (banks and companies) whose securities it is supposed to objectively rate. This conflict pervades every aspect of Moody's operations, Harrington says. It incentivizes everyone at the company, including analysts, to give Moody's clients the ratings they want, lest the clients fire Moody's and take their business to other ratings agencies.

In short, Harrington describes a culture of conflict that is so pervasive that it often renders Moody's ratings useless at best and harmful at worst.

Harrington believes the SEC's proposed rules will make the integrity of Moody's ratings worse, not better. He also believes that Moody's recent attempts to reform itself are nothing more than a pretty-looking PR campaign.

We've included highlights of Harrington's story below. Here are some key points:

  • Moody's ratings often do not reflect its analysts' private conclusions. Instead, rating committees privately conclude that certain securities deserve certain ratings--and then vote with management to give the securities the higher ratings that issuer clients want.
  • Moody's management and "compliance" officers do everything possible to make issuer clients happy--and they view analysts who do not do the same as "troublesome." Management employs a variety of tactics to transform these troublesome analysts into "pliant corporate citizens" who have Moody's best interests at heart.
  • Moody's product managers participate in--and vote on--ratings decisions. These product managers are the same people who are directly responsible for keeping clients happy and growing Moody's business.
  • At least one senior executive lied under oath at the hearings into rating agency conduct. Another executive, who Harrington says exemplified management's emphasis on giving issuers what they wanted, skipped the hearings altogether.
More Highlights

That was just a small sample of highlights. Here are 30 more highlights of Harrington's accusation against Moody's.

The Business Insider article confirms what I said earlier today about the rating agency corruption. The difference is we now have a whistle-blowing insider telling the story.

In this case, the SEC cannot sweep it under the rug as they did with fraud investigation of banks: SEC Destroys 9,000 Fraud Files Involving Wells Fargo, Bank of America, Citigroup, Goldman Sachs, Credit Suisse, Deutsche Bank, Morgan Stanley, Lehman

For my take on the rating agency whores and more importantly what should be done to fix the problem, please see In Praise of Timely, Blatant Incompetence

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com
Click Here To Scroll Thru My Recent Post List
Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

Bernie Sanders Releases CFTC Data Including Names of Oil Speculators to Ire of CFTC; Who is to Blame for Oil Speculation?

Posted: 19 Aug 2011 10:34 AM PDT

The CFTC and Wall Street are upset about a leak by Bernie Sanders that shows exactly who held what positions when crude futures topped $140 in 2008.

Yahoo!Finance reports U.S. oil speculative data released by Senator sparking ire. However the article does not disclose the participants.
Senator Bernie Sanders, a staunch critic of oil speculators, leaked the information to a major newspaper in a move that has unsettled both regulators and Wall Street alike.

In a June 16 e-mail reviewed by Reuters, a senior policy adviser to Sanders discusses how his office received private data with the names and positions of traders and forwarded it exclusively to a Wall Street Journal reporter.

The e-mail, which also attaches two files with the data, was sent to Public Citizen's Tyson Slocum asking him to review it and speak with the newspaper about his observations.

The leaked information has sparked concern at the Commodity Futures Trading Commission, which is legally prohibited from releasing confidential information that identifies trader positions and identities.

The leak also raises broader questions as U.S. regulators gear up to collect massive new amounts of private data from market players on everything from swaps and hedge funds to blueprints for how large financial firms can be liquidated. The breach of data could make Wall Street less reluctant to hand over sensitive information if they fear it is not appropriately safeguarded.

"This type of incident will have a chilling effect on derivatives trading in the U.S. because market participants will be reluctant to take the risk that their positions will be exposed to the public-and their competitors," John Damgard, president of the Futures Industry Association, said in a statement sent to Reuters.

People familiar with the matter say the data later obtained by Sanders was first formally requested by the U.S. House Energy Committee. From there it somehow migrated over to the U.S. Senate.
From Bernie Sanders' Website
August 19th, 2011

Inquiring minds may be interested in what Bernie Sanders' Website has to say about the matter.

Rampant Oil Speculation Data revealing rampant oil speculation in 2008, supplied by Sen. Sanders' office, emerges as the Commodity Futures Trading Commission (CFTC) comes under mounting pressure to complete new rules that would set much tougher limits on speculative trading in energy and metals markets, Reuters and The Calgary Herald reports. Sanders said he felt the data needed to be publicly aired.

"The CFTC has kept this information hidden from the American public for nearly three years," Sanders said. "This is an outrage. The American people have a right to know exactly who caused gas prices to skyrocket in 2008 and who is causing them to spike today."
Oil Speculator Scapegoats

Many debate whether speculators can influence the price of oil. I think they can. However, blame should not go to oil speculators, but rather to the Fed for injecting massive amounts of liquidity seeking a home.

The Fed wants to support housing prices and foster jobs creation. However, the Fed can only supply liquidity, it cannot dictate where it goes or if it goes anywhere at all.

Another piece of the blame goes to public unions and their ludicrous pension plan assumptions.

Most public pension plans need 8.5% annualized returns and they are not going to get it from US treasuries or US equities. This encourages the funds speculate in commodities, accumulating a rising number of oil futures over time.

Looking back to the Greenspan era, one can blame the Fed for the holding interest rates too low, too long. On a continual basis, one can blame both Congress and the Fed for actively debasing the US dollar, thereby fueling the desire of market participants to hold hard assets.

In this sense, increased speculation is not a cause of rising oil prices but rather a symptom of other fundamental problems.

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com
Click Here To Scroll Thru My Recent Post List

Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

Yet Another 2.5 Hours, 2.3% Hour Rally, This Time on Silly Rumors Eurobonds Back in Play

Posted: 19 Aug 2011 09:10 AM PDT

If someone woke up at 10:00 AM Central and looked at a stock market quotes, they would have seen the stock markets essentially flat. Once again appearances would be deceiving.

US Futures and Select Equities Approximately 10:15 Central



click on chart for sharper image

At roughly 10:00 Central things were essentially flat. Yet the ride in the S&P 500 futures shows a 26 point move from bottom to top (from 1117.50 to 1153.25)

S&P 500 Futures



click on chart for sharper image

Notes:
The purple circle is roughly 10:00 AM.
The first green bar in the second frame is today's open.

For such action to occur occasionally is perfectly normal. That such action is now typical is not. There are massive distortions in the markets where every tiny piece of news, some of it complete nonsense, sends shares in whatever direction.

European Stocks Reverse 3.6% Decline

Volatility is the norm globally. Bloomberg reports European Stocks Resume Earlier Decline; Stoxx 600 Retreats for Second Day
The Stoxx Europe 600 Index lost 0.1 percent to 226.42 at 3:30 p.m. in London, paring an earlier drop of 3.6 percent. The gauge has tumbled 22 percent from this year's peak in February amid concern that Europe will fail to contain its sovereign-debt crisis and that the economic recovery in the U.S. will falter.

The Stoxx 600 pared an earlier loss as the European Commission said it may present draft legislation on joint bond sales by euro-area nations when completing a report on the feasibility of common debt sales, putting pressure on Germany to drop its opposition.
Someone is going to write a feasibility study and that caused a reversal?

I do not know what if anything caused today's glorious reversal, but if it is a "feasibility study" then prepare for more down because such a study is meaningless as long as Germany and France will not go along.

Heck, the nature of the proposal is such that all Eurozone members would have to approve it so even Finland could reject it.

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com
Click Here To Scroll Thru My Recent Post List
Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.

In Praise of Timely, Blatant Incompetence

Posted: 19 Aug 2011 02:10 AM PDT

Tonight I am going to do something different, openly praise blatant incompetence. I will list my reasons later but first let me sing the praises of sheer incompetence at Moody's, Fitch, and the S&P (the big 3 rating agencies).

Downgrade of U.S. Debt Long Overdue

Many are in shock that the S&P downgraded debt of the US from AAA. Not me. It was long overdue.

However, the S&P proved it was incompetent in the way it made the downgrade. Pray tell how can a rating agency make a $2 trillion error? The answer is obvious: sheer incompetence.

The irony is Moody's and Fitch proved they are incompetent by not downgrading U.S. debt.

If you need a myriad of reasons, I highly recommend Issues and Solutions for Restoring Credibility to the Credit Rating Agencies and Rehabilitating the Alternative Banking System by Janet M. Tavakoli, President, Tavakoli Structured Finance, Inc.

Also note that Egan-Jones downgraded US debt on July 18 from AAA to AA+ and nobody batted an eye.

"We are taking a negative action not based on the delay in raising the debt ceiling but rather our concern about the high level of debt to GDP in excess of 100% compared to Canada's 35%."

NRSRO Certification

The SEC certifies Egan-Jones as one of 10 NRSROs "Nationally Recognized Statistical Rating Organizations".

The "Big 3" NRSROs are Moody's, Fitch, and the S&P.

Most have never heard of Egan-Jones or any rating agencies but the big 3, and that explains why nobody howled when Egan-Jones did its downgrade, yet everyone howled like rabid wolves over the S&P's action.

The wolves demanded action and action they got.

S&P Under Justice Department Investigation

The Associated Press reports Justice Department investigating Standard and Poor's mortgage securities ratings .
The Justice Department is investigating whether the Standard & Poor's credit ratings agency improperly rated dozens of mortgage securities in the years leading up to the financial crisis, The New York Times reported Wednesday.
S&P Investigated for Insider Trading

The Wall Street Journal reports SEC Asking About Insider Trading at S&P
The post-downgrade backlash against S&P seems to be gathering strength.

The FT is reporting, citing anonymous sources, that the SEC is investigating whether there was any insider trading done by employees of Standard & Poor's ahead of their downgrade of the US a week ago.

Dow Jones Newswires writes:

"The U.S. Securities and Exchange Commission has asked Standard & Poor's to disclose who within its ranks knew of the recent decision to downgrade U.S. debt before it was announced, the Financial Times newspaper reported Thursday on its website, citing unnamed people familiar with the matter."

Remember, rumors of a post-bell downgrade were rampant on Wall Street very early on Friday, rumors that turned out to be true. It sure sounded like a leak, though the leak could have come from either S&P or Treasury. It seemed inevitable there would be an investigation, though it could be hard to find anything.

MarketWatch points out that, according to the 2006 Credit Rating Agency Reform Act, S&P could have its license revoked if it leaked word of the downgrade.
Nearly universal sentiment was that treasury yields would rise on a downgrade. I said "no effect". Yields plunged in spite of the downgrade, so clearly the decision had no effect.

Rumors float all the time. The S&P gave a date as to when they would announce. They even hinted at a downgrade in my opinion. So, how tough is it for someone to start a rumor that had a 50% chance of being correct?

Excuse me for asking, but as long as prostitutes are under investigation, what about an investigation of the other two whores, Moody's and Fitch, or better yet all of them for reasons far more serious than unfounded witch-hunts, like outright fraud.

AAA Rated CDOs, CDOs-Squared, and Other Garbage

By now nearly everyone realizes Moody's, Fitch, and the S&P were grossly incompetent and fueled the mortgage crisis by rating pure garbage mortgages in numerous forms as AAA.

But was it gross incompetence or purposeful fraud by the big 3 to see who could collect the most "paying johns", damn the consequences?

Regardless, why is only the S&P under investigation?

The answer is the big three whores are supposed to do what the government says they are supposed to do, and the S&P didn't. So the S&P is under investigation. And that serves as a warning to Moody's and Fitch.

Explanation of Whores

I have used the term "whores" twice now so it needs an explanation. What I mean is the big 3 rating agencies arguably sold themselves to the highest bidder, essentially granting an AAA rating to damn near anything for a fee.

The Rating Agency Model, as it now exists, pays raters on the basis of how much volume they do, not on how well they rate anything. If you are willing to rate pure garbage as AAA no matter what it is really worth, you get a lot more "action" and make a lot more money.

And action the "big 3" got. And everyone, turned a blind eye to the process, because no one likes to end a party, especially a party that whores are throwing with Greenspan and Bernanke cheerleading like a pair of pom-pom girls at the big game.

The SEC Caused this Mess

There is just one more detail I need to point out before we get to the proper solution. That detail pertains to the question "Who Caused this Mess?"

I have talked about this on numerous occasions actually, but perhaps now is the time someone will listen.

Flashback September 28, 2007: Time To Break Up The Credit Rating Cartel
The rating agencies were originally research firms. They were paid by those looking to buy bonds or make loans to a company. If a rating company did poorly it lost business. If it did poorly too often it went out of business.

Low and behold the SEC came along in 1975 and ruined a perfectly viable business construct by mandating that debt be rated by a Nationally Recognized Statistical Rating Organization (NRSRO).

Establishment of the NRSRO did three things (all bad):

  1. It made it extremely difficult to become "nationally recognized" as a rating agency when all debt had to be rated by someone who was already nationally recognized.
  2. In effect it created a nice monopoly for those in the designated group.
  3. It turned upside down the model of who had to pay. Previously debt buyers would go to the ratings companies to know what they were buying. The new model was issuers of debt had to pay to get it rated or they couldn't sell it. Of course this led to shopping around to see who would give the debt the highest rating.

Spotlight on Prostitutes

There is nothing new here. I have been talking about this for years.

But finally rating agencies are in the spotlight of Congress, of foreign governments, of investors, and of pension fund managers stupid enough to buy AAA rated garbage stamped by paid prostitutes.

Solutions

Some of the proposed solutions to this mess are horrific. There is a massive 400 page bill in Congress to address the problem.

Tavakoli's report, cited above, is 50 pages long. I agree with most of her analysis. I cannot endorse the ending paragraph.
The solution is to raise one or more rating agencies up to standard to merit the NRSRO label. Meanwhile, rating agencies can continue to issue ratings but must commit to coming up to standard. Those that cannot should have the privilege of issuing ratings completely revoked. The second part of the solution is to develop global third party benchmarks and global third party rating scales and make accurate ratings the only measurement of success.
No, that is NOT the Solution

The government does not have and never has had a need to have a NRSRO label. Moreover, there is no need to require all debt be rated. Indeed, the act of mandating that all debt be rated by designated rating agencies is what led to the escalating problem of everything being rated AAA in the first place.

Finally, it is complete silliness to suggest some committee can determine who merits NRSRO and to wait until rating agencies come up to standard.

THE Solution

  1. End immediately the NRSRO label. Neither the government nor the SEC has any business handing a monopoly business to anyone.
  2. End immediately the requirement that all debt be rated. The market will sort this out in a flash.
By immediately I mean 6 months, 8 months or whatever time is appropriate for debt-buyers to decide who they want to use as opposed to some committee deciding who should be approved.

People buying debt will have to do homework, but that is far better than trusting an AAA rating placed on garbage by prostitutes paid to place a label.

Over time, Moody's, Fitch, and the S&P will do a better job, or they will cease to exist. Simply put, those who rate debt accurately will flourish, those who don't will go out of business.

What's wrong with that?

So Why Do I Praise Blatant Incompetence?

I praise blatant, timely incompetence because it takes massive force (in this case universally recognized blatant incompetence at precisely the right time), before there is any chance of getting change.

There is a small window of opportunity here.

The time to take advantage is now. Instead of silly Congressional investigations of the S&P in regards to the timing of their announcement, Congress simply needs to write a bill eliminating the NRSRO label and the requirement that debt be rated. Yes, it's as simple as that.

S&P, I salute your gross incompetence. Your timing was perfect. Whether anything sensible happens remains to be seen, but at least there is a small chance for reasonable voices to be heard.

Contact Congress

Please send your congressional representatives an email or fax and tell them to scrap the NRSRO "Nationally Recognized Statistical Rating Organizations" rating entirely, ending the monopoly of Moody's, Fitch, and the S&P.

Mike "Mish" Shedlock
http://globaleconomicanalysis.blogspot.com
Click Here To Scroll Thru My Recent Post List
Mike "Mish" Shedlock is a registered investment advisor representative for SitkaPacific Capital Management. Sitka Pacific is an asset management firm whose goal is strong performance and low volatility, regardless of market direction. Visit http://www.sitkapacific.com/account_management.html to learn more about wealth management and capital preservation strategies of Sitka Pacific.
Publicat de 1 la 23:13 0 comentarii

Damn Cool Pics

Damn Cool Pics

Horsemaning From Around The World

Posted: 19 Aug 2011 02:51 PM PDT

Originated in America, horsemaning, more and more people around the world now adopt the new entertainment activity as a Horsemaning which is the same sh*t as planking, owling etc but it looks more fun. Here are the most weird and shocking pics of horsemaning from all around the world.

Related Post:
Horsemaning is the New Planking

In Paris


In Belgium


In Budapest


In Holland


In Melbourne, Australia


In Turkey


In Thailand


Still In Thailand


At The Nike Store


In Fischamend, Austria


On Google+


At Cold Stone Cremery


At The Exotic Huffington Post Offices (NYC)


At The MTV Offices (Times Square, NYC)


At A Deli (Location Unknown)


On The Beach (South Of France)


On The Radio


In The Style Of The 18th Century


At NBC Studios (NYC)


At BuzzFeed HQ (Soho, NYC)


On The Planet Kamino?


In A Basement, While Playing Call Of Duty: Black Ops (Location Unknown)


In A Pool (Location Unknown)


In An Office…With A Giant Knife


Rural Nebraska?


American People Vs. Japanese People

Posted: 19 Aug 2011 02:46 PM PDT

Fun comic introducing some curious differences between the Japanese and the Americans. I'm sure it was created by someone from Japan.

Click on Image to Enlarge.

Source: 9gag
Publicat de 1 la 19:06 0 comentarii

SEOmoz Daily SEO Blog

SEOmoz Daily SEO Blog

Introducing SEOmoz PRO Perks

Posted: 19 Aug 2011 12:06 AM PDT

Posted by AndrewDumont

Howdy, SEOmoz family! I'm Andrew, the new Business Development Director over at SEOmoz, and this is my first post on the blog. Great relationships are all about kicking things off on the right foot... or at least that's what I hear. That in mind, SEOmoz PRO members, we've got a nice little treat for you. 

If you've spent any time poking around the PRO Dashboard, you've probably run into something called the "PRO Member Discount Store." Sound familiar? Initially, the idea was to pack it full of amazing discounts on web services we loved, exclusively for PRO members. Due to a lack of vetting on our part, the store quickly turned into a catch all that didn't provide much value at all, and that ain't right. We figured it was time to revive that property, with its oh so good intent.

But before we get into the unveiling, we first wanted to talk about goals. In rethinking the discount store, we knew we had to keep the following principles at the core:

  • Quality - With the previous store, we lost track of the most important thing; quality. For service discounts to provide value, they need to be 1) competitive and 2) available for services we know and love. In the new store, you'll notice the quality in services and discounts the moment you hit the page. 
  • Exclusivity - Without focusing on exclusivity, we weren't able to offer up the type of competitive discounts we knew you wanted. In the new store, we cut back on the number of services, and in turn increased the size of discounts offered. 
  • Simplicity - When you're on the verge of a great deal, you want to take advantage of it right away. In the new store, we removed all of the guess work, making the path to coupon redemption simple, and only one step away.

With that, ladies and gents, we give you SEOmoz PRO Perks.

Jammed packed with some of the hottest services on the web, you'll quickly see why we're calling them perks. We went out and talked to our favorite tools in the social monitoring, affiliate management, customer service and on page optimization space, to bring you some serious heat right off the bat. To provide a few highlights, here are some of our current perks.  

  • 50% Off Unbounce
  • 50% Off Geckboard
  • 40% Off Invoicera 
  • 40% Off WooThemes
  • 30% Off SnapEngage
  • 30% Off ViralHeat
  • 20% Off Wistia 
  • 20% Off Get Satisfaction
  • And much more...

Piqued your interest? Head on over to SEOmoz's new PRO Perks store, and indulge as you wish -- being an SEOmoz PRO definitely has its perks. If you're still not an SEOmoz PRO, there's no better time than the present to give our 30-day free trial a go. 

Have some ideas for other services you'd like to see in our PRO Perks store? Drop me a line at andrew[at]seomoz.org, or let us know in the comment section below. We want to make sure that our PRO Perks are consistenly bringing that hotness. 


Do you like this post? Yes No

Statistics: Don't Make These Mistakes - Whiteboard Friday

Posted: 18 Aug 2011 02:05 PM PDT

Posted by Aaron Wheeler

 Statistics can be very powerful tools for SEOs, but it can be hard to extract the right information from them. People understandably make a lot of mistakes in the process of taking raw data and turning it into actionable numbers. If you know what potential mistakes can be made along the way, you're less likely to make them yourself - so listen up! Will Critchlow, the co-founder and chief strategist at Distilled, has a few tips on how to use valid and sound statistics reliably. Use a lot of statistics yourself? Let us know what you've learned in the comments below!

 

Video Transcription

Howdy, Whiteboard fans. Different look this week. Will Critchlow from Distilled. I am going to be talking to you about some ways you can avoid some common statistical errors.

I am a huge fan of the power of statistics. I studied it. I have forgotten most of the technical details, but I use it in my work. We use it a lot at Distilled. But it is so easy to make really easy to avoid mistakes. Most of it comes from the natural way that humans aren't really very good at dealing with numbers generally, but statistics and probability in particular.

The example I like to use to illustrate that is imagine that we have a disease that we are testing for, a very rare disease, suppose. So, we have a population of people, and some small, some tiny proportion of these people have this rare disease. There is just this tiny, tiny sliver at the top. Maybe that is 1 in 10,000 people, something like that. We have a test that is 99% accurate at diagnosing when people have or don't have this disease. That sounds very accurate, right? It's only wrong 1 time in 100. But let's see what happens.

We run this test, and out of the main body of the population, I am going to exaggerate slightly, most people are correctly diagnosed as not having the disease. But 1% of them, this bit here, are incorrectly diagnosed as having the disease. That's the 99% correct but 1% incorrect. Then we have the tiny sliver at the top, which is a very small number of people, and again 99% correct, a small percent are incorrectly told they don't have it. Then, if we just look at this bit in here, zoom in on there, what we see is actually of all the people who are diagnosed as having this disease, more of them don't have it than do. Counterintuitive, right? That's come from the fact that, yes, our test is 99% accurate, but that still means it is wrong 1 in 100 times, and we're actually saying it is only 1 in 10,000 people who have this disease. So, if you are diagnosed as having it, it is actually more likely that is an error in the diagnosis than that you actually have this very rare disease. But we get this wrong. Intuitively people, generally, everyone would be likely to get this kind of question wrong. Just one example of many.

Some things that may not be immediately intuitively obvious, but if you are working with statistics, you should bear in mind.

Number one, independence is very, very important. If I toss a coin 100 times and get 100 heads, then if those were independent coin flips, there is something very, very odd going on there. If that is a coin that has two heads on it, in other words, they're not in fact independent, the chance of me getting a head is the same on every one, then they're completely different results. So, make sure that whatever it is you are testing, if you are expecting to do analysis over the whole set of trials, that the results are actually independent. The common ways this falls down are when you are dealing with people, humans.

If you want reproducible results, if you accidently manage to include the same person multiple times, their answers to a questionnaire, for example, will be skewed the second time they answer it if they have already seen the site previously, if you are doing user testing or those kinds of things. Be very careful to set up your trials, whatever it is that you are testing, for independence. Don't over worry about this, but realize that it is a potential problem. One of the things we test a lot is display copy on PPC ads. Here you can't really control who is seeing those, but just realize there is not a pure analysis going on there because many of the same people come back to a site regularly and have therefore seen the ad day after day. So there is a skew, a lack of independence.

On a similar note, all kinds of repetition can be problematic, which is unfortunate because repetition is kind of at the heart of any kind of statistical testing. You need to do things multiple times to see how they pan out. The thing I am talking about here particularly is you often have seen confidence intervals given. You have seen situations where somebody says we're 95% sure that advert one is better than advert two or that this copy converts better than that copy or that putting the checkout button here converts better than putting the checkout button there. That 95% number is coming from a statistical test. What it is saying is, it is assuming a whole bunch of independence of the trials, but it is essentially saying the chance of getting this extreme a difference in results by chance if these two things were identical is less than 5%. In other words, fewer than 1 in 20 times would this situation arise by chance.

Now, the problem is that we tend to run lots of these tests in parallel or sequentially. It doesn't really matter. So, imagine you are doing conversion rate optimization testing and you tweak 20 things one after another. Each time you test it against this model and you say, first of all I am going to change the button from red to green. Then I am going to change the copy that is on the button. Then I am going to change the copy that is near the button. Then I am going to change some other thing. You just keep going down the tree. Each time it comes back saying, no, that made no difference, or statistically insignificant difference. No, that made no difference. No, that made no difference. You get that 15 times, say. On the 16th time, you get a result that says, yes, that made a difference. We are 95% sure that made a difference. But think about what that is actually saying. That is saying the chance of this having happened randomly, where the two things you are testing between are actually identical, is 1 in 20. Now, we might expect something that would happen 1 in 20 times possibly to come up by the 16th time. There is nothing unusual about that. So actually, our test is flawed. All we've shown is we just waited long enough for some random occurrence to take place, which would have happened definitely at some point. So, you actually have to be much more careful if you are doing those kinds of trials.

One thing that works very well, which scuppers a lot of these things, is be very, very careful of this kind of thing. If you run these trials sequentially and you get a result like that, don't go and tell your boss right then. Okay? I've made this mistake with a client, rather than the boss. Don't get excited immediately because all you may be seeing is what I was just talking about. The fact that you run these trials often enough and occasionally you are going to find one that looks odd just through chance. Stop. Rerun that trial. If it comes up again as statistically significant, you are now happy. Now you can go and whoop and holler, ring the bell, jump and shout, and tell your boss or your clients. Until that point, you shouldn't because what we very often see is a situation where you get this likelihood of A being better than B, say, and we're like we're 95% sure here. You go and tell your boss. By the time you get back to your desk, it has dropped a little bit. You're like, "Oh, um, I'll show in a second." By the time he comes back over, it has dropped a little bit more. Actually, by the time it has been running for another day or two, that has actually dropped below 50% and you're not even sure of anything anymore. That's what you need to be very careful of. So, rerun those things.

Kind of similar, don't train on your sample data. If you are looking for a correlation between, or suppose you're trying to model search ranking factors, for example. You're going to take a whole bunch of things that might influence ranking, see which ones do, and then try to predict some other rankings. If you get 100 rankings, you train a model on those rankings, and then you try and predict those same rankings, you might do really well, because if you have enough variables in your model, it will actually predict it perfectly because it has just learned, it has effectively remembered those rankings. You need to not do that. I have actually made this mistake with a little thing that was trying to model the stock market. I was, like, yes, I am going to be rich. But, in fact, all it could do was predict stock market movements it had already seen, which it turns out isn't quite as useful and doesn't make you rich. So, don't train on your sample data. Train on a set of data over here, and then much like I was saying with the previous example, test it on a completely independent set of data. If that works, then you're going to be rich.

Finally, don't blindly go hunting for statistical patterns. In much the same way that when you run a test over and over again and eventually the 1 in 20, the 1 in 100 chance comes in, if you just go looking for patterns anywhere in anything, then you're definitely going to find them. Human brains are really good at pattern recognition, and computers are even better. If you just start saying, does the average number of letters in the words I use affect my rankings, and you find a thousand variables like that, that are all completely arbitrary and there is no particular reason to think they would help your rankings, but you test enough of them, you will find some that look like they do, and you'll probably be wrong. You'll probably be misleading yourself. You'll probably look like an idiot in front of your boss. That's what this is all about, how not to look like an idiot.

I'm Will Critchlow. It's been a pleasure talking to you on Whiteboard Friday. I'm sure we'll be back to the usual scheduled programming next week. See you later.

Video transcription by Speechpad.com


Do you like this post? Yes No

Publicat de 1 la 17:11 0 comentarii
Abonați-vă la: Postări (Atom)