Central Perk: Seth's Blog : Measuring without measuring

sâmbătă, 1 iunie 2013

Seth's Blog : Measuring without measuring

Measuring without measuring

As an organization grows and industrializes, it's tempting to simplify things for the troops. Find a goal, make it a number and measure it until it gets better. In most organizations, the thing you measure is the thing that will improve.

Colleges decided that the SAT were a useful shortcut, a way to measure future performance in college. And nervous parents and competitive kids everywhere embraced the metric, and stick with it, even after seeing (again and again) that all the SAT measures is how well you do on the SAT. It's easier to focus on one number than it is to focus on a life.

Paypal and Chase and countless other organizations do precisely this: they figure out a metric, decide it's important and then create a department to improve that metric.

Consider the Chase Fraud Prevention department. It costs a credit card company a lot of money when fraudulent charges are made, because they often have to eat the cost. So this department of thousands of people works to make the number of fraudulent charges go down at the same time they keep expenses low. Which sounds great until you realize that the easiest way to do this is to flag false positives, annoy honest customers and provide little or no fallback when a mistake is made.

Simple example: I regularly get an automated phone call from the bank with an urgent warning. But even when I answer the phone, the system doesn't let me ring through to an operator. Instead, I have to write every detail down, then call, wait on hold, prove it's me, type in all the information, and THEN explain to them that in fact, the charge was mine.

And this department has no incentive to fix this interaction, because 'annoying' is not a metric that the bosses have decided to measure. Someone is busy watching one number, but it's the wrong one.

Or consider the similar problem at Paypal. Stories of good (or great) customers being totally shut down, sometimes to the point of bankruptcy, are legion. There may be people at Paypal who care about this, but the security people don't. That's because they're not measuring the right thing.

Measurement is fabulous. Unless you're busy measuring what's easy to measure as opposed to what's important.