Saturday, February 28, 2015

Before You Pivot, Run The Test Again

No doubt you've heard insanity defined as "doing the same thing over and over and expecting different results." And yet we know from science that advances often come when experiments fail to replicate a result. Einstein himself, said to be the aphorism's author, often repeated experiments. After all, experiments sometimes produce false results. You don't have to be Einstein to know it is a good idea to run the test again.

Yet if you are in business, you probably live by the insanity aphorism - insisting that no test be repeated. How often have you said "But we already tried that, and look how it turned out!" Calling others insane is an effective way to shut down further experimentation (and thinking).

Fortunately for us all, the world re-runs experiments all the time, and often gets different results. Webvan failed. Now Amazon, Google, and others are delivering to your door. EachNet (and its acquirer, eBay) failed to make cash-on-delivery work in Chinese C-to-C e-commerce. Now Taobao's cash-on-delivery system is thriving. The failures of Alta Vista, Excite, Lycos, and others led many to conclude that internet search could not be a business. Now, well, you know.

You're probably already trying to explain the differences in all these examples. Slow down; the broader issue here is the problem of false results.

Sometimes experiments generate false negatives - they tell you "no" when the real answer is "yes." And sometimes experiments generate false positives, telling you "yes" when the real answer is "no." You of course know about false positives and false negatives in medicine. We worry about them a lot, which is why we often go back for a re-test when things get medically serious. But for whatever reason, we don't think about false results nearly enough in business.

For instance, I recall one of the early movers in digital medical diagnostic imaging. Their system was rapidly adopted by several hospitals, leading to a lot of excitement, including executives quitting their jobs and joining the company. Then growth abruptly ended. It turns out the early wins were a false positive. (Many more examples of false positives can be found in Geoffrey Moore's "Crossing the Chasm" books, enough to have created a consulting juggernaut.)

False negatives in business are common too, as in the examples of search, delivery, and Chinese COD - but they are often harder to spot. The problem is that false positives are self-correcting, but false negatives are not. When you get a positive result from a business experiment, typically you'll keep at it. If it turns out to have been a false result, the world will make that clear enough. But if you get a false negative, you'll be inclined to "pivot." And you'll never know that you were on to something good - unless somebody else tries it again.

Perhaps you are thinking: "Hey, in all your examples, there were some variables that changed. A good test would take into account all the variables that matter." Sure, Amazon and Google now know about some things that Webvan did not, and they have adjusted those variables accordingly and that's why their experiments are working when Webvan's did not.

Here's the problem: Often we don't know all the variables that matter. This problem is well understood in science. Good scientists know that two seemingly identical experiments can produce different results, since often there are variables operating that are unknown to the scientist at the time. In fact, even random chance can produce odd results. That's why a good scientist knows that there is a lot she does not know; so she runs the test again.

The lesson: Insane though it may seem, run the test again.


A rigorous treatment of this problem, sometimes called the "hot stove effect" is by Jerker Denrell and James March.

Sunday, February 15, 2015

The Leader's Lens

It was time to run an online survey of the employees at a large technology company. My work with their leadership team had raised some interesting research questions, so one of the vice presidents asked her assistant to help me make it happen. She said to her assistant, who I will refer to here as Amelia, "Please help Bill to get access to everything he needs. We want to get everything arranged pretty quickly."

Amelia is a remarkable assistant - very thorough. But often she is asked to organized social events. So that was the lens through which she understood her boss' request. I was thinking "survey;" she was thinking "social event." 

As we began working on the project, Amelia asked, "when will it take place?"

"As soon as we can set everything up," I responded, adding, "I would like to host it here at Stanford." Although I did not explain this to Amelia, my concern was that if we hosted the survey on the company's servers, I might not be able to analyze the data on my own computer for security reasons.

"Host it at Stanford!" she responded. 

"Sure," I said. "We do this all the time."

She asked, "How many employees will be involved?"

"All of them," I replied.

"ALL OF THEM?" Amelia was dumbfounded.

"Of course!" I said. "We can use our server and--"

"But, Bill," she interrupted, "it makes no sense to host it at Stanford. We host these all the time here at our company. And we can get servers, of course!"

"Servers?" I questioned, "really, Amelia, one server should be more than enough."

"ONE SERVER, for the ENTIRE company?" Exclaimed a flabbergasted Amelia.

"No problem." I said. "These days servers are extremely efficient. And the tasks should not be too intensive."

As the discussion went on, we ultimately realized the misunderstanding and had a good laugh. But this story nicely illustrates the importance of one's "interpretive lens," the assumptions that shape how we understand information. Amelia and I were hearing all the same words, but they meant very different things to each of us because we were looking through different interpretive lenses.

Effective leaders understand the importance of the interpretive lens. I remember at one company, their sales head in Europe went around the approved product and price list, creating a solution for a customer that was not approved by the corporate marketing organization. He won the customer, and in fact was responsible for growing the business considerably through such tactics. The CEO called him back to the firm's silicon valley headquarters; some thought it would be a reprimand. But the CEO brought him into a top leadership meeting to applaud him for being entrepreneurial and customer focused. The manager received a promotion and a raise. This story spread through the company's employees quickly. The CEO's interpretive lens saw the manager as an innovator. Others saw him as a rule breaker. Both interpretations were correct, but the CEO wanted the "innovator" lens to win the day. By making his interpretation clear to everyone, he helped to shape their interpretive lenses. 

Every day at work, alternative lenses compete. Is a failed project shameful, or a healthy sign of experimentation? Is an outspoken employee insubordinate, or is she showing leadership? A great leader shapes the lenses through which her employees interpret what happens.

Look around you at work. Do you like the lens being used to interpret what happens? If not, what does this say about your leadership? 


The sociological research on this topic is reviewed by Robert Benford and David Snow.