The Process of Theory Building

I started working at The Clayton Christensen Institute and my job is to help develop the theory of disruptive innovation.

In order to do this I need to understand at least two concepts:

  • The process of theory building
  • Disruptive innovation theory

I’m more comfortable with the latter–having been a student (and victim) of it for more than a decade–but the the process of theory building is a new concept. At least to me but also, I believe, to many. The belief that a theory is fully cooked when first conceived is not the way science developed and the idea that business management theories are singular ideas rather than processes is symptomatic of an immaturity in the field.

So here are the basics of theory building as put forward by Clay Christensen and David Sundahl:

Definition: A theory is a statement of what causes what, and why, and under what circumstances. A theory can be a contingent statement or a proven statement. That is all.

Many managers shy away from using the word “theory” because it is associated with the term theoretical which suggests impractical. But managers use theory every day. They make decisions on some basis of cause and effect, often without being specific about their reasoning.

Process: First comes observation. Second, description. Third categorization. Fourth comes analysis and a statement of what causes what and why. This analysis can be simply an observation of a pattern or a more rigorous correlation analysis.

But that’s not the end of the process. The causal statement needs to be tested by predictions whose validity is tested with further observations and confirmation or denial of the statement. If the statement is denied we need to decide if it’s an anomaly that expands the theory or whether it contradicts the theory making it less useful.

The anomaly allows a new categorization to take shape. Getting the categories right is the key to the usefulness of the theory. The discovery of anomalies can thus make a theory stronger. The discovery of anomalous phenomena is the pivotal element in the process of building an improved theory.

This iteration between prediction/confirmation/anomaly handling can go for quite some time. As anomalies are accounted for on a regular basis then they can either be exhausted or depleted enough that a robust enough categorization emerges and the predictive power is nearly complete.

Example: In my reading of Apple’s financial statements I observed that Capital Expenditures were rising dramatically after the company began to sell iPhones. The observations were made over a few years. The pattern observed showed some correlation between spending and shipments of units.

The company’s spending was then compared with a group of other technology companies. These observations suggested that spending varied according to business model and strategy and that Apple seemed to be transitioning from one type of spending (on infrastructure) to another (on manufacturing equipment.)

Then a statement was made that Apple was using capital expenditures to not only ensure supply of components but also of component manufacturing equipment. This was borne of necessity but had the side effect of creating competitive advantage as its unibody devices and Macs were unique and differentiated.

As the more data came in, by the prediction was made that capital expenditures– which are incurred before production starts and which are pre-announced on a fiscal year basis — indicate new product ramps or new product introductions.

A few anomalies were experienced when spending increased but production didn’t and vice versa. These were studied and explained by shifts in technology (mainly screens) which required “out-of-phase” investment. Additionally, the companies in the cohort also varied their spending on the basis of opportunities in the short term.

As it stands, the theory that Apple uses capital investment in tooling to manage its quality and quantity of production and that in doing so it integrates deeply into its supply chain creating competitive lock-outs is holding up. It is not sufficiently precise to forecast actual production volumes for individual product lines but the growth in the business is broadly foretold by the growth in capital expenditures.

Indeed the share price generally reflects this:

Screen Shot 2014-09-30 at 5.31.38 PM

Proposition: At a basic (micro) level, the process of theory building is something we do instinctively. We observe patterns, make statements that A causes B and carry on with the theory as an assumption. The challenge is more on a macro level. Few theories are built rigorously about the causes of success or failure of business systems. This includes understanding why large, powerful firms fail when confronted with small, weak competitors. Why, how and when industries disappear. How resources are allocated and how priorities are set. It’s as if individuals behave with far more intuitive insight than firms.

That is what must change.

Because firms are increasingly determining the prosperity and sustainability of nations and the world. We can’t afford mismanagement.

The counter-point to this quest is that large systems are intractable and business is inherently chaotic, unpredictable. It may be, but much of what we know as science today was once thought of as impossibly mysterious and unknowable. I have faith that as the physical universe, the affairs of men have laws which govern them.

  • stefnagel

    Sweet. Can’t wait to see what you get into with your new job.

    “… tested by predictions…” That’s what most misuses of the term theory overlook. Theory must be measurable, testable, and ultimately repeatable.

  • ronin48

    It seems a bit presumptuous to assume that theory and theory building need to be defined or redefined.

    In science and medicine at least, theories (and even hypotheses) that are empirically contradicted become rejected. This is a basic tenet of the scientific method. And when a theory fails badly and often enough to be rejected, what replaces it is not a new version but a new and different theory altogether.

    Disruptive innovation, whether it can truly be called a theory or was never more than a hypothesis, has been empirically contracted in many ways. It has failed badly and often. It should be rejected.

    There’s a very reasoned analysis of disruptive innovation in a recent article in The New Yorker. I highly recommend it:

  • santoscork

    Complicated topic to say the least, chaos theory might lend a helping hand. I will not claim to be an expert on the topic by any stretch but theories themselves are difficult subject matter that can provoke deep debate and discussion.

    The first section of your post reminded me instantly of Rupert Sheldrake and his banned TedTalk – titled “The Science Delusion” presented at a TEDx event called “Challenging Existing Paradigms.”

    I recommend reading the article over the viewing the talk (not on as it’s more dramatic. He is very articulate and his, defence as it were, bestows much deserved thought and reflection.

    I very much doubt that I am presenting useful material for the task you have at hand but I believe there is still something valuable, lessons to learn when courage seeks to challenge dogma — Sheldrake might have won him more praise in the short term by not having observed anything at all but there is some respect attained by asking hard questions.

    From my vantage point Apple appears to routinely challenge the ‘status quo’. The “Think Different” ad campaign is but one example of Apple’s energy and force in expressing their will to go beyond the conventional. Since then Apple had to continue on a path of significant disruption to achieve their current status.

  • Walt French

    Not to dismiss the “scientific method” that works so well in the physical sciences, but while the history of them is strong, the present can look like a bumbling affair.

    A recent post on Ars Technica about dark matter highlights that there are many competing explanations for excess mass in the universe, but for many, we don’t even know how we’d collect evidence to support or contradict them. “Nature is sly,” the author states.

    Ditto for disruptions, I’ll guess. Both over-determined (too many potential theories/explanations for the few consistent pieces of data) and under-determined (missing information about intermediating mechanisms). Modern macro-economics is plagued with model specifications that can lead to multiple equilibria — the same starting conditions and forces, plus that damn butterfly in Colorado, lead to very different outcomes. Disruption theory, too, would be plagued by sometimes surprising side-effects. Who’d have thought NYC taxi revenues would be slashed in 2014, because of iPhones? What sensible model would’ve have predicted it, and ALSO rejected a good fraction of OTHER crazy ideas that haven’t happened? Who could even enumerate all the possibilities, well enough to say that a theory in 2014 was operating with better than coin-flip accuracy, say on a Chi-squared test basis?

    Not to discourage you; I love big ambitions. I think you’re doing the right thing by setting up the criteria by which you can say a theory is helpful or not. Good luck!

    • stefnagel

      Along these same lines, I peek at what Nassim Taleb is working at. He talks about the underestimation of randomness. And Black Swans. So there’s that.

      And Taleb does a lot of his work in math. Which makes sense as math makes the invisible visible. Horace seems a dab hand at math. But how mathy must he get to further articulate a theory on disruption?

      And, as someone once said about Bonnie Raitt and the blues, will he get so far into numbers that he never comes out the other side?

      • GlennC777

        Math quickly runs up against limits in even simple systems. The three-body problem in gravitation still doesn’t have a single complete solution, and the Shroedinger wave equation can only be solved for limited specific instances. That is quantum mechanics and gravity, right there, unsolvable except in the simplest cases.

        So any system with any real complexity needs to be modeled, which means approximated, and the approximations only work when the assumptions made about the behavior being modeled are correct, and are always limited in precision.

        I doubt any of this would apply to human behavior except in the most limited ways, anyway.

      • Walt French

        It’s too bad that Taleb’s many good points are often buried underneath his self-promoting bloviation.

        But yes, it’s relevant. Maybe in the very long run, disruptive innovation is just a force that rises, saturates, and is replaced, while in the short run, a coin flip determines whether BlackBerry hangs on long enough to beat back the innovation, much as Apple recovered after failing against Microsoft. (Yes, a very strongly biased coinflip.)

        The more I think about this, the more I think making one- to five-year forecasts for disruptive innovators is a fraught enterprise. Apple appears to be very comfortable with its iPhone, Mac and iPad business lines, so is willing to make a very large — huge, maybe — shift towards non- or low-tech wearables, shown by Beats and the gold watch that has 90% of its value in something Apple has never done before.

        Long ago, Horace made the point that by the time you have hard data, it’s too late to use it strategically. Never more true. Cook & Co are flying by a reckoning essentially invisible to the outside. Not wrong, just impossible for me to know.

    • stefnagel

      As another example of complex systems: How wolves make rivers.

    • charly

      Taxi revenue in NYC hasn’t been slashed in 2014 if you include gypsy cabs.

  • Christian Peel

    Congratulations! What’s your position at the Christensen Institute? Your name is not on the list of team members at the moment.

    In what ways is “Theory Building” different than the scientific method?

    • Horace Dediu

      The scientific method is a more detailed information sharing (publishing and review) process. Theory building is a “meta-level” process for how the information obtained from scientific processes is analyzed.

  • chano1

    But Horace, as with the universe and Man’s affairs, there are more than just definable laws at work. There are so many other imponderables that can have significant impacts: insight, intuition, chance/serendipity, hunger:complacency ratio, management culture usw…..

  • equanimous

    A laudable effort and I wish you success. However, one suggestion I might make is to avoid using Physics as an analogy. A closer analogy would be your favorite sport – humans don’t behave as repeatably as atoms and molecules and are not as bound by behavioral law.

    Each business situation must be analyzed using rigor but beware of inductive reasoning – which is what some business theories tend to engender after they’ve been ‘established’

  • Nalini Kumar Muppala

    You might want to include Paul Carlile as an author of the original paper. Also, a link to the original paper would be fitting.

  • emaansingh

    Hi Horace, you could look at some literature from the History and Philosophy of Science. Thomas Kuhn should give you a view of how theories are made, some literature from the Sociology of Scientific Knowledge would also inform you on the notion of how theory is made (Bruno Latour) and challenged. Positivism (try Karl Popper for positivism) has many informed critiques and an understanding of those will help you formulate your own position.
    Best of Luck.

    • stefnagel

      Thomas Kuhn. Excellent.

  • deemery

    It might be worth considering the “OODA Loop” theory of John Boyd for decision-making. After all, isn’t a ‘theory’ itself a kind of decision?

  • Accent_Sweden

    While I enjoy and look forward to your thoughts on the topic, a warning flag shot up for me on your last line: “I have faith that as the physical universe, the affairs of men have laws which govern them.” I must ask you to be careful of hinting at or confusing natural laws of the physical world with how humans interact and their dynamics. That road is paved by the bones of hundreds of millions of individuals. Faith is the key word here. Faith and proof, never shall and never should the twain meet. There is no “If this, then that” laws in the “affairs of men”. There are “If this, sometimes that, but perhaps also that, and maybe that, then again could be that”. Theories to explain events are useful but are not to be confused with laws that are unchanging. I realise you probably wrote the last line as a nice sounding ending and likely didn’t mean it as I interpreted it, but it struck an admittedly overly sensitive liberal arts nerve and so I take to my keyboard to say my peace.

    • Walt French

      Suggests at least a brief look at the science of history (where “science” takes on its much more general historical meaning of a body of thought).

    • jinglesthula

      “It may be, but much of what we know as science today was once thought of as impossibly mysterious and unknowable.”

      Humans are not conducive to having all their variables controlled for in a sterile experiment. That does not mean we should shy away from seeking theories to aid in understanding their affairs. Nor does it mean that we should consider all theory/belief/faith on them completely useless until we believe we have proof.

      Theories about the physical universe are anything but fixed. Yet we have faith that there really are underlying laws. Our continuous hammering on theories about natural laws is our quest to understand them. I believe Horace’s work demonstrates the value of seeking for useful theories about people and human systems as well.

    • stefnagel

      Spot on. “… the affairs of men have laws that govern them.” Every other word in that statement seems loaded and burdened. And unfun.

      Why, as Horace says so often, do disruptions come from the small org, the new org, the innocent, the ignorant? And why do the disruptive products seem to be toys, not tools, at first?

      I would look at the ideas of play, children, learning, and creativity to find disruption. Says Blake in his songs of innocence:

      Little lamb, who made thee? …
      Little lamb, I’ll tell thee:
      He is called by thy name,
      For He calls Himself a Lamb.
      He is meek, and He is mild,
      He became a little child.

  • Boudica

    I believe Hari Seldon already solved your challenge some time ago (or was it some time from now?).

    • fahirsch

      Two problems with Hari Seldon’s Theory: a) We never saw the equations; b) a mutant happened.

      • Sacto_Joe

        Without The Mule, how boring and predictable things would be….

    • stefnagel

      Wasn’t Hari’s insight that we don’t so much make choices as recognize choices that have been already made for us?

  • obarthelemy

    You seem to be describing the scientific process in general. Observe a behaviour, formulate a theory about it, test the theory. But, there’s a reason why social sciences in general are called “soft”, and economics in particular “the dismal science”. I never found Asimov’s “Foundation” very believable (though mildly thought-provoking).
    Also, I think “intelligence” is not simply understanding complex systems. It is also being able to hold several incompatible or orthogonal understandings of a given system.

    • GlennC777

      It is quite possible that such systems are non-understandable. It isn’t enough to say that there are too many variables: each element of the system has the capacity to introduce new variables, and the effects of a given variable can be fundamentally unpredictable (chaotic).
      Where the hard sciences can model the behavior of a complex system with known rules, for example the behavior of gases, even this has modest limits, i.e. weather, which is literally impossible to model beyond a certain threshold of accuracy. The uncertainties begin at the quantum level and expand to the macro. Imagine trying to model the behavior of a gas when each molecule can change its behavior at will in unpredictable ways.

      I have long believed in the notion that the course of human events is chaotic in nature and only seems predictable in hindsight. What will the effects of the difficulties in the Ukraine be, or in Hong Kong? What if the Manhattan Project had been unsuccessful, Hitler had been hit by a bus in his youth or FDR had not contracted polio and so on and so on? Cultural and social movements can succeed or die based on ephemeral momentum effects, leaders are chosen or win their positions by hair-splitting margins, and the unknown future course of history bifurcates dramatically on a dime.
      Many social scientists disagree with this, maybe partly because it would greatly limit what could be achieved in their disciplines. Perhaps they are right. My bet is against them.

  • Nalini Kumar Muppala

    Effort estimation comes to mind. Like Theory development the estimation process constantly evolves and learns from new inputs and anomalies. Tasks are different – some in subtle ways and others in big ways. The cost/effort estimation process is important as it determines the profitability of companies that take on fixed cost jobs. At the end of the day the algorithm or heuristic that a person or team develops needs to evolve and learn with time. Without such a heuristic the estimator is missing on past experiences and is at the whim of emotions. Theory helps to estimate/predict what could happen but reality is unknown until the fact. Without Theory one has few arrows in the quiver to attack the problem of prediction/estimation.


  • berult

    How could the open mind, which mingled with the Universe as though its physical overarch it owned, have frowned upon the boundless teaching exerted on a-void-ance by a single sub-atomic particle?

    How could the open mind, which maddeningly devises architectures for a “Philosophiæ Naturalis Principia Mathematica” solution to disruption, have foretold, on a blitzkrieg errand, Steve Jobs’ iPhone-advent…to glitter as a star-construct of a theoretical non sequitur?

    An event in the life of the Universe elevates modesty to the power of zen. Apologies for the slight German accent.

  • stefnagel

    Here’s my wish for Horace’s endeavor, in the words of Peter Drucker: “The best way to predict the future is to create it.”

  • Sacto_Joe

    How “disruptive” is the AppleWatch? It’s roots, I’m convinced, are in the lowly wristband-mounted iPod Nano (which I’ve owned and used). In that sense, it’s a disruption that made an initial appearance years back. And it has already spawned spin-offs, including alternative smart-watches by Apple’s competitors. True, Apple has, once more, gone to the well for a new interface, improving the ability to interact with a tiny device. And that will in turn increase the usefulness of that device. But really, this is just the natural process one would expect as portable computing and miniaturization continue to evolve.

    The “driver”, therefore, of this wave of disruption is, simply, the evolution of computer tech. Which explains why Apple is benefitting so greatly while others are losing out. But are there other potential disruptors out there? That’s the really interesting question….

    • charly

      The core purpose of the AppleWatch is not an ipod nano but a wallet so i seroiusly doubt that the nano is its root.

    • Walt French

      “Roots,” I suppose. But the OS is a hundred times more complex than what the iPod OS would support. My money would be on the watch running a very small subset of iOS. Apple is looking for developers to boost the watch. They can’t be too enthused about learning a *third,* and radically different, environment (most do both iOS and MacOSX apps), let alone having to learn an entirely different programming approach.

      Apple (Jobs?) long ago observed that maintaining multiple, incompatible OS’s was a quick route to failure. Ergo, the substantial overlap in developer tools, frameworks and support services for iOS and MacOS. Also, the near-complete absence of third-party apps for iPodOS.

      Which, per Wikipedia, was quite different from anything we use nowadays: “…PortalPlayer’s reference platform based on two ARM cores. The platform had rudimentary software running on a commercial microkernel embedded operating system.”

  • def4

    The affairs of men do have some laws but they are surely not like the laws of the physical universe.
    Humans are driven by emotions. Your acceptance or denial of this fundamental fact will determine whether you are chasing a deeper truth or the next management fad.
    You will be lied to. A lot. Sometimes even by yourself.

    • jinglesthula

      Are you suggesting that human emotions are not governed by laws?