Analysts' categorical failure

Gartner explicitly explained so in its press release: “Gartner’s PC group does not track media tablet sales in this PC shipment data, so iPad sales are not included in these results

via Is Apple the real U.S. PC market share leader — or soon will be? | Betanews.

The reluctance of  industry analysts to measure the iPad as a computer is a fascinating and vivid symptom of how analysts conspire with their customers to smother visibility of impending failure.

Before we dive into the motivation to ignore iPads, we need to understand how analysts in general and companies in particular group products into comparable piles. This process of grouping is called categorization. It should be distinguished from segmentation which groups buyers in a market. Categorization is essential for competitive analysis and measurement of the performance of a product (relative to other products).

Categorization is a challenging problem for most companies. To illustrate why, consider four possible methods for categorizing products:

  1. By product attributes. The assumption here is that products with similar attributes compete in the same market. As an example hard disk drives might be categorized by dimensions of the cases and sub-categorized by capacity and perhaps transfer speeds.
  2. By user attributes. This method is similar to segmentation where products are grouped by the type of buyer. Magazine titles might be properly categorized by the age and gender of the readers. This can also cover geographic data.
  3. By user behavior. This method, sometimes called psychographics is a more advanced method than demographics where user personality, values, attitudes, interests or lifestyles are measured. You might have seen examples where users are defined with a “prototype” person and the products are attached to this prototype. This is a common way to categorize mobile phones (Nokia is a known leader in this method).
  4. By modes of consumption. Determine what products are hired to do the same job. In this case you have to have real insight into what people actually do with their products and try to look broadly at what they substitute. The classic example of this is to understand that a power drill competes with products what make holes in walls not just with other drills.

When reading this list, it’s easy to see that it’s ranked in order of difficulty. The easiest thing to do is group according to things you can measure easily. Products are easier to measure than demographics which are easer to measure than psychographics and that’s easier to capture than observed behavior.

Consider how you would structure a study to capture this data.

  1. In the first case, you need an analyst to collect products (or just their descriptions) and record values in a spreadsheet.
  2. In the second case, users need to be surveyed for basic data. This is hard because you have to find users and then you have to get a statistically significant unbiased sample.
  3. In the psychographic study you need to ask a lot more questions about lifestyle in your survey and hope that the subject answers honestly. You still need a large sample.
  4. In the case of consumption analysis you need to observe a subject use a product for a prolonged period of time in the usual context where the product is used and be able to interject questions when certain behavior is encountered. The observation of one person can take one research person-day per subject. To get statistically significant data requires literally man-years of work.

Now it should be clear why an analyst will only categorize according to product attributes. It’s not only economically and temporally expensive to perform research on methods 2 to 4 but it also requires interpretation and insight into study development.

A few companies go beyond buying analyst market data and do their own research. As far as I know user studies are more commonly done by consumer products companies. But it’s extremely rare to see this research done by low-margin technology companies. They don’t have the budgets or skill sets to do any research.

However, in my experience, nobody performs the fourth, most difficult, form of categorization.

The result is that good, smart managers trained to always base their decisions on hard facts, pick up these lowest-deonominator analyst reports to help decide which products to develop. And it should be obvious why they are surprised when products that did not show up on any market study suddenly take all their profits away.

Thanks Gartner.