Econoblogging becomes a college course
American Liberals Sneaking Across Border Into Canada

Super Crunchers review

About three months later than promised, here’s my review of a (relatively) new book by Yale economist and lawyer, Ian Ayres, titled Super Crunchers. I thank Jason D. Gordon, President of Precedent Media, Unlimited for sending me a copy.

As an old joke begins, “There’s good news and there’s bad news . . .”

The good news.

This could well be an interesting book for a high school or college student who is curious about applications of statistics. This could well be a useful book for a businessman who is unacquainted with statistics and who wants to know how experiments might help him run his business. For everybody else, there are a handful of good stories. The stories are about how, according to Ayres, “We are in a historic moment of horse-versus-locomotive competition, where intuitive and experimental expertise is losing out time and time again to number crunching. . . . Business and government professionals are relying more and more on databases to guide their decisions. . . . What is Super Crunching? It is statistical analysis that impacts real-world decisions.” (p. 10)

Here are the stories that I think are interesting.

—To the utter dismay of the “grand world of oenophiles”, noted economist Orley Ashenfelter constructed a statistical model that predicts—years in advance—the quality of Bordeaux wines (pp. 1-6). It’s an impressive story, although I think Ayres should have presented more detail about how accurate Ashenfelter’s predictions have been. The single sentence on page 9 is intriguing but insufficient.

—Wal-Mart is well-known for its expert number-crunching. The book recounts a small but striking instance (p. 29): “Before Hurricane Ivan hit Florida in 2004, Wal-Mart already had started rushing strawberry Pop-Tarts to stores in the hurricane’s path. Analyzing sales of other stores in areas hit by hurricanes, Wal-Mart was able to predict that people would be yearning for the gooey comfort of Pop-Tarts, finger food that doesn’t require cooking or refrigeration.”

—Ayres discusses how companies such as CapOne, Monster.com, and Jo-Ann Fabrics are determining elements of their marketing strategies through randomized experiments (pp. 46-49; pp. 52-54). And he asserts that as the Internet lowers the cost of such experimenting, more firms will probably begin to do it.

—Don Berwick, president of the Institute for Healthcare Improvement, carefully analyzes data using statistics, including data from randomized experiments, to suggest simple ways for hospitals to improve care, improve care to the point of saving thousands of lives. Simple ways like elevating the heads of patients on ventilators and applying antiseptics to the skins of patients with central line catheters (pp. 83-87).

—Over the last few decades, social scientists have documented that people are prone, ordinarily, to systematic biases in their perceptions and in their decision-making. Models derived from statistical analysis can thus be expected, in some circumstances, to forecast better and to decide better than humans.  The book presents this example (pp. 110-11): “. . . purchasing managers couldn’t outperform a simple statistical formula to predict timeliness of delivery, adherence to the budget, or purchasing satisfaction.” (See also the discussion of decision support software, pp. 95-102.)

In addition to these examples, Ayres also adroitly addresses a question that readers may have after learning about the wonders of Super Crunching: “What’s Left for Us [Humans] to Do?” (pp. 124-28) Ayres answers that statistics and computers, by themselves, can’t develop theories with testable implications; they can’t develop causal explanations. And we still very much need such theories and explanations if we want to try to understand the world.

The non-so-good news.

These examples plus some additional detail plus the material needed to glue it together could have comprised a useful, tightly-focused essay of forty pages or less. What Ayres has given us instead is a book of 217 pages. (The reader may wonder if this is just sour grapes on my part: I’ve had few—if any—ideas worth forty pages, let alone a book. Could be. But I press on . . .)

It wouldn’t be bad if the other material were simply less interesting or less important. But I think the additional material weakens the book more than that. There are a number of places where Ayers’s presentation is superficial and questionable.

Here are some examples.

—On page 57 Ayres tells us that charities have started using randomized experiments to help them raise money. One such experiment concluded that an offer to match contributions dollar-for-dollar increased contributions by 19 percent. But surprisingly, “two-for-one and three-for-one matches didn’t generate any higher giving than the one-for-one match.” In the next paragraph Ayres crows, “One thing we’ve seen over and over is that decision makers overestimate the power of their own intuitions.”

Excuse me? The charitable giving study is surprising not because it contradicts intuition but because it seems to contradict a fundamental part of economic theory, the Law of Demand. The Law of Demand states that a lower price, other things equal, increases quantity demanded. A higher match rate effectively lowers the price of charitable giving, so the Law implies that the higher match rate should have definitely increased giving. (The theory doesn’t predict that giving would necessarily increase a lot, just that it would increase.)

I’m sorry, but I’m unwilling to let a single empirical study, randomized or not, trump a well-verified and extremely useful piece of economics.

—Ayres wonders (p. 60) why more firms aren’t doing randomized experiments. “Why isn’t Wal-Mart running [them]?” He repeats the question (p. 62): “Why aren’t more firms doing this already? Of course, it may be because traditional experts are just defending their turf.”

My question is how the heck does Ayres know whether firms are doing randomized experiments or not? If they are as useful as he contends, why would firms want to inform their competitors? And if they’re not doing them, there could well be sound economic reasons why not. Ayres, himself, even provides one possible reason a few pages earlier: firms discovered to be making randomly different offers can hugely anger their customers, as Amazon infamously learned (p. 59).

—Ayres writes (p. 136): “. . . the norm of sharing datasets has become increasingly grounded in academics. The premier economics journal in the United States, the American Economics Review, requires that researchers post to a centralized website all the data backing up their empirical articles.”

Ayres’s depiction of the data sharing “norm” is, currently at least, way too rosy. He should have at least cited the work of Drexel economist Bruce D. McCullough and co-authors. See, in particular, “Do Economics Journal Archives Promote Replicable Research?

—On pp. 145-49 Ayres gushes over a company called Epagogix and its alleged success, using a neural network model, at forecasting the gross revenues of movies.

But whether neural network models—or any statistical model—can reliably forecast movie success runs counter to the seminal work on the economics of movie-making by economist Art DeVany. Ayres doesn’t note DeVany’s work. For a brief, non-technical introduction to DeVany’s research that appeared in the New York Times, see this. For DeVany’s withering attack on a movie-revenue neural network model—granted, not necessarily resembling the one used by Epagogix (and in a brief search I couldn’t find that DeVany has commented specifically on Epagogix’s model)—see this.

—Ayres devotes just one paragraph (I think) to the important statistical problem of overfitting (p. 144).

It’s a O.K. paragraph, but how many people in the book’s intended audience—which I presume to be people who are not statisticians or quantitative analysts—can begin to understand the problem from just this brief mention? And I would have liked Ayres to refute—at least a little—the possibility that his study of what causes a law review article to be more frequently cited might be a swell example of overfitting. (He notes that his “central statistical formula had more than fifty variables.” )

This relates to a larger problem with most of the book: there aren’t enough qualifications about number-crunching for a presumably non-specialist audience. On p. 116 Ayres writes, “In sharp contrast to traditional experts, statistical procedures not only predict, they also tell you the quality of the prediction.” Sure, if the investigator has a random sample and if he or she also estimates the true model. Even given the random samples in the studies Ayres focuses on, model (specification) uncertainty is usually a substantial problem. And all our cheap computing power is making the important problem of overfitting or “data mining” ever more important.

(Further down on p. 116 Ayres does note “. . . data analysis can itself be riddled with errors. Later on, I’ll highlight examples of data-based decision making that have failed spectacularly.” But the “highlight[ed] examples” seem to be mainly an attack on John Lott’s concealed-carry research—more on this below—and three pages, pp. 187-90, that focus on attrition bias in random samples. This is quite a bit less than needed to give readers some understanding of the specification and data-mining problems.)

One might reply that an author writing a book for a mainstream audience shouldn’t be expected to provide a lot of detailed qualifications about a technical subject. Particularly when the author is an economist, lest he remind readers of the old joke about economists that has the punch line, “Somebody find me a one-armed economist!”

As a teacher I’m aware that there’s a difficult tradeoff between presenting material in a simple and comprehensible way and presenting material extremely carefully with attention to qualifications, exceptions, and details. But I would make two points. One, Ayres chose to write a book with eight chapters and about twenty-six pages of endnotes. How costly would it have been to provide another half dozen pages of notes, or another chapter, to further discuss the important limits of number-crunching?

And two, Ayres certainly doesn’t seem averse to a “on the one hand, but on the other hand” argument. Early in the book he argues that way about the welfare consequences of Super Crunching—“To some, this kind of manipulation is the science of diabolically separating as many dollars from a customer as possible on a repeated basis. To others, it is the science of improving customer satisfaction and loyalty—and of making sure the right customers get rewarded. It’s actually a bit of both.” (p. 31)—and he returns to this argument several times, to the point of excess. (Compare p. 173 on which he argues that firms are going to take advantage of number-crunching to practice more, and more effective, price discrimination such that “. . . consumers are going to have to search more to make sure that the offered price is fair” to the following page on which he notes that “Firms like Farecast.com, E-loan, Priceline, and Realrate.com allow customers to comparison shop more easily. In effect, they do the heavy lifting for you and help level the playing field with the price-crunching sellers.”)

—Finally, let me note that Ayres’s treatment of the research of my graduate-school classmate, John Lott, is cursory and distorted . On pages 14 through 16, Ayres touts his paper (co-authored with Steven D. Levitt) about a supposed LoJack externality. But he doesn’t refute, or even mention, John’s substantial criticism of that paper. (Interested readers can see John's brief comment; a bit longer discussion appears in his book Freedomnomics, pp. 43-44.)

Ayres does discuss Lott and some of his work on pages 180 through 185. The first page and half, however, is a completely irrelevant discussion of some of John’s internet postings and personality traits. The last half page imprudently discusses a legal case that was in progress at the time Ayres wrote the book and includes this odd, smirking remark, “John is such a tenacious adversary that I’m a little scared to mention his name here in this book.” (p. 184) In between those two parts, readers get an extremely unbalanced discussion of the alleged failings of John’s concealed-carry research. Even if Ayres were completely right on the merits, he had other, much better examples from which to make his point and he certainly should have scrapped the strange ad hominem details.

My conclusion: if you can get a Readers’s Digest version of Ayres’s book, or you can hear him give a brief summary of it, it might be worth your time. I am more doubtful that, for many readers, it’s worth even Amazon's discounted price of $16.50.

Comments