Econometric Modeling as Junk Science

Zimring and Hawkins were right. Within a year, two determined econometricians, Dan Black and Daniel Nagin (1998) published a study showing that if they changed the statistical model a little bit, or applied it to different segments of the data, Lott and Mustard's findings disappeared. Black and Nagin found that when Florida was removed from the sample there was “no detectable impact of the right-to-carry laws on the rate of murder and rape.” They concluded that "inference based on the Lott and Mustard model is inappropriate, and their results cannot be used responsibly to formulate public policy."

Myth Two: Imprisoning More People Cuts Crime

The Lott and Mustard case was exceptional only in the amount of public attention it received. It is quite common, even typical, for rival studies to be published using econometric methods to reach opposite conclusions about the same set of data. In one exceptionally frank statement of frustration with this state of affairs, two highly respected criminologists, Thomas Marvell and Carlisle Moody (1997: 221), reported on the reception of a study they did of the effect of imprisonment on homicide rates. They reported that they:

“widely circulated [their] findings, along with the data used, to colleagues who specialize in quantitative analysis. The most frequent response is that they refuse to believe the results no matter how good the statistical analysis. Behind that contention is the notion, often discussed informally but seldom published, that social scientists can obtain any result desired by manipulating the procedures used. In fact, the wide variety of estimates concerning the impact of prison populations is taken as good evidence of the malleability of research. The implication, even among many who regularly publish quantitative studies, is that no matter how thorough the analysis, results are not credible unless they conform with prior expectations. A research discipline cannot succeed in such a framework.”

To their great merit, Marvell and Moody frankly acknowledged the problems with multiple regression, and made some suggestions for improvement. This, however, is more the exception than the rule with econometricians, who often become so immersed in their models that they lose track of how arbitrary they are. Many of them come to believe that their models are more real, more valid, than the messy, recalcitrant, "uncontrolled" reality they purport to explain.

Myth Three: Executing People Cuts Crime.

In 1975 The American Economic Review published an article by a leading economist, Isaac Ehrlich of the University of Michigan, who estimated that each execution deterred eight homicides. Before Ehrlich, the best known specialist on the effectiveness of capital punishment was Thorsten Sellen, who had used a much simpler method of analysis. Sellen prepared graphs comparing trends in different states. He found little or no difference between states with or without the death penalty, so he concluded that the death penalty made no difference. Ehrlich, in an act of statistical one-upmanship, claimed that his analysis was more valid because it controlled for all the factors that influence homicide rates.

Even before it was published, Ehrlich's work was cited by the Solicitor General of the United States in an amicus curiae brief filed with the United States Supreme Court in defense of the death penalty. Fortunately, the Court decided not to rely upon Ehrlich's evidence because it had not been confirmed by other researchers. This was wise, because within a year or two other researchers published equally sophisticated econometric analyses showing that the death penalty had no deterrent effect.

The controversy over Ehrlich's work was so important that the National Research Council convened a blue ribbon panel of experts to review it. After a very thorough review, the panel decided that the problem was not just with Ehrlich's model, but with the use of econometric methods to resolve controversies over criminal justice policies. They (Manski, 1978: 422) concluded that:

“because the data likely to be available for such analysis have limitations and because criminal behavior can be so complex, the emergence of a definitive behavioral study lying to rest all controversy about the behavioral effects of deterrence policies should not be expected.”