Econometric Modeling as Junk Science

What Lott and Mustard were doing was comparing trends in Idaho and West Virginia and Mississippi with trends in Washington, D.C. and New York City. What actually happened was that there was an explosion of crack-related homicides in major eastern cities in the 1980s and early 1990s, most of them among people who were quite well armed despite the lack of gun permits. Lott's whole argument came down to a claim that the largely rural and western "shall issue" states were spared the crack-related homicide epidemic because of their "shall issue" laws. This would never have been taken seriously if it had not been obscured by a maze of equations.

Why Regression Fails: Lack of Predictive Testing.

The acid test in statistical modeling is prediction. A useful model should make predictions that are better than random guessing. Only in this way can cause and effect be distinguished and causal predictions be tested. Regression modelers often do this with historical data, in effect using data from the more distant past to predict the more recent past. The problem with this is that, when the outcome is already known, it is too easy to adjust the model to fit the known outcome. This is like using the day before yesterday's weather to predict yesterday's weather, or the day before yesterday's stock prices to predict yesterday's prices. This may be useful as a way of learning, but the only really persuasive test is to predict tomorrow's weather or stock prices. This criterion, success in prediction, is used to evaluate models of financial markets, the weather, medical outcomes, population trends and many other phenomena. These models all work imperfectly, and regression gives us a good measure of just how imperfectly.

Unfortunately, researchers who use econometric techniques to evaluate social policies generally do not subject their models to predictive tests. They could do so, either by making future predictions and waiting to see what happens, or, if that would take too long, by developing their model with data from one population and then using it to predict results with another population. But most researchers simply do not do this, or if they do the models do not work so the results are never published. (The Urban Institute did make a prediction, but they did not wait for the outcome data to publicize their conclusions. When the data showed that their model did not work, they simply took it down from their WEB site.)

The journals that publish these studies do not require predictive testing, which suggests that the editors and reviewers have low aspirations for their fields. Instead, researchers take data for a fixed period of time and keep fine tuning and adjusting their model it until they can "explain" the trends that have already happened. There are always a number of ways to do this, and with modern computers it is not terribly hard to keep trying until you find something that fits. At that point, the researcher stops, writes up the findings, and sends the paper off for publication. Later, another researcher may adjust the model to obtain a different result. This fills the pages of social science journals and helps young professors get tenure. Everybody pretends not to notice that little or no progress is being made.

The Alternative: Insist on Intelligible Graphs and Tables

When junk science is released to the media by scholars at prestigious universities, and published in reputable refereed journals, people become understandably skeptical about the value of social science research. A few years ago The Economist (May 13, 1995) published a tongue-in-cheek editorial announcing that “74.6% of sociology is bunk.” Cynics wondered if the estimate might not have been low. But it is important not to throw out the baby with the bath water. There is good solid work being done in sociology, criminology and other social sciences, although it may not make it into the journals that value statistical complexity over reliable findings. The most reliable work uses simpler statistical techniques that do not require so much adjustment and standardizing of the data. This has the great advantage that it the work can be read and used by people who have not devoted years of their lives to learning obscure econometric techniques.

Studies that make extensive use of graphics, such as those of Sellin (1959) and Blumstein and Wallman (2000) have been much more successful and informative than regression studies. As an example of the power of simple graphical techniques, we can graph some of the data from John Lott's gun control data set. When a data set is so huge, it may be necessary to select a small part of it for graphing, but this can be quite informative if the selection is done well. In reviewing Lott's data, I found that in the state of Pennsylvania, a "shall issue" law was passed in 1989, but the city of Philadelphia was exempted from it. This provided an excellent opportunity for "natural experiment, " comparing trends in two metropolitan areas that differed on a key variable. The graph that follows compares trends in Philadelphia, which is a city and a county, with those in Allegheny County, which includes Pittsburgh. The graph shows that murder rates are generally higher in Philadelphia than in Allegheny County, but the passage of a law giving citizens the right to get permits to carry concealed weapons did not have the positive effect posited by John Lott. In fact, the Allegheny County murder rate was declining prior to the passage of the law, then increased slightly. In Philadelphia, the murder rate had been increasing, then it leveled off despite the fact that the new law did not apply in that city. The violent crime statistics for the same two counties show the same pattern. Disaggregating the data in this way allows us to draw on our qualitative, historical knowledge in interpreting statistical trends. To discredit this kind of finding, concealed weapons advocates would have to show how other factors somehow compensated for the failure of the shall issue law to have any apparent effect.

Conclusions