Econometric Modeling as Junk Science

Although they seem complicated, regression models are actually simplifications of the real world. To simplify the mathematics, regression uses linear equations. This means it assumes that if you plot the relationship between any two variables on a graph, the trend will look like a straight line. Regression models also assume that variables are distributed according to a classic bell-shaped normal curve. And it assumes analysts know which variables are causes and which are effects. Of course, regression analysts know that the real world does not fit their assumptions, and they make various adjustments to the data to compensate. But the adjustments create other problems. The only valid way to test a model after all these adjustments is to show that it works to predict future trends. Regression models that have not been demonstrated to work with fresh data, other than the data used to create them, are junk science.

Why Regression Fails: Linear Models of a Nonlinear World.

When faced with the nonlinearity of the real world, the first instinct of the regression modeler is to standardize and control the data. In doing this, they minimize or eliminate the most interesting and important historical events. They end up analyzing a standardized and idealized world that bears little relationship to reality. For example, consider the trends in prison growth and homicide that Marvell and Moody (1997) sought to explain. Their paper begins with a graph showing trends in prisoners per 100,000 people and homicides per 1,000,000 people in the United States. This very interesting and useful graph is reproduced below from their data.

The interesting things about the trends the graph portrays are the turning points, the places where the trends diverge from linearity. Homicide rates increased sharply from the mid 1960s to the early 1970s, then leveled off. The number of prisoners shot up markedly beginning in the 1970s, as the United States built more prisons in response to the increasing crime rate. The homicide rate leveled off in the 1980s and remained stable thereafter.

Instead of trying to explain these important turning points, Marvell and Moody used multiple regression techniques to "control" for it. They introduced controls for every measurable variable they could think of, including (Marvell and Moody, 1997: 209) "age structure, economic factors, public relief, race, and variables marking World War II and the crack epidemic."

All these controls purged the striking historical changes from their data. This led them to the conclusion that a 10% increase in prison populations leads to roughly 13% fewer homicides. But a simple inspection of their graph shows that the promised 13% decline in the homicide rate for each 10% increase in imprisonment since 1975 simply did not occur.

Marvell and Moody were troubled that the expected reduction did not take place, but it was not enough to cause them to abandon their econometric methods. They, after all, were not discussing the real world but a world simplified and purified by a long series of mathematical adjustments. Confronted with the historical facts, they argued that, had imprisonment not increased, homicide would have gotten a lot worse. They went on, however, to observe that this would never really have happened because the government would have taken other actions to prevent it.

But what is the value of an analysis that leads to implications that the authors realize could never actually take place? How valid can the theory underling the multiple regression analysis be if it leaves out key variables, such as political constraints, simply because they cannot be quantitatively measured? How much do the results depend on arbitrary decisions about which control variables to introduce and how to measure them? Marvell and Moody were left with statistics that purported to tell us what might have happened if nothing that actually happened had happened.

Why Regression Fails: The World is Not a Bell-Shaped Curve.