Peter Frase

The present crisis looks as though it may bring about a long-delayed moment of reckoning in the field of economics. Macro-economics has been plunged into turmoil now that many of its leading practitioners stand exposed as dogmatists blithely clinging to absurd pre-Keynesian notions about the impossibility of economic stimulus and the inherent rationality of markets, who have nothing at all to say about the roots of the current turmoil. Micro-economics, meanwhile, has seen Freakonomics run its course, as long-standing criticisms of the obsession with "clean identification" over meaningful questions spill over into a new row over climate-change denialism.

Joining the pile-on, Jon Elster has an article in the electronic journal Capitalism and Society on the "excessive ambitions" of the social sciences. Focusing on economics--but referring to related fields--he criticizes three main lines of inquiry: rational choice theory, behavioral economics, and statistical inference.

Although I agree with most of the article's arguments, much of it seemed rather under-argued. At various points, Elster's argument seems to be: "I don't need to provide an example of this; isn't it obvious?" And with respect to his claim that "much work in economics and political science is devoid of empirical, aesthetic, or mathematical interest, which means that it has no value at all", I'm inclined to agree. But it's hard for me to say that Elster is contributing a whole lot to the discussion. I'm also a bit skeptical of the claim that behavioral economics has "predictive but not prescriptive implications", given the efforts of people like Cass Sunstein to implement "libertarian paternalist" policies based on an understanding of some of the irrationalities studied in behavioral research.

But the part of the essay closest to my own interests was on data analysis. Here Elster is wading into the well-travelled terrain of complaining about poorly reasoned statistical analysis. He himself admits to being inexpert in these matters, and so relies on others, especially David Freedman. But he still sees fit to proclaim that we are awash in work that is both methodogically suspect and insufficiently engaged with its empirical substance.

The criticisms raised are all familiar. The specter of "data snooping . . . curve-fitting . . . arbitrariness in the measurement of . . . variables", and so on, all fit under the rubric of what Freedman called "data driven model selection". And indeed these things are all problems. But much of Elster's discussion suffers from his lack of familiarity with the debates. He refers repeatedly to the problem of statistical significance testing--both the confusion of statistical and substantive significance, and the arbitrariness of the traditional 5% threshold for detecting effects. While I wouldn't deny that these abuses persist, I think that years of relentless polemics on this issue from people like Deirdre McCloskey and Jacob Cohen have had an impact, and practice has begun to shift in a more productive direction.

Elster never really moves beyond these technical details to grapple with the larger philosphical issues that arise in applied statistics. For example, all of the problems with statistical significance arise from an over-reliance on the null hypothesis testing model of inference--even though as Andrew Gelman says, the true value of a parameter is never zero in any real social science situation. Simply by moving in the direction of estimating the magnitude of effects and their confidence intervals, we can avoid many of these problems.

And although Freedman makes a number of very important criticisms of standard practice, the article that Elster relies upon relies very heavily on the weakness of the causal claims made about regression models. As a superior model, Freedman invokes John Snow's analysis of cholera in the 1850's, which used simple methods but relied upon identifying a natural experiment in which different houses received their water from different sources. In this respect, the article is redolent of the time it was published (1991), when the obsession with clean identification and natural experiments was still gaining steam, and valid causal inference seemed like the most important goal of social science.

Yet we now see the limitations of that research agenda. It's rare and fortuitous to find a situation like Snow's cholera study, in which a vitally important question is illuminated by a clean natural experiment. All too often, the search for identification leads researchers to study obscure topics of little general relevance, thereby gaining internal validity (verifiable causality in a given data set) at the expense external validity (applicability to broader social situations). This is what has led to the stagnation of Freakonomics-style research. What we have to accept, I think, is that it is often impossible to find an analytical strategy which is both free of strong assumptions about causality and applicable beyond a narrow and artificial situation. The goal of causal inference, that is, is a noble but often futile pursuit. In place of causal inference, what we must often do instead is causal interpretation, in which essentially descriptive tools (such as regression) are interpreted causally based on prior knowledge, logical argument and empirical tests that persuasively refute alternative explanations.**

This is, I think, consistent with the role Elster proposes for data analysis, in the closing of his essay: an enterprize which "turns on substantive causal knowledge of the field in question together with the imagination to concoct testable implications that can establish 'novel facts'". And Elster gives some useful practical suggestions for improving results, such as partitioning data sets, fitting models on only one half, and not looking at the other half of the cases until a model is decided upon. But as with many rants against statistical malpractice, it seems to me that the real sociological issue is being sidestepped, which is that the institutional structure of social science strongly incentivizes malpractice. To put it another way, the purpose of academic social science is not, in general, to produce valid inference about the world; it is to produce publications. As long as that is the case, it seems unlikely that bad practices can be definitively stamped out.

**Addendum: Fabio Rojas says what I wanted to say, rather more concisely. He notes that "identification is a luxury when you have an abundance of data and a pretty clear idea about what casual effects you care about.". Causal inference where possible, causal interpretation where necessary, ought to be the guiding principle. Via the Social Science Statistics blog, there is also a very interesting paper by Angus Deaton on the problems of causal inference. Of particular note is the difficulty of testing the assumptions behind instrumental variables methods, and the often-elided distinction between an instrument that is external to the process under investigation (that is, not caused by the system being studied) and one that is truly exogenous (that is, uncorrelated with the error term in the regression of the outcome on the predictor of interest.)

I identify with Jeremy Freese to an unhealthy degree. When the other options are to a) have a life; or b) do something that advances his career, he chooses to concoct a home-brewed match between GSS respondents in 2006 and their 2008 re-interviews. I would totally do this. I still might do this.

And then he drops the brutal insight that provides my title. Context.

UPDATE: And then Kieran Healy drops this:

The real distinction between qualitative and quantitative is not widely appreciated. People think it has something to do with counting versus not counting, but this is a mistake. If the interpretive work necessary to make sense of things is immediately obvious to everyone, it’s qualitative data. If the interpretative work you need to do is immediately obvious only to experts, it’s quantitative data.

Have you seen this man?

[Cross-posted from The Activist]

The talk of blog-land this week is Ryan Lizza's long New Yorker profile of Obama administration economist-guru Larry Summers. It's an interesting and useful piece, although I think it's far too forgiving of Summers and the Obama administration for all the reasons Dean Baker provides here. I would only add that it's shameful to write a discussion of Summers' years at Harvard without mentioning the Andrei Shleifer affair, an episode of raw gangsterism in which Summers connived to protect a man who had enriched himself by immiserating the Russian people and wrecking their country.

However, what I want to focus on in the Lizza piece is the curious case of one Jared Bernstein. Formerly an economist at the very progressive Economic Policy Institute, Bernstein is now the chief economic adviser to Vice President Biden. More importantly, he is the only high-ranking economist in the Obama White House whom one could plausibly describe as "on the left." And in the photo that accompanies the Lizza profile, he appears alongside Summers, Tim Geithner, Christina Romer and Peter Orszag, a group that is described in a caption as "the advisers who meet with the President daily to discuss the economy".

In the actual article, however, he only appears three times in passing. The first is in December 2008, when the Obama economic team meets in Chicago to discuss their response to the economic crisis. Although we are given an extensive description of the contributions from Romer, Orszag and others in this conversation, we do not hear what, if anything, Bernstein had to say. More significantly, Lizza reports that at the end of the meeting, "Summers, Romer, Geithner, Orszag, Emanuel, and Jason Furman huddled in the corner to lock down" the amount of the stimulus package. Bernstein, apparently, was politely dismissed before this little summit.

Bernstein reappears in a discussion of Summers' alleged conversion from gung-ho deregulator to chastened Keynesian, averring that "I was reading Larry’s articles in the Financial Times over the past couple of years, and thought, Wow, it’s all too rare that you see the thinking of such a prominent economist move like that." But here he's being brought in as an outside commentator whose job is to burnish Summers' lefty cred (and give Lizza an excuse to repeat the often-told fact that Bernstein used to be some kind of hippie musician). The fact that he works in the White House is essentially irrelevant. His only other appearance in the piece is at the end, where Lizza reports that there is "a half-hour meeting each morning, in which Obama is briefed by the top members of his economic team: Summers, Geithner, Romer, Orszag, and Bernstein." From Obama's policies and Lizza's reporting, it's not hard to conclude that one of these five is pulling significantly less weight with the President than the other four.

I bring up all of this not just because I like Jared Bernstein and wish he had more influence than Larry Summers, but to raise a larger point about the relationship between the Obama administration and the left. Thus far, it seems to me that in every case where a solid progressive has entered the administration, they would have been better off staying outside. Bernstein, I suspect, would be more effective if he could do what Dean Baker and Robert Reich are doing: criticize the administration's economic policy from the left. Instead, he's in a position where he can't honestly speak his mind because of his official post, and yet he still seems to have minimal impact on policy-making. A similar case, I think, can be made about Van Jones, whose Glenn Beck-inspired defenestration from the White House may turn out to be a blessing if it frees him up to be a consistent and outspoken progressive about environmental justice issues. And the list goes on: Hilda Solis, for example, seems to have vanished without a trace.

There's long been a debate among leftists in this country about the relative merits of "inside" and "outside" strategies for moving the state and the Democratic party to the left. I believe, as do many in DSA, that a successful strategy has to incorporate both. But the case of Jared Bernstein is, I think, a cautionary tale: being "inside" a Democratic administration means nothing if you don't have a voice there. In that case, it amounts to nothing more than a co-opting of progressive voices by a fundamentally conservative political project.