A Note on the Uses of Official Statistics

| Gabriel |

They are ourselves, I replied; and they see only the shadows of the images which the fire throws on the wall of the den; to these they give names, and if we add an echo which returns from the wall, the voices of the passengers will seem to proceed from the shadows.  — Plato

One of the points I like to stress to my grad students is that data is not an objective (or even unbiased) representation of reality but the result of a social process. The WSJ had a story recently on how we get the “jobs created or saved” figures around the stimulus bill and it makes me want to burn my Stata dvd, take a two-hour shower, and then switch to qualitative methods where at least I know that I would be responsible for any validity problems in my work.

The idea of “jobs created or saved” by a government policy is a meaningful concept in principle but in practice it’s essentially impossible to reckon with any certainty. It’s the kind of problem you might be able to approach empirically if it happened many times and there was some relatively exogenous instrument, but in a single instance you’re probably better off using an answer derived from theory than actually trying to measure it. Nonetheless the political process demands that it be answered empirically and the results are absurd.

The way the government has tried to measure “jobs created or saved” by the stimulus is by simply asking contractors or subcontractors how many jobs were created or saved in their firm by the contract. This involves both false positives of contractors exaggerating the number of jobs they created or saved and false negatives of firms that were not direct beneficiaries of contracts but increased or retained production in expectations of benefitting from the multiplier. In the case covered by the WSJ, a shoe store that sold nine pairs of boots for $100 each to the Army Corps of Engineers didn’t know what else to put and so said they saved nine jobs. When asked about this by the WSJ the shoe store owner’s daughter/bookkeeper replied

“The question, I would like to know is: How do you answer that? Did we create zero? Is it creating a job because they have boots and go out and work for the Corps? I would be really curious to hear how somebody does create a job. The formula is out there for anyone to create, and it’s just so difficult,” she said.

Who’d a thunk it, but apparently FA Hayek was reincarnated as a shoe store worker in Kentucky.

(h/t McArdle)

Fool’s Gold and Organizational Inertia

| Pierre |

Daniel Beunza at Socializing Finance links to Donald MacKenzie’s LRB review of Fool’s Gold, Gillian Tett’s new book on the financial crisis. I just finished reading the book, and I can only recommend it. Tett is an editor at the Financial Times; she also has a PhD in anthropology from Cambridge, which probably explains why the book somehow reads more like an econ-soc analysis of the crisis than as a journalistic account. In his review piece, MacKenzie gives a clear and detailed overview of the book’s main argument — and as such, his review is one of the best and most accessible accounts of the recent developments in structured finance I’ve read so far. But there’s a point that bothered me in the book, and which MacKenzie doesn’t seem to touch on:

Tett tells us about the crisis mostly from the standpoint of bankers and credit derivative specialists at J.P. Morgan — a bank which, by and large, stayed out the mortgage-backed securities mess and emerged relatively unscathed from the crisis. There’s nothing suprising about this: it is probably easier to find people willing to be interviewed on the crisis at J.P. Morgan nowadays than at, say, Citigroup or AIG. But this angle is precisely what makes the story so fascinating. The book starts from a simple but puzzling observation: the credit instruments at the center of the crisis (those pools of loans that were sliced into multiple tranches with distinct risk levels, which were in turn sold with overly optimistic ratings, aka collateralized debt obligations or CDOs) originated not from the home mortgage market but from the corporate bond market. Arguably, the crisis was largely caused by the belief that the same structured products could easily be used to transfer home mortgage risk away from banks’ balance sheets—although estimating the risk of CDO tranches turned out to be much more complex for mortgages than for corporate debt (in particular, there wasn’t any reliable data allowing to estimate the correlation of default probabilities). But surprisingly, the pioneer and leader in corporate debt CDOs, J.P. Morgan, decided not to further their advantage in structured finance: instead of moving into the mortgage-backed securities market and applying the same recipes they had just developed for corporate debt on a much wider scale, J.P. Morgan largely stayed out the market. Incentives were there: J.P. Morgan had expertise in such structured products; investment banks could collect enormous fees for underwriting CDOs and the market was booming; but JP Morgan executives, Tett observes, stayed on the sidelines and were even puzzled by the development of the market. So, what happened?

Tett’s account is essentially an organizational story. She argues that the prevailing culture at J.P. Morgan (a rather old-fashioned, boring and elitist institution) favored more prudent risk management strategies and more effective oversight than at other banks. This is an interesting hypothesis, but it may be, in part, the product of Tett’s methodology: the evidence supporting this argument comes mostly from interviews with J.P. Morgan executives — which are of course subject to response bias, recall bias and so forth. MacKenzie seems to generally agree with Tett’s thesis: what made J.P. Morgan different from other banks was the foresight of its management. Maybe reading too much organizational sociology has made me more cynical than I should be, but there’s an alternative explanation that Tett never fully explores or dismisses: organizational inertia. At the begining of the housing bubble, J.P. Morgan was specialized in corporate debt, and had no experience with home mortgages unlike BoA or Citi (J.P. Morgan was later absorbed by Chase Manhattan — but even then, only Chase got involved in mortgage CDOs). I do not doubt that organizational culture played an important role, but I did not see much evidence in the book that J.P. Morgan’s corporate culture (coupled with its expertise in structured finance) made its management more aware of the fragility of the mortgage CDOs than its competitors — if this were truly the case, one would have expected the bank to bet against the market by massively shorting CDO indexes, as a few hedge funds did. The alternative hypothesis, of course, is that organizational culture contributes to organizational inertia — and while it did not necessarily make J.P. Morgan’s executives more prudent or more aware of the risks inherent in the mortgage market, it may have prevented the bank from taking positions (long or short) in a segment of the industry it did not belong to.