| Gabriel |
There has been a tremendous amount of hype over the last few years about universal pre-K as a magic bullet to solve all social problems. We see a lot of talk of return on investment at rates usually only promised by prosperity gospel preachers and Ponzi schemes. Unfortunately, two recent large-scale studies, one in Quebec and one in Tennessee, showed small negative effects for pre-K. An article writing up the Tennessee study in New York advises fear not, for:
These are all good studies, and they raise important questions. But none of them is an indictment of preschool, exactly, so much as an indictment of particular approaches to it. How do we know that? Two landmark studies, first published in 1993 and 2008, demonstrate definitively that, if done right, state-sponsored pre-K can have profound, lasting, and positive effects — on individuals and on a community.
It then goes on to explain that the Perry and Abecedarian projects were studies involving 123 and 100 people respectively, had marvelous outcomes, and were play rather than drill oriented.
The phrase “demonstrate definitively” is the kind of phrase you have to very careful with and it just looks silly to say that this definitive knowledge comes from two studies with sample size of about a hundred. Tiny studies with absurdly large effects sizes are exactly where you would expect to find publication bias. Indeed, this is almost inevitable when the sample sizes are so underpowered that the only way to get β/se>1.96 is for β to be implausibly large. (As Jeremy Freese observed, this is among the dozen or so major problems with the PNAS himmicane study).
The standard way to detect publication bias is through a meta-analysis showing that small studies have big effects and big studies have small effects. For instance, this is what Card and Krueger showed in a meta-analysis of the minimum wage literature which demonstrated that their previous paper on PA/NJ was only an outlier when you didn’t account for publication bias. Similarly, in a 2013 JEP, Duncan and Magnuson do a meta-analysis of the pre-K literature. Their visualization in figure 2 emphasizes the declining effects sizes over time, but you can also see that the large studies (shown as large circles) generally have much smaller β than the small studies (shown as small circles). If we added the Tennessee and Quebec studies to this plot they would be large circles on the right slightly below the x-axis. That is to say, they would fall right on the regression line and might even pull it down further.
This is what publication bias looks like: old small studies have big effects and new large studies have small effects.
I suppose it’s possible that the reason Perry and Abecedarian showed big results is because the programs were better implemented than those in the newer studies, but this is not “demonstrated definitively” and given the strong evidence that it’s all publication bias, let’s tentatively assume that if something’s too good to be true (such as that a few hours a week can almost deterministically make kids stay in school, earn a solid living, and stay out of jail), then it ain’t.
| Gabriel |
Today the Economist posted a graph showing the patrons of factions in various civil wars in the Middle East. The point of the graph is that the alliances don’t neatly follow balance theory, since it is in fact sometimes the case that the friend of my enemy is my friend, which is a classic balance theory fail. As such, I thought it would be fun to run a Spinglass model on the graph. Note that I could only do edges, not arcs, so I only included positive ties, not hostility ties. One implication of this is ISIS drops out as it (currently) lacks state patronage.
Here’s the output. The second column is community and the third is betweenness.
> s Graph community structure calculated with the spinglass algorithm Number of communities: 4 Modularity: 0.4936224 Membership vector:  4 4 3 2 2 2 4 3 4 3 1 4 1 3 3 4 2 4 2 > output b [1,] "bahrain_etc" "4" "0" [2,] "egypt_gov" "4" "9.16666666666667" [3,] "egypt_mb" "3" "1.06666666666667" [4,] "iran" "2" "47.5" [5,] "iraq_gov" "2" "26" [6,] "iraq_kurd" "2" "26" [7,] "jordan" "4" "6.73333333333333" [8,] "libya_dawn" "3" "1.06666666666667" [9,] "libya_dignity" "4" "0.333333333333333" [10,] "qatar" "3" "27.5333333333333" [11,] "russia" "1" "0" [12,] "saudi" "4" "4" [13,] "syria_gov" "1" "17" [14,] "syria_misc" "3" "31.0333333333333" [15,] "turkey" "3" "6.83333333333333" [16,] "uae" "4" "4" [17,] "usa" "2" "74.4" [18,] "yemen_gov" "4" "74.3333333333333" [19,] "yemen_houthi" "2" "0"
So it looks like we’re in community 2, which is basically Iran and its clients, though in fairness we also have high betweenness as we connect community 2 (Greater Iran), community 3 (the pro Muslim Brotherhood Sunni states), and community 4 (the pro Egyptian government Sunni states). This is consistent with the “offshore balancing” model of Obama era MENA policy.
Here’s the code:
library("igraph") setwd('~/Documents/codeandculture') mena <- read.graph('mena.net',format="pajek") la = layout.fruchterman.reingold(mena) V(mena)$label <- V(mena)$id #attaches labels plot.igraph(mena, layout=la, vertex.size=1, vertex.label.cex=0.5, vertex.label.color="darkred", vertex.label.font=2, vertex.color="white", vertex.frame.color="NA", edge.color="gray70", edge.arrow.size=0.5, margin=0) s <- spinglass.community(mena) b <- betweenness(mena, directed=FALSE) output <- cbind(V(mena)$id,s$membership,b) s output
And here’s the data:
*Vertices 19 1 "bahrain_etc" 2 "egypt_gov" 3 "egypt_mb" 4 "iran" 5 "iraq_gov" 6 "iraq_kurd" 7 "jordan" 8 "libya_dawn" 9 "libya_dignity" 10 "qatar" 11 "russia" 12 "saudi" 13 "syria_gov" 14 "syria_misc" 15 "turkey" 16 "uae" 17 "usa" 18 "yemen_gov" 19 "yemen_houthi" *Arcs 1 18 2 9 2 18 4 5 4 6 4 13 4 19 7 2 7 14 7 18 10 3 10 8 10 14 10 18 11 13 12 2 12 9 12 18 15 3 15 8 15 14 16 2 16 9 16 18 17 5 17 6 17 14 17 18
| Gabriel |
The transformations of the television industry are an endlessly fascinating subject that I spend a lot of time ruminating on but haven’t ever, you know, actually published on. We can start with a few basic technological shifts, specifically the DVR and broadband internet. Both technologies have the effect that people are watching fewer commercials. From this we can infer that advertisers will have a pronounced preference for “DVR-proof” advertising.* One form of this is product shots, which are indeed a big deal nowadays, especially in the reality competition genre. Of course product shots are inherently cumbersome and are pretty much the antithesis of the scatter advertising market insofar as they require commitments during pre-production which is even more extreme than up-fronts and which is why we long ago got past the age of Texaco Star Theatre. So basically, the 30 second spot you will always have with you. Or rather, the demand for the 30 second spot you will always have with you and the question is can we find a type of programming where people watch the ads. (Note that the recent Laureate Jean Tirole did work on this issue, as explained by Alex Tabarrok at MR).
In practice getting people to watch spot advertising means programming that has to be watched live and in practice that in turn means sports.** Thus it is entirely predictable that advertisers will pay a premium for sports. It is also predictable that the cable industry will pay a premium for sports because must-watch ephemera is a good insurance policy against cord-cutting. Moreover, as a straight-forward Ricardian rent type issue, we would predict that this increased demand would accrue to the owners of factor inputs: athletes, team owners, and (in the short-run) the owners of cable channels with contracts to carry sports content. Indeed this has basically all happened. You’ve got ESPN being the cash cow of Disney, ESPN and TNT in turn signing a $24 billion deal with the NBA, an NBA team selling for $2 billion, and Kobe Bryant making $30 million in salary. Basically, there’s a ton of money in DVR-proof sports, both from advertising and from the ever-rising carriage fees that get passed on in the form of ever rising basic cable rates. (I imagine a Johnny Cash parody, “how high’s the carriage fees mama? 6 bucks per sub and rising.”).
Here’s something else that is entirely predictable from these premises: we should have declining viewership for sports. Think about it, you have widget A and widget B. Widget A has a user experience that’s the same as it’s always been (ie, you got to watch it when it’s on and sit through the ads) but the price is rapidly increasing (it used to be you could get it over broadcast or just from a basic cable package that was relatively cheap). In contrast you have widget B which has a dramatically improved user experience (you can watch every episode ever on-demand whenever you feel like it without ads and do so on your tv, tablet, or whatever) and a rapidly declining price (if you’re willing to wait for the previous season, scripted content is practically free). If you’re the marginal viewer who ex ante finds sports and scripted equally compelling, it seems like as sports get more expensive and you keep having to watch ads, whereas scripted gets dirt cheap, ad-free, and generally more convenient, the marginal viewer would give up sports, watch last season’s episodes of Breaking Bad on Netflix, be blissfully unaware of major advertising campaigns, and pocket the $50 difference between a basic cable package and a $10 Netflix subscription. Of course you wouldn’t predict that the kinds of guys who put body paint on their naked torsos would give up on sports just because Netflix has every season of Frasier, but you would predict that at the population level interest in sports would decline slightly to moderately.
The weird thing is that this latter prediction didn’t happen. During exactly the same period over which sports got more expensive in absolute terms and there was declining direct cost and hassle for close substitutes, viewership for sports increased. From 2003 to 2013, sports viewership was up 27%. Or rather, baseball isn’t doing so great and basketball is holding its own, but holy moly, people love football. If you look at both the top events and top series on tv, it’s basically football, football, some other crap, and more football. (Also note that football doesn’t appear in the “time-shifted” lists, meaning that people do watch the ads). And it’s not just that people have always liked football or that non-football content is weakening, but football is growing in absolute popularity.
That this would happen in an era of DVRs and streaming is nuts, and kind of goes contrary to the whole notion of substitutes. I mean, I just can’t understand how when one thing gets more expensive and something else that’s similar gets a lot cheaper and lower hassle, that you see people flocking to the thing that is more money in absolute terms and more hassle in relative terms.*** Maybe we just need to keep heightening the contradictions and then eventually the system will unravel, but this doesn’t explain why we’ve seen a medium-run fairly substantial rise in sports viewership instead of just stability with a bit of noise.
I’m sure one of my commenters is smarter than me and can explain why either my premises or logic is incorrect, but at least to me this looks like an anomaly. And even if we can ultimately find some auxiliary hypothesis that explains why of course we’d predict a rise in sports viewership if we only considered that [your brilliant ex post explanation goes here],**** let’s keep in mind that this is all ex post, and adjust down our confidence about making social scientific predictive inferences accordingly. A theory like decline in total cost of widget B will lead to substitution of widget B for widget A is a pretty good theory and if its predictions don’t hold in the face of something like bigger linebackers or more exciting editing for instant replay, then you have to wonder how much any theory can get us.
*If we’re a bit more creative we could also infer that the market information regime for audience ratings will see a lot of contentious changes.
**It is interesting that the tv networks aggressively promote Twitter in order to promote live viewing of scripted content and news, but at this point the idea that networks will hashtag their way to a higher “C3” ratings is pretty niche/speculative.
*** The closest parallel I can think of is that it’s the easy-going mainline Protestant churches that have seen especially steep declines in attendance/membership and the more personally demanding churches that are relatively strong. I may have to rethink this point though after I fully digest the new Hout & Fischer.
**** Your ex post explanation better speak to the (extensive) marginal fan and not just the intensity of hardcore fans, since my understanding is total number of football viewers is up, and so the explanation can’t be anything like the growth in fantasy leagues leads hardcore fans to watch 20 hours a week instead of 3 hours a week.
| Gabriel |
Like most native speakers of Stata, the most natural thing in the world is to start every script or session with a log file. Typing “log using” is like brushing your teeth, it’s the first thing you do and you just feel gross if you haven’t done it. Judging by what you get if you Google “logging in R” this seems to be something of a cultural eccentricity peculiar to Stata users as R users seems not to understand the question. In particular most responses to the question say something like “use sink(),” which ignores that to a Stata user a log file is neither the command history nor the output, but the two interpolated together so that you can see what command elicited what output.
However, much as the frustrated tourist abroad will occasionally find someone who understands what they mean in asking for a Western toilet, one great StackOverflow user speaks sufficient Stata to direct us to what we were hoping to find. Specifically, the library “TeachingDemos” includes a “txtStart()” function that behaves almost exactly like a Stata log file by default, but where you also have various options such as to suppress commands/output or to use Markdown format.
To install TeachingDemos:
Thereafter invoke it start a log file, do your work, and close it:
library(TeachingDemos) txtStart('mylogfile.txt') # this is similar to "log using mylogfile.txt" in Stata #insert code here to load your data, analyze it, etc txtStop() # this is similar to "log close" in Stata
| Gabriel |
[cross-posted from TAS]
The Supreme Court recently ruled in favor of Hobby Lobby but among the many things that are not widely understood is that the decision did not actually result in the firm’s employees losing insurance coverage for IUDs. The actual result is that the employees will still have coverage for IUDs, but the insurance processor rather than Hobby Lobby will have to pay for it (at least in theory). That is, Hobby Lobby was seeking to take advantage of the Obama administration’s own proposal for faith-based nonprofits. As Julian Sanchez at Cato observed, the entire case turns on an entirely symbolic issue of whether the Greens explicitly have to pay for IUDs or are allowed to wink at an obfuscation in which their insurance company bears the cost (at least theoretically).
I found this interesting not only because it’s a much discussed case but also because it’s a close fit with my article published a few months ago in Sociological Theory. (Here’s an ungated version that lacks the benefit of some really good copy-editing). In the article I talk about situations where a moral objection gets in the way of a transaction but the transaction nonetheless occurs through the expedient of obfuscating that a transaction is occurring at all. I describe three mechanisms for accomplishing this and the nonprofit exemption which now also applies to Hobby Lobby is characterized by two of them, brokerage and bundling. That is, the employer does not buy theIUD for the employee but rather pays a broker (the insurance processor) who in turn provides the IUD. Moreover, the IUD is bundled together with other health coverage. The third model which is not at issue in Hobby Lobby but which I describe in the paper is gift exchange, where explicit quid pro quo is replaced with tacit reciprocity.
Of course for an exchange to be morally objectionable or for it to be koshered is entirely subjective. Most obviously in Hobby Lobby there is a range of opinions about the moral acceptability of birth control and abortifacients and where to draw the line between the two. More interesting to me is that opinions vary on what counts as “buying” the contested commodity and whether to seize on obfuscation and denounce it. On this issue the irony is that while the Obama administration itself came up with this obfuscation for nonprofits it opposed extending it to for-profit firms. At a general level, obfuscation doesn’t objectively exist but rather it creates a permission structure that actors can choose to consent to.
This becomes clear when we contrast Hobby Lobby to Little Sisters of the Poor. Whereas the owners of Hobby Lobby sued to avail themselves of the obfuscatory accomodation, the Little Sisters of the Poor who (as a nonprofit) already have this obfuscation available to them but are suing to denounce it as mere obfuscation and completely remove themselves from even obfuscated provision of all birth control. Specifically, the Little Sisters are refusing to fill out EBSA Form 700 stating their objection to providing contraceptive coverage since to do so would trigger provision through their insurer and they see this as involving themselves in something morally objectionable. That is, while Hobby Lobby would be delighted to wink and nod (and the Obama administration was reluctant to allow them to do so) the Little Sisters are adamantly opposed to a fig leaf (and the Obama administration would be delighted were they to play along with the face-saving obfuscation).
| Gabriel |
I’m as excited as anybody about Sociological Science as it promises a clean break from the “developmental” model of peer review by moving towards an entirely evaluative model. That is, no more anonymous co-authors making your paper worse with a bunch of non sequiturs or footnotes with hedging disclaimers. (The journal will feature frequent comment and replies, which makes debate about the paper a public dialog rather than a secret hostage negotiation). The thing is though that optimistic as I am about the new journal, I don’t think it will replace the incumbent journals overnight and so we still need to fix review at the incumbent journals.
So how did peer review get broken at the incumbent journals?
You and I broke it.
Your average academic’s attitude towards changes demanded in R&R is like the Goofy cartoon “Motor Mania.”
In this cartoon Goofy is a meek pedestrian dodging aggressive drivers but as soon as he gets behind the wheel himself he drives like a total asshole. Similarly, as authors we all feel harassed by peer reviewers who try to turn our paper into the paper they would have written, but then as reviewers ourselves we develop an attitude of “yer doin it wrong!!!” and start demanding they cite all our favorite articles and with our favorite interpretations of those articles. (Note that in the linked post, Chen is absolutely convinced that she understands citations correctly and the author has gotten them wrong out of carelessness, without even considering the possibility that the interpretive flaw could be on her end or that there might be a reasonable difference of opinions).
So fixing peer review doesn’t begin with you, the author, yelling at your computer “FFS reviewer #10, maybe that’s how you would have done it, but it’s not your paper” (and then having a meeting with your co-authors that goes something like this):
And then spending the next few months doing revisions that feel like this:
And finally summarizing the changes in a response memo that sounds like this:
Nor, realistically, can fixing peer review happen from the editors telling you to go ahead and ignore comments 2, 5, and 6 of reviewer #6. First, it would be an absurd amount of work for the editors to adjudicate the quality of comments. Second, from the editor’s perspective the chief practical problem is recruiting reviewers and getting timely reviews from them and so they don’t want to alienate the reviewers by telling them that half their advice sucks in their cover letter any more than you want to do that in your response memo.
Rather, fixing peer review has to begin with you, the reviewer, telling yourself “maybe I would have done it another way myself, but it’s not my paper.” You need to adopt a mentality of “is it good how the author did it” rather than “how could this paper be made better” (read: how would I have done it). That is the whole of being a good reviewer, the rest is commentary. That said, here’s the commentary.
Do not brainstorm
Responding to a research question by brainstorming possibly relevant citations or methods is a wonderful and generous thing to do when a colleague or student mentions a new research project but it’s a thoroughly shitty thing to do as a peer reviewer. There are a few reasons why the same behavior is so different in two different contexts.
First, many brainstormed ideas are bad. When I give you advice in my office, you can just quietly ignore the ideas I give you that don’t work or are superfluous. When I give you advice as a peer reviewer there is a strong presumption that you take the advice even if it’s mediocre which is why almost every published paper has a couple of footnotes along the lines of “for purposes of this paper we assume that water is wet” or “although it has almost nothing to do with this paper, it’s worth noting that Author (YEAR) is pretty awesome.” Of course some suggestions are so terrible that the author can’t take them in good conscience but in such cases the author needs to spend hours or days per suggestion writing an apologetic and extremely deferential memo apologizing for not implementing the reviewer’s suggestions.
Second, many brainstormed ideas are confusing. When I give you advice in my office you can ask follow-up questions about how to interpret and implement it. When I give advice as a peer reviewer it’s up to you to hope that you read the entrails in a way that correctly augurs the will of the peer reviewers. As a related point, be as specific as possible. “This paper needs more Bourdieu” is a not terribly useful comment (indeed, “cite this” comments without further justification are usually less about any kind of intellectual content than they are about demanding shibboleths or the recitation of a creedal confession) whereas it might actually be pretty helpful to say “your argument about the role of critics on pages 4-5 should probably be described in terms of restricted field from Bourdieu’s Field of Cultural Production.” (Being specific has the ancillary benefit that it’s costly to the reviewer which should help you maintain the discipline to thin the mindfart herd stampeding into the authors’ revisions.)
Third, ideas are more valuable at the beginning of a project than at the end of it. When I give you advice about your new project you can use it to shape the way the project develops organically. When I give it to you as a reviewer you can only graft it on after the fact. My suggested specification may check the robustness of your finding or my suggested citation may help you frame your theory in a way that is more appealing, but they can’t help you develop your ideas because that ship has sailed.
That’s not to say that you shouldn’t give an author advice on how to fix problems with the paper. However it is essential to keep in mind that no matter how highly you think of your own expertise and opinions, you remember that the author doesn’t want to hear it. When you give advice, think in terms of “is it so important that these changes be made that I upset the author and possibly delay publication at a crucial career point.” Imagine burning a $20 bill for every demand you make of the author and ask yourself if you’d still make it. Trust me, the author would pay a lot more than $20 to avoid it — and not just because dealing with comments is annoying but because it’s time-consuming and time is money. It usually takes me an amount of time that is at least the equivalent of a course release to turn-around an R&R and at most schools a course release in turn is worth about $10,000 to $30,000 if you’re lucky enough to raise the grants to buy them. If you think about the productivity of science as a sector then ask yourself if your “I’m just thinking out loud” comment that takes the author a week to respond to is worth a couple thousand dollars to society. I mean, I’ve got tenure so in a sense I don’t care but I do feel a moral obligation to give society a good value in exchange for the upper middle class living it provides me and I don’t feel like I’m getting society its money’s worth when I spend four months of full-time work to turn around one round of R&R instead of getting to my next paper. This brings me to my next point…
Distinguish demands versus suggestions versus synapses that happened to fire as you were reading the paper
A lot of review comments ultimately boil down to some variation on “this reminds me of this citation” or “this research agenda could go in this direction.” OK, great. Now ask yourself, is it a problem that this paper does not yet do these things or are these just possibilities you want to share with the author? Often as not they’re really just things you want to share with the author but the paper is fine without them. If so, don’t demand that the author do them. Rather just keep it to yourself or clearly demarcate these as optional suggestions that the author may want to consider, possibly for the next paper rather than the revision of this paper.
As a related issue, demonstrate some rhetorical humility. Taking a commanding and indignant tone doesn’t mean you know what you’re talking about. On a recent review I observed, I noticed that one of the reviewers whose (fairly demanding) comments seemed to reflect a deep understanding of the paper nonetheless used a lot of phrases like “might,” “consider,” “could help,” etc whereas another reviewer who completely missed the point of the paper was prone to phrases like “needs to” and “is missing.”
There’s wrong and then there’s difference of opinion
On quite a few methodological and theoretical issues there is a reasonable range of opinion. Don’t force the author to weigh in on your side. It may very well be appropriate to suggest that the author acknowledge the existence of a debate on this subject (and perhaps briefly explore the implications of the alternative view) but that’s a different thing from expecting that the author completely switch allegiances because error has no rights. Often such demands are tacit rather than explicit, just taking for granted that somebody should use, I don’t know, Luhmann, without considering that the author might be among the many people who if told “you can cite Luhmann or you can take a beating” would ask you “tell me more about this beating? will there be clubs involved?”
For instance, consider Petev ASR 2013. The article relies heavily on McPherson et al ASR 2006, which is an extremely controversial article (see here, here, and here). One reaction to this would be to say the McPherson et al paper is refuted and ought not be cited. However Petev summarizes the controversy in footnote 10 and then in footnote 17 explains why his own data is a semi-independent (same dataset, different variables) corroboration of McPherson et al. These footnotes acknowledge a nontrivial debate about one of the article’s literature antecedents and then situates the paper within the debate. No matter what your opinion of McPherson et al 2006, you should be fine with Petev relying upon and supporting it while parenthetically acknowledging the debate about it.
There are also issues of essentially theoretical nature. I sat on one of my R&R for years in large part because I’m using a theory in its original version while briefly discussing how it would be different if we were to use a schism of the theory while one of the reviewers insists that I rewrite it from the perspective of the schismatic view. Theoretical debates are rarely an issue of decisive refutation or strictly cumulative knowledge but rather at any given time there’s a reasonable range of opinions and you shouldn’t demand that the author go with your view but at most that they explore its implications if they were to. Most quants will suggest robustness checks to alternative plausible model specifications without demanding that these alternative models are used in the actual paper’s tables, we should have a similar attitude towards treating robustness or scope conditions to alternative conceptions of theory as something for the footnotes rather than a root and branch reconceptualization of the paper.
There are cases where you fall on one side of a theoretical or methodological gulf and the author on another to the extent that you feel that you can’t really be fair. For instance, I can sometimes read the bibliography of a paper, see certain cites, and know instantly that I’m going to hate the paper. Under such circumstances you as the reviewer have to decide if you’re going to engage in what philosophers of science call “the demarcation problem” and sociologists of science call “boundary work” or you’re going to recuse yourself from the review. If you don’t like something but it has an active research program of non-crackpots then you should probably refuse to do the review rather than agreeing and inevitably rejecting. Note that the managing editor will almost always try to convince you to do the review anyway and I’ve never been sure if this is them thinking I’m giving excuses for being lazy and not being willing to let me off the hook, them being lazy about finding a more appropriate reviewer, or an ill-conceived principle that a good paper should be pleasing to all members of the discipline and thus please even a self-disclaimed hostile reader. Notwithstanding the managing editor’s entreaties, be firm about telling him or her, “no, I don’t feel I could be fair to a paper of type X, but please send me manuscripts of type Y or Z in the future.”
Don’t try to turn the author’s theory section into a lit review.
The author’s theory section should motivate the hypotheses. The theory section is not about demonstrating basic competence or reciting a creedal confession and so it does not need to discuss every book or article ever published on the subject or even just the things important enough to appear on your graduate syllabus or field exam reading list. If “AUTHOR (YEAR)” would not change the way we understand the submission’s hypotheses, then there’s no good reason the author needs to cite it. Yes, that is true even if the “omitted” citation is the most recent thing published on the subject or was written by your favorite grad student who you’re so so proud of and really it’s a shame that her important contribution isn’t cited more widely. If the submission reminds you of a citation that’s relevant to the author’s subject matter, think about whether it would materially affect the argument. If it would, explain how it would affect the argument. If it wouldn’t, then either don’t mention it at all or frame it as an optional suggestion rather than berating the author for being so semi-literate as to allow such a conspicuous literature lacuna.
By materially affect the argument I mostly have in mind the idea that in light of this citation the author would do the analysis or interpret the analysis differently. This is not the same thing as saying “you do three hypotheses, this suggests a fourth.” Rather it’s about this literature shows that doing it that way is ill-conceived and you’re better off doing it this way. It’s simplest if you think about in terms of methods where we can imagine a previous cite demonstrates how important it is for this phenomena that one models censorship, specifies a particular form for the dependent variable, or whatever. Be humble in this sort of thing though lest it turn into brainstorming.
Another form of materially affecting the argument would be if the paper is explicitly pitched as novel but it is in fact discussing a well understood problem. It is not necessarily a problem if the article discusses an issue in terms of literature X but does not also review literature Y that is potentially related. However it is a problem if the author says nobody has ever studied issue A in fashion B when there is in fact a large literature from subfield Y that closely parallels what the author is pitching. More broadly, you should call the authors on setting up straw man lit review, where one special case of that would be “there is no literature.” (Note to authors: be very careful with “this is unprecedented” claims). Again, be humble in how you apply this lest it turn into a pretext for demanding that every article not only motivate its positive contribution, but also be prefaced with an exhaustive review that would be suitable for publication in ARS.
There is one major exception to the rule that a paper should have a theory section and not a lit review, which is when the authors are importing a literature that is likely to be unfamiliar to their audience and so they need more information than usual to get up to speed. Note though that this is an issue best addressed by the reviewers who are unfamiliar with the literature and for whom it is entirely appropriate to say something like “I was previously unfamiliar with quantum toad neuropathology and I suspect other readers will be as well so I ask that rather than assuming a working knowledge of this literature that the author please add a bit more background information to situate the article and point to a good review piece or textbook for those who want even more background.” Of course that’s rarely how the “do more lit review” comments go. Rather such comments tend to be from people with a robust knowledge of theory X and they want to ensure that the authors share that knowledge and gavage it into the paper’s front end. I’m speaking from personal experience as on several occasions I have used theories that are exotic to sociologists and while several of the reviewers said they were glad to learn about this new-to-them theory and how it fits with more mainstream sociology like peanut butter and chocolate, nobody asked for more background on it. And I’m cool with that since it means my original drafts provided sufficient background info for them to get the gist of the exotic theory and how it was relevant. Of course, I did get lots of “you talk about Bourdieu, but only for ten pages when you could easily go for twenty.” That is, nobody wants to know more about something they didn’t know before and need a little more background knowledge to get up to speed, but everybody wants to yell “play Freebird!” This is exactly backwards of how it should be.
Don’t let flattery give you a big head
It is customary for authors to express their gratitude to the reviewers. You might take from this to think, “ahhh, Gabriel’s wrong about R&Rs being broken,” or more likely “that may be true of other reviewers, but I provide good advice since, after all, they thank me for it.” Taking at face value an author who gushes about what a wonderful backseat driver you are is like watching a prisoner of war saying “I would like to thank my captors for providing me with adequate food and humane treatment even as my country engages in unprovoked imperialist aggression against this oppressed people.” Meanwhile he’s blinking “G-E-T-M-E-O-U-T-O-F-H-E-R-E” in Morse code.
Appreciate the constraints imposed on the author by the journal:
Many journals impose a tight word count. When you ask an author to also discuss this or that, you’re making it very difficult for them to keep their word count. One of the most frustrating things as an author is getting a cover letter from the editor saying “Revise the manuscript to include a half dozen elaborate digressions demanded by the reviewers, but don’t break your word count.”
Some journals demand that authors include certain material and you need to respect that. ASR is obsessed with articles speaking to multiple areas of the discipline. This necessarily means that an article that tries to meet this mandate won’t be exclusively oriented towards your own subfield and it may very well be that its primary focus is on another literature and its interest in your own literature being secondary. Don’t view this as an incompetent member of your own subfield but as a member of another subfield trying (under duress) to build bridges to your subfield. Similarly some journals demand implications for social policy or for managers. Even if you would prefer value neutrality (or value-laden but with a different set of values) or think it’s ridiculous to talk as if firms will change their business practices because somebody did a t-test, appreciate that this may be a house rule of the journal and the author is doing the best she can to meet it.
Stand up to the editors:
You can be the good guy. Or if necessary, you can demand a coup de grace. But either way you can use your role as a reviewer to push the editors and your fellow reviewers towards giving the authors a more streamlined review process.
First, you can respond to the other parts of the reviews and response memo from the previous round. If you think the criticisms were unfair or that the author responded to them effectively, go ahead and say so. It makes a big difference to the author if she can make explicit that the other reviewers are with her.
Second, you can cajole the editors to make a decision already. In your second round R&R review tell the editors that there’s never going to be a complete consensus among the reviewers and they should stop dicking the authors around with R&Rs. You can refuse to be the dreaded “new reviewer.” You can refuse to review past the first round R+R. You can tell the editors that you’re willing to let them treat your issues as a conditional accept adjudicated by them rather than as another R&R that goes back to you for review.
Just as importantly as being nice, you can tell the editors to give a clean reject. Remember, an R&R does not mean “good but not great” or “honorable mention” but “this could be rewritten to get an accept.” Some flaws (often having to do with sampling or generalizability) are of a nature that they simply can’t be fixed so even if you like the other aspects of the paper you should just reject. Others may be fixable in principle (often having to do with the lit review or model specification) but in practice doing so would require you to rewrite the paper for the authors and it benefits nobody for you to appoint yourself anonymous co-author. Hence my last point…
Give decisive rejections
I’ve emphasized how to be nice to the authors by not burdening them with superfluous demands However it’s equally important to be decisive about things that are just plain wrong. I have a lot of regrets about my actions as a peer reviewer and if I were to go through my old review reports right now I’d probably achieve Augustinian levels of self-reproach. Many of them of are of the nature of “I shouldn’t have told that person to cite/try a bunch of things that didn’t really matter because by so doing I was being the kind of spend-the-next-year-on-the-revisions-to-make-the-paper-worse reviewer I myself hate to get.” However, I don’t at all regret, for instance, a recommendation to reject that I wrote in which I pointed out that the micro-mechanisms of the author’s theory were completely incompatible with the substantive processes in the empirical setting and that the quantitative model was badly misspecified. Nor do I regret recommending to reject a paper because it relied on really low quality data and its posited theoretical mechanism was a Rube Goldberg device grounded in a widely cited but definitively debunked paper. Rather my biggest regret as a reviewer is that I noticed a manuscript had a grievous methodological flaw that was almost certainly entirely driving the results but I raised the issue in a hesitant fashion and the editor published the paper anyway. As I’ve acquired more experience on both sides of the peer review process, I’ve realized that being a good peer reviewer isn’t about being nice, nor is it about providing lots of feedback. Rather being a good reviewer is about evaluating which papers are good and which papers are bad and clearly justifying those decisions. I’m honored to serve as a consulting editor for Sociological Science because that is what that journal asks of us but I also aspire to review like that regardless of what journal I’m reviewing for and I hope you will too. (Especially if you’re reviewing my papers).
| Gabriel |
1st of March, 1600 Anno Domini
To his majesty Felipe Hapsburg III,
His majesty Christian den Fjerde, king of Denmark-Norway, sends you greetings and congratulations on your recent ascension and wishes your reign to extend peace and prosperity to Spain.
We are writing to alert you that your realm’s “New Spain” project infringes on methods belonging to the Kingdom of Denmark-Norway. We wish to offer you an opportunity for licensing, including back-royalties for past activities of New Spain.
In particular, the kingdom of Denmark-Norway lays claim to various patents and other intellectual property stemming from its “Vinland” project with various filings contained in the Íslendingasögur. Although the Vinland project is no longer an ongoing project, in bankruptcy its assets have reverted to the crown and among these are various techniques infringed upon by New Spain. The most important of these patents is for “Growth in Commerce and Enrichment of the Realm Through Expansion to Lands Across the Ocean.” However New Spain also infringes on various other of the Vinland patents, including “A Method for Enslavement of Skrælingjar” and “Export of Agricultural Commodities Derived From Exotic Plants.”
We would therefore like to offer you the opportunity to license the Vinland patents to cover your activities in New Spain. A licensing deal must account not only prospectively but retrospectively for all such stuffs exported back into Christendom. It has come to our attention that New Spain has to date exported approximately 17,000 tons of silver in addition to several hundred tons of gold and many more tons of exotic useful things such as tabac, maize, cocoa, and various spices.
Please send an embassy to discuss licensing terms and royalties immediately so as to avoid a cease and desist on all activities of New Spain.
His Majesty King Christian IV