Yesterday I got the “Sociology” and “Research Methods, Statistics, & Evaluation” catalogs from Sage and I noticed something interesting about them. Let’s take a look at a page together and see if you notice what interested me. (click to get a full-size pdf)

Here’s a hint, they starts with “$” and they ain’t there. The vast majority of the titles in the catalog don’t have any prices. This is odd. Generally one of the most salient pieces of information about any market commodity is the price and that includes books. If you search Amazon for these same titles you will see that Amazon makes the prices pretty conspicuous. Nor is there some unique delicacy of academics that Amazon is boorishly overlooking. If you look at the sociology catalogs for PUP or UC you’ll see that there are prices throughout. So why doesn’t Sage provide prices?

If you understand the book market you may be able to guess, but let’s take a look at another page from the Sage catalog for really conclusive proof.

This page has prices. As you may have noticed the tab on the edge helpfully notes that these are books “For Your Bookshelf.” In contrast, the (much more numerous) books that are described as “Recommend to Your Students” or “Textbooks” do not have prices. We can fit PUP and UC into this pattern insofar as they’re academic presses, not textbook publishers, meaning that in Sage’s terminology every book offered by PUP or UC is “For Your Bookshelf” and thus deserves to have a price attached. (Academic presses like PUP and UC certainly don’t object to course adoption, but the books are primarily written for an audience of colleagues and course adoption is a valuable crossover market).

Generally, the pattern is that when somebody else is paying you don’t care about the price. When I adopt a book for course adoption it’s my students who pay. Of course it’s almost inevitable that I’m going to be less price sensitive when choosing a book for course adoption than if I were buying 120 copies out of pocket. What amazes me is how brazen Sage is in not even putting the price in the catalog when my students are paying but doing so when I pay, a not so subtle hint that I should be completely indifferent to how much my students pay although obviously I’ll care when I pay. This is not unlike the critique of health insurance underlying HSAs: third party payment in medicine, as in textbooks, discourages price sensitivity and by extension leads to cost creep. This is sometimes taken to the extreme that doctors never quote patients the price of an MRI (or whatever it is they are referring/prescribing) and often can’t provide price information even if the patient asks. Even if the third party payment problem is intrinsic (I can hardly imagine the pedagogical nightmare of students choosing their own textbooks), we can at least hope for the people making allocation decisions to be responsible proxies. It doesn’t help when the sellers don’t even reveal the price and the proxies evidently don’t care that this information is absent.

I recently read Graeber’s Debt: The First Five Thousand Years and found it to be very impressive and thought-provoking. As an indication of how impressed I was, I’ll just say that: I’ve recommended it to several people, I’m citing it in one of my next papers, I’m very seriously considering assigning it as a text when I prep econ soc sometime in the next couple years, and it inspired me to go back and read The Gift by Mauss and several journal articles. I’d long been vaguely aware of the anthropology of exchange through my exposure to Zelizer and Fiske, but had never read very deeply in it and so I learned a lot. Debt is written as “big history” (not unlike JLM’s Social Structures and Fukuyama’s OoPO) and this gives it a different character than the more cross-sectional approaches taken by Mauss, Zelizer, or Fiske.

Large parts of the book could better be called Commerce: The First 5,000 Years or Exchange: The First 5,000 Years. However there are a few places where Graeber explicitly tackles debt. The most widely known of these (in part because Graeber opened a can of whup-ass on some guy at Mises who tenaciously stuck to the discredited view) is the origins of money. The received view from Adam Smith which still persists in econ textbooks is that primordial exchange takes the form of barter but that barter suffers from the “double coincidence of wants” problem wherein A can only trade with B if A has something B desires and vice versa. In this model, money lubricates exchange by creating a universal medium of exchange. As Graeber shows, this model of pre-monetized exchange assumes arms-length transactions whereas almost all documented exchange in primitive cultures is thoroughly embedded and takes one form or another of communal sharing or gift exchange. That is, delayed reciprocity is typical and this avoids the problem of coincidence of wants. From the economist’s perspective delayed reciprocity introduces further issues of trust, time preference, etc and thus is a more complex form of exchange than barter, but this is because (at least on this issue) economists are acting as arm chair philosophers of the social contract and it is the anthropologists who are (at least on this issue) being good empiricists.

The important exception to the “barter is a myth” point is that Graeber argues that arms-length exchange does occur in primitive cultures, but only between and not within meaningful social units and that such arms-length exchange is somewhat sketchy and dangerous. More broadly, one of the central points of the book is that arms length exchange in general and market economies in particular require the disembedding of people and commodities from their social context. Graeber sees this process as often violent and he makes a powerful argument that this originates with slavery, both in antiquity and in early modern Africa.

Other interesting points he makes on debt are various ways that it becomes a moral obligation such that debtors are seen as sinners and religious salvation is seen as a spiritual analog to redemption. This helps explain something I never completely understood when watching The Sopranos, which is why gangsters first go to the trouble of getting someone to incur an illegal debt before shaking them down? It turns out that the point of loan-sharking instead of mere naked extortion is the victim feels a certain moral obligation to repay the debt and so loan sharks exploiting gambling addicts has the same logic as how many grifts (e.g., 419 advanced-fee fraud, the fiddle game, etc.) first involve the victim as co-conspirator in a crime against a real or imagined third party. Moreover, Graeber makes the bold point towards the end of the book that debt can drive people to do things that they otherwise would be morally averse to, with his example being the conquistadores.

This is all fascinating but it depends a lot on how much you trust Graeber’s empirical claims. For instance, was it really true that everyday economic life in early modern Britain was largely cashless and instead used a combination of token currencies, informal credit, and asynchronous barter? Maybe, I really don’t know. I’d like to trust Graeber on this but I don’t know if I can since he gets some things pretty wrong, or at least dubious. At Unfogged there’s a review (and a very funny comments thread) pointing out that the following sentence contains six factual claims all of which are incorrect:

Apple Computers is a famous example: it was founded by (mostly Republican) computer engineers who broke from IBM in Silicon Valley in the 1980s, forming little democratic circles of twenty to forty people with their laptops in each other’s garages.

This is not exactly stuff written in the cuneiform of Mesopotamian diplomacy, the barbarian law codes of mediaeval Ireland, or the field notes of Victorian anthropologists, but something that occurred in suburban California around the time I was born and concerns the extremely well documented origins of one of the world’s biggest firms. If Graeber gets this wrong, how can we trust him about the stuff that’s harder to check, like all that business about barbarian law codes.

The thing that really bothers me though (because it’s more than an isolated sentence) is the last few chapters, which argue that America’s current account deficit constitutes military tribute. He means this literally. For instance, he suggests that the Iraq War was punishment for Iraq switching to the euro — meanwhile back in reality the euro area itself overlaps pretty closely with NATO and several eurozone countries invaded Iraq together with the United States. I guess we’ve just been too busy punishing Iraq for using euros to get around to dropping a few bombs on the European Central Bank which actually issues those euros. (This is pretty strange since the ECB is just a few minutes of flight time from a massive USAF base, so bombing it would be a very convenient way to ensure the continued flow of tribute).

When I first read this military tribute argument in the early 1990s (in Chomsky’s Deterring Democracy) it made a lot of sense to me, but two things were different then:

1. I was a lot younger, less informed about economics, and more paranoid in my political thinking.
2. In the early 1990s the US government’s major foreign debt holders were countries that could plausibly be described as military protectorates (Japan, South Korea, Saudi Arabia, etc.). Now the single largest holder of US government debt is the People’s Republic of China. For those of you following at home, China is most certainly not a US military protectorate but our major geostrategic rival against whom a post-GWOT DOD is orienting its strategic doctrine.

Graeber addresses problem #2 head on and tries to explain this away by some convoluted argument that I can’t even reproduce but I find his argument much less plausible than the more parsimonious explanation that the Chinese are buying t-bills (a) as a store of value (b) as a medium of exchange and (c) as a tacit export subsidy that suits their domestic politics. That is, they buy t-bills for basically the same reasons as everybody else, including those countries where we have Air Force or Navy bases. This deliberate obtuseness about how a reserve currency works and the paranoid understanding that it is provincial tribute is by far the worst part of the book. I’m trying to draw a fact/value distinction between my lack of sympathy for his political positions and his empirical claims as I’d like to think that when reading someone with whom I disagree I can distinguish between their empirical claims that are well-supported, debatable, and downright nuts. That is to say I don’t think these chapters upset me because they are normatively “anti-American” but because as an empirical matter they badly fail to understand how (for better or worse) American power works.

This business about tribute is at the end so I’d like to say that I recommend the book but that you stop on page 365, right before he gets his Chomsky on, but I honestly worry whether I can trust the parts of the book I’m not as informed about. This is the 13th chime of the clock, the brown M&Ms in the Van Halen dressing room; pick your metaphor, but this business about Apple computer and especially about Chinese t-bill holdings ultimately makes me take a “trust but verify” attitude towards a book that I found both extremely enjoyable and intellectually inspirational as I was reading it. My recommendation is that people interested in exchange read the book, but do so with an appropriate degree of skepticism and look to see reactions from historians and anthropologists who are qualified to assess the empirical claims. Also see other reviews at Understanding Society and OrgTheory.

Lewis, Gonzalez, and Kaufman have a forthcoming paper in PNAS on “Social selection and peer influence in an online social network.” The project uses Facebook data from the entire college experience of a single cohort of undergrads at one school in order to pick at the perennial homophily/influence question. (Also see earlier papers from this project).

Overall it’s an excellent study. The data collection and modeling efforts are extremely impressive. Moreover I’m very sympathetic to (and plan to regularly cite) the conclusion that contagious diffusion is over-rated and we need to consider the micro-motives and mechanisms underlying contagion. I especially liked how they synthesize the Bourdieu tradition with diffusion to argue that diffusion is most likely for taste markers that are distinctive in both sense of the term. As is often the case with PNAS or Science, the really good stuff is in the appendix and in this case it gets downright comical as they apply some very heavy analytical firepower to trying to understand why hipsters are such pretentious assholes before giving up and delegating the issue to ethnography.

The thing that really got me thinking though was a claim they make in the methods section:

Because data on Facebook are naturally occurring, we avoided interviewer effects, recall limitations, and other sources of measurement error endemic to survey-based network research

That is, the authors are reifying Facebook as “natural.” If all they mean is that they’re taking a fly on the wall observational approach, without even the intervention of survey interviews, then yes, this is naturally occurring data. However I don’t think that observational necessarily means natural. If researchers themselves imposed reciprocity, used a triadic closure algorithm to prime recall, and discouraged the deletion of old ties; we’d recognize this as a measurement issue. It’s debatable whether it’s any more natural if Mark Zuckerberg is the one making these operational measurement decisions instead of Kevin Lewis.

Another way to put this is to ask where does social reality end and observation of it begin? In asking the question I’m not saying that there’s a clean answer. On one end of the spectrum we might have your basic random-digit dialing opinion survey that asks people to answer ambiguously-worded Likert-scale questions about issues they don’t otherwise think about. On the other end of the spectrum we might have well-executed ethnography. Sure, scraping Facebook isn’t as unnatural as the survey but neither is it as natural as the ethnography. Of course, as the information regimes literature suggests to us, you can’t really say that polls aren’t natural either insofar as their unnatural results leak out of the ivory tower and become a part of society themselves. (This is most obviously true for things like the unemployment rate and presidential approval ratings).

At a certain point something goes from figure to ground and it becomes practical, and perhaps even ontologically valid, to treat it as natural. You can make a very good argument that market exchange is a social construction that was either entirely unknown or only marginally important for most of human history. However at the present the market so thoroughly structures and saturates our lives that it’s practical to more or less take it for granted when understanding modern societies and only invoke the market’s contingent nature as a scope condition to avoid excessive generalization of economics beyond modern life and into the past, across cultures, and the deep grammar of human nature.

We are, God help us, rapidly approaching a situation where online social networks structure and constitute interaction. Once we do, the biases built into these systems are no longer measurement issues but will be constitutive of social structure. During the transitional period we find ourselves in though, let’s recognize that these networks are human artifices that are in the process of being incorporated into social life. We need a middle ground between “worthless” and “natural” for understanding social media data.

Last time I described using the twitteR library for R. In that post I had R itself read over a list to loop. In this post I make the have looping occur in Bash with argument passing to R through the commandArgs() function.

First, one major limitation of using Twitter is that it times you out for an hour after 150 queries. (You can double this if you use OAuth but I’ve yet to get that to work). For reasons I don’t really understand, getting one feed can mean multiple queries, especially if you’re trying to go far back in the timeline. For this reason you need to break up your list into a bunch of small lists and cron them at least 80 minutes apart. This bit of Bash code will split up a file called “list.txt” into several files. Also, to avoid trouble later on, it makes sure you have Unix EOL.

split -l 50 list.txt short_tw
perl -pi -e 's/\r\n/\n/g' short_tw*

The next thing to keep in mind is that you’ll need to pass arguments to R. Argument passing is when a script takes input from outside the script and processes it as variables. The enthusiastic use of argument passing in Unix is the reason why there is a fine line between a file and a command in that operating system.

In theory you could have R read the target list itself but this crashes when you hit your first dead URL. Running the loop from outside R makes it more robust but this requires passing arguments to R. I’d previously solved this problem by having Stata write an entire R script, which Stata understood as having variables (or “macros”) but which from R’s perspective was hard-coded. However I was recently delighted to discover that R can accept command-line arguments with the commandArgs() function. Not surprisingly, this is more difficult than $1 in Bash, @ARGV in Perl, or `1′ in Stata, but it’s not that bad. To use it you have to use the “–args” option when invoking R and then inside of R you use the commandArgs() function to pass arguments to an array object, which behaves just like the @ARGV array in Perl.

Here’s an R script that accepts a Twitter screenname as a command-line argument, uses the twitteR library to collect that feed, and then saves it as a tab-delimited text file of the same name. (It appends if there’s an existing file). Also note that (thanks to commenters on the previous post) it turns internal EOL into regular spaces. It’s currently set to collect the last 200 tweets but you can adjust this with the third line (or you could rewrite the script to make this a command-line argument as well).

args <- commandArgs(trailingOnly = TRUE)
library(twitteR)
howmany <- 200 #how many past tweets to collect

user <- args[1]
outputfile <- paste('~/project/feeds/',user,'.txt',sep="")
print(user)
print(outputfile)

tmptimeline <- userTimeline(user,n=as.character(howmany))
tmptimeline.df <- twListToDF(tmptimeline)
tmptimeline.df$text <- gsub("\\n|\\r|\\t", " ", tmptimeline.df$text)
write.table(tmptimeline.df,file=outputfile,append=TRUE,sep="\t",col.names=FALSE)

quit()

To use the script to get just a single feed, you invoke it like this from the command-line.

R --vanilla --args asanews < datacollection.R

Of course the whole reason to write the script this way is to loop it over the lists. Here it is for the list “short_twaa”.

for i in `cat short_twaa`; do R --vanilla --args $i < datacollection.R ; done

Keep in mind that you’ll probably want to cron this, either because you want a running scrape or because it makes it easier to space put the “short_tw*” files so you don’t get timed out.

Previously I’d discussed scraping Twitter using Bash and Perl. Then yesterday on an orgtheory thread Trey mentioned the R library twitteR and with some help from Trey I worked out a simple script that replaces the twitterscrape_daily.sh and twitterparse.pl scripts from the earlier workflow. The advantage of this script is that it’s a lot shorter, it can get an arbitrary number of tweets instead of just 20, and it captures some of the meta-text that could be useful for constructing social networks.

To use it you need a text file that consists of a list of Twitter feeds, one per line. The location of this file is given in the “inputfile” line.

The “howmany” line controls how many tweets back it goes in each feed.

The “outputfile” line says where the output goes. Note that it treats it as append. As such you can get some redundant data, which you can fix by running this bash code:

sort mytweets.txt | uniq > tmp
mv tmp mytweets.txt

The outputfile has no headers, but they are as follows:

#v1 ignore field, just shows number w/in query
#v2 text of the Tweet
#v3 favorited dummy
#v4 replytosn (mention screenname)
#v5 created (date in YMDhms)
#v6 truncated dummy
#v7 replytosid
#v8 id
#v9 replytouid
#v10 statussource (Twitter client)
#v11 screenname

Unfortunately, the script doesn’t handle multi-line tweets very well, but I’m not sufficiently good at R to regexp out internal EOL characters. I’ll be happy to work this in if anyone cares to post some code to the comments on how to do a find and replace that zaps the internal EOL in the field tmptimeline.df$text.

library(twitteR)
howmany <- 30 #how many past tweets to collect
inputfile <- "~/feedme.txt"
outputfile <- "~/mytweets.txt"

feeds <- as.vector(t(read.table(inputfile)))
for (user in feeds) {
	tmptimeline <- userTimeline(user,n=as.character(howmany))
	tmptimeline.df <- twListToDF(tmptimeline)
	write.table(tmptimeline.df,file=outputfile,append=TRUE,sep="\t",col.names=FALSE)
}

Finally, if you import it into Stata you’ll probably want to run this:

drop v1
ren v2 text
ren v3 favorited 
ren v4 replytosn
ren v5 created
ren v6 truncated 
ren v7 replytosid
ren v8 id
ren v9 replytouid
ren v10 statussource
ren v11 screenname
gen double timestamp=clock(subinstr(created,"-","/",.),"YMDhms")
format timestamp %tc

I recently read Frédéric Martel’s book Mainstream and my initial reaction was shock to discover that I can still read French this many years out from lycée. After I got over that I was able to have a more substantive reaction which was that I really liked it. First there is the sheer scale of the book. He did hundreds of interviews, many of them with extremely high-placed people who I can’t even imagine trying to get access to. He covers pretty much every media industry you can think of and gives a good overview of each. (I know most of the industries he describes pretty well and concur with his descriptions). The first half of the book is primarily concerned with the US media industry and the second half with globalization. In addition to the sheer wealth of detail there is a consistent thesis which is that there is such a thing as a global mass culture centered in America, but there is also a multi-polar nature to this with things like telenovelas, Bollywood, and K-pop circulating within their  respective regional spheres.

My copy of the book visiting Disneyland

On the subject of theme, it’s worth noting that different editions of the book have different subtitles. The original subtitle translates to “inquiry into the culture that pleases everyone” (actually, it’s a pun that can also be translated as “inquiry into the culture that pleases the whole world”). The paperback’s subtitle is “inquiry into the global war over culture and media.” The subtitles of various translated editions follow one or the other. I find the original subtitle to be a better description of the book, especially the first half. However throughout the book there is an appreciation for the politics of media globalization. For instance, there is lengthy discussion of US trade policy and piracy, censorship in various countries, Chinese film import quotas (which can be effective leverage for censorship), etc. This interest in both the comparative and IR politics of cultural production is not only important but Frédéric is in a good position to understand and explain it, having been a cultural attache posted to Boston and remaining a cultural policy intellectual associated with the socialist party.

Even in my semi-literacy with the French language I was able to pick up on how good the writing style is. In a lot of ways Frédéric reminds me of Tom Wolfe. That is, they are both PhD sociologists by background but this perspective is implicit rather than explicit and their writing style is exuberant, with the narrative jumping from location to location every two pages and with an occasional pause to savor absurdity. Likewise, in both cases the thesis gradually swirls into view out of a vortex of detail.

The book has been translated into several languages already and I hope it gets an English translation, but if you can read French and are interested in the culture industries and/or globalization you might want to check out the original rather than waiting. (I found it well worth it to read the original even though I read about half as fast in French as I do in English).

FWIW, Frédéric is a friend of mine and gave me the book when I visited him in June. I’ve been looking forward to this book since I saw him in LA a few years ago when he was doing the field work for the book and in particular I remember the “WRITER” t-shirt he mentions in the book.

Finally, there seems to be something of an arbitrage opportunity for selling foreign books in the US. The paperback goes for 9 euros in France, but the remailers who list on Amazon.com offer it for $30. I would be surprised if the marginal cost of forwarding a book from France to the US is actually $18. I doubt that it’s inventory costs either as they make you wait 10 days which implies that it’s an on-demand service.

| Trey |

Recently, Gabriel pointed me to a piece in Ad Age (and original press release) about IBM researchers correlating the conversations of fashion bloggers with the state of the economy (make sure you file away the accompanying graph for the next time you teach data visualization). Trevor Davis, a “consumer-products expert” with IBM, claimed that as economic conditions improve, the average height of high heels mentioned by these bloggers decreases. Similarly, as economic conditions worsen, the average height would increase. As Gabriel pointed out, these findings seemed to lack any sort of face validity — how likely does it seem that, at any level of economic performance, the average high heel is seven inches tall (even among fashionistas)? I’ll return to the specific problems posed by what I’ll call the “seven-inch heel problem” in a moment, but first some background on the methods that most likely went into this study.

While amusing, if not very credible, the IBM study is part of a growing area (dubbed by some “computational social science”) situated at the intersection of natural language processing and machine learning. By taking advantage of the explosion of available digital text and computing power, researchers in this area are attempting to model structure in and test theories of large-scale social behavior. You’ve no doubt seen some of this work in the media, ranging from “predicting the Arab Spring” to using Twitter to predict GOP primary frontrunners. Many of these works hew towards the style end of the style-substance divide and are not typically motivated by any recognizable theory. However, this is changing as linguists use Twitter to discover regional dialect differences and model the daily cycle of positive and negative emotions.

Much of this work is being met by what I perceive to be reflexive criticism (as in automatic, rather than in the more sociological sense) from within the academy. The Golder and Macy piece in particular received sharp criticism in the comments on orgtheory, labeled variously “empiricism gone awry”, non-representative, and even fallacious (and in which yours truly was labeled “cavalier”). Some of this criticism is warranted, although as is often the case with new methods and data sources, much of the criticism seems rooted in misunderstanding. I suspect part of this is the surprisingly long-lived skepticism of scholarly work on “the internet” which, with the rise of Facebook and Twitter, seems to have been reinvigorated.

However, sociologists are doing themselves a disservice by seeing this work as research on the internet qua internet. Incredible amounts of textual and relational data are out there for the analyzing — and we all know if there’s one thing social scientists love, it’s original data. And these data are not limited to blog posts, status updates, and tweets. Newspapers, legislation, historical archives, and more are rapidly being digitized, providing pristine territory for analysis. Political scientists are warming to the approach, as evidenced by none other than the inimitable Gary King and his own start-up Crimson Hexagon, which performs sentiment analysis on social media using software developed for a piece in AJPS. Political Analysis, the top-ranked journal in political science and the methodological showcase for the discipline, devoted an entire issue in 2008 to the “text-as-data” approach. Additionally, a group of historians and literary scholars have adopted these methods, dubbing the new subfield the “digital humanities.”

Sociologists of culture and diffusion have already warmed to many of these ideas, but the potential for other subfields is significant and largely unrealized. Social movement scholars could find ways to empirically identify frames in wider public discourse. Sociologists of stratification have access to thousands of public- and private-sector reports, the texts of employment legislation, and more to analyze. Race, ethnicity, and immigration researchers can model changing symbolic boundaries across time and space. The real mistake, in my view, is dismissing these methods as an end in and of themselves rather than as a tool for exploring important and interesting sociological questions. Although many of the studies hitting the mass media seem more “proof of concept” than “test of theory,” this is changing; sociologists will not want to be left behind. Below, I will outline the basics of some of these methods and then return to the seven-inch heels problem.

The use of simple scripts or programs to scrape data from the web or Twitter has been featured several times on this blog. The data that I collected for my dissertation were crawled and then scraped from multiple English and Arabic news outlets that post their archives online, including Al Ahram, Al Masry Al Youm, Al Jazeera, and Asharq al Awsat. The actual scrapers are written in Python using the Scrapy framework.

Obtaining the data is the first and least interesting step (to sociologists). Using the scraped data, I am creating chains of topic models (specifically using Latent Dirichlet Allocation) to model latent discursive patterns in the media from the years leading up to the so-called “Arab Spring.” In doing so, I am trying to identify the convergence and divergence in discourse across and within sources to understand how contemporary actors were making sense of their social, political, and economic contexts prior to a major social upheaval. Estimating common knowledge prior to contentious political events is often problematic due to hindsight biases, because of the problems of conducting surveys in non-democracies, and for the obvious reason that we usually don’t know when a major social upheaval is about to happen even if we may know which places may be more susceptible.

Topic modeling is a method that will be look familiar in its generalities to anyone who has seen a cluster analysis. Essentially, topic models use unstructured text — i.e., text without labeled fields from a database or from a forced-choice survey — to model the underlying topical components that make up a document or set of documents. For instance, one modeled topic might be composed of the words “protest”, “revolution”, “dictator”, and “tahrir”. The model attempts to find the words that have the highest probability of being found with one another and with the lowest probability of being found with other words. The generated topics are devoid of meaning, however, without theoretically informed interpretation. This is analogous to survey researchers that perform cluster or factor analyses to find items that “hang together” and then attempt to figure out what the latent construct is that links them.

Collections of documents (a corpus) are usually represented as a document-term matrix, where each row is a document and the columns are all of the words that appear in your set of documents (the vocabulary). The contents of the individual cells are the per-document word frequencies. This produces a very sparse matrix, so some pre-processing is usually performed to reduce the dimensionality. The majority of all documents from any source are filled with words that convey little to no information — prepositions, articles, common adjectives, etc. (see Zipf’s law). Words that appear in every document or in a very small number of documents provide little explanatory power and are usually removed. The texts are often pre-processed using tools such as the Natural Language Toolkit for Python or RTextTools (which is developed in part here at the University of Washington) to remove these words and punctuation. Further, words are often “stemmed” or “lemmatized” so that the number of words with common suffixes and prefixes but with similar meanings is reduced. For example, “run”, “runner”, “running”, and “runs” might all be reduced to “run”.

This approach is known as a “bag-of-words” approach in that the order and context of the words is assumed to be unimportant (obviously, a contentious assumption, but perhaps that is a debate for another blog). Researchers that are uncomfortable with this assumption can use n-grams, groupings of two or more words, rather than single words. However, as the n increases, the number of possible combinations and the accompanying computing power required grows rapidly. You may be familiar with the Google Ngram Viewer. Most of the models are extendable to other languages and are indifferent to the actual content of the text although obviously the researcher needs to be able to read and make sense of the output.

Other methods require different assumptions. If you are interested in parts of speech, a part-of-speech tagger is required, which assumes that the document is fairly coherent and not riddled with typos. Tracking exact or near-exact phrases is difficult as well, as evidenced by the formidable team of computer scientists working on MemeTracker. The number of possible variations on even a short phrase quickly becomes unwieldy and requires substantial computational resources — which brings us back to the seven-inch heels.

Although IBM now develops the oft-maligned SPSS, they also produced Watson. This is why the total lack of validity of fashion blogging results is surprising. If one were seriously going to track the height of heels mentioned and attempt to correlate it with economic conditions, in order to have any confidence that you have captured a non-biased sample of mentions, at least two necessary steps would include:

• Identifying possible combinations of size metrics and words for heels: seven-inch heels, seven inch heels, seven inch high heels, seven-inch high-heels, seven inch platforms, etc. And so on. This is further complicated by the fact that many text processing algorithms will treat “seven-inch” as one word.
• Dealing with the problem of punctuational abbreviations for these metrics: 7″ heels, 7″ high heels, 7 and a 1/2 inch heels, etc. Since punctuation is usually stripped out, it would be necessary to leave it in, but then how to distinguish quotation marks that appear as size abbreviations and those that appear in other contexts?
• Do we include all of these variations with “pumps?” Is there something systematic such as age, location, etc. about individuals that refer to “pumps” rather than “heels?”
• Are there words or descriptions for heels that I’m not even aware of? Probably.

None of these is an insurmountable problem and I have no doubt that IBM researchers have easy access to substantial computing power. However, each of them requires careful thought prior to and following data collection; the combination of them together quickly complicates matters. Since IBM is unlikely to reveal their methods, though, I have serious doubts as to the validity of their findings.

As any content analyst can tell you, text is a truly unique data source as it is intentional language and is one of the few sources of observational data for which the observation process is totally unobtrusive. In some cases, the authors are no longer alive! Much of the available online text of interest to social scientists was not produced for scholarly inquiry and was not generated from survey responses. However, the sheer volume of the text requires some (but not much!) technical sophistication to acquire and make sense of and, like any other method, these analyses can produce results that are essentially meaningless. Just as your statistics package of choice will output meaningless regression results from just about any data you feed into it, automated and semi-automated text analysis produces its own share of seven-inch heels.

The WSJ has a very interesting table of the unemployment and wage distributions for various majors. There’s lots to talk about, particularly the STEM/humanities/social/vocational divide, but one thing that struck me was that the highest and lowest unemployment rates were dominated by tiny majors. In general, small populations tend to have more widely varying outcomes just as a function of standard error, which is why you should always ignore headlines about big jumps in the crime rate for small towns. Anyway, I downloaded the data, generated some plots, and yup, it’s your classic funnel.

Here’s unemployment by rank popularity. Because low rank means popular, the funnel is backwards.

WSJ only provides rank, but I approximated raw size as the inverse of log rank plus 1 and this gives us the typical funnel.

Moral of the story, don’t change your major from clinical psych to actuarial science just yet. On the other hand, nursing, elementary education, and general education really do appear to be real deal outliers of low unemployment.

Here’s the code.

insheet using ~/Documents/codeandculture/majors.txt, clear
drop v7
gen unemploymentpercent_real=real(subinstr(unemploymentpercent,"%","",.))
twoway scatter unemploymentpercent_real  popularity, xtitle(Rank Order Popularity) ytitle(% Unemployed)
graph export majors_unemployed_rank.png, replace

gen size=1/(log(1+popularity))
corr unemploymentpercent_real popularity size

lab def size 0 "Obscure" 2 "Ubiquitous"
lab val size size
twoway (scatter unemploymentpercent_real size), xlabel(#2, labels angle(forty_five) valuelabel) xtitle(Approximate Raw Size) ytitle(% Unemployed)
graph export majors_unemployed_size.png, replace

*have a nice day

Major Field	Unemployment Percent	25th % Earnings	Median % Earnings	75th % Earnings	Popularity
ACCOUNTING	5.4%	$41,000	$61,000	$94,000	3
ACTUARIAL SCIENCE	0.0%	$52,000	$81,000	$116,000	150
ADVERTISING AND PUBLIC RELATIONS	6.1%	$36,000	$50,000	$74,000	41
AEROSPACE ENGINEERING	3.6%	$60,000	$84,000	$111,000	105
AGRICULTURAL ECONOMICS	1.3%	$30,000	$57,000	$99,000	122
AGRICULTURE PRODUCTION AND MANAGEMENT	3.0%	$32,000	$48,000	$71,000	75
ANIMAL SCIENCES	5.7%	$26,000	$40,000	$60,000	67
ANTHROPOLOGY AND ARCHEOLOGY	6.9%	$30,000	$40,000	$60,000	55
APPLIED MATHEMATICS	4.1%	$52,000	$71,000	$100,000	131
ARCHITECTURAL ENGINEERING	5.8%	$50,000	$71,000	$96,000	140
ARCHITECTURE	10.6%	$37,000	$60,000	$85,000	33
AREA ETHNIC AND CIVILIZATION STUDIES	5.7%	$34,000	$48,000	$76,000	66
ART AND MUSIC EDUCATION	4.2%	$32,000	$41,000	$51,000	48
ART HISTORY AND CRITICISM	6.9%	$33,000	$45,000	$71,000	81
ASTRONOMY AND ASTROPHYSICS	0.0%	$56,000	$62,000	$101,000	170
ATMOSPHERIC SCIENCES AND METEOROLOGY	1.6%	$40,000	$68,000	$101,000	146
BIOCHEMICAL SCIENCES	7.1%	$30,000	$48,000	$80,000	87
BIOLOGICAL ENGINEERING	6.8%	$39,000	$60,000	$94,000	126
BIOLOGY	5.6%	$35,000	$51,000	$76,000	14
BIOMEDICAL ENGINEERING	5.9%	$45,000	$68,000	$101,000	137
BOTANY	6.9%	$26,000	$40,000	$55,000	147
BUSINESS ECONOMICS	5.0%	$44,000	$71,000	$101,000	80
BUSINESS MANAGEMENT AND ADMINISTRATION	6.0%	$38,000	$56,000	$85,000	1
CHEMICAL ENGINEERING	3.8%	$60,000	$86,000	$117,000	49
CHEMISTRY	5.1%	$39,000	$59,000	$85,000	36
CIVIL ENGINEERING	4.9%	$55,000	$76,000	$101,000	32
CLINICAL PSYCHOLOGY	19.5%	$25,000	$40,000	$61,000	168
COGNITIVE SCIENCE AND BIOPSYCHOLOGY	4.5%	$36,000	$43,000	$91,000	167
COMMERCIAL ART AND GRAPHIC DESIGN	8.1%	$31,000	$45,000	$69,000	21
COMMUNICATION DISORDERS SCIENCES AND SERVICES	3.3%	$32,000	$41,000	$50,000	98
COMMUNICATION TECHNOLOGIES	6.7%	$33,000	$50,000	$73,000	89
COMMUNICATIONS	6.3%	$35,000	$50,000	$81,000	7
COMMUNITY AND PUBLIC HEALTH	4.1%	$31,000	$46,000	$70,000	110
COMPOSITION AND SPEECH	7.7%	$30,000	$40,000	$61,000	99
COMPUTER ADMINISTRATION MANAGEMENT AND SECURITY	9.5%	$39,000	$52,000	$75,000	114
COMPUTER AND INFORMATION SYSTEMS	5.6%	$44,000	$62,000	$86,000	31
COMPUTER ENGINEERING	7.0%	$58,000	$81,000	$102,000	47
COMPUTER NETWORKING AND TELECOMMUNICATIONS	5.2%	$35,000	$53,000	$76,000	97
COMPUTER PROGRAMMING AND DATA PROCESSING	6.2%	$39,000	$55,000	$84,000	121
COMPUTER SCIENCE	5.6%	$50,000	$77,000	$102,000	10
CONSTRUCTION SERVICES	5.4%	$49,000	$65,000	$101,000	76
COSMETOLOGY SERVICES AND CULINARY ARTS	7.3%	$26,000	$41,000	$60,000	115
COUNSELING PSYCHOLOGY	5.2%	$23,000	$34,000	$42,000	133
COURT REPORTING	4.9%	$36,000	$55,000	$81,000	151
CRIMINAL JUSTICE AND FIRE PROTECTION	4.7%	$36,000	$50,000	$73,000	13
CRIMINOLOGY	5.2%	$35,000	$50,000	$71,000	92
DRAMA AND THEATER ARTS	7.1%	$28,000	$40,000	$60,000	45
EARLY CHILDHOOD EDUCATION	4.1%	$28,000	$37,000	$45,000	50
ECOLOGY	5.2%	$31,000	$43,000	$60,000	109
ECONOMICS	6.3%	$42,000	$69,000	$108,000	16
EDUCATIONAL ADMINISTRATION AND SUPERVISION	0.0%	$41,000	$65,000	$89,000	171
EDUCATIONAL PSYCHOLOGY	10.9%	$28,000	$35,000	$51,000	156
ELECTRICAL AND MECHANIC REPAIRS AND TECHNOLOGIES	8.4%	$30,000	$44,000	$68,000	134
ELECTRICAL ENGINEERING	5.0%	$60,000	$86,000	$111,000	17
ELECTRICAL ENGINEERING TECHNOLOGY	5.5%	$42,000	$65,000	$91,000	65
ELEMENTARY EDUCATION	3.6%	$32,000	$40,000	$49,000	8
ENGINEERING AND INDUSTRIAL MANAGEMENT	9.2%	$50,000	$71,000	$98,000	127
ENGINEERING MECHANICS PHYSICS AND SCIENCE	6.5%	$40,000	$67,000	$101,000	132
ENGINEERING TECHNOLOGIES	5.3%	$40,000	$60,000	$91,000	117
ENGLISH LANGUAGE AND LITERATURE	6.7%	$32,000	$48,000	$75,000	11
ENVIRONMENTAL ENGINEERING	2.2%	$54,000	$67,000	$90,000	144
ENVIRONMENTAL SCIENCE	5.0%	$40,000	$52,000	$76,000	60
FAMILY AND CONSUMER SCIENCES	5.1%	$30,000	$40,000	$58,000	29
FILM VIDEO AND PHOTOGRAPHIC ARTS	7.3%	$30,000	$45,000	$71,000	54
FINANCE	4.5%	$44,000	$65,000	$101,000	12
FINE ARTS	7.4%	$28,000	$44,000	$65,000	22
FOOD SCIENCE	6.9%	$34,000	$71,000	$101,000	129
FORESTRY	3.1%	$38,000	$50,000	$73,000	104
FRENCH GERMAN LATIN AND OTHER COMMON FOREIGN LANGUAGE STUDIES	5.9%	$32,000	$48,000	$67,000	43
GENERAL AGRICULTURE	3.0%	$28,000	$44,000	$68,000	71
GENERAL BUSINESS	5.3%	$38,000	$59,000	$91,000	2
GENERAL EDUCATION	4.2%	$31,000	$41,000	$53,000	9
GENERAL ENGINEERING	5.9%	$47,000	$73,000	$101,000	24
GENERAL MEDICAL AND HEALTH SERVICES	5.8%	$35,000	$50,000	$71,000	74
GENERAL SOCIAL SCIENCES	8.2%	$34,000	$50,000	$74,000	68
GENETICS	7.4%	$33,000	$71,000	$99,000	163
GEOGRAPHY	6.1%	$40,000	$54,000	$81,000	62
GEOLOGICAL AND GEOPHYSICAL ENGINEERING	0.0%	$56,000	$73,000	$101,000	166
GEOLOGY AND EARTH SCIENCE	5.7%	$41,000	$60,000	$93,000	73
GEOSCIENCES	3.2%	$36,000	$52,000	$71,000	153
HEALTH AND MEDICAL ADMINISTRATIVE SERVICES	4.3%	$36,000	$51,000	$76,000	63
HEALTH AND MEDICAL PREPARATORY PROGRAMS	5.2%	$40,000	$60,000	$86,000	130
HISTORY	6.5%	$34,000	$50,000	$81,000	18
HOSPITALITY MANAGEMENT	5.8%	$32,000	$49,000	$71,000	38
HUMAN RESOURCES AND PERSONNEL MANAGEMENT	6.6%	$40,000	$55,000	$85,000	40
HUMAN SERVICES AND COMMUNITY ORGANIZATION	6.9%	$29,000	$38,000	$50,000	77
HUMANITIES	8.4%	$30,000	$45,000	$62,000	118
INDUSTRIAL AND MANUFACTURING ENGINEERING	5.6%	$50,000	$75,000	$100,000	59
INDUSTRIAL AND ORGANIZATIONAL PSYCHOLOGY	10.4%	$45,000	$62,000	$81,000	135
INDUSTRIAL PRODUCTION TECHNOLOGIES	3.1%	$46,000	$67,000	$91,000	82
INFORMATION SCIENCES	5.9%	$48,000	$71,000	$95,000	69
INTERCULTURAL AND INTERNATIONAL STUDIES	6.6%	$32,000	$50,000	$71,000	100
INTERDISCIPLINARY SOCIAL SCIENCES	6.3%	$31,000	$40,000	$50,000	96
INTERNATIONAL BUSINESS	8.5%	$38,000	$52,000	$87,000	72
INTERNATIONAL RELATIONS	5.8%	$40,000	$57,000	$93,000	79
JOURNALISM	7.0%	$34,000	$50,000	$79,000	25
LANGUAGE AND DRAMA EDUCATION	5.0%	$32,000	$41,000	$50,000	58
LIBERAL ARTS	7.6%	$32,000	$48,000	$71,000	20
LIBRARY SCIENCE	15.0%	$23,000	$36,000	$49,000	159
LINGUISTICS AND COMPARATIVE LANGUAGE AND LITERATURE	10.2%	$30,000	$44,000	$70,000	90
MANAGEMENT INFORMATION SYSTEMS AND STATISTICS	4.2%	$47,000	$71,000	$96,000	44
MARKETING AND MARKETING RESEARCH	5.9%	$40,000	$59,000	$90,000	6
MASS MEDIA	6.9%	$32,000	$46,000	$69,000	35
MATERIALS ENGINEERING AND MATERIALS SCIENCE	7.7%	$57,000	$84,000	$105,000	136
MATERIALS SCIENCE	4.7%	$65,000	$81,000	$106,000	161
MATHEMATICS	5.0%	$42,000	$63,000	$95,000	28
MATHEMATICS AND COMPUTER SCIENCE	3.5%	$55,000	$91,000	$151,000	158
MATHEMATICS TEACHER EDUCATION	3.4%	$34,000	$42,000	$56,000	108
MECHANICAL ENGINEERING	3.8%	$60,000	$81,000	$106,000	23
MECHANICAL ENGINEERING RELATED TECHNOLOGIES	6.6%	$38,000	$65,000	$87,000	123
MEDICAL ASSISTING SERVICES	2.9%	$34,000	$51,000	$71,000	95
MEDICAL TECHNOLOGIES TECHNICIANS	1.4%	$44,000	$58,000	$74,000	51
METALLURGICAL ENGINEERING	3.9%	$50,000	$86,000	$110,000	152
MICROBIOLOGY	5.2%	$40,000	$60,000	$86,000	94
MILITARY TECHNOLOGIES	10.9%	$81,000	$86,000	$126,000	173
MINING AND MINERAL ENGINEERING	4.3%	$71,000	$101,000	$121,000	162
MISCELLANEOUS AGRICULTURE	3.7%	$31,000	$46,000	$67,000	160
MISCELLANEOUS BIOLOGY	5.3%	$31,000	$50,000	$69,000	125
MISCELLANEOUS BUSINESS & MEDICAL ADMINISTRATION	5.3%	$35,000	$52,000	$81,000	64
MISCELLANEOUS EDUCATION	3.7%	$33,000	$46,000	$65,000	61
MISCELLANEOUS ENGINEERING	7.4%	$42,000	$71,000	$91,000	106
MISCELLANEOUS ENGINEERING TECHNOLOGIES	6.0%	$45,000	$65,000	$91,000	88
MISCELLANEOUS FINE ARTS	16.2%	$26,000	$40,000	$49,000	164
MISCELLANEOUS HEALTH MEDICAL PROFESSIONS	3.3%	$35,000	$45,000	$62,000	93
MISCELLANEOUS PSYCHOLOGY	10.3%	$30,000	$45,000	$71,000	120
MISCELLANEOUS SOCIAL SCIENCES	3.8%	$38,000	$52,000	$85,000	143
MOLECULAR BIOLOGY	5.3%	$32,000	$50,000	$76,000	124
MULTI-DISCIPLINARY OR GENERAL SCIENCE	4.6%	$36,000	$55,000	$81,000	26
MULTI/INTERDISCIPLINARY STUDIES	5.5%	$34,000	$42,000	$50,000	107
MUSIC	5.2%	$30,000	$45,000	$67,000	37
NATURAL RESOURCES MANAGEMENT	6.9%	$36,000	$50,000	$71,000	78
NAVAL ARCHITECTURE AND MARINE ENGINEERING	1.7%	$60,000	$96,000	$117,000	145
NEUROSCIENCE	7.2%	$34,000	$52,000	$76,000	154
NUCLEAR ENGINEERING	4.1%	$65,000	$96,000	$138,000	149
NUCLEAR INDUSTRIAL RADIOLOGY AND BIOLOGICAL TECHNOLOGIES	2.2%	$47,000	$64,000	$81,000	142
NURSING	2.2%	$48,000	$60,000	$80,000	4
NUTRITION SCIENCES	6.4%	$35,000	$51,000	$71,000	101
OCEANOGRAPHY	3.3%	$40,000	$50,000	$79,000	148
OPERATIONS LOGISTICS AND E-COMMERCE	4.7%	$45,000	$65,000	$97,000	102
OTHER FOREIGN LANGUAGES	6.4%	$32,000	$45,000	$76,000	111
PETROLEUM ENGINEERING	4.4%	$83,000	$127,000	$178,000	138
PHARMACOLOGY	0.0%	$48,000	$60,000	$101,000	169
PHARMACY PHARMACEUTICAL SCIENCES AND ADMINISTRATION	3.2%	$78,000	$105,000	$121,000	53
PHILOSOPHY AND RELIGIOUS STUDIES	7.2%	$30,000	$42,000	$65,000	42
PHYSICAL AND HEALTH EDUCATION TEACHING	4.5%	$34,000	$46,000	$59,000	39
PHYSICAL FITNESS PARKS RECREATION AND LEISURE	4.8%	$33,000	$45,000	$61,000	27
PHYSICAL SCIENCES	2.5%	$36,000	$51,000	$68,000	157
PHYSICS	4.5%	$39,000	$68,000	$101,000	70
PHYSIOLOGY	4.6%	$30,000	$48,000	$68,000	113
PLANT SCIENCE AND AGRONOMY	2.7%	$28,000	$42,000	$71,000	85
POLITICAL SCIENCE AND GOVERNMENT	6.0%	$38,000	$57,000	$91,000	15
PRE-LAW AND LEGAL STUDIES	7.9%	$32,000	$45,000	$69,000	91
PSYCHOLOGY	6.1%	$30,000	$43,000	$65,000	5
PUBLIC ADMINISTRATION	6.9%	$36,000	$50,000	$78,000	112
PUBLIC POLICY	2.2%	$47,000	$65,000	$101,000	141
SCHOOL STUDENT COUNSELING	0.0%	$18,000	$20,000	$42,000	172
SCIENCE AND COMPUTER TEACHER EDUCATION	5.0%	$36,000	$47,000	$58,000	116
SECONDARY TEACHER EDUCATION	3.8%	$35,000	$43,000	$59,000	57
SOCIAL PSYCHOLOGY	8.8%	$32,000	$45,000	$60,000	155
SOCIAL SCIENCE OR HISTORY TEACHER EDUCATION	3.0%	$35,000	$45,000	$58,000	83
SOCIAL WORK	6.8%	$30,000	$39,000	$51,000	30
SOCIOLOGY	7.0%	$33,000	$45,000	$67,000	19
SOIL SCIENCE	4.9%	$43,000	$64,000	$81,000	165
SPECIAL NEEDS EDUCATION	3.6%	$34,000	$42,000	$50,000	52
STATISTICS AND DECISION SCIENCE	6.9%	$50,000	$76,000	$108,000	128
STUDIO ARTS	8.0%	$25,000	$37,000	$57,000	84
TEACHER EDUCATION: MULTIPLE LEVELS	1.1%	$30,000	$38,000	$48,000	86
THEOLOGY AND RELIGIOUS VOCATIONS	4.1%	$25,000	$38,000	$54,000	46
TRANSPORTATION SCIENCES AND TECHNOLOGIES	4.4%	$42,000	$68,000	$98,000	56
TREATMENT THERAPY PROFESSIONS	2.6%	$40,000	$62,000	$81,000	34
UNITED STATES HISTORY	15.1%	$30,000	$50,000	$96,000	139
VISUAL AND PERFORMING ARTS	9.2%	$20,000	$36,000	$52,000	103
ZOOLOGY	6.7%	$33,000	$55,000	$81,000	119	

I was intrigued by the FT infographic on cell phone patent suits and decided to reformat it with F-R layout to get a big picture. A few things leap out. First, there is some pretty serious reciprocity (aka, counter-suits) going on, especially with Apple. On the other hand, Microsoft seems to be pretty good at attacking people who aren’t in a position to fight back. (*cough*trolls*cough*). Second, Google is at the periphery of the network which is pretty strange since many of these suits are actually about Android. This highlights the litigation strategy of picking off the weak members of the Android herd rather than taking the fight directly to Google itself. Furthermore, it suggests a data issue that there are omitted ties from the network, specifically positive ties between firms in the form of alliances (especially the Open Handset Alliance), and that reading the graph without these positive ties is misleading.

Anyway, here’s the graph. Click on it for a scalable PDF. Click here for the data in “.net” format. Code to produce the graph is below the fold.

setwd("~/Documents/codeandculture/patents/")
library(igraph)
patents <- read.graph("patents.net", c("pajek"))
V(patents)$label <- V(patents)$id #attaches labels

pdf("patents.pdf")
la = layout.fruchterman.reingold(patents)  #create layout for use on several related graphs
plot.igraph(patents, layout=la, vertex.size=1, vertex.label.cex=0.5, vertex.label.color="darkred", vertex.label.font=2, vertex.color="white", vertex.frame.color="NA", edge.color="gray70", edge.arrow.size=0.5, margin=0)
dev.off()
png("patents.png")
plot.igraph(patents, layout=la, vertex.size=1, vertex.label.cex=0.5, vertex.label.color="darkred", vertex.label.font=2, vertex.color="white", vertex.frame.color="NA", edge.color="gray70", edge.arrow.size=0.5, margin=0)
dev.off()

#have a nice day

USC Annenberg posted video of my talk in which I discuss the genre chapter of Climbing the Charts. In the chapter/talk I discuss how genre conventions structure diffusion, using as examples crossover between radio formats and the institutionalization of reggaetón with the growth of the “hurban” format.