Memetracker into Stata

| Gabriel |

A few months ago I mentioned the Memetracker project to scrape the internet and look for the diffusion of (various variants of) catchphrases. I wanted to play with the dataset but there were a few tricks. First, the dataset is really, really, big. The summary file is 862 megabytes when stored as text and would no doubt be bigger in Stata (because of how Stata allocates memory to string variables). Second, the data is in a moderately complicated hierarchical format, with “C” specific occurrences, nested within “B” phrase variants, which are in turn nested within “A” phrase families. You can immediately identify whether a row is A, B, or C by the numer of leading tabs (0, 1, and 2, respectively).

I figured that the best way to interpret this data in Stata would be two create two flat-files, one a record of all the “A” records that I call “key”, and the other a simplified version of all the “C” records but with the key variable to allow merging with the “A” records. Rather than do this all in Stata, I figured it would be good to pre-process it in perl, which reads text one line at a time and thus is well-suited for handling very large files. The easy part was to make a first pass through the file with grep to create the “key” file by copying all the “A” rows (i.e., those with no leading tabs).

Slightly harder was to cull the “C” rows. If I just wanted the “C” rows this would be easy, but I wanted to associate them with the cluster key variable from the “A” rows. This required looking for “A” rows, copying the key, and keeping it in memory until the next “A” row. Meanwhile, every time I hit a “C” row, I copy it but add in the key variable from the most recent “A” row. Both for debugging and because I get nervous when a program doesn’t give any output for several minutes, I have it print to screen every new “A” key. Finally, to keep the file size down, I set a floor to eliminate reasonably rare phrase clusters (anything with less than 500 occurrences total).

At that point I had two text files, “key” which associates the phrase cluster serial number with the actual phrase string and “data” which records occurrences of the phrases. The reason I didn’t merge them is that it would massively bloat the file size and it’s not necessary for analytic purposes. Anyway, at this point I could easily get both the key and data files into Stata and do whatever I want with them. As a first pass, I graphed the time-series for each catchphrase, with and without special attention drawn to mentions occurring in the top 10 news websites.

Here’s a sample graph.

Here’s the perl file:

#!/usr/bin/perl
#mt_clean.pl by ghr
#this script cleans the memetracker.org "phrase cluster" data
#http://snap.stanford.edu/data/d/quotes/Old-UniqUrls/clust-qt08080902w3mfq5.txt.gz
#script takes the (local and unzipped) location of this file as an argument
#throws out much of the data, saves as two tab flatfiles
#"key.txt" which associates cluster IDs with phrases
#"data.txt" which contains individual observations of the phrases
# input
# A:  <ClSz>  <TotFq>  <Root>  <ClId>
# B:          <QtFq>   <Urls>  <QtStr>  <QtId>
# C:                   <Tm>    <Fq>     <UrlTy>  <Url>
# output, key file
# A:  <ClSz>  <TotFq>  <Root>  <ClId>
# output, data file
# C:<ClID>	<Tm>	<UrlTy>	<URL>
# make two passes.

use warnings; use strict;
die "usage: mt_clean.pl <phrase cluster data>\n" unless @ARGV==1;

#define minimum number of occurences a phrase must have
my $minfreq = 500;

my $rawdata = shift(@ARGV);
# use bash grep to write out the "key file"
system("grep '^[0-9]' $rawdata > key.txt");

# read again, and write out the "data file"
# if line=A, redefine the "clid" variable
# optional, if second field of "A" is too small, (eg, below 100), break the loop?
# if line=B, skip
# if line=C, write out with "clid" in front
my $clid  ;
open(IN, "<$rawdata") or die "error opening $rawdata for reading\n";
open(OUT, ">data.txt") or die "error creating data.txt\n";
print OUT "clid\ttm\turlty\turl\n";
while (<IN>) {
	#match "A" lines by looking for numbers in field 0
	if($_=~ /^\d/) {
		my @fields = split("\t", $_); #parse as tab-delimited text
		if($fields[1] < $minfreq) { last;} #quit when you get to a rare phrase
		$clid = $fields[3]; #record the ClID
		$clid =~ s/\015?\012//; #manual chomp
		print "$clid ";
	}
	#match "C" lines, write out with clid
	if ($_ =~ m/^\t\t/) {
		chomp;
		my @fields = split("\t", $_);
		print OUT "$clid\t$fields[2]\t$fields[4]\t$fields[5]\n";
	}
}
close IN;
close OUT;
print "\ndone\n";

And here’s the Stata file:

clear
set mem 500m
set more off
cd ~/Documents/Sjt/memetracker/
*import key, or "A" records
insheet using key.txt, clear
ren v1 clsz
ren v2 totfq
ren v3 root
ren v4 clid
sort clid
lab var clsz "cluster size, n phrases"
lab var totfq "total frequency"
lab var root "phrase"
lab var clid "cluster id"
compress
save key, replace
*import data, or "C" records
insheet using data.txt, clear
drop if clid==.
gen double timestamp=clock(tm,"YMDhms")
format timestamp %tc
drop tm
gen hostname=regexs(1) if regexm(url, "http://([^/]+)") /*get the website, leaving out the filepath*/
drop url
gen blog=0
replace blog=1 if urlty=="B"
replace blog=1 if hostname=="blog.myspace.com"
gen technoratitop10=0 /*note, as of 2/3/2010, some mismatch with late 2008 memetracker data*/
foreach site in huffingtonpost.com engadget.com gizmodo.com mashable.com techcrunch.com boingboing.net gawker.com corner.nationalreview.com thedailybeast.com tmz.com {
	replace technoratitop10=1 if hostname=="`site'"
}
gen alexanews10=0 /*as w technorati, anachronistic*/
foreach site in news.yahoo.com bbc.co.uk cnn.com news.bbc.co.uk news.google.com nytimes.com msnbc.msn.com foxnews.com {
	replace alexanews10=1 if hostname=="`site'"
}
drop urlty
sort clid timestamp
contract _all /*eliminate redundant "C" records (from different "B" branches)*/
drop _freq
save data, replace
*draw a graph of each meme's occurrences
levelsof clid, local(clidvalues)
foreach clid in `clidvalues' {
	disp "`clid'"
	quietly use key, clear
	quietly keep if clid==`clid'
	local title=root in 1
	quietly use data, clear
	histogram timestamp if clid==`clid', frequency xlabel(#5, labsize(small) angle(forty_five)) title(`title', size(medsmall))
	graph export graphs/`clid'.png, replace
	twoway (histogram timestamp if clid==`clid') (line alexanews10 timestamp if clid==`clid', yaxis(2)), legend(off) xlabel(#5, labsize(small) angle(forty_five)) title(`title', size(medsmall))
	graph export graphs_alexa/`clid'.png, replace
}
*have a nice day

Pete

| Gabriel |

The culture section listserv sent out an announcement this morning that Richard Peterson died yesterday. Pete of course was probably the single most important figure in laying out the production of culture paradigm in the mid-1970s as a process-oriented alternative to the functionalist and Marxist approaches that predominated until then. At a personal level, like more than a few production of culture scholars I knew Pete well and co-authored with him (and in fact still have an incomplete manuscript with him on the omnivore hypothesis that he always optimistically referred to as “the ASR version” as compared the “book chapter” version we published last year). Aside from my advisor, he was my biggest mentor when I was in grad school and he wasn’t even at my institution. Working with Pete ultimately resulted in both good work and mentorship more broadly but could be frustrating at times. For instance, he was so curious about what was in the data and how it spoke to various (orthogonal) theoretical and substantive problems that he’d ask for dozens of cross-tabs that had no chance of ending up in the paper. His personality was cynical in principal but generous in practice. Ultimately what counts from the perspective of the discipline is that it remains popular to analyze culture using the tools of economic sociology, both among the many people who worked with him and also among those who did not. From the perspective of humanity it’s probably more important that he was a good family man, always telling you about his grown children’s accomplishments and inviting you to have lunch with him and Claire.

Soc of Mass Media, week 5

| Gabriel |

Monday’s lecture was on the Peterson + Berger model of creative destruction in recorded music. There’s no Wednesday lecture because I’m giving the midterm (hint: the answer to the second question is “just because”).

Next week is artistic careers, including starving artists, Baumol’s disease, deHavilland, and Galenson’s experimentalist/conceptualist dichotomy.

For your consideration

| Gabriel |

So the Oscar nominees were announced at dawn this morning and half of the ten best picture nominees had done appreciable box office. Likewise, many of the other categories had nominees from self-important movies that nobody saw like “Invictus.” Of course, the reason we have ten best picture nominees is that the Academy realized that the last few years’ Oscars were of no interest to the average moviegoer because the total box office of all the nominees combined was less than the Kelly Blue Book value of a three year old Honda Civic. You could just see the Academy board of governors thinking, “If only we had nominated ‘The Dark Knight,’ people might have tuned in to see the vaudevillian razzamatazz of Hugh Jackman!” Of course, there are two reasons why Dark Knight wasn’t nominated. One is that “Oscar bait” now has highly stylized genre conventions (like close-ups of Sean Penn looking constipated) which the Dark Knight didn’t meet (it was more interested in being watchable).

The other is that Dark Knight didn’t really need an Oscar nomination as it already made plenty of money and had good word of mouth. In contrast, your classic Oscar bait movie extends its theatrical run and/or sells more dvds after a nomination. Oscars are an essential resource for Oscar bait films, which nobody will watch until they are consecrated. When something is valuable, people tend to pursue it, and hence we have the aggressive “for your consideration” Oscar campaigning, as discussed at length a few weeks ago on The Business.

According to industry legend, this began in a really serious way when Miramax elbowed its way to a best picture for “Shakespeare in Love,” displacing the obvious favorite of “Saving Private Ryan.” I realized that I could test this by looking at my data to see signs of the development of Oscar performativity. One of the clearest examples of the film industry organizing itself around the Oscars is the concentration of Oscar bait in December. Late released films are both more salient to the Oscar voters and thus more likely to be nominated and still in theaters in February and thus more likely to be able to exploit nomination. By looking at the interaction between decade of release and day of the year, we can thus track the development of Oscar performativity. As seen below, there was basically no Oscar performativity in the late 30s and early 40s. From the late 40s through the early 90s there was a steady but low level of performativity. Then in the late 90s and early aughts the performativity gets really strong.

So basically, it looks like it really was “Shakespeare in Love” that set the precedent for all the artsy movies being crammed into December. Thanks Bob and Harvey!

Here’s the code:

set matsize 1000
gen decade=year
recode decade 1900/1935=. 1936/1945=1 1946/1955=2 1956/1965=3 1966/1975=4 1976/1985=5 1986/1995=6 1996/2005=7
capture lab drop decade
lab def decade 1 "1936-1945" 2 "1946-1955" 3 "1956-1965" 4 "1966-1975" 5 "1976-1985" 6 "1986-1995" 7 "1996-2005"
lab val decade decade
replace date=round(date)
logit actor_nom female major FPY g_drama centrality pWnom pDnom c.date##decade
esttab, se
*have a nice day

Here are the results.

actor_nom
female              0.924***
                 (0.0570)

major               0.538***
                 (0.0841)

FPY00              -0.326***
                 (0.0555)

g_drama             1.814***
                 (0.0892)

centrality         0.0635***
                (0.00363)

pWnom               0.253***
                 (0.0423)

pDnom               1.026***
                 (0.0609)

date              0.00185**
               (0.000691)

1b.decade               0
                      (.)

2.decade           -0.597*
                  (0.266)

3.decade           -0.450
                  (0.275)

4.decade          -0.0529
                  (0.269)

5.decade           -0.220
                  (0.274)

6.decade           -1.051***
                  (0.302)

7.decade           -1.994***
                  (0.354)

1b.decade#~e            0
                      (.)

2.decade#c~e      0.00362***
                (0.00104)

3.decade#c~e      0.00442***
                (0.00107)

4.decade#c~e      0.00325**
                (0.00104)

5.decade#c~e      0.00348***
                (0.00104)

6.decade#c~e      0.00465***
                (0.00112)

7.decade#c~e      0.00766***
                (0.00125)

_cons              -12.43***
                  (0.412)
----------------------------
N                  147908
----------------------------
Standard errors in parentheses
* p<0.05, ** p<0.01, *** p<0.001

Shufflevar [update]

| Gabriel |

I wrote a more flexible version of shufflevar (here’s the old version) and an accompanying help file. The new version allows shuffling an entire varlist (rather than just one variable) jointly or independently and shuffling within clusters. The easiest way to install the command is to type:
ssc install shufflevar
For thoughts on this and related algorithms, see my original shufflevar post and/or the lecture notes on bootstrapping for my grad stats class.
Here’s the code.

*1.0 GHR January 29, 2010
capture program drop shufflevar
program define shufflevar
	version 10
	syntax varlist(min=1) [ , Joint DROPold cluster(varname)]
	tempvar oldsortorder
	gen `oldsortorder'=[_n]
	if "`joint'"=="joint" {
		tempvar newsortorder
		gen `newsortorder'=uniform()
		sort `cluster' `newsortorder'
		foreach var in `varlist' {
			capture drop `var'_shuffled
			quietly gen `var'_shuffled=`var'[_n-1]
			quietly replace `var'_shuffled=`var'[_N] in 1/1
			if "`dropold'"=="dropold" {
				drop `var'
			}
		}
		sort `oldsortorder'
		drop `newsortorder' `oldsortorder'
	}
	else {
		foreach var in `varlist' {
			tempvar newsortorder
			gen `newsortorder'=uniform()
			sort `cluster' `newsortorder'
			capture drop `var'_shuffled
			quietly gen `var'_shuffled=`var'[_n-1]
			quietly replace `var'_shuffled=`var'[_N] in 1/1
			drop `newsortorder'
			if "`dropold'"=="dropold" {
				drop `var'
			}
		}
		sort `oldsortorder'
		drop `oldsortorder'
	}
end

And here’s the help file.

{smcl}
{* 29jan2010}{...}
{hline}
help for {hi:shufflevar}
{hline}

{title:Randomly shuffle variables}

{p 8 17 2}
{cmd:shufflevar} {it:varlist}[, {cmdab: Joint DROPold cluster}({it:varname})]

{title:Description}

{p 4 4 2}
{cmd:shufflevar} takes {it:varlist} and either jointly or for each variable
shuffles {it:varlist} relative to the rest of the dataset. This means any
association between {it:varlist} and the rest of the dataset will be random.
Much like {help bootstrap} or the Quadratic Assignment Procedure (QAP), one
can build a distribution of results out of randomness to serve as a baseline
against which to compare empirical results, especially for overall model-fit
or clustering measures.

{title:Remarks}

{p 4 4 2}
The program is intended for situations where it is hard to model error
formally, either because the parameter is exotic or because the application
violates the parameter's assumptions. For instance, the algorithm has been
used by Fernandez et. al. and Zuckerman to interpret network data, the author
wrote this implementation for use in interpreting {help st} frailty models
with widely varying cluster sizes, and others have suggested using the metric
for adjacency matrices in spatial analysis.

{p 4 4 2}
Much like {help bsample}, the {cmd:shufflevar} command is only really useful
when worked into a {help forvalues} loop or {help program} that records the
results of each iteration using {help postfile}. See the example code below to
see how to construct the loop.

{p 4 4 2}
To avoid confusion with the actual data, the shuffled variables are renamed
{it:varname}_shuffled.

{p 4 4 2}
This command is an implementation of an algorithm used in two papers that used
it to measure network issues:

{p 4 4 2}
Fernandez, Roberto M., Emilio J. Castilla, and Paul Moore. 2000. "Social
Capital at Work: Networks and Employment at a Phone Center." {it:American Journal of Sociology} 105:1288-1356.

{p 4 4 2}
Zuckerman, Ezra W. 2005. "Typecasting and Generalism in Firm and Market: Career-Based Career Concentration in the Feature Film Industry, 1935-1995." {it:Research in the Sociology of Organizations} 23:173-216.

{title:Options}

{p 4 8 2}
{cmd:joint} specifies that {it:varlist} will be keep their actual relations to
one another even as they are shuffled relative to the rest of the variables.
If {cmd:joint} is omitted, each variable in the {it:varlist} will be
shuffled separately.

{p 4 8 2}
{cmd:dropold} specifies that the original sort order versions of {it:varlist}
will be dropped.

{p 4 8 2}
{cmd:cluster}({it:varname}) specifies that shuffling will occur by {it:varname}.

{title:Examples}

{p 4 8 2}{cmd:. sysuse auto, clear}{p_end}
{p 4 8 2}{cmd:. regress price weight}{p_end}
{p 4 8 2}{cmd:. local obs_r2=`e(r2)'}{p_end}
{p 4 8 2}{cmd:. tempname memhold}{p_end}
{p 4 8 2}{cmd:. tempfile results}{p_end}
{p 4 8 2}{cmd:. postfile `memhold' r2 using "`results'"}{p_end}
{p 4 8 2}{cmd:. forvalues i=1/100 {c -(}}{p_end}
{p 4 8 2}{cmd:. 	shufflevar weight, cluster(foreign)}{p_end}
{p 4 8 2}{cmd:. 	quietly regress price weight_shuffled}{p_end}
{p 4 8 2}{cmd:. 	post `memhold' (`e(r2)')}{p_end}
{p 4 8 2}{cmd:. }}{p_end}
{p 4 8 2}{cmd:. postclose `memhold'}{p_end}
{p 4 8 2}{cmd:. use "`results'", clear}{p_end}
{p 4 8 2}{cmd:. sum r2}{p_end}
{p 4 8 2}{cmd:. disp "The observed R^2 of " `obs_r2' " is " (`obs_r2'-`r(mean)')/`r(sd)' " sigmas out on the" _newline "distribution of shuffled R^2s."}{p_end}

{title:Author}

{p 4 4 2}Gabriel Rossman, UCLA{break}
rossman@soc.ucla.edu

{title:Also see}

{p 4 13 2}On-line:
help for {help bsample},
help for {help forvalues},
help for {help postfile},
help for {help program}

Soc Mass Media, week 4

| Gabriel |

Monday’s lecture was about the business logic of large media corporations and Marxist hegemony theory. Wednesday’s lecture was about the Hotelling model of product variety and monopoly. Next week there’s only one lecture because I’m giving a midterm. The lecture will be on Peterson and Berger (and the Lopes/Dowd extensions of the same) on product variety and monopoly, which I make a case is in it’s mature form really a niche partitioning model with Schumpeterian twist.

Sorry for the light-posting for the last couple weeks. Between a really nasty and persistent cold, giving a guest lecture to our pro-seminar, and having to read 30 graduate applications a week I’ve been exhausted.

Soc of Mass Media, week 3

| Gabriel |

There was no lecture on Monday for MLK. Wednesday’s lecture was on media ownership policy. The next three lectures will be about the consequences of corporate media ownership.

Age, Period, Flim Flam

| Gabriel |

A few months ago, The Chronicle had a very interesting article on generation gurus, who claim insight into the “millenials,” or as actual social scientists boringly call them, “the 1980s and 1990s birth cohorts.” Lots of organizations, including college admissions boards, are really interested in these gurus’ advice on how to understand the kids these days. (Which reminds me of the obnoxious creative team of Smitty and Kurt, who were brought on to Sterling Cooper to sell Martinson’s coffee to the Pepsi generation).

I remember way back when I was in high school reading a long-form magazine article (The Atlantic?) on Howe and Strauss and I thought it was a great theory, in part because some of the details seemed like they were (or ought to be) true and in part because the generational dialectic struck me as plausible. Basically they say that idealistic generations are followed by cynics who in turn are followed by pragmatic workhorses who are in turn followed by idealists, with the mechanism being that each generation reacts against its parents’ excesses. According to this schema, the reason I grew up listening to Nirvana was as a reaction to the “All you need is love” stuff of the boomers.

When I got all growed up and actually started dealing with, you know, systematic data, I was more than a little disappointed that while cohort change is not always linear, it is basically monotonic and it is definitely not cyclical or dialectical. I’m primarily an orgs guy rather than a people guy, but I’ve still done some moderately extensive age/period/cohort stuff with the GSS and SPPA and on everything I looked at (mostly social attitudes and cultural consumption), there’s absolutely no evidence whatsoever for the Howe and Strauss dialectic. So for instance, if you look at strong preference for opera and classical you first have to limit the data to BA or higher education (less educated people don’t like this music regardless of cohort) and then you see a clear trend that the music is popular with educated people born before 1950 and unpopular with educated people born after 1950. There is no distinction between “boomers” and “gen X” in the data, and in fact older boomers are still into high culture. The only issue that I’m aware of that even vaguely approximates the Howe and Strauss model is abortion attitudes, but a) the cohort effects on abortion attitudes are weak and b) the effects of cohort on other sex/reproduction opinions, like gay marriage, are monotonic.

So given that the empirical evidence for these ideas is so weak, why are college administrators, marketers, etc, so into it? I think the answer has to be that it was facially plausible and more importantly that it was pretty clear. A money quote from the article is:

Amid this complexity, the Millennials message was not only comforting but empowering. “It tickled our ears,” says Palmer H. Muntz, director of admissions and an enrollment-management consultant at Lincoln Christian University, in Illinois. “It packaged today’s youth in a way that we really wanted to see them. It gave us a formula for understanding them.”

This is reminiscent of the argument that John Campbell gave for explaining the popularity of supply side economics. His argument is basically that the idea gained popularity not because it had especially powerful theory or empirics behind it, but because it was comprehensible and gave a tractable guide to action. In theory, Lafferism is contingent on the important question of where the current tax regime lies relative to the curve’s maximum, but in practice this contingency was elided and people took it to mean “always cut taxes.” That is the appeal of the idea was not so much that we had good reasons to think it reflected reality (or more specifically, that it was applicable to current circumstances), but because it clearly prescribed action — and I think it’s worth adding, actions that were desirable in a free lunch kind of way. In the same way, if you read Howe and Strauss, they are relentlessly positive about the millennials, portraying them as a dialectically-generated reproduction of the go-getters who first stormed the beaches of Normandy and then nested into Levittown. Victory, affluence, swing music, what’s not to love about these kids?

You can see similar wishful thinking in the eagerness of municipal officials to throw consulting contracts at Richard Florida. Florida’s basic shtick is that if Methenburg, PA wants to develop they should just rezone old warehouses and put up a sign reading “Methenburg Arts District,” this will attract artists, who in turn will attract engineers, who in turn will turn Methenburg into the next Silicon Valley. I always imagine after Florida gives his powerpoint, the city councilmen or county selectmen are enthusiastically coming up with ideas about how to be “cool” like Murray Hewitt on Flight of the Conchords. It sounds like a perfect plan: Methenburg get to be “cool,” we get development, and it doesn’t require either making expenditures or forgoing revenues to any appreciable extent.

If only it were true.

Misc links

• Rufus Pollock uses the occasion of Argentinian copyright extension to take extremely long copyright terms to the woodshed. He shows that once you consider the decay in demand for cultural products (most of which are essentially ephemera) and net present value discounting, the marginal incentive effects of further extending already long copyright terms are absolutely infinitesimal. The most plausible alternative explanation is that extremely long copyright terms represents public choice more than public policy. Although he doesn’t say so, I think one obvious way to demonstrate this is that the extension is retroactive. Even if another twenty years is basically worthless in prospect to a new market entrant, it’s worth a lot to those incumbents who own those few works that have proven to be of lasting appeal. Also worth noting is that Pollock posts his code.
• In oral arguments at the SCOTUS, a lawyer used the word “orthogonal.” Roberts and Scalia were fascinated by the word and seemed to want to make it the secret word of the day. I myself am fond of the word as it’s a pretty clear way to get across concepts like “lack of interaction effects,” which is a more subtle concept than merely “uncorrelated.” For instance, see my discussion of weighting and when it’s ok to omit controls.
• In another SCOTUS case, the NFL gave exclusive merchandise rights for every team in the league to Reebok. A small hat company is suing the NFL on antitrust grounds and so the court has to decide if the the teams are independent firms and the NFL is a trade group (the plaintiff’s theory) or the NFL is a firm and the teams are franchisees (the NFL’s theory). Given how expansion teams and things like that work, it seems the latter. There’s a related question about whether a “Rams” hat is in competition with a “Saints” hat (the plaintiff’s theory) or with a “Cardinals” hat (the NFL’s theory). Given that the price of NFL merchandise rose by 50% after the exclusive concession deal, it seems like the former is the obvious answer. Even though I’ve never watched a football game in my life, I still find this really interesting as it involves a lot of fairly subtle question about monopolistic competition, substitution/categorization, strong externalities, etc.

Soc of Mass Media, week 2

| Gabriel |

I now have a real page on ITunes U. I suggest this version as (unlike the Bruincast RSS), I’ve edited the meta-text to have informative “name” and “comments” fields.

On Monday’s lecture I talked about the superstar effect. On Wednesday’s lecture I talked about promotion, including payola. Next Monday is MLK day but on Wednesday I’m doing antitrust.