One should always play fairly when one has the winning cards: electoral viability and Twitter sentiment

With Jorge Fábrega, we we are currently working in a really interesting project to understand how public opinion surveys can be related to social media discussions. In a nutshell, we are interested in comparing public opinion data to what we can observe from Twitter. As a first step, we have prepared a draft chapter for an edited volume on digital methods for the social sciences, currently under review. The paper is still in a very early draft version, but I just wanted to share some of the early results.

The argument goes as follows. There is a great deal of research trying to use social media data (mainly Twitter) to forecast elections, and some research on how to compare Twitter to candidate’s support on opinion polls. In the case of the former, most attempts have failed, while on the latter,  Nick Beauchamp has managed to produce some interesting results. Our approach is slightly different: instead of looking at candidate support, we focus on support for certain policies. And instead of trying to predict support, we take one step back and think on other ways to relate both elements.

What came to our minds as a useful starting approach was to look at the sentiment on Twitter and how it relates to candidate support. We had two initial hypotheses:

  1. Supporters of candidates with higher electoral viability tend to tweet with a more positive tone overall.
  2. There is a correspondence between support for a policy and the tone used to tweet.

We used Chile as our case, and had two data sources: The CEP survey from October 2013, and Twitter data we collected previous to Chile’s presidential election last year.

Chile’s last election was quite competitive in terms of the number of candidates, but not in the vote share. President Bachelet had to go to a run-off election (something expectable with 9 candidates) against Evelyn Matthei, but she managed to get 62% of the votes on that round. A record share in the country’s recent history. In terms of electoral viability, 78% of the respondents of the CEP survey (2 months before the election) believed that Bachelet would win the presidency, with only 5% of people thinking that Matthei could get the job.

With that in mind, we processed our Twitter data to estimate two things: the political position of Twitter users in Chile, and the tone in which they tweet about 5 controversial topics:

  1. the possibility of constitutional reform through an assembly;
  2. a change in the current electoral system;
  3. the approval of an equal marriage law;
  4. abortion; and
  5. the ownership of the copper mines (they used to belong to the State).

To estimate the political position, we used retweet networks. In simple words, we setup an initial group of Twitter users with known preference for a candidates, and then performed some social network analysis of those who retweeted them. From there, we could move on to assign a probability for each user to support one of the candidates. We focused only on the four main candidates (Bachelet, Matthei, Enríquez-Ominami and Parisi), since they were the ones that generated more attention during the campaign.

To estimate the tone, we translated the lexicon from Wilson et al. and crafted our own version of the sentiment R package (used for sentiment analysis, of course). Each tweet was deconstructed into single words, which were then compared to the lexicon. Each word in the lexicon had assigned a polarity (positive or negative), and whenever there was a match between the words from the tweets and the lexicon, we assigned that polarity to the word. Then, the package uses the voter algorithm to determine that, given the number of matches, that tweet was itself positive or negative.

We first calculated the effect of candidate support on support for the policies mentioned above. our results (shown in the figure below) show that most respondents who supported one of the four candidates, in comparison to those who supported another or none, also support the policies. This was statistically significant for all the cases with a * in the figure.

Table_twt_cep

 

In the case of Twitter, since our sampling was not probabilistic, we were more concerned about the direction of the relationship rather than the significance. Our dependent variable was the likelihood of a tweet having a positive tone. The results can be seen in the figure, and the most surprising feature is the decrease in probability for a positive tone when talking about equal marriage.

As we expected, supporters of Bachelet had a higher tendency to tweet with a positive tone, overall. Also, whenever there is a statistically significant relationship between supporting Bachelet and supporting a policy (e.g. electoral reform, abortion or copper mines ownership) that had a correlate with an increase in the probability of a positive tone on Twitter.

However, our second hypothesis suffered from a different fate. As you can observe, there is some decoupling between the support for a policy and the likelihood of a positive tone. Let’s look at the case of supporters of Enríquez-Ominami. aAlthough they seem keen to support all of the reforms, the tone they use when they tweet is not positive in most cases. We see a similar behaviour in some of the supporters for other candidates as well

We are still in the stage of making sense of these results, and thinking about possible expansions of this research. In the meantime, this is a nice starting point, and provides some insightful information on how to use Twitter data, not to replace, but to complement classical public opinion research.

APSA 2013 Twitter report

 

Following last year’s humble attempt to provide some insight from the twitter conversations around #APSA2012 (specially considering the last minute cancellation of the conference) – and given that other duties restricted me from attending APSA this year 🙁 – I will be collecting and displaying some data from this year’s conversations. There will be more updates throughout the conference. If you want to follow the chronological reports, you need to start from bottom to top.

Short methods note: Edges are created by mentions, replies or re-tweets. Nodes are coloured according to the components, and their size is scaled according to eigenvector centrality. Isolates (ie. people not talking to anyone but using the hashtag #APSA2013) are not included.

BONUS TRACKS:

1. DATA: Someone asked me for the data I used to produce this post, and I strongly believe in the importance or replication. Here it is a list of all the tweet IDs I used. Sorry, but that’s the only way I can share it without violating Twitter’s TOS –> DATA

2. I plotted all the geotagged tweets against the map of Chicago. This gives a better sense of where the tweets where concentrated around the city.

chicago_plot

UPDATE 10 (AND FINAL): A few comments before I introduce the data. This exercise had two purposes. First, I wanted to freshen up my skills on Twitter data collection and analysis. After spending part of the summer learning a lot on Python, R and SNA (mainly thanks to the International Summer School 2013 “Social Network Analysis: Internet Research”), I decided that an extension of last year’s analysis on the APSA tweets would be a good opportunity. In total honesty, I hope you enjoyed it too. Second, my research agenda uses extensively this type of social media data to draw inferences about political behaviour. Although this particular exercise was extremely self-centred, since I’m focusing on the interactions in a Political Science conference, it provides some insight on what social media data can tell us, and how can we use it to make sense of bigger issues. That’s why I decided to write this other post on Obama’s speech this week, to show some “real life” examples. Also, I realise that I’m not new on this field, and that there are amazing people working on these issues for a long time (most of them with much more sophisticated analyses than mine). I believe in building community, so I tried to attribute their work where appropriate and link to their own websites and Twitter accounts. I extremely recommend you to follow them and their research. Finally, this post will eventually become a paper-like longer post, with more descriptive data and some interesting questions to test. I can’t promise when, but it will come.

Ok, now let’s go to the data analysis. Joshua Tucker (NYU) tweeted today his excitement for being in the “top ten vertices” list from a Twitter SNA made by Marc Smith, using NodeXL. I’ve used NodeXL in the past (and I believe is an amazing off-the-shelf tool for Windows user), but its reliance on the Search API made me realise that I could get better results by downloading the data via the streaming API for the full duration of the conference. It requires more time and resources, but the results are much more informative. Then, I decided to create my own top ten, but using eigenvector centrality instead of betweenness centrality (as in the NodeXL list). The reason is simple: the former relies on the relative importance of the connections of a node. That is, if the people I interact with are more “important” (or central) in the network, I become more important too. Betweenness centrality, on the other hand, focuses on who are the bridges across different nodes, who is more able to connect the rest. Although that is usually an important question in network analysis (actually, I co-authored a paper with Jorge Fábrega where we use it extensively), in substantive terms eigenvector centrality seems more appropriate for the type of network we have here. With that info in mind, here are the winners:

Table: Top 10 accounts according to their Eigenvector Centrality.

RANK DAY 1 DAY 2 DAY 3 DAY 4 OVERALL
1 @APSAtweets @andrew_chadwick @dandrezner @APSAtweets @APSAtweets
2 @texasinafrica @davekarpf @25lettori @dandrezner @dandrezner
3 @abuaardvark @rasmus_kleis @rasmus_kleis @mqsawyer @andrew_chadwick
4 @APSAMeetings @kreissdaniel @andrew_chadwick @LarrySabato @ezraklein
5 @dfreelon @OUPAcademic @APSAtweets @TerriGivens @texasinafrica
6 @CambridgeJnls @insidehighered @mikejjensen @insidehighered @APSAMeetings
7 @andrew_chadwick @FUNGLODE @ezraklein @j_a_tucker @abuaardvark
8 @ProfCaraJones @Worse_Reviewer @StephanieCarvin @monkeycageblog @davekarpf
9 @dandrezner @j_a_tucker @TerriGivens @washingtonpost @raulpacheco
10 @zizip @abuaardvark @raulpacheco @APSAMeetings @ProfCaraJones

In terms of volume, day 4 was the smallest one. With only 114 nodes and 141 edges, the conversations were less frequent. A possible explanation is that most of the delegates had already gone by then, and only those who had panels on that day were staying around the conference venues. The clusters are a bit more institutional, with high prominence form APSA’s official accounts, along with some blogs and websites (such as @monkeycageblog and @insidehighered). A new addition is Larry Sabato, from U. Virginia

day4

 

The cumulative network does not show many differences from yesterday. This is not surprising, because most of the activity took place before, and most of the communications were between people who already tweeted each other before. The new interactions might have added some weight to the already existing edges, but not much more. In any case, here is the final network of the APSA 2013 Annual Meeting:

APSA 2013 final graph - 868 nodes and 1794 edges.
APSA 2013 final graph – 868 nodes and 1794 edges.

UPDATE 9: Day 3 was clearly quieter than the precious two. A bit of it might be the classical effect of people leaving after they present, or simply wandering around Chicago. It might also be that the panels are becoming increasingly more interesting, and people prefer to pay attention to the presentations instead of tweeting ;). In any case with all the fuss around President Obama’s speech on Syria (Hint: I recently published a quick report on that), I was expecting that IR crowd attending the conference would be very active. Well, just by simple observation of their accounts, they were, but did not necessarily use the #apsa2013 hashtag to express their views. That said, @dandrezner and @ezraklein are some of the “stars” of today’s network, with a high level of eigenvector centrality. The Political Communications cluster remains active with @andrew_chadwick, @rasmus_kleiss, and @25lettori leading the way (clearly a clique around Royal Holloway’s New Political Communication Unit).

Day r of #APSA2013. 253 nodes and 325 edges.
Day 3 of #APSA2013. 253 nodes and 325 edges.

Moving on to the cumulative graph, the network is not becoming much bigger (832 nodes in total). This reflects the lower number of conversations from day 3, but also that some ties are already established and some people keep talking to each other. The APSA team is doing really well in driving the conversation, with @APSAtweets and @APSAmeetings as really central nodes in the network. As expected, those who were central yesterday, remain so today, so no news on that regard. All in all, the network seems to be coming to a point of “convergence” or “stability”, with conversations taking place among the same members and with no significant cliques outside the big group. The question of inter-field dialogue remains open, as some relevant nodes in the network belong to different components (such as @ezraklein in comparison with the rest of the bigger component).

Cumulative network at day3. 832 nodes and 1667 edges
Cumulative network at day3. 832 nodes and 1667 edges

 

(QUICK) UPDATE 8: Using Pablo Barberá’s StreamR package (along with ggplot2), I mapped the tweets that had location data in them (only 19 out of 1321). Not surprisingly, most of them are highly concentrated in Chicago, but a couple appear to be somewhere else in the US. This goes towards question whether people not attending the conference are getting any benefit by tweeting about it. There were no geolocated tweets outside the US, in case you were wondering.

Geolocated tweets in red.
Geolocated tweets in red.

UPDATE 7: This is the final summary of day 2. The next 2 days I aim to produce just one daily report, so you’ll have to bear with me. Again, I present two graphs. The first one is the full network for all the days of the conference (including pre-conference events). The second one contains all tweets captured at day 2 until 7.30pm.

The cumulative network shows again a big component in pink, but the network is becoming much more diverse than in previous iterations. More clusters appear, while others that were disconnected (such as the one lead by @funglode) are now connected to the bigger network. The usual suspects remain as key actors in the network, and depending on the volume of tweets over the weekend, they will probably remain in that position. Some well-known IR scholars do not belong to the bigger component, which is an interesting phenomenon. If we look at @ezraklein or @SlaughterAM, they are connected to the big network, but form clusters around them (perhaps the cross-field conversations are not as clear as I thought) The Political Communications group is highly active, especially @andrew_chadwick@zizip and @davekarpf (who also shared a widely tweeted panel today, which might also account for their relevance in the network).

An important notice is that this exercise is, in some way, a performative process. While I publish these networks, some people become aware of their own position and the people they interact with. That is always something to take into consideration when doing the analysis, which brings some epistemological discussions to the table (this is like Schrödinger’s cat reporting on its own experiment).

Cumulative network at August 30, 7.30pm. 704 nodes and 1384 edges
Cumulative network at August 30, 7.30pm. 704 nodes and 1384 edges
Network for day 2 (30 August). 375 nodes and 635 edges.
Network for day 2 (30 August). 375 nodes and 635 edges.

 

UPDATE 6: This time I’m bringing two graphs. The first one corresponds to the cumulative network. That is, the Twitter conversations from the pre-conference events until the last update. The second graph corresponds only to the conversations taking place during day 2 until 1pm CT. As you can notice, there are similarities among the networks, such as the existence of a big component in the middle (the cumulative network uses strongly connected components to colour the nodes). However, the central actors vary a bit. There are some accounts that remain relevant and central to the network, such as @apsatweets@dandrezner@texasinafrica@raulpacheco@ezraklein and @j_a_tucker. However, we can observe some new actors coming into the scene, such as @heathbrown and the institutional account for @insidehighered. Also, there is an interesting cluster formed by @funglode and @anniavaldez, formed mainly by Spanish-speaking users.

The field boundaries seem more diffused now, which brings questions about whether conferences actually create the opportunity for cross-field dialogue. There are several panels trying to analyse the overall role of Political Science, and how can we communicate better with our audiences. Maybe that is driving a lot of the conversations. That’s an interesting hypothesis to test. Another interesting fact is that some central nodes are people who are not attending APSA this year (such as myself :)). This also brings a question about who benefits from the conference, and if it is necessary to attend to obtain some basic returns from it. Obviously, we need to get data from other sources outside Twitter to find that out. In the meantime, this has become more than a simple exercise of mapping APSA.

Cumulative network from the pre-conference events until August 30 at 1pm. 603 nodes and 1113 edges.
Cumulative network from the pre-conference events until August 30 at 1pm. 603 nodes and 1113 edges.
Network for day 2 (August 30) at 1pm. 223 nodes and 323 edges.
Network for day 2 (August 30) at 1pm. 223 nodes and 323 edges.

UPDATE 5: This graph is a lot bigger than the previous one, as it brings together the data form the pre-conference events plus the day 1 (August, 29). Thanks again to @jorgefabrega for the help using the Search API to retrieve that data (I know, the search API might not be the best option to get an accurate picture, but it’s the only one I had available. If you want a thorough discussion of the representativeness of the different Twitter APIs – mainly the Streaming API – I would definitely encourage you to look at Mostatter et al. 2013)

Back to business. I made some small changes to the visualisation this time. I used strongly connected components instead of weakly connected components. First, it made more sense since the network is directed. Second, with the weakly connected component we got a big group in the middle where almost everyone was connected, which is not true. Also, one of my goals is to analyse the networks and try to make a comparison by sections/fields affiliation (if anyone is interested in helping with that, please let me know in the comments section!). This time we have 479 nodes and 823 edges.

I’m currently collecting data from today’s sessions, and will provide a daily graph and an accumulated one. Let’s see how that works. As usual, feedback is more than welcome.

day0_1

UPDATE 4: Last graph of the day (it’s pretty late here in London). This corresponds to an accumulated network of the entire first day of #APSA2013 until 7pm, Chicago time. Now the network is much bigger than the previous one (it seems that conversations take some time to build up) with 327 nodes and 489 edges. The clusters we saw in the previous graph are much more diffused now. We can observe a big central component (in green) that connects most members of the network. However, it is possible to observe some patterns in the conversation that can be attributed to different fields or the type of Twitter accounts (oddly enough, publishers’ accounts tend to mention and re-tweet each other).

Tomorrow morning I will aim to produce a larger accumulated network with info from the pre-conference events (thanks to Jorge Fábrega for his help on getting that data). Also, I aim to produce the accumulated version and a daily one. Let’s see if we can get something from these dynamic networks. I hope you are enjoying the conference, stay tuned!

apsa2013_4
Full network of day 1 of APSA (August 29)

UPDATE 3: At 3pm Chicago time, things got much more complex and ‘networked’ (pun intended). At this point there are 167 nodes (ie. Twitter accounts) and 261 edges (defined by mentions, replies or re-tweets. We can observe a big cluster in the middle (in dark orange) where the APSA official account (@APSAtweets), alongside some well recognised political science/tweeters, such as @dandrezner@raulpacheco, and the recently acquired by the WaPo, @monkeycageblog. Another recognisable cluster (in pink) is the one formed by the Political Communications scholars, such as @zizip@andrew_chadwick@davekarpf, and @abuaardvark)

apsa2013_3
Network at 29 August, 3pm CT

UPDATE 2: This is the network at 12pm. As you can see, the groups are getting bigger and tighter as the conference evolves.

apsa2013_2
Network at 29 August, 12pm CT

UPDATE: At 9am in Chicago, this is how the network looks like.

apsa2013_1
Network at 29 August, 9am CT

Note: Thanks to Alex Hanna for his small – yet crucial – advice on how to build the networks.

Quick report: Obama’s speech on Syria

President Obama made a speech today explaining the US position on attacking Syria (more details here, here, and here). Luckily, I was collecting data on the APSA 2013 conference, so I managed to run a small script and collect some tweets during the speech. It’s a bit early to get a good idea of the tone and the substantive info we can get from there, but for now, let me show you how the tweets are geographically located. Out of ~5000 tweets I managed to collect, only 1% of them had location coordinates, which is pretty much the usual rate. I plotted all of them against a world map, and here is the result. tweets_obama_syria   Now, if we perform some basic network analysis using the data, most of the nodes with bigger centrality were news outlet and the official account of Barack Obama (@BarackObama). Most of the edges correspond to people re-tweeting mainstream media accounts, while others were simply making their own comments. The network shows how all these people interacted during the first 8 minutes of the speech.

Network of tweets mentioning 'Obama' or 'Syria'. Ties represent mentions, replies or RTs, colours correspond to weakly connected components, and the size of the nodes reflects the eigenvector centrality score of each account.
Network of tweets mentioning ‘Obama’ or ‘Syria’. Ties represent mentions, replies or RTs, colours correspond to weakly connected components, and the size of the nodes reflects the eigenvector centrality score of each account.

Finally, I perform some (really) basic sentiment analysis on the tweets of the first 8 minutes. The method was designed by Alex Hanna, from U. Wisconsin – Madison, and I used the list of words developed by Neal Caren, from UNC – Chapell HIll. This also means that words in other languages than English were not coded. The scores are calculated by minute, and they all stay very close zero. However, the sentiment was more negative at the beginning of the speech and ended up being positive. Uncertainty measures are not provided by this (very brute) way of calculating the sentiment, so it’s not possible to know if they are significantly different than zero (this was a boring caveat, but an important one). sentiment   This is all for now. As you can see, there is nothing about IR or geopolitics in this post. Is mainly a way to show how Twitter data can give us a fast (and sometimes overwhelming) way to analyse current events.