Twitter’s Misguided Barriers for Researchers

(Bloomberg Opinion) -- Last month, Twitter’s chief executive, Jack Dorsey, told the Senate Intelligence Committee he believes “the people use Twitter as they would a public square and they often have the same expectations that they would have of any public space.”

But for all his vows of openness, the platform is making it increasingly difficult and expensive for academics to access tweets to research important questions from how information about diseases like Zika spread to how social movements like Black Lives Matter work to how social media can be used to promote democracy.

Here’s the problem: It’s not possible to use Twitter’s advanced search function to pull all tweets related to a certain subject within a date range. For example, when I tried searching recently for use of the #MeToo hashtag during the first three days of the movement, I got a subset of tweets, but not all of them, and there was no way for me to know how many tweets there had been in aggregate. Astonishingly, when I searched for use of the #MeToo hashtag on Oct. 15, 2017 — the first day of the movement — I got zero results.

And Twitter’s policies have made it increasingly challenging and costly for academics like me to access historical tweets in other ways.

Justin Littman, a software developer at Stanford University Libraries, explained that researchers with advanced software skills used to be able to sign up to write their own software and get the data they needed from Twitter’s Application Programming Interface (API). In July, the company announced that researchers would have to first request access to the API for such projects order to prevent malicious use of the application. The company also limits how data sets obtained this way can be shared. Users who sign up to use the developer platform also have to agree to Twitter’s policies. One of these imposes restrictions on studying a range of subjects, including political beliefs and racial and religious topics.

It’s great that Twitter is trying to prevent its data from being used for nefarious purposes, such as interfering in elections. But it’s disturbing that the company requires researchers using their API to agree to policies that include restrictions on studying topics like identity politics. A Twitter spokesperson pointed out that access to the basic API is free, but Littman noted that it only provides historical tweets going back a few days.  That’s unhelpful for many academics whose studies have longer time frames.

Academics can get access to any past tweets except deleted tweets via third parties but only if they’re willing to pay. While companies like Crimson Hexagon, a business intelligence firm, provide access to archives of tweets, they target their services primarily to corporations for business intelligence or marketing. Littman said the only product that was focused on providing historical data to academic researchers was Sifter, which stopped operating on Sept. 29.

Stu Shulman, the founder and chief executive of Texifter, the creator and operator of Sifter, said this was because Twitter rejected its application to renew its contract to use Twitter data for academic purposes and said that only use of Twitter data for businesses would be acceptable. Shulman also said Twitter made the pricing prohibitive. A spokesperson for Twitter told me it was the company that decided not to renew the agreement. Littman said that another challenge of getting data from third parties is that Twitter does not allow people using such outside services to download more than 50,000 tweets a day. Therefore, these services usually don’t let academics export tweets in bulk, which makes it much harder for those working with large data sets,

Researchers can also request data sets directly from Twitter at a price. Shulman said a dozen different researchers told him that when they approached the company directly to request data, they were quoted prices starting at $1,250 – a much larger fee than his company used to typically charge for small projects, which is prohibitive for many academics. When asked what they charge for such data, a Twitter spokesperson said requests need to be discussed with the company’s sales team.

Facebook is also making it tough for academics to access data. Littman said researchers can request academic partnerships with Facebook  but, again, it’s problematic that a social network should have a say in what academics may and may not study. And, in April, Facebook changed policies about access to its API in order to “better protect people’s information.” Littman said that, due to the changes, NetVizz, an application that allowed academics to get data from Facebook, stopped operating. Now, he said, Facebook provides a set of data so limited that “it’s not useful.” For example, he wasn’t able to use it to pull public posts by all U.S. senators.

Littman has a number of theories about why social networks seem to be making it harder for academics to access data. “Research by academic institutions is clearly perceived as a liability post-Cambridge Analytica,” he said. (Last year, Facebook claimed that a researcher “lied to us” by sharing data with Cambridge Analytica, a political consulting firm that worked for President Donald Trump’s 2016 campaign. Littman also pointed out that “while there’s clearly a huge societal benefit to this research, it’s not necessarily research that benefits social media companies directly. It’s easier to say no than to figure out how to handle it properly.” Another issue, he said, is that Twitter apparently wants to further monetize the use of its data.

It’s one thing to sell data to companies. But social media sites whose business models are predicated on selling advertising shouldn’t be trying to squeeze dollars out of researchers who are trying to gather data that could make tremendous contributions to society. And it shouldn’t be up to Twitter or Facebook to decide whether professors are allowed to study subjects like identity politics. They should make access to their data available for free to academic researchers. The social networks should either make all historical posts available on their websites through advanced searches or, if they want companies to have to pay for such data, they should set up a separate way for academics to access past posts for free without having to justify their projects.

Academics would never be blocked from accessing what happened in a public space. Twitter’s role is “helping to not only serve that public conversation so that everyone can benefit,” Dorsey said, “but also to increase the health of that conversation as well, and in order to do that, we need to be able to measure it.”

Twitter needs to stop thwarting the very people who can help it do that  and provide solutions to many other social problems at the same time.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

Kara Alaimo is an assistant professor of public relations at Hofstra University and author of “Pitch, Tweet, or Engage on the Street: How to Practice Global Public Relations and Strategic Communication.” She previously served in the Obama administration.

©2018 Bloomberg L.P.