Due to Twitter’s limited API, researchers may be left in the dark

Twitter announced earlier this month that it would restrict free use of its API, the programming interface that enables developers to communicate with Twitter. While the change undoubtedly has an impact on startups and independent developers who are creating tools to make the platform entertaining and safe, it also poses a challenge for academics and students who use Twitter to conduct research on a variety of topics.

The Elon Musk-led company last week informed developers in an email that the Essential, Elevated, and Academic Research access tiers would be replaced by the $100 per month “low-level usage” tier for the Twitter API. There isn’t much information available right now about what developers can do with that $100 per month. Platformer Casey Newton estimates that a low-level enterprise API access could run as much as $42,000 per month.

Not good news. Just in from @TwitterDev , which confirms the elimination of the academic API, and the replacement with a "low level of API usage" for $100/month. pic.twitter.com/o6IHy5yR7l

— 🇺🇦 David Lazer (@davidlazer) February 10, 2023

I worked really hard and for months on a tool to be used for data gathering in research projects, for academia, journalists, and OSINT people, and I would have released it on GitHub.
In 9 days my work is now useful for just a small minority willing to pay for the service, and https://t.co/0aORMWN8sr

— Alberto Olivieri (@AlbertoOlivie13) February 11, 2023

Researchers’ ability to afford it
Spending hundreds or thousands of dollars every month might not be feasible for many members of the research community.

“Earlier this week, a HateLab undergraduate dissertation student had to alter his thesis design because he lacked the money to pay for Twitter data collection. Millions of people worldwide and thousands of people across the nation will share his experience. The Twitter researcher pipeline from undergraduate to professional has been disrupted by this change, which is incredibly disruptive and will have a significant impact on the research ecosystem dependent on this data, according to Professor Matthew Williams of HateLab, which is a part of Cardiff University’s social data science lab and researches online hate speech.

Ironically, Twitter’s developer portal lists HateLab as a success story for using research for good. On the platform, it has published a number of studies on hate speech. The agency has been utilizing the social network’s API for scholarly research. But in the new version, that might disappear.

Twitter hasn’t taken into account the possibility that $100 per month for basic access might be excessive for researchers based in underdeveloped countries.

“Researchers who study hate speech online, particularly independent researchers and those in developing countries, will be significantly impacted by the decision to end free access to Twitter API. In India, where hate speech is spreading on Twitter at an alarming rate, its effects will be felt most acutely. It is a significant financial burden for them to pay 100 USD monthly and 1,200 USD annually, according to Raqib Hameed Naik, the founder of Hindutva Watch, a research group with offices in India.

Even for organizations that might be willing to spend money, the procedure is not entirely clear. The Institute of Data, Democracy & Politics’ Rebekah Tromble reported that when they attempted to submit the form for enterprise-level access, they were routed to the academic research program instead. She also mentioned that the company’s Twitter contact was no longer employed there.

We’re being directed to the Enterprise ($$$$$) option. The only problem? Filling out their form results in an auto-reply directing researchers back to the Academic Research Program. And the contact person listed no longer works at Twitter. 2/

— Rebekah Tromble | rebekahtromble@federate.social (@RebekahKTromble) February 10, 2023

Effects of the API research shutdown
Independent research has played a significant role in making Twitter more beneficial and less harmful. The company has demonstrated a number of projects using Twitter data in fields like healthcare, online hate speech, and climate change.

In order to study the platform’s governance issues, the company earlier this year established the Twitter Moderation Research Consortium (TMRC). Members from academia, civil society, nongovernmental organizations, and journalism were invited to join. However, the program has stalled and those who worked on it have quit since Musk took over.

During the public fights that resulted in the company being acquired, the Tesla CEO himself used data from Botometer, a tool to measure bot followers on Twitter accounts. The Indiana University Observatory on Social Media created the tool. But the new API announcement may put the tool’s future in danger.

Many research projects consider a sizable number of tweets. To research various topics, they send the platform hundreds of queries. What might be included in the $100 per month tier has not been disclosed by Twitter. But for the majority of the projects, it probably won’t be enough. For comparison, the Twitter API used to grant access to 50 requests every 15 minutes per app and 10 million tweets per month under the academic research track.

One of the Botometer researchers, Kaicheng Yang, expressed concern about Twitter’s decision to end academics’ access to its free API.

My guess, this "basic access" is not enough for most research projects, the $100 monthly fee is too high for 95% of the researchers.

If we want to keep the whole Twitter research body and community alive, we will have to start considering alternatives.#NoResearchWithoutAPI

— Kevin Yang (@yang3kc) February 10, 2023

The NYU Center for Social Media and Politics’ co-director Joshua Tucker recently released a paper on Russian disinformation campaigns that took place on Twitter during the 2016 U.S. presidential election campaign. He claimed that the campaign examined information from thousands of tweets, making it challenging to conduct large-scale research if the social network charges academics for the information.

“This [Twitter’s action] is fundamentally a step in the wrong direction. The decision by Twitter will only make access to data harder for outside researchers at a time when significant legislative efforts are being made to make it easier for outside researchers to access data on a global scale. “As a result, policymakers, the press, civil society, and the business community will have more blind spots regarding the impact of the platforms on society,” he said.

Many researchers have noted in recent days that free Twitter API access aids in crisis management in the wake of natural disasters like the recent devastating earthquake in Syria and Turkey. A collection of unaffiliated researchers asked Twitter to maintain the free API access in an open letter they sent to the company earlier this month.

Couldn't come at a worse time. Most analysts and programmers that are building apps and functions for Turkey earthquake aid and relief, and are literally saving lives, are reliant on Twitter API. Any limit/structure/architecture change will make everyone's life difficult. https://t.co/mpwMnWmSPh

— Akin Unver (@AkinUnver) February 8, 2023

Elon Musk, Twitter’s new CEO, has pledged to increase transparency on the site and lessen the prevalence of spam and manipulative accounts. These priorities are commendable, and we support them, the letter said.

In fact, many of the most innovative methods for managing bots have been created by the independent research community. For that work, API access has been a crucial resource. The new data access restrictions on Twitter will lessen the transparency that both the platform and our societies so desperately need.

Without impartial research, the company risked losing knowledge of issues with false information and hate speech on the platform. The company has already received a “yellow card” from the EU for providing false information in its report. The Code of Practice, which includes a promise to give data to researchers among other things, has been signed by numerous businesses.

Although breaking the research code doesn’t have any legal repercussions, it might after the Digital Services Act (DSA) goes into effect the following year. Separately, Twitter was criticized for limiting access to its platform by charging for its API by the bloc’s high commissioner Josep Borrell. The effects of that change are more widespread than anticipated, and a short-term gain in revenue might become a longer-term problem of greater significance.