Measuring Journalism’s Impact — What We Know Now That We Didn’t Before

Journalism outlets around the world try to measure the impact of what they do. Each organization can have different ideas about how to define impact, and use a wide range of metrics to measure the larger effects of their reporting.

We’ve been spending the last few years looking at how to measure impact, surveying journalists and trying to add to our community’s understanding of the effects of investigative reporting. In all, our research identified roughly 90 different ways to measure impact, but the reality is that newsrooms usually look at just a few.

Agência Mural de Jornalismo das Periferias, in Brazil, breaks down the metrics they use into four categories: Real life change; amplification by other outlets; audience engagement; and influence on the public debate. The Marshall Project, a nonprofit news organization that covers criminal justice and race in the US, measures its reporting impact using these three criteria:

• Impact on policymakers: did reporting contribute to change a law, policy or practice?
• Advocates and experts: did the reporting provide useful information for advocates?
• Other media: Does The Marshall Project set a higher standard and inspire other media to cover criminal justice better?

The Daily Maverick in South Africa focuses on giving its audiences the information it needs on key topics that affect their lives. For instance, lack of quality education is a big problem in South Africa and so Daily Maverick teamed up with Report for the World (RFW), a nonprofit that helps fund and train newly hired reporters who cover education, global health, and climate in newsrooms in low and middle-income countries. That group is helping fund an education reporter at Daily Maverick, and editor Jillian Green says the coverage is essential for audiences that care about the state of local schools. “Civil society knows the problems in this area and have been trying to galvanize. We’re giving them information to use in that fight. We want to make sure the parents have the information they need and can hold government to account,” Green said in an interview.

The South African publication Daily Maverick teamed up with Report for the World (RFW), a nonprofit that helps fund and train reporters covering education, global health, and climate in newsrooms in low and middle-income countries. Image: Screenshot, Daily Maverick

As well as measuring effects on policy and laws, most of the outlets we’ve surveyed say they use social media or Google Analytics to measure story impact, but the reasons vary. Some newsrooms prioritize audience engagement, others like Frontier Myanmar — an outlet that focuses on in-depth reportage of news, conflict, economic, and political affairs across Myanmar — appreciate the ease with which these numbers can be tracked. Some mediums, like podcasts, generally have smaller audiences than more traditional news media, which makes pageviews or listenership alone an imperfect metric for the effects of their work.

Changing Government Policy, Albeit Slowly

One traditional definition of journalistic impact involves affecting government policy or prompting officials to address a problem. But to spur the public outcry or regulatory embarrassment necessary for action often requires consistent reporting across a much longer timeline.

This kind of slowly-gestating impact typically happens in three phases: individual, deliberative, and substantive. In the classic books “The Journalism of Outrage” and “Democracy’s Detectives,” the authors explain that an individual impact could be a corrupt policeman getting fired, a deliberative example could involve a parliamentary investigation or congressional hearing, and substantive impact could result in a law or policy being changed.     

Image: Screenshot, Harvard University Press

But global muckrakers throughout history have found that civil society advocacy is a key precondition for journalistic impact. “The mobilization model doesn’t work unless there is civil society support or partnership. Change is slow and results are slow and there is no change without civil society pushing it.” Mark Lee Hunter said in a recent interview.

With the rise of so-called democratators, attacks on journalism have ramped up and made it that much harder to prompt a government response to policy issues or official wrongdoing revealed by accountability reporting.

Indeed, an editor in India, where the Modi government has come down hard on the media, told us that it’s now much harder to galvanize government response to an issue than it was 20 years ago. “The gold standard of impact, that we always looked forward to as journalists, doesn’t happen anymore. No one in the government or position of authority responds to our work.”

Understanding Broader Effects on Audiences

Some have broadened the definition of impact to include “any discernible effects” on media audiences, which can include public awareness of a problem and better understanding. Economists and political scientists have used large datasets and natural experiments to show that the presence of news outlets improves governance, reduces corruption, affects voter turnout, and redirects government distribution of resources. These desirable effects have been linked to the watchdog role of media outlets, the fact that they can shine a spotlight, or name and shame.

Andrey Boborykin, executive director of Ukrainska Pravda, one of Ukraine’s largest independent news organizations, said the country’s ongoing war with Russia is, of course, more top of mind for Ukranians than, say, topics like climate change and the environment. However, he noted that while a story about global warming may not get much attention “audiences are more likely to read and share a story about a former national park being deteriorated due to the war. This kind of article affects their lives and emotions. They feel infuriated and these stories get more traction,” Boborykin said in an interview.

When we surveyed International Consortium of Investigative Journalists (ICIJ) partners about their collaboration on the  Pandora Papers, they said they firmly believed that their reporting affected the beliefs and attitudes of citizens. Thirty-three out of the 39 media outlets (85%) reported receiving notable responses and feedback from audiences regarding that ICIJ investigation. Of those 33 media outlets, 23 outlets reported a positive audience reception (70% of responses) while 4 outlets said that publication generated some form of public criticism or backlash (12% of responses). The survey we did of Report for the World outlets found similar responses, with 70% saying their reporting shifted the perceptions of their audiences and 85% saying audience knowledge increased as a result of the reporting.

The Pandora Papers investigation involved 600 journalists from 150 media outlets working together across borders. Image: Screenshot, ICIJ

Inside the Newsroom

When we first started looking at cross-border collaborations, we didn’t think as much about the internal impact they would have on newsrooms. But then we realized innovation is a common by-product of an outlet partnering with groups like ICIJ or Report for the World.

“Having an RFW reporter changed the mentality at Agência Mural,” said Izabela Moi, cofounder and executive director of Agência Mural de Jornalismo das Periferias, which focuses on underreported areas in Sao Paulo, Brazil.

In fact, our surveys of ICIJ members and RFW editors found that work on collaborative projects helped reporters gain new skills, including document analysis, data scraping, database analysis, and transnational coordination. In addition, we discovered that collaboration begets more collaboration.

For example, 37 out of 55 media outlets (69%) who worked with ICIJ on the Pandora Papers reported that they either collaborated somewhat more or much more than usual during that investigation. An overwhelming 52 out of 55 media outlets (95%) reported that the ICIJ partnership led them to increase collaboration with other organizations.

Impact’s Dark Side: Risks to Reporter Safety, Opportunity Costs 

Predictably, sensitive investigations carry risks for participating journalists. Five out of 55 media outlets (9% of the total) reported that participating journalists faced either legal intimidation or threats of bodily harm due to their work. One outlet reported that their participating journalist had to go into temporary hiding following publication of the Pandora Papers, while other outlets reported receiving death threats, legal threats, and smear campaigns.

We also looked at the opportunity costs caused by major cross-border collaborations and asked what stories could not be covered as a result of time spent working on a transnational ICIJ investigation. Sixteen out of 55 media outlets (29%) indicated that they were unable to report on other stories due to the time and resources that they dedicated to the Pandora Papers. Stories they couldn’t cover included reporting on coal, deforestation, local corruption, human rights violations, and human trafficking.

Methods for analyzing the impact of investigative journalism have become far more sophisticated. Having economists and political scientists research questions of media impact has added to the methodological sophistication and the analytics now available from social media platforms have also been revealing. Our recent research suggests that the effects of investigative and collaborative reporting are even more far-reaching on the journalism community than we had imagined when these projects began decades ago.

Dr. Anya Schiffrin is the director of the media, technology and communications specialization at Columbia  University’s School of International and Public Affairs. 

Dr. Dylan Groves is an assistant professor of history at Lafayette College and received his Ph.D. from Columbia  University’s political science department. 

Their co-authors on the Understanding Journalism Impact paper include Adjunct Associate Professor André Corrêa d’Almeida, Dr. Lindsay Green-Barber,  Audrey Hatfield (Columbia University Masters of Public Administration, 2023), and Ph.D. candidate Adelina Yankova. Schiffrin is now working on a follow-up project with a team of SIPA students as well as Rania Itani, Preethi Nallu, and Raghavi Sharma from Report for the World.