Why Reporters Need to Carefully Assess the Evidence on COVID-19

Reporters investigating the coronavirus pandemic are being confronted with a mountain of medical research papers, statistical models, and government figures that purport to be true.

But experts insist that even journalists without health backgrounds can readily assess their truthfulness and relevance by making sure they see the full picture when reading research papers, using available resources, recognizing major gaps in scientific knowledge — and by looking to the past.

In the seventh webinar in GIJN’s series Investigating the Pandemic, two epidemiologists and an investigative journalist steeped in evidence-based methodology shared tips and resources to help reporters navigate the deluge of data.

Carl Heneghan, a professor of evidence-based medicine at Oxford University in the United Kingdom, said investigative reporters everywhere should be on the lookout for “collateral damage” deaths associated with COVID-19.

Heneghan said recent, highly accurate data for the UK showed that one-third of “excess deaths” – total mortality above the average for that period of the year – were non-COVID deaths. A contributor to Oxford’s COVID-19 Evidence Service, Heneghan said 38,544 deaths beyond the norm had been recorded for England and Wales alone in the past six weeks.

“There are considerable concerns (among medical professionals) that there are people in the home setting dying from other causes, and particularly those not presenting to health care (facilities),” he said.

Heneghan warned that reporters should avoid the temptation to over-focus on hospital capacity and intensive care and critical care settings, given that data was increasingly suggesting far greater problems at nursing homes and other facilities for the elderly. He said one-third of all care homes in the UK had reported COVID-19 outbreaks.

Heneghan said reporters should also make an effort to use the date of death in evaluating mortality data, and not the date those deaths were reported by authorities, which is a typical format for government data.

“The day of reporting is highly inaccurate, because we are getting deaths reported today that go back a month, and even as far back as March 11,” he said.

Serena Tinari, a Switzerland-based investigative reporter who specializes in public health, offered three imperatives for reporters which are especially important for journalists without previous health beats: read journal studies, assess the quality of medical evidence, and compare elements within today’s crisis with those of past pandemics.

The co-founder of Re-Check – an NGO for investigating health issues – Tinari suggested that reporters adopt an evidence-based medicine (EBM) mindset, so they can pick out red flags in studies, and not only evaluate the answers provided by researchers, but the questions they have as well. To illustrate the need for that mindset, she cited the recent media hype around a study of the experimental COVID-19 drug remdesivir in The New England Journal of Medicine.

Tinari said a careful read of the full study showed that there were only 53 patients involved with the trial, which was actually fewer than the number of authors on the paper.

Second, she said the study featured no “control arm,” which means a comparison to a group not taking the drug. She said the study’s cohort parameters fell below the highest levels of quality, which include meta-analysis or at least one randomized control trial in experimental research.

Third, the study’s disclosure section revealed that the company producing remdesivir, Gilead Sciences, had also funded the study, presenting a potential conflict of interest.

Finally, a separate, subsequent study – a randomized control trial – found that the drug had shown no statistically significant clinical benefits.

“These are red flags,” she said. “And this is a drug that has already been authorized for use on patients.”

Tinari recommended four online resources to help reporters analyze the evidence:

• To assess news reports and press release claims about drugs, tests and procedures: HealthNewsReview.org, a site founded by renowned medical investigative journalist Gary Schwitzer.
• To find the design and intention behind almost all clinical studies around the world: ClinicalTrials.gov which is run by the National Institutes of Health in the United States.
  
• To understand the landscape around poor quality research and lax oversight, and to understand “EBM thinking”: a recent in-depth piece by medical investigative reporter Jeanne Lenzer, titled Pandemic Science Out of Control.
• To educate yourself if you lack experience covering the health beat: Ivan Oransky’s guide to covering medical stories when research is moving at high speed. 

A third speaker at the GIJN webinar, Rome-based clinical epidemiologist Tom Jefferson, also a contributor to the Oxford COVID-19 Evidence Service, emphasized that there are significant gaps in scientific knowledge about this pandemic.

For instance, he said that despite widespread government calls for the public to wear face masks, very little is known about the efficacy of masks in the general public, and very little is being done to find out.

“As far as I know, there is only one (trial) going on, in Denmark,” Jefferson said.

This, by itself, could be fodder for investigative reporters, he added.

But Jefferson emphasized that COVID-19 beat reporters should also look to the past, including recent coronavirus outbreaks, such as SARS in 2003 and MERS in 2012.

He recommended that they review a 166-year-old case, when the world’s first modern epidemiological investigation happened in London’s Soho. A local anesthetist, John Snow, integrated data from a cholera pandemic into a map of Soho, and hypothesized that the outbreak originated from dirty water at a public pump at the center of the area experiencing the worst mortality rates.

Snow’s idea clashed with the prevalent theory that such diseases were transmitted through foul odors, and appeared to be undermined when a 59-year-old woman more than four miles away, in the upmarket suburb of Hampstead, died of the same disease. So Snow walked to the woman’s home and found that she had ordered water to be collected from that same Soho pump via horse and cart.

Jefferson said the famous lessons of “the widow of Hampstead” had often not been applied in tracking the coronavirus pandemic. For instance, he said that Italy’s twin index cases — the first COVID-19 cases recorded in the country — had appeared on the same day, February 20, despite being almost 150 miles apart, and having no apparent relation to each other, or to the outbreak in China.

“These cases should have been investigated,” he said. “It would shed light on the means of transmission, not just for today, but for the future.”

GIJN’s next webinar is Thursday May 14 at 9am EDT. Sign up here for Investigating the Pandemic: The Threat to Africa.

Rowan Philp is a reporter for GIJN. Rowan was formerly chief reporter for South Africa’s Sunday Times. As a foreign correspondent, he has reported on news, politics, corruption, and conflict from more than two dozen countries around the world.