Knowing where to look for data — and accessing it via scraping data from websites — can be a powerful force multiplier for investigative journalists.
DocumentCloud now includes many more cutting-edge functions — which include extracting personal identification information embedded in large files, importing data from programs like Google Drive, transcribing YouTube audio, and even peering through weak blackout redactions.
Social media platforms are among the most difficult sites to scrape for data across the internet. A recent session at NICAR23 unveiled several dynamic new tools — including Junkipedia, a possible CrowdTangle replacement — that can perform a wealth of social media monitoring tasks, from tracking down who is behind harmful ads to identifying conspiracy groups or influencers spreading disinformation.
How investigative journalist Olanrewaju Oyedeji, from Nigeria’s Dataphyte, exposed corruption in government notebook contracts by analyzing data from the state’s online procurement portal.
In this edition of GIJN Toolbox, we examine the latest advancements from the IRE22 conference on data extraction and optical character recognition (OCR) tools for turning unwieldy documents into searchable spreadsheets.
How widespread is mask use in your country? Our NodeXL #ddj mapping from July 13 to 19 finds The New York Times mapping the odds of people encountering other mask wearers in the United States, two university professors quantifying the number of interruptions a parent suffers on average every hour while working from home, the Committee to Protect Journalists talking to data journalists about the struggles of reporting on COVID-19, and openDemocracy documenting cases of mistreatment of women in labor around the world since the pandemic started.
The coronavirus pandemic has upended the lives of people around the world, but some communities are especially hard hit. Our NodeXL #ddj mapping from July 6 to 12 finds The New York Times analyzing data that reveals Black and Latino people have been disproportionately affected by COVID-19, The Washington Post highlighting that business relief funds for the pandemic have gone to the rich and well-connected, and Bloomberg looking at more than 120 US businesses that say the coronavirus helped force them into bankruptcy.
What’s the global data journalism community tweeting about this week? Our NodeXL #ddj mapping from February 12 to 18 finds @MattLWilliams discussing the ethics of publishing Twitter content, @MaryJoWebster explaining several common “dirty data” problems and @MediaShiftOrg showing examples of the powerful impact of small data teams in newsrooms.
Buzzfeed News wanted to get a better idea of where President-elect Donald Trump gets his information. So they analyzed everything he has tweeted since he launched his campaign and took a look at the links he has shared and the news sources they came from. Open Lab Fellow Lam Thuy Vo walks you through their process.
Scraping the Heathrow Airport operational data website revealed that since the start of 2016, an average of 72 people have complained about the level of noise pollution every day. Here’s how BBC Online data journalist Daniel Wainwright did it.
