How can you search and analyze collections of documents on your own computers with simple tools? At DataHarvest, Robert Gebeloff and I ran a workshop to answer that question. As people were seemed interested, here’s a write-up of the two key tools we worked with: Apache Tika for content extraction and regular expressions in Sublime Text as an advanced search tool.
When it’s time to start a new investigation, journalists prepare themselves in different ways: from doing the research to pitching the story to building the narrative. It’s not about a single formula, but about integrating different resources and strategies. Here, we present a selection of the tips presented at the 2016 conference of Investigative Reporters and Editors (#IRE16), useful for starting and developing investigations.
In an extract from his book Finding Stories with Spreadsheets Paul Bradshaw explains how to use basic cleaning functions in spreadsheets to make it easier to combine data, including a case study where the same functions were used to speed up a research process for a story.
Before the emergence of digital tools, recording and (especially) transcribing an interview was a tedious affair. The little microcassette tapes were of dubious reliability—and yes, I once had one fail on me during a crucial and contentious encounter. Transcribing was worse, as you’d sit there constantly hitting the “play” and “rewind” buttons, an imprecise process that risked damage to the tape.
Alongside the advantages available for criminals of operating on a global scale, making it harder to track them down, there are also disadvantages that the clever journalist or law enforcement official can exploit to expose them. How do we do this? Firstly, through data: more data means more transparency, provided the quality of information is there and supported by tools that allow proper analysis. Secondly, by journalists using advanced techniques.
With the aim of contributing to the promotion of data-based investigations and asserting its vision of journalism as an essential service to democracy, OjoPúblico has published “La navaja suiza del reportero. Herramientas de investigación en la era de los datos masivos” (“The Swiss Army Knife Journalist: Digital Research Tools in the Era of Big Data”), a resource for Hispanic reporters who want to become familiar with the world of data journalism and, above all, to understand its meaning and relevance in Latin America and the world.
Some of the best tips and techniques come out of the annual meetings by our colleagues in Scandinavia, and this year is no different. More than 600 participants attended the SKUP conference in Tønsberg, Norway, on April 8-10. Among the tip sheets presented was this gem of 25 practical tips that reporters in even the smallest newsrooms can use to good effect, focused on finding great characters and cases to bring your story to life.
People regularly ask what tools to use or what programming language to learn for data-driven journalism (ddj). There is no right answer for it, especially considering that technology and tools available are evolving quickly in the field.
A Round-up of Recent Debates about Community Support and Culture Change in Newsrooms. A study released earlier this month suggested that 40% of Americans would buy digital newspaper subscriptions if they were presented with a persuasive argument. It is worth spending a minute unpacking what that “persuasive argument” might look like.
In part one of this blog I discussed the need for journalists to save evidence they find online and some of the techniques for recording that information. In part two, I’ll focus on some of the issues journalists face when it comes to saving information from social media and mobile devices.
As journalists we are used to saving information. We securely store documents, keep meticulous notes, save and back up important emails on our computers. But what about the information we find online during the course of our investigations?
Data literacy is a never-ending process. Going to workshops and hands-on practice are important, but to really become acquainted with the “culture” of data literacy, you’ll have to do a lot of reading. Don’t worry, we’ve got your back: below is a curated list of 16 blogs to follow in 2016 if you want to: improve your data-visualisation skills; see the best examples of data journalism; discover the methodology behind the best data-driven projects; and pick-up some essential tips for working with data.
Spotlight is without a doubt the most compelling, most insightful movie on investigative journalism since All the President’s Men, the 1975 classic on the Watergate Scandal. This is great story-telling that takes viewers inside the Spotlight investigative team at The Boston Globe as it dived into one of the more notorious crimes of our time – the systematic tolerance and cover-up of thousands of cases of pedophilia by the Catholic Church. At a time when investigative journalists are under fire around the world, here is a public tutorial on why in-depth, watchdog reporting is so important to social accountability and democracy.
We’re back from the wonderful Global Investigative Journalism Conference in Lillehammer with a new roundup of resources for you. Thanks to everyone who came to and shared the presentation I did there with Margot Williams of The Intercept, 100 Best Databases for Internet Research. You can find links to all the resources that Margot and I talked about in this post.
The Internet is an amazing research resource but often discovery of quality reference tools and reports can be challenging, especially when you think about how quickly new resources appear and often disappear. I hope that these columns not only alert you to useful material but will help you build up your own collection of reference resources, which you can save on your own computer, or utilize a tool like The Wayback Machine to archive it for yourself and others.
Time for a new collection of tools and reports. This week we’ve got a human rights database, file conversion for 208 formats, and nine new reports from research organizations, ranging from terrorism and corruption to European migration. Got a suggestion for The Research Desk? Write me at gprice@mediasourceinc.com.
Time for a new collection of resources and research reports. Today’s roundup features two online databases — one on nuclear reactors, the other on health — and the latest international research reports from Brussels, London, and Washington. Global Health Facts from the Kaiser Family Foundation is an organized and frequently updated collection of browsable and searchable data.
Web scraping is a way to extract information presented on websites. As I explained it in the first installment of this article, web scraping is used by many companies. It’s also a great tool for reporters who know how to code, since more and more public institutions publish their data on their websites.
With web scrapers, which are also called “bots,” it’s possible to gather large amounts of data for stories. But what are the ethical rules that reporters have to follow while web scraping?
$8 billion in just a few hours earlier this year? It was because of a web scraper, a tool companies use—as do many data reporters. A web scraper is simply a computer program that reads the HTML code from webpages, and analyze it. With such a program, or “bot,” it’s possible to extract data and information from websites.
Just as an article or a gallery of photos can shine a light on an issue for the public, so can these in-person gatherings. Some media organizations are putting on full-fledged festivals in the same vein as South by Southwest and TED. These gatherings include panels of experts, one-on-one conversations with major newsmakers and presentations that explore ground-breaking topics. In other words, they’re an entirely new way of informing and providing information — undoubtedly journalistic functions.
It’s time for another roundup of resources and research reports that we’ve curated from around the web. Most of these selections are new on the web in the past few weeks. Several of the reports listed below are full of data including visualizations and can be thought of as reference tools for both current and future research.
Information stored on your computer or mobile is at risk. You could leave it on a train; it could be seized at an airport security checkpoint; or by the police or courts. And of course hackers can access your data. You need to be aware of all the risks and ways to protect your information, your sources and yourself.
In this just-released video, investigative reporter Mark Schapiro goes in-depth on how to use investigative techniques in probing often complex environmental issues. Schapiro, a veteran of the original Center for Investigative Reporting, gave this talk in Hamburg at NR15, the July 2015 annual conference of Netzwerk Recherche, Germany’s investigative journalism association.
For the last seven years I have been interviewing and profiling successful entrepreneurial journalists in various countries of various socioconomic classes. I’ve talked to publishers and editors with staffs of as many as a hundred as well as some one-man/one-woman bands. The ones that survive and thrive after several years share some common practices.
