Online Research Tools

Online Research Tools and Investigative Techniques by the BBC’s ace online sleuth Paul Myers has long been a starting point for online research by GIJN readers. His website, Research Clinic, is rich in research links and “study materials.” Here’s a tipsheet about finding people online that Myers presented at a 2019 GIJN webinar. And a GIJN article about his presentation at GIJC19, 4 Questions for Online Super-Sleuth Paul Myers.

Using Twitter to Find People at the Scene of a Breaking Story

Customise your Browser: Using Add-ons for your Web Research

Browser Add-ons (Part 2): Traveling Back in Time

GIJN’s Investigative Toolbox, a column by GIJN’s Alastair Otter, explores selected topics:

SpyOnWeb, VirusTotal, and SpiderFoot HX  (By Brian Perlman)

Tracking Names and Websites, Verifying Video, a Clustering Search Engine

Digging for People, Trawling the Web and Keeping Yourself Safe

Backgrounding People and Companies

Mine Twitter and Monitor Website Updates

Beyond Spreadsheets and Deep Searching the Web

Other Toolboxes

Bellingcat’s Online Investigative Toolkit  includes a wide variety of resources on maps, geo-based searches, images, social media, transport, data visualization, experts and more.

Malachy Browne’s Toolkit. More than 80 links to open source investigative tools by one of the best open-source sleuths in the business. When this New York Times senior story producer flashed this slide at the end of his packed GIJC17 session, nearly everyone requested access.

Jake Creps keeps up a steady stream of useful tips on his blog. He also hosts a podcast, osintpodcast.com. Creps is an open source intelligence analyst with experience in the public and private sector.

OSINTcurio.us features weekly podcasts, webcasts and “10 minute tips” on video covering many aspects of doing open source investigations. It’s a community project begun in late 2018 by about 10 contributing experts.

Online Methods to Investigate the Who, Where, and When of a Person. Another great list by Internet search expert Henk Van Ess. Also see Who Posted What, “a Facebook keyword search generator with which you can search for posts on Facebook who have they keyword in it, on a specific date or between two dates,” a Van Ess idea developed by Daniel Endresz.

The Open Source Intelligence Framework has a very detailed and ever-growing list of digital investigative tools.

Google Searching Tips

Use Google’s Advanced Search Page and their instructions on Advanced Search. Also, Google has training classes for journalists, including a lesson on searching, a “power searching” course, and more.

For a list of key commands, see Google’s Advanced Search Syntax by Paul Myers.

For good basics, see Google Search Tips for Journalists by ExpertiseFinder

From there, try:

Daniel Russell’s SearchResearch and his Google Advanced Search Operators.

37 Advanced Google Search Tips for Smarter Searching from Eric Melillo at COFORGE.

Google Advanced Search: Tips and Operators for Better Searches by David Vranicar of Oberlo.

Google Advanced Search: How to Find Better Info (2X Faster) from Keyword Tool Blog.

Google Search Operators: The Complete List (42 Advanced Operators) by Joshua Hardwick in ahrefs.

Advanced Google Search from the library at the University of Texas-San Antonio.

Research Buzz wrote Going Old School to Solve A Google Search Problem to describe a method to exclude certain sites from search results.

17 Great Search Engines You Can Use Instead of Google, published by Search Engine Journal in 2020.

Dark Web Tools

Searching the Deep Web, by Giannina Segnini. Beginning with advanced tips on sophisticated Google searches, this presentation at GIJC17 by the director of Columbia University Journalism School’s Data Journalism Program moves into using Google as a bridge to the Deep Web using a drug trafficking example. Discusses tracking the container, the ship, and customs. Plus, Facebook research and more.

How to Become a Deep Web Super Sleuth. This GIJN article describes advice given by Albrecht Ude, a German journalist, researcher, and trainer at the 11th Global Investigative Journalism Conference. “Search engines are completely useless for finding any content on the deep web,” said Ude, whose first bit of advice is to think abstractly, followed by more tips. Also see his related tipsheet: Finding and using databases. 

Step by step guide to safely accessing the darknet and deep web, an article by Paul Bischoff in VPN & Privacy, a publication of Comparitech.com, a UK company that aims to help consumers make more savvy decisions when they subscribe to tech services such as VPNs.

Even More Tools

Digital investigations for journalists: Content for a four module course from the Knight Center for Journalism in the Americas.

OSINT Essentials by Eoghan Sweeney provides links to mostly free tools and services that are useful for online verification, digital journalism, and open-source intelligence work.

Sector035, self-described as “Just a shadowy nerd,” provides a variety of tips weekly, especially strong on geolocation. Publishes Week in OSINT and Quiztime.

First Draft, a global non-profit that supports journalists, condenses a lot in its Advanced Toolkit. Also see First Draft’s Essential Guide to Verifying Online Information (PDF) (2019) and related training materials.

Tools, Useful Links & Resources, by Raymond Joseph, a journalist and trainer with South Africa’s Southern Tip Media. Six packed pages of information on Twitter, social media, verification, domain and IP information, worldwide phonebooks, and more. In a related GICJ17 presentation, Joseph described “How to be Digital Detective.” 

Exposing the Invisible has a variety of guides on its site, including on “googleDorking,” a technique to query search engines for information hidden on public websites and vulnerabilities exposed by public servers. Other guides address finding metadata and using WHOIS.

Knowing Your Way Around The Command Line, by Max Harlow, Senior Newsroom Developer at The Financial Times, was presented at GIJC19. What’s possible? He summarizes: Find who owns a website, Analyze data at lightning speed, Match up names written differently, Link up your data to online databases, Bulk search over hundreds of documents.

AML RightSource, a private US firm “solely focused on Anti-Money Laundering (AML)/Bank Secrecy Act (BSA) and financial crimes compliance solutions,” put together a packed sheet of resources.

Investigate with Document Cloud, by Doug Haddix, Executive Director, Investigative Reporters and Editors. A guide to using 1.6 million public documents shared by journalists, analyzing and highlighting your own documents, collaborating with others, managing document workflows and sharing your work online.

An Investigative Guide to LinkedIn created by Nathan Patin for Bellingcat in 2019. “This guide aims to provide helpful tools and techniques for identifying LinkedIn profiles and for extracting information that will then allow you to pivot to other social media profiles belonging to the target.”

A Guide to Open Source Intelligence (OSINT) by Michael Edison Hayden for the Tow Center for Digital Journalism at the Columbia Graduate School Journalism School (2019) covers a variety of topics:

• The Difference between Open and Closed Networks
• Searching the Open Web
• Verifying the Authenticity of Social Media Accounts
• Verifying Images and Videos
• Exploring Fringe Websites
• Using Archives, Saving Your Work
• Learning New Platforms and Interacting with Hostile Communities

Finding Former Employees, by James Mintz. “10 Tips on Investigative Reporting’s Most Powerful Move: Contacting Formers,” according to veteran private investigator Mintz, founder and president of The Mintz Group.

Tools and tips for digging into Facebook from two investigative journalists A 2019 interview with Brooke Williams, an award-winning investigative reporter and Associate Professor of the Practice of Computational Journalism at Boston University and Henk van Ess, lead investigator for Bellingcat.com.

Investigative Research Links from research editor Margot Williams offers an array of suggestions, from “Effective Google Searching” to a list of “Research Guru” sites. Also see her GIJC19 tipsheet: Research Plan for Investigative Project.

Bureau Local collaborative tools is a crowd-sourced spreadsheet of more than 80 listings, some of them research tools, sponsored by the UK-based Bureau of Investigative Journalism.

Tools for Reporters by Samantha Sunne, provides regular e-mails on new tricks, such as Finding emails when they don’t want to be found.

Fundamental search for journalists This series of online classes from Datajournalism.com, taught by Vincent Ryan, covers advanced searching, verification, visualization and geo-mapping.

SPJ Journalist’s Toolbox from the Society of Professional Journalists in the US, curated by Mike Reilley. Includes an extensive list of, well, tools.

IntelTechniques is a site by Michael Bazzell, a former US government computer crime investigator and now an author and trainer. It hosts his podcast, The Privacy, Security, & OSINT Show.

World.192.com has a list of international telephone directories.

Using Phone Contact Book Apps For Digital Research, an instruction 2019 posting by Aric Toler of Bellingcat.

Use Chrome Developer Tools to View Masked Phone Numbers for Free on a Popular People Search Site, by Sean Lawson, a University of Utah professor, was published in 2019.

How to find an academic research paper, by David Trilling, a staff writer for Journalist’s Resource, based at Harvard’s Shorenstein Center on Media, Politics and Public Policy.

Using deep web search engines for academic and scholarly research, an article by Chris Stobing in VPN & Privacy, a publication of Comparitech.com, a UK company that aims to help consumers make more savvy decisions when they subscribe to tech services such as VPNs.

Research Beyond Google: 56 Authoritative, Invisible, and Comprehensive Resources, a resource from Open Education Database, a US firm that provides a comprehensive online education directory for both free and for-credit learning options.

The Engine Room, a US-based international NGO, created an Introduction to Web Resources, that includes a section on making copies of information to protect it from being lost or changed.

Awesome Public Datasets, a very large community-built compilation organized by topic.

Resources from Startme.com A multitude of resources from a company that created an integrated bookmarks manager. The free library contains pages with hundreds of links:

Addons
Databases
Search Engines
Tools
Verification Toolkit
Tutorials

Databases, for example, is a page with links to a wide range of material, about stolen property, weather, hacking, media, traffic, analytics, domain registration, buildings, animals, drones, finance and more.

Bates InfoTips, by Mary Ellen Bates, is a good source of up-to-date ideas, such as Limiting Google News searches by time and Finding people through Facebook Graph Search. Bates is ” founder and principal of Bates Information Services Inc. and a long-time info geek.”

MakeUseOf regularly features good tips, such as 4 Anonymous Web Browsers That Are Completely Private.

6 Unique & Free Keyword Research Tools You Didn’t Know You Needed, from Search Engine Journal.

This Tool Shows Exposed Cameras Around Your Neighbourhood, an article in Motherboard explaining how Kamerka can take an address, landmark, or coordinates and display exposed internet connected cameras on a map.

Open-Source Intelligence (OSINT) Reconnaissance, a long article by “researcher-writer-hacker-tinkerer” Ian Barwise in Medium, covers techniques such as “Google dorking.”

After the GDPR: researching domain name registrations  A look at the impact of the European General Data Protection Regulation, which makes researching domain name registrations harder, necessitating different tools and techniques, according to OSINTCurious.

How to Write Facebook Graph Search Queries a 2020 WikiHow guide on using Facebook’s graph search syntax to find posts, comments, people, and other entities on Facebook.

The Most Comprehensive TweetDeck Research Guide in Existence (Probably) by Charlotte Godart, an investigator, and trainer for Bellingcat (2019).

Finding Hidden Business Resources A presentation at the Special Libraries Association 2019 conference by Mary Ellen Bates on finding materials on the deep web.

Our Search for the Best OCR Tool, and What We Found, by Ted Han and Amanda Hickman in Source (2019).

Instagram Location Search See Instagram description of other search options.

Don’t forget the old-school research methods too. Here’s a thorough guide by Tactical Tech on Navigating Libraries and Archives for Investigations.