Facial Recognition Made Easy

Methodology Note: I included screenshots to capture how the examples looked at the time I conducted the searches. These results may change over time, however.

When disaster strikes, corruption occurs, or social injustice happens, facial recognition technology can expose hidden truths. The technology can, however, provide false information as well. A thorough validation, verification, and cross-referencing exercise is essential to ensuring that the information presented is accurate and fair. Also, the tools have ethical and legal ramifications.

Face recognition helped to establish that a military video was Russian propaganda. It was also an essential tool to identifying who was behind a notorious Venezuelan fake news website. Bellingcat identified the separatists linked to the downing of MH17 with the help of facial recognition and explained how it can be used to geolocate sites in old photos.

Conventional search engines are ill-equipped to identify faces. They must strike a delicate balance between image search capabilities and privacy considerations. To achieve effective facial recognition, dedicated tools are necessary.

What options are available and what functionalities do they offer? Moreover, what are the ethical implications associated with this technology? Let’s have a look at tracking people in a crowd, finding similar photos of a person and comparing older with newer photos. For each application, I will give an example and the links to the tools. And I will discuss legal ramifications and best practice cases.

Tracking People in a Crowd

This form of facial recognition technology is used to monitor individuals within a large group of people. Even without knowing a person’s name, the technology can identify and track the movements and actions of specific individuals in real-time. This can be particularly useful in situations like protests or large public gatherings, where it’s important to monitor the activities of certain individuals for investigative purposes.

In the context of investigative journalism, this form of people tracking provides valuable insights into the dynamics and influence of participants during important meetings, conferences, or public forums like protests or demonstrations. Understanding the frequency of an individual’s contributions can shed light on their level of involvement, their role within the group, and their potential impact on decision-making processes.

Example

Tracking the movements of individuals involved in the Jan. 6, 2021 attack on the US Capitol was a valuable investigative technique in understanding the sequence of events, identifying connections, and unraveling the broader narrative. While it may not provide direct identification, this approach helped to piece together the movements and actions of individuals, shedding light on their roles within the event.

The Washington Post created a video timeline of the Capitol riot using hours of video footage, some of which was exclusively obtained. The timeline shows how rioters broke into the Capitol building, forcing lawmakers to evacuate. The Post also used facial recognition to identify some of the rioters and assess how close they came to lawmakers. The timeline shows the chaos and violence that took place on that day. Open source information was the most critical investigative method in identifying and prosecuting Capitol rioters.

Using videoindexer.ai to follow the people involved in the January 6 United States Capitol Attack. No names were shown, but it was possible to follow them in the crowd. Image: Screenshot

Tools

Azure AI Video Indexer from Microsoft (free for up to 40 hours of footage) and various professional vendors.

How It Works

For Video Indexer, you upload a video to the site. Each minute of video will take about two minutes to process. When complete, you will be able to extract a lot of interesting information. For example, it verifies what locations are mentioned, and transcribes the full text.

If you click “Play next” it will show you each time that person speaks:

Images: Screenshot, Azure AI Video Indexer

It will also show you a full transcript of the video:

Images: Screenshot, Azure AI Video Indexer

You can download this transcript and then use specialized tools like Wordtune or ChatGPT to summarize the conversation.

Images: Screenshot, WordTune

Finding Similar Photos of the Same Person

Searching vast image databases to find pictures of a particular individual aids in the identification of unknown persons. By uncovering multiple images of a person, you can gather more details about their identity, actions, and connections. Analyzing photos and videos enables you to create timelines, reconstruct events, and develop a comprehensive understanding of the story’s context.

Example

Image: Ekanskie Vestia, via Knack

The facial recognition tool PimEyes was used by the Belgian fact checkers at Knack to perform a reverse search on the face of a man in a military uniform (outlined in the yellow box, above), who was falsely identified in various US and UK media outlets as a retired Russian major general called back to duty to serve in the war in Ukraine. The tool didn’t find any results for the man’s face, but it did identify two other individuals in the photo. Further investigation revealed that the man was actually a former Russian border guard.

Tools

PimEyes (paid), Facecheck  (slow), Search for Faces on VK or TikTok (infested with ads).

All the mentioned services lack transparency about their methods and their location. This lack of openness raises concerns regarding data privacy and security. Users might be unaware of how their uploaded photos or personal information are processed, stored, or shared. It’s essential for users to be cautious while using such services, especially when dealing with sensitive data or personal information.

How It Works

All four services want you to upload a picture. Here I’ve chosen an image of Ramzan Kadyrov, head of the Chechen Republic.

Image: Screenshot, FaceCheck

Image: Screenshot, FaceCheck

PimEyes requires a subscription and charges for source photo access after three free searches. Payment options include monthly, yearly, or one-time fees.

Comparing Old Photos with New Photos

Facial recognition technology can also be used to compare older photographs with recent ones to determine if they represent the same individual. This is particularly useful in investigations involving missing persons or fugitives, where the person’s appearance may have changed over time. By comparing historical and recent photographs, journalists can confirm whether a recently identified individual is indeed the same person depicted in an earlier image.

Image: Screenshot, PimEyes

Example

In Dead Body in Paraguay Is Likely Serbian Drug Smuggler by Milica Vojinović (KRIK) and Stevan Dojčinović (OCCRP/KRIK), facial recognition technology was used to identify a deceased individual found in Paraguay as a Serbian drug smuggler.

Tools

Betaface (free), Kairos (paid)

How It Works

Image: Screenshot, OCCRP

In Betaface, you first have to upload two photos of the person you think is the same. When done, click inside the second face and select Compare Faces. Photo matches are rewarded with a score from 0% to 100%. Only those above 80% are worth double-checking.

Professional Solutions for Visual Recognition

Azure Face API, Amazon Rekognition, Google Cloud Vision AI (US), Face++, and Tencent Cloud (Chinese) offer advanced capabilities for image and facial recognition. They allow developers to integrate powerful visual recognition features into their applications.

In 2020 GIJN held a webinar with Henk van Ess on how to conduct visual search and verification investigations. You can watch the full webinar here:

Ramifications

Facial recognition technology presents several challenges including privacy concerns, accuracy issues, and potential misuse. It can collect personal data without consent, leading to privacy and data protection issues. In the European Union, most of the mentioned tools conflict with the region’s General Data Protection Regulation, according to some IT legal advice.

Some ethical and legal problems are:

1. Privacy Concerns: Facial recognition tools often collect and analyze personal data without the individual’s consent.
2. Accuracy and Reliability: Facial recognition technology is not always accurate and reliable, especially when it comes to women and people of color. And false identifications can have serious consequences. You need to be aware of the limitations of the technology and ensure that they are not relying solely on facial recognition to identify individuals.
3. Potential for Misuse: Journalists need to be aware of the potential for misuse and ensure that they are using the technology ethically and responsibly.

When Do I Use the Tools?

Here are four scenarios for using the tools.

1. Uncovering Hidden Truths: By responsibly — and within legal and ethical boundaries — utilizing these tools, reporters reveal secret misconduct or shed light on critical issues. Example: Debunking war propaganda.
2. Exposing Patterns and Networks: Here, reporters can assist in identifying potentially illicit patterns of behavior, connections, and networks. This can help educate the public on the underlying dynamics of events, uncover hidden agendas, or expose systemic issues that require public attention and scrutiny. Example: US Capitol insurrection.
3. Accountability and Holding Power to Account: Facial recognition tools can be deployed by reporters for purposes of accountability, providing evidence of specific individuals’ involvement in criminal activities or other wrongdoing. Example: Documenting a relationship between politicians and criminals pictured in the same photo.
4. Public Interest and Transparency: When employed responsibly and within legal and ethical boundaries, reporters can use these tools to uncover crucial information that informs public discourse, fosters understanding, and encourages transparency from those in positions of power. Example: Fact-checking hoaxes in social media. 

As always, consult your editor or employer about how you plan to use facial recognition in any reporting project and discuss what is appropriate for you individual situation and region.

GIJN welcomes reposts of this article for non-commercial use, but we ask that you follow our Creative Commons license agreement. 

Dutch-born Henk van Ess teaches, talks, and writes about open source intelligence with the help of the web and AI.  The veteran guest lecturer and trainer travels around the world doing internet research workshops. His projects include Digital Digging (AI & research), Fact-Checking the Web, Handbook Datajournalism (free download), and speaking as a social media and web research specialist.