Facial Forensics: The Next Generation of Tools for Solving Crimes

In 2015, in Jacksonville, Florida, an African-American man sold $50 worth of crack cocaine to an undercover officer. But, not wanting to blow their cover, the officer delayed in making an arrest; instead, he took pictures of the suspect. Local authorities then ran those images through a statewide database, where a proprietary algorithm from a company with the Marvel Villain name of MorphoTrust proposed a list of candidate suspects. An analyst looked over those candidates, and selected William Allen Lynch as the most likely match. Lynch was arrested, convicted, and is now serving an eight-year sentence. Prosecutors wanted 30.

Egregious sentencing matters aside (Lynch was a repeat offender who defended himself for a portion of his time in court), the case stirred up serious questions regarding the use of facial forensics in law enforcement.

In legal papers he wrote himself, by hand, in jail, Lynch argued that the jury should see more results from the police’s facial recognition database—more candidate matches—so that they could consider how his face had been selected. His request was denied, but it’s central to his appeal, which, mercifully, is being headed up by a public defender and not Lynch himself. This appeal could be the first time a US court addresses the concepts of due process and evidentiary standards in facial recognition.

Florida has one of the largest and most widely used facial recognition systems in the US. Based out of Pinellas County, it’s called FACES (Face Analysis Comparison & Examination System), and it’s powered by French-designed algorithms, which scan more than 30 million images from driver’s licenses, mugshots, and juvenile booking photos.

But to the people using it, it’s largely a black box: when deposed, the analyst who selected Lynch’s picture from a trove of candidate matches admitted to not understanding how the system worked, or even which type of rating provided by the system suggested the strongest facial match. And, even though research has shown facial recognition to be significantly less reliable than fingerprinting, no oversight or audits are conducted on Florida’s 8,000 facial recognition searches per month.

How Facial Recognition Tech Works

Since Woody Bledsoe first developed facial recognition software back in the 1960s, it’s become practically inescapable. Today, it’s found in everything from Apple Face ID, to Facebook’s auto-tag feature, to biometric passport control points in international airports.

For law enforcement, facial forensics is in dead body identification, partial face recognition, and identification of suspects from surveillance footage. It’s helped Taylor Swift identify stalkers at her concerts and it’s allowed Maryland authorities to identify a mass shooting suspect. Soon, facial recognition software might even be built into police body cameras, offering the prospect of real-time identification.

It’s not a perfect technology. Most facial recognition software still relies on 2D geometric mapping: using the distinguishing characteristics of individual landmarks (nose, mouth, eyes) and the distance between them to create a numerical code, called a faceprint. While it’s adept at comparisons to mugshots and passport photos, 2D mapping isn’t good at handling different lighting, angles, and facial expressions—particularly those found in grainy surveillance footage.

3D facial recognition software, like Apple Face ID, isn’t as easily fooled. By adding depth to the equation, a 3D faceprint can include contours, curves, and more nuanced forms of distance. This depth is achieved through the projection of invisible spectrums of light onto a face, and then, gauging the distance of varied points of that light from the camera itself using sensors. Apple Face ID uses 30,000 infrared dots to map a face. To compare that faceprint to 2D images, however, requires the faceprint to be down-converted to 2D, too.

For the most accuracy, facial forensics needs a human point of view.

How Facial Forensics Experts Do Their Job

Forensic facial experts fall into three general categories: examiners, reviewers, and super-recognizers:

• Examiners have extensive training in a rigorous, detailed, and time-consuming facial identification process. Their determinations are often transferred into official documentation that can then be used in court proceedings.
• Reviewers are trained to perform under quicker and less rigorous conditions, and their determinations may be used to help generate leads in criminal cases (such as Lynch’s).
• Super-recognizers are individuals with no professional training who have shown high natural ability for facial identification; they’re particularly useful to law enforcement in severely time-sensitive settings.

Within each category, there are different types of facial comparison: a one-to-one comparison is for identity verification, a one-to-many comparison for identifying an unknown face, and a many-to-many comparison for finding multiple database entries for one person under different names.

Some forms of facial identification, such as when a border guard compares a passport photo to the face of the person holding it, are quick, holistic, and non-technical. Other forms, such as attempting to identify a suspect from a sketch or partial photo, are slow, analytical, and technical.

No standardized method of facial recognition exists, and that cuts down on accountability and accuracy in the process. Some groups, like the Facial Identification Scientific Working Group (FISWG) are working to change that. FISWG proposes a morphological analysis, which is a systematic process that can be taught, documented, and scaled. But even this system has its flaws, and only limited studies have been done on its accuracy and reproducibility.

According to a study published by the National Academy of Sciences, professional facial examiners are the best human solution to the problem of face identification. The study also found that the best machine algorithms do about as well as the best humans, but optimal results are only achieved when the two work in collaboration: a single examiner paired with an algorithm performs better than two examiners paired together.

The Debate Around Facial Forensics

A 2016 study by Georgetown University’s Center on Privacy & Technology found that half of American adults were in a law enforcement agency’s facial recognition database—and most without their knowledge.

Even more alarming, a 2012 study found facial recognition to be particularly bad at identifying African Americans, and government tests in 2019 found that the type of facial recognition systems used in police investigations produce more false-positives when evaluating African American women.

The FBI’s Next Generation Identification system has access to 411 million photos in its database that uses a MorphoTrust facial recognition algorithm. The system has info-sharing agreements with several state and federal agencies. But its own privacy impact assessment found that the system “may not be sufficiently reliable to accurately locate other photos of the same identity, resulting in an increased percentage of misidentifications.” The US Government Accountability Office has called on the FBI to better ensure the privacy and accuracy of its NGI system; it’s unclear what, if anything, has been done to that end.

These problems aren’t isolated to a particular form of facial forensics; they’re endemic. Amazon’s facial recognition software, Rekognition, is used both by municipal police agencies and federal customs and immigration officers. But, according to an MIT study, it still shows gender and racial bias, particularly among women with darker skin: its rate of error for light skinned men was 0.8 percent, but over 34 percent for dark-skinned women. The Electronic Frontier Foundation has reported similar findings with other forms of facial recognition software.

Another issue for many is that it’s close to impossible to opt-out of facial recognition databases: Facebook forked over $550 million after losing a class-action lawsuit for identifying people without their consent. Even tech-savvy San Francisco has banned law enforcement from using facial recognition tools, citing issues of privacy and false positives.

Civil rights groups like the ACLU have called for an end to the use of facial recognition in policing, saying it can lead to false arrests or phony excuses to target minority communities; an ACLU study from 2018 showed Rekognition incorrectly matching 28 members of Congress with criminal mugshots.

The Future for Facial Recognition

New York City’s police commissioner believes facial recognition makes cities safer, and without violating anyone’s civil rights. NYPD has used it not only to make arrests, but also to identify Alzheimer’s patients and identify disfigured victims. And, according to the Innocence Project, 71 percent of its documented false convictions have resulted from wrongful witness identification; facial forensics could help here as well.

Other facial recognition tech advocates are keen to point out the altruistic potential of the tech, too. Amazon highlights Rekognition’s use by nonprofits like the International Centre for Missing & Exploited Children. But, after recent protests, they’re less public about their use by US Immigration and Customs Enforcement.

Still, the lack of clear legislation and oversight has caused tech giants like Microsoft and Google to refrain from handing their own versions of the tech to law enforcement authorities. They’re right to be wary: Woody Bledsoe’s pioneering research into facial recognition back in the 1960s wasn’t funded by altruistic sources, but by a still-unnamed intelligence agency. Today, more accountable oversight is needed.

The forensics, processes, and tech around facial recognition are still evolving. Government tests in 2010 found the average error rate of facial recognition algorithms for purely 2D mugshots to be 8 percent, but in 2018 that had dropped to 0.3 percent.

Today, deep convolutional neural networks (DCNNs), which are trained on millions of face images from thousands of people, can recognize faces in highly-variable, low-quality images. But modern facial forensics won’t become an equitable and acceptable practice until the tech—and the people behind the tech—acknowledge their shortcomings head-on.