Interesting article from Popular Mechanics. Well not really an article, more of an op-ed hit piece with little to no actual research into real forensics. Instead just the regurgitation of old-tired attacks on forensics based on half-truths. Can’t say I agree with much of it. I point out factual errors, and make my own comments inline below.
Original article posted here:
CSI Myths: The Shaky Science Behind Forensics
Forensic science was not developed by scientists. It was mostly created by cops, who were guided by little more than common sense. And as hundreds of criminal cases begin to unravel, many established forensic practices are coming under fire. PM takes an in-depth look at the shaky science that has put innocent people behind bars.
By Brad Reagan
Published in the August 2009 issue.On Jan. 11, 1992, the jury in the murder trial of Roy Brown heard from a dentist named Edward Mofson. To establish his credentials, Dr. Mofson testified that he was certified in forensic odontology, belonged to six related professional organizations and did forensic consulting throughout New York state. He then explained that several months earlier he was called to the morgue in Cayuga County, New York, to analyze the body of 49-year-old Sabina Kulakowski.
Kulakowski’s corpse was found by a volunteer firefighter on a dirt road some 300 yards from the farmhouse where she lived, which had burned to the ground in the night. She was severely beaten and stabbed, and there were multiple bite marks on her body. Brown was a natural suspect in the grisly murder. The week before the crime, the hard-drinking 31-year-old had been released from jail on charges of threatening to “wipe everybody out” at the social services office where Kulakowski worked; the agency had put his daughter into foster care. In addition to the motive, the district attorney at trial produced other circumstantial evidence, including testimony from Brown’s two ex-wives that he had bitten them. But Mofson, now deceased, was the centerpiece of the prosecution.
Mofson testified that seven bite marks found on Kulakowski were “entirely consistent” with dental impressions taken from Brown. It was the only physical evidence tying Brown to the crime. Although a defense expert disputed Mofson’s findings, the jury convicted Brown of second-degree murder. He was sentenced to 25 years to life in prison.
As the years ticked by, few listened as Brown proclaimed his innocence from his cell in the Elmira Correctional Facility. Then Brown got an unusual lucky break. His stepfather’s house burned down, taking with it all of his records from the trial. To replace his documents, Brown submitted an open records request to the county. The sheriff who processed Brown’s request mistakenly sent him the entire investigative file. It revealed another suspect: Barry Bench, the firefighter who discovered Kulakowski’s body. Bench’s brother had dated Kulakowski up until two months before the murder and Bench was reportedly upset that she continued to live in the family farmhouse. On the day before Christmas in 2003, Brown sent a letter to Bench letting him know he was seeking DNA testing. “Juries can make mistakes,” he wrote. But, “DNA is God’s creation, and God makes no mistakes.” Soon after receiving the message, Bench committed suicide by jumping in front of an Amtrak train. DNA tests confirmed that Bench was guilty of Kulakowski’s murder, and Brown was set free.
It’s interesting that the premise here is that forensics is “faulty” because it was developed by cops. Yet the example used is a faulty bite-mark identification made by a dentist. I don’t know of any cops that are dentists
The faulty identification that sent Brown to prison for 15 years may seem like a rare glitch in the U.S. criminal justice system. It wasn’t. As DNA testing has made it possible to re-examine biological evidence from past trials, more than 200 people have had their convictions overturned. In approximately 50 percent of those cases, bad forensic analysis contributed to their imprisonment.
It would be nice to have more evidence to back up this 50% claim. Usually these cases have to do when comparing serological results versus DNA results. Serology is a technique where different biological factors were compared to see if a suspect could be included or excluded. Things like A-B-O typing for example. An analyst 20 years ago concluding the suspect had AB– blood, and the defendant had AB– blood includes the suspect as someone who could have donated the blood to the scene. It does not mean “he’s guilty.” And 20 years later when a more advanced technique is developed, which is purportedly capable of individualizing biological material to one person, says the defendant in the first trial didn’t do it, doesn’t mean the original forensic testing was “faulty”, it just means the newer technique is better able discriminate between individuals.
On television and in the movies, forensic examiners unravel difficult cases with a combination of scientific acumen, cutting-edge technology and dogged persistence. The gee-whiz wonder of it all has spawned its own media-age legal phenomenon known as the “CSI effect.” Jurors routinely afford confident scientific experts an almost mythic infallibility because they evoke the bold characters from crime dramas. The real world of forensic science, however, is far different. America’s forensic labs are overburdened, understaffed and under intense pressure from prosecutors to produce results. According to a 2005 study by the Department of Justice, the average lab has a backlog of 401 requests for services. Plus, several state and city forensic departments have been racked by scandals involving mishandled evidence and outright fraud.
But criminal forensics has a deeper problem of basic validity. Bite marks, blood-splatter patterns, ballistics, and hair, fiber and handwriting analysis sound compelling in the courtroom, but much of the “science” behind forensic science rests on surprisingly shaky foundations. Many well-established forms of evidence are the product of highly subjective analysis by people with minimal credentials—according to the American Society of Crime Laboratory Directors, no advanced degree is required for a career in forensics. And even the most experienced and respected professionals can come to inaccurate conclusions, because the body of research behind the majority of the forensic sciences is incomplete, and the established methodologies are often inexact. “There is no scientific foundation for it,” says Arizona State University law professor Michael Saks. “As you begin to unpack it you find it’s a lot of loosey-goosey stuff.”
It’s true there is no requirement for advanced degrees in most labs. There are a few forensic positions that do require masters level or better. But so what? Up until recently there haven’t been many colleges that offered course work in forensics. People who got into the field were degreed chemists and similar. It’s only since the OJ case, television shows like CSI, and the popularization of forensics that many universities have even started offering forensic programs.
Which when you think about it, is a bit of a hypocrisy. The same University scholars who claim forensics is a crock, and unscientific are now making money training students on how to be forensic scientists. Is the problem the universities didn’t get their share of the money, and that’s the real problem with forensics? Look at what else ASU is charging students for now, and you’ll see a probably angle on where the trash-talking about forensics is coming from.
And by the way. A college course based on forensic firearms identification? Please, most universities’ policies on firearms would make such a program impossible. A forensic scientist at the Phoenix Police Department did some important research in forensic firearms identification while enrolled at U.C. Davis. He and his instructor had to provide their own firearms, and do all test firing on their own time away from campus due to school rules. No universities aren’t ready to actually even begin looking at fully embracing forensic science yet.
Not surprisingly, a movement to reform the way forensics is done in the U.S. is gaining momentum. The call for change has been fueled by some embarrassing failures, even at the highest levels of law enforcement. After the 2004 train bombings in Madrid, Spain, the FBI arrested Oregon lawyer Brandon Mayfield and kept him in jail for two weeks. His incarceration was based on a purported fingerprint match to a print found on a bag of detonators discovered near the scene of the crime. As a later investigation by the Justice Department revealed, the FBI’s fingerprint-analysis software never actually matched Mayfield to the suspect fingerprint, but produced him as an “unusually close nonmatch.” Lacking any statistical context for how rare such similarities are, investigators quickly convinced themselves that Mayfield was the prime suspect.
This is true. It showed a problem called “Institutional Bias” that’s frequency is unknown. But it does show the need to pull forensic labs out of police, law enforcement, and prosecutor’s agencies.
As far as lacking “statistical context”, I hope that doesn’t mean that in order for something to be reliable, it needs statistics to back it up. I’ve had enough statistics courses in my life to know there are lies, damn lies, and statistics.
After getting my degree is toxicology, with an emphasis on analytical chemistry, I never had a class where after I performed my experiment, and got my end product, I had to show a statistical likely-hood ratio that my end product is what it’s supposed to be.
The next year, 2005, Congress commissioned the National Academy of Sciences (NAS) to examine the state of forensics in U.S. law enforcement. The result was a blistering report that came out this February, noting “serious deficiencies” in the nation’s forensic science system and advocating extensive reforms. It specifically noted that apart from DNA, there is not a single forensic discipline that has been proven “with a high degree of certainty” to be able to match a piece of evidence to a suspect. The obvious implication is the sobering possibility that more Roy Browns are currently locked up based on shoddy science. Then there’s the flip side: A lot of bad guys who should be in prison still roam free. A study by the Innocence Project of the prisoners exonerated by DNA found that the real perpetrators were identified in 103 cases—roughly half. In all but one, the perpetrator committed at least one serious crime after the innocent person was jailed.
The scientific method is instrumental to our understanding of the physical world. To scientists, the process is sacrosanct: Research your topic, generate a hypothesis, test the hypothesis, analyze your data and then publish the results for peer review. Forensic science, however, was not developed by scientists. It was created by cops—often guided by little more than common sense—looking for reliable ways to match patterns from clues with evidence tied to suspects. What research has been done understandably focuses on finding new techniques for putting criminals in jail.
I just have to disagree with the notion that forensic science was created by cops. Every forensic discipline I’m trained in, as I was trained, employs the scientific method. Every case, every ballistic comparison I perform I employ the scientific method.
In the academic community the legal sciences get a comparative trickle of federal funding. In 2007, the National Institute of Justice awarded 21 grants for forensic research (excluding DNA) totaling $6.6 million; the National Institutes of Health awarded 37,275 grants totaling $15 billion. And without a wealth of statistically defensible research to back up their evidence, forensic examiners generally rely upon their own intuition and the experience of their colleagues. “You can’t take a few case studies and say, ‘Oh, it worked on these people; it must be reliable,’” says Karen Kafadar, an Indiana University statistics professor and a member of the NAS committee. “That is hardly a placebo-controlled, double-blind randomized trial.”
I think the real rub here (it all started when the Bush administration really started to push federal grant money to police crime labs) is the fact that the academics, the university professors, the doctors of philosophy, the college research projects, all started getting less funding, and forensics started getting more.
Once again, the call of the academic “where’s the statistics?”
The FBI’s errors in the Madrid bombing case were particularly surprising because they called into question one of the gold standards of evidence—fingerprints. In recent years, legal experts have become deeply concerned about the accuracy of the “friction ridge analysis” central to fingerprint identification. Fingerprints are believed to be unique, but the process of matching prints has no statistically valid model. And forensic examiners are often working in an imperfect world, where prints taken in a police station on an ink pad are compared to prints left at a crime scene, which may be smudged or partially captured. Yet, as University of California–Los Angeles law professor Jennifer Mnookin has written, “fingerprint examiners typically testify in the language of absolute certainty.”
Regardless of what scientific discipline you engage in, there is an certain amount of subjectivity. What makes it scientific is the fact that the process can be done again, and the results should be reproducible. In the Madrid case, once “institutional bias” was removed, the error was discovered.
Regardless of what type of scientific testing there is, due to the subjectivity, there will errors. Scientists are not “gods”, and they are not always correct. Once again, another reason why forensic labs should be moved out of the police departments, and prosecutor’s offices. So there is less “institutional bias”, and there is a greater chance of policies being put in place to allow for blind re-analysis.
A 2006 study by the University of Southampton in England asked six veteran fingerprint examiners to study prints taken from actual criminal cases. The experts were not told that they had previously examined the same prints. The researchers’ goal was to determine if contextual information—for example, some prints included a notation that the suspect had already confessed—would affect the results. But the experiment revealed a far more serious problem: The analyses of fingerprint examiners were often inconsistent regardless of context. Only two of the six experts reached the same conclusions on second examination as they had on the first.
Ballistics has similar flaws. A subsection of tool-mark analysis, ballistics matching is predicated on the theory that when a bullet is fired, unique marks are left on the slug by the barrel of the gun. Consequently, two bullets fired from the same gun should bear the identical marks. Yet there are no accepted standards for what constitutes a match between bullets. Juries are left to trust expert witnesses. “‘I know it when I see it’ is often an acceptable response,” says Adina Schwartz, a law professor and ballistics expert with the John Jay College of Criminal Justice.
No properly trained forensic firearms examiner testifies that two bullets fired from the same barrel should bear identical marks. We learn this the first time we begin training in comparison analysis. We see first hand that subsequent test firings do not have identical marks.
Ms. Schwartz background is in philosophy, and I highly doubt she’s spent much time behind a comparison microscope, let alone gone through the normal 2 — 3 year training period. She is famous for writing scathing articles that get published in legal journals with no actual scientific peer-review. Her opinions are touted as fact by attorneys who’s clients are in trouble based on firearms evidence, and of course by other academics (probably upset about lost grant funding).
What they don’t talk about is the repeated research over the past many decades that show the reliability of firearms identification analysis. Including the most recent where over 500 different firearms examiners across the globe completed a “worst case scenario” test with 10 consecutively manufactured barrels, and not one of them had a “wrong” answer. It wasn’t an easy test, I know, I participated in it.
Not all forensic disciplines are in dispute.
Techniques that grew out of organic chemistry and microbiology have a strong scientific foundation. For example, chromatography, a method for separating complex mixtures, enables examiners to identify chemical substances in bodily fluids—evidence vital to many drug cases. The evolution of DNA analysis, in particular, has set a new scientific standard for forensic evidence. But it also demonstrates that good science takes time.
I don’t know how drug identification can be “reliable.” I mean it was drug identification cases that brought us the Melendez-Diaz decision. And I know when I was performing drug identifications, I never had any “statistical context” I looked at to “prove” my conclusions.
The double-helix structure of DNA was discovered in the 1950s, but it wasn’t until 30 years later that sample analysis became sophisticated enough for positive ID. In 1987, a serial rapist by the name of Tommie Lee Andrews was the first person convicted in the U.S. using DNA. Nevertheless, for several years scientists continued to research and debate what constitutes a satisfactory match. The resulting process is broadly accepted and quantifiable (when using the most advanced analysis, there is a one in more than a quadrillion chance of a random match of two strangers’ nuclear DNA).
Unless you have a case like mine. A few years ago I had a case where the DNA analyst, due to subjective error, had to change the “one in a quadrillion” chance, to around “one in a hundred.” Kind of makes one question the “perfectness” of DNA analysis. That analyst actually was one of those individuals was an “advanced degree” the article was so quick to point out the importance of.
But DNA constitutes less than 10 percent of the case load at U.S. crime labs. The goal going forward, everyone agrees, is to make the rest of forensics more rigorous and statistically grounded. Promising work is already being done: Sargur Srihari, a pattern-recognition expert with the State University of New York at Buffalo, is developing software to help quantify the certainty of fingerprint matches. And, Nicholas Petraco, a chemist and mathematician at John Jay, is working on a database of microscopic tool marks to give statistical significance to the identification of burglars’ tools.
The NAS report recommends the establishment of an independent entity—a National Institute of Forensic Science—which would be the central authority responsible for funding research as well as creating and promulgating the standards of evidence and certification for experts. If such a system worked properly, juries would only hear from experts who are certified in their fields and examiners who work in accredited laboratories.
It’s likely that the microscope of serious scientific scrutiny will turn disciplines such as fingerprint and ballistics analysis, which have long histories and large sample sizes, into stronger standards of evidence. But many other forensic disciplines may be classified as far less sound. Bite marks, footprints, tire tracks, handwriting, bloodstain patterns and other forms of analysis that suffer from multiple confounding variables could end up being used as exclusionary evidence or as qualified supporting evidence only. Some types of evidence may be completely discredited. That’s what happened with voiceprint analysis and lead analysis of bullets, which were popular forensic techniques until studies showed significant error rates.
Here is another gross misstatement in this article (did they even do any research before they wrote this hit piece?), bullet let analysis was never “popular.” Only the FBI performed it. One lab in hundreds or thousands does not “popular” make.
Within the forensic community, the reaction to the mounting criticism is mixed. Some are offended and blame the “propaganda” of defense attorneys and the snobbery of academics. Dean Gialamas, president of the American Society of Crime Laboratory Directors, says most techniques have “a strong foundation in science” even if they have not been subject to the type of applied research needed to satisfy critics. And he notes that his organization has long advocated more standardization and stronger ethics rules, so hired guns can’t pollute courtrooms with biased testimony. At the end of the day, Gialamas and most other forensic experts say they are confident their methods will ultimately be validated by further research. Even critics of the current system say forensics should remain a critical part of law enforcement. “Let’s just give it to people as completely and honestly as we possibly can,” Saks says.
It will take years to fully reconcile the rigors of the scientific method with the needs and processes of the judicial system. But in the meantime, questionable forensic science will continue to tip the scales of justice. And when bad decisions are made in the courtroom, an innocent person’s entire life can be swept right out from under him. It happened to Steven Barnes 20 years ago. Then 23 years old, he was brought to trial for the rape and murder of a 16-year-old girl. He had never been arrested before and was confident he’d be cleared. Yet he watched as forensics expert Elaine Pagliaro testified that two hairs found in Barnes’s pickup were microscopically similar to the victim’s. Pagliaro also noted that soil samples taken from the truck were consistent with dirt from the crime scene and even that a distinctive pattern from the victim’s jeans was similar to an imprint left on the truck.
Due largely to her testimony, Barnes was sentenced to 25 years to life in prison. Last year, he was cleared by DNA and released. He’d never been on the Internet or used a cellular phone, and his girlfriend, who initially stuck by him after he went to prison, had long ago married another man. Barnes told Popular Mechanics that he works hard not to be overwhelmed by bitterness, even toward the jurors. “They must have thought, ‘[Pagliaro] knows what she is talking about.’”
And saying the hairs were similar, and the soil was similar, is not damning evidence for the accused, nor does there appear to be any factual errors in the testing or testimony of the forensic scientist. It’s how the lawyers misrepresent what the results of the analysis are is the problem.
Pagliaro, a veteran analyst with the Connecticut State Police, has recently co-authored a book called The Real World of a Forensic Scientist. “I think this scrutiny is actually good,” she says. “It’s important for the public to have a realistic expectation of what the science can do.” As for the Barnes case, there is no suggestion of impropriety regarding her testimony, but none of the evidence she presented was based on statistically validated science. “You feel awful someone spent all that time in jail,” she says. “All you can do is look back and say, ‘Was that the best we could do?’”
Ms Pagliaro is absolutely correct. The problem isn’t with forensic science. It certainly isn’t wrong because it wasn’t “developed” in the ivory towers of academia. The problem is the public miss perception of forensic science.
Just because an analyst says two fibers, or two hairs are similar, doesn’t mean they have a common source. It just means that chemically and physically there are not differentiable by current scientific testing techniques. But 20 years from now, where there is a “DNA” technique for carpet fibers who knows? Technology evolves, but it does not mean all previous work performed is faulty.
Imagine if for as long as you can remember, you wake up in the morning, and notice the sun rising in the east. Later in the day you always notice the sun setting in the west. Based on your observation, you form the hypothesis that tomorrow the sun will rise in the east and set in the west. Lo and behold, the next day your hypothesis is proved correct. In this analogy, lawyers and academics would argue that because you have never performed any statistical analysis, your hypothesis is wrong, and your conclusions that the sun’s rising and setting phenomena are not based on reliable scientific methodology.
No related posts.
[…] in forensic testimony aren’t what the scientist testified to, but rather how the lawyers misrepresented the scientists statements later in the […]
[…] evidence has been questioned in this blog before, specifically in an article by Popular Mechanics related to the NAS report, and the article on recent congressional […]
[…] a problem with the “academia” criticizing the field of forensic science. (Just read my thoughts on the recent Popular Mechanics article where the academics try to shoot down forensic scie.…) Constructive criticism is always appreciated, but complaints and attacks by people with no actual […]
Given a lawyers follow a process with not more than an ounce of scientific process. The Law doesn’t follow a process designed to send only the guilty to jail and release the innocent with the minimum cost– almost the completely the opposite. The whole area of witnesses is full of flaws as many pschology experiments have demonstrated.