GAO Report echoes NAS report that FBI' s study of anthrax letters leaves much to be desired, including lack of validation of the methods used/GAO

Here is the GAO report released today, two years in the making, a study done for unnamed "congressional requesters." In past years, the requesters have been listed, but in this case, they are not. This most likely reflects the sensitivity of this matter. Why so sensitive?

Because the FBI botched its investigation, never had even the shred of a case against Ivins, and this is not the first time they were called out on it. The 2011 National Academy of Science (NAS) report pointed out the same things as this GAO report: that FBI and its contractors developed methods that were not validated. They never proved that the anthrax in the letters actually grew from parent spores in Ivins' flask, as FBI purports. This failed claim forms the linchpin of the FBI's entire case.

Here is the 2011 WaPo editorial on the need for a comprehensive investigation, following release of the NAS report. Today's GAO investigation did not fulfill the WaPo's request to include an investigation of the case against Ivins himself.

Following the NAS report, in October 2011, veteran investigative reporters at McClatchey (Greg Gordon, who is amazing), PBS' Frontline and ProPublica dug deeper into the case than anyone so far had done, and created a TV show, a huge online cache of documents from the case, and several journalistic pieces, like this one.  Their work revealed new pieces of evidence that further undercut the FBI's case, like this and this. One article explored the validity of the FBI's scientific evidence.

UPDATE: These reporters write about the new GAO report here.

The National Academy of Science panel couched its conclusions in extremely polite, even deferential language toward the FBI. The NAS committee created new definitions for old words that could only have been intended to confuse readers of the report. (See my detailed discussion of this, below)

If you actually say, out loud, that the FBI faked its search for the anthrax criminal(s), flushed $100 million down the toilet in its most expensive case to date, and deliberately avoided conducting a credible investigation to find the anthrax letters perpetrators, then you question the entire edifice of US law enforcement and imply a conspiracy around an anthrax letters coverup at the highest levels of government, as suggested in Professor Graeme MacQueen's recent book.  And that, ladies and gentlemen, you are simply not permitted to do.

Step back from the abyss, get a limited GAO report that still took 2 years of review to see the light of day, and close the door on the anthrax letters case. Got it?


Here is what I wrote about the NAS report following its 2011 release:

Muddying the waters: contradictory NAS Report interpretations, and how the report itself asks to be read

Professor Paul Keim, an anthrax genetics expert at Northern Arizona University and FBI contractor, is claiming the NAS report supports the FBI's case.  It doesn't.  Please review the report and judge for yourself.

The NAS report has many confusing aspects.  But it basically says:

• The FBI totally screwed up its data collection of anthrax samples, and it should not be considered comprehensive.
• FBI failed to give the committee needed information, although it provided some response (sometimes "tersely") to every request for information.
• FBI found a number of morphologically unusual mutants ("morphotypes") in some letters, then chose some morphotypes to study further, but FBI provided no explanation for why only some morphotypes and not all letters were selected. 
• NAS pointed out that one could not say what expertise and equipment were required, nor how long it would take to produce the amount of spores used, absent that information.  (This conclusion challenged FBI claims that Ivins had the equipment and expertise needed, and also challenged the importance made of Ivins' late nights in the lab.)
• The assertion that Ivins tried to fool the FBI with the samples he submitted is unsupported. 
• FBI was chided for failing to use the newest molecular techniques, which could have speeded up the research and helped to clarify the relationship between the letter spores and Ivins' flask of spores. 

For example, from page 26 of the report:

"No written explanatory materials were provided with these documents that would fully
inform the committee as to why the analyses were done and how these documents contributed to the FBI investigations and conclusions. The material regarding analyses of the FBIR specimens was coded, often with different numbers for the same sample set. Consequently, the committee spent a considerable amount of time sorting through and attempting to interpret the available materials before it could begin to evaluate the science and consider the scientific conclusions. In addition, much of the information provided to the committee was compartmentalized and sections of some documents were redacted.
When the committee posed questions to the FBI for clarification, the agency was always
responsive; however, responses to questions were sometimes minimal or terse, or were deflected as intruding into the criminal investigation and beyond the purview of the committee despite the committee’s explanation that the questions were of a scientific nature."

And excerpts from page 119:

"The first challenge with the repository was the lack of independence among samples and an incomplete understanding of the provenance of samplesdue to the known history of sharing...  FBI scientists and investigators sought to determine the history of shipments among institutions and the genealogical relationships among samples in the repository, but they never obtained a complete record.

Another challenge with the repository was that, since the importance of the mutant genotypes was not fully understood when the subpoena protocol was written, the document was vague (e.g., “use an inoculum taken across multiple colonies”), and was not written in a way that would maximize the chance that variant genotypes in a mixed stock population would be submitted... After the importance of the mutant genotypes became known, there was no request for additional samples using a revised protocol that might have improved the sampling.
 
A final challenge was that the repository collection process was based on the integrity of the individuals asked to provide samples. If the motive for the repository was to identify the source of the letter material, standards of custody of evidence would dictate that agents of the FBI should have obtained the samples. In most instances, holders of the material were asked to provide samples and send them in. The sender could have been the instigator and may not have complied with instructions, as the FBI alleges with respect to Dr. Ivins."

Yet the report was otherwise couched in the most conciliatory language.  FBI was praised whenever possible.

The committee was barred from commenting directly on the guilt or innocence of suspects.  In order to get around this restriction and create a report that complied with its contract, while being as specific as possible about whether the science indicted Ivins, the NAS report included tables that presented FBI and DOJ statements, verbatim.  Then the report commented on whether the committee agreed with the statements.

Using this method, NAS' report was able to say (page 15):

The results of the genetic analyses of the repository samples were consistent with the finding that the spores in the attack letters were derived from RMR-1029, but the analyses did not definitively demonstrate such a relationship. The scientific data alone do not support the strength of the government’s repeated assertions that “RMR-1029 was conclusively identified as the parent material to the anthrax powder used in the mailings” (USDOJ, 2010, p. 20), nor the role suggested for the scientific data in arriving at their conclusions, “the scientific analysis coordinated by the FBI Laboratory determined that RMR-1029, a spore-batch created and maintained at USAMRIID by Dr. Ivins, was the parent material for the anthrax used in the mailings” (USDOJ, 2010, p. 8).

The report created new definitions to specify strength of association.  This goes to the heart of the report's meaning.  Here's what the report says, rather oddly, about how its language conveys the strength of an association (see page 41):

"Quantifying an association, as well as the degree of certainty (or uncertainty) in that association, involves statistical methods (see Chapter 6). Common language involves qualifiers, rather than quantifiable measures, of this association and the degree of confidence in it, which can cause confusion among practitioners from different fields that use the terms. Since the interpretation of these qualifiers and the ways in which they are used differ across disciplines (e.g., statistics, science, law, common language), their use by the committee is clarified here. In the chapters that follow, the committee uses the following four qualifiers of association, listed in order of increasing certainty (decreasing uncertainty):

• consistent with an association

• suggest an association

• indicate an association

• demonstrate an association

The expression “consistent with” is frequently used in this report and conveys the weakest level of certainty (greatest amount of uncertainty). In general, when the term “consistent with” is used, it means that an association may or may not be present; the available data can neither rule out nor confirm an association. The term “suggests” denotes a greater level of certainty for an association than “consistent with,” but even here the normal use of the word in science denotes a weaker level of certainty than is implied by the word in everyday parlance. That is, the potential for an association is stronger, and the evidence for the absence of an association is weaker, but both are still possible. In contrast, the terms “indicate” and “demonstrate” denote higher degrees of certainty and these are usually reserved for strong scientific conclusions (i.e., less uncertainty, or less likelihood of an absence of an association). All four levels could potentially be quantified with measures of “statistical significance,” but the committee does not assign such measures in most instances because the data at hand are generally not appropriate for such precise quantification of the degree of uncertainty. 

In summary, the reader is cautioned to consider carefully the terminology in this report in light of the fact that the qualifiers of certainty used here are those used most commonly in the scientific literature and that these words can carry different weight in common language and in the courtroom." 

So the term "consistent with an association" in the NAS-FBI context, implies the weakest possible association.  Got that?   Keim and the FBI have taken "consistent with" to mean the NAS Report supports the FBI claims, when the report's own definitions state that "consistent with" implies "the greatest amount of uncertainty" about the association.  UPDATE:   Keim is quoted in the 2/17/11 Global Security Newswire with the following (disingenuous) statement:

Keim disputed news reports suggesting that this week's analysis questions the FBI for naming Ivins as the perpetrator of the attacks.  "The committee isn't saying that. ... All the major conclusions that the FBI came to, the committee said, 'Yeah, the evidence is consistent with that.'"

[Sorry for all the mumbo-jumbo, but this level of detail helps explain how opposing sides may each cite this report to claim victory--Nass]