Muddying the waters: contradictory NAS Report interpretations, and how the report itself asks to be read

Professor Paul Keim, an anthrax genetics expert at Northern Arizona University and FBI contractor, is claiming the NAS report supports the FBI's case.  It doesn't.  Please review the report and judge for yourself.

The NAS report has many confusing aspects.  But it basically says:

• The FBI totally screwed up its data collection of anthrax samples, and it should not be considered comprehensive.
• FBI failed to give the committee needed information, although it provided some response (sometimes "tersely") to every request for information.
• FBI found a number of morphologically unusual mutants ("morphotypes") in some letters, then chose some morphotypes to study further, but FBI provided no explanation for why only some morphotypes and not all letters were selected. 
• NAS pointed out that one could not say what expertise and equipment were required, nor how long it would take to produce the amount of spores used, absent that information.  (This conclusion challenged FBI claims that Ivins had the equipment and expertise needed, and also challenged the importance made of Ivins' late nights in the lab.)
• The assertion that Ivins tried to fool the FBI with the samples he submitted is unsupported. 
• FBI was chided for failing to use the newest molecular techniques, which could have speeded up the research and helped to clarify the relationship between the letter spores and Ivins' flask of spores. 

For example, from page 26 of the report:

"No written explanatory materials were provided with these documents that would fully
inform the committee as to why the analyses were done and how these documents contributed to the FBI investigations and conclusions. The material regarding analyses of the FBIR specimens
was coded, often with different numbers for the same sample set. Consequently, the committee
spent a considerable amount of time sorting through and attempting to interpret the available
materials before it could begin to evaluate the science and consider the scientific conclusions. In
addition, much of the information provided to the committee was compartmentalized and
sections of some documents were redacted.
When the committee posed questions to the FBI for clarification, the agency was always
responsive; however, responses to questions were sometimes minimal or terse, or were deflected
as intruding into the criminal investigation and beyond the purview of the committee despite the
committee’s explanation that the questions were of a scientific nature."

And excerpts from page 119:

"The first challenge with the repository was the lack of independence among samples and an incomplete understanding of the provenance of samples due to the known history of sharing...  FBI scientists and investigators sought to determine the history of shipments among institutions and the genealogical relationships among samples in the repository, but they never obtained a complete record.

Another challenge with the repository was that, since the importance of the mutant genotypes was not fully understood when the subpoena protocol was written, the document was vague (e.g., “use an inoculum taken across multiple colonies”), and was not written in a way that would maximize the chance that variant genotypes in a mixed stock population would be submitted... After the importance of the mutant genotypes became known, there was no request for additional samples using a revised protocol that might have improved the sampling.
 
A final challenge was that the repository collection process was based on the integrity of the individuals asked to provide samples. If the motive for the repository was to identify the source of the letter material, standards of custody of evidence would dictate that agents of the FBI should have obtained the samples. In most instances, holders of the material were asked to provide samples and send them in. The sender could have been the instigator and may not have complied with instructions, as the FBI alleges with respect to Dr. Ivins."

Yet the report was otherwise couched in the most conciliatory language.  FBI was praised whenever possible.

The committee was barred from commenting directly on the guilt or innocence of suspects.  In order to get around this restriction and create a report that complied with its contract, while being as specific as possible about whether the science indicted Ivins, the NAS report included tables that presented FBI and DOJ statements, verbatim.  Then the report commented on whether the committee agreed with the statements.

Using this method, NAS' report was able to say (page 15):

The results of the genetic analyses of the repository samples were consistent with the finding that the spores in the attack letters were derived from RMR-1029, but the analyses did not definitively demonstrate such a relationship. The scientific data alone do not support the strength of the government’s repeated assertions that “RMR-1029 was conclusively identified as the parent material to the anthrax powder used in the mailings” (USDOJ, 2010, p. 20), nor the role suggested for the scientific data in arriving at their conclusions, “the scientific analysis coordinated by the FBI Laboratory determined that RMR-1029, a spore-batch created and maintained at USAMRIID by Dr. Ivins, was the parent material for the anthrax used in the mailings” (USDOJ, 2010, p. 8).

The report created new definitions to specify strength of association.  This goes to the heart of the report's meaning.  Here's what the report says, rather oddly, about how its language conveys the strength of an association (see page 41):

"Quantifying an association, as well as the degree of certainty (or uncertainty) in that
association, involves statistical methods (see Chapter 6). Common language involves qualifiers,
rather than quantifiable measures, of this association and the degree of confidence in it, which
can cause confusion among practitioners from different fields that use the terms. Since the
interpretation of these qualifiers and the ways in which they are used differ across disciplines
(e.g., statistics, science, law, common language), their use by the committee is clarified here. In
the chapters that follow, the committee uses the following four qualifiers of association, listed in
order of increasing certainty (decreasing uncertainty):

consistent with an association
suggest an association
indicate an association
demonstrate an association

The expression “consistent with” is frequently used in this report and conveys the
weakest level of certainty (greatest amount of uncertainty). In general, when the term “consistent
with” is used, it means that an association may or may not be present; the available data can
neither rule out nor confirm an association. The term “suggests” denotes a greater level of
certainty for an association than “consistent with,” but even here the normal use of the word in
science denotes a weaker level of certainty than is implied by the word in everyday parlance.
That is, the potential for an association is stronger, and the evidence for the absence of an
association is weaker, but both are still possible. In contrast, the terms “indicate” and
“demonstrate” denote higher degrees of certainty and these are usually reserved for strong
scientific conclusions (i.e., less uncertainty, or less likelihood of an absence of an association).
All four levels could potentially be quantified with measures of “statistical significance,” but the
committee does not assign such measures in most instances because the data at hand are
generally not appropriate for such precise quantification of the degree of uncertainty.
In summary, the reader is cautioned to consider carefully the terminology in this report in
light of the fact that the qualifiers of certainty used here are those used most commonly in the
scientific literature and that these words can carry different weight in common language and in
the courtroom."

So the term "consistent with an association" in the NAS-FBI context, implies the weakest possible association.  Got that?   Keim and the FBI have taken "consistent with" to mean the NAS Report supports the FBI claims, when the report's own definitions state that "consistent with" implies "the greatest amount of uncertainty" about the association.  UPDATE:   Keim is quoted in the 2/17/11 Global Security Newswire with the following (disingenuous) statement:

Keim disputed news reports suggesting that this week's analysis questions the FBI for naming Ivins as the perpetrator of the attacks.  "The committee isn't saying that. ... All the major conclusions that the FBI came to, the committee said, 'Yeah, the evidence is consistent with that.'"

[Sorry for all the mumbo-jumbo, but this level of detail helps explain how opposing sides may each cite this report to claim victory--Nass]