David Kirby: HuffPost Report on CDC’s Vaccine Safety Datalink Uninformative and Completely MisleadingPublished June 21st, 2008 in Autism, Child Health, Fallacious Medical Reporting, Infant Health, MMR Vaccine, Medical & Epidemiological Studies, Medical Reporting, Quacks, Vaccines
“Medical reporter” David Kirby has delivered a potentially explosive report to his unfortunate and misinformed minions at the Huffington Post, in which he shows a startling string of misunderstandings and complete lack of knowledge of basic epidemiologic design and methods. Furthermore, he writes that Dr. Julie Gerberding “admits to a startling string of errors in the design and methods used in the CDC’s landmark 2003 study that found no link between mercury in vaccines and autism, ADHD, speech delay or tics,” when, in fact, the CDC report admitted no such thing about the 2003 study.
Gerberding was responding to a 2006 Report of the Expert Panel on Thimerosal Exposure in Pediatric Vaccines: Feasibility of Studies Using the Vaccine Safety Datalink to the National Institute of Environmental Health Sciences (NIEHS). Nowhere in the 2006 report, however, did the NIEHS panel conclude that the CDC’s 2003 thimerosal safety study was riddled with “several areas of weaknesses” that combined to “reduce the usefulness” of the study. In fact, in the NIEHS panel meeting that generated the 2006 report, the quality of the CDC’s 2003 thimerosal safety study was not even discussed. This can be seen clearly if you carefully read the NIEHS Report of the Expert Panel.
In addition, earlier this week Epi Wonk had a long discussion with one of the Expert Panel Members (who adamantly insisted that he/she remain nameless), who confirmed three things for me:
(1) The purpose of the NIEHS Expert Panel was exactly as stated in the report:
“It has been proposed that the Vaccine Safety Datalink could be used to look at the association between autistic disorder (AD) or autism spectrum disorders (ASD) by means of an ECOLOGIC ANALYSIS (emphasis mine) that would compare rates before and after the removal of thimerosal from most childhood vaccinations. To determine the feasibility and potential contribution and/or drawbacks of such a study, and to consider alternative study designs that could be conducted using the VSD database, the NIEHS convened a panel of experts…
(2) The quality of previous epidemiological studies of the association between thimerosal and autism was not discussed.
(3) The overall quality of the 2003 Verstraeten et al. study was not discussed. Indeed, in the section of the report in which the expert panel considered research panels other than ecologic analyses, which they did dismiss as riddled with “several areas of weaknesses” that combined to “reduce the usefulness” of ecologic studies, the expert panel “…recommended that further consideration be given to conducting an extension of the Verstraten study that would include additional years for follow up, would add more managed care organizations and reexamine the criteria for exclusion of births and/or take a sensitivity analyses approach to examining the impact of various exclusion criteria.”
In the HuffPost story, David Kirby quotes Julie Gerberding as writing that her agency “does not plan to use” the Vaccine Data Safetylink (VSD) for any future “ecological studies” of autism. “In fact”, Kirby continues, “Gerberding’s report said, any continued use of the VSD for continued ecological studies of vaccines and autism ‘would be uninformative and completely misleading.’”
Well, yes, that’s what the CDC thinks about using the VSD for ecologic analyses. I couldn’t agree more. At this point I obviously need to step back and explain about ecologic analyses. Fortunately, I taught epidemiologic design and methods for about 35 years, I had some students almost as clueless as David Kirby, but I’m a patient teacher. Another interesting fact is that there has only been one ecologic study published using the VSD, and I’ve written extensively about the study on this blog. Guess what? It wasn’t done by the CDC, who knew better long before the 2006 NIEHS Expert Panel. I’m speaking of the infamous Young-Geier Autism Study. So let me paraphrase from my explanation of “ecologic” in my previous critique of that paper:
KIRBY: “The final NIEHS report was a serious and thoughtful critique of where the CDC went wrong in its design, conduct and analysis of the study. The NIEHS panel “identified several serious problems” with the CDC’s effort.
FACT: The final NIEHS report was a serious and thoughtful critique of “using the VSD to look at the association between autistic disorder (AD) or autism spectrum disorders (ASD) by means of an ecologic analysis that would compare rates before and after the removal of thimerosal from most childhood vaccinations, to determine the feasibility and potential contribution and/or drawbacks of such a study, and to consider alternative study designs that could be conducted using the VSD database.” The NIEHS panel “identified several serious problems that were judged to reduce the usefulness of an ecologic study design using the VSD to address the potential association between thimerosal and the risk of AD/ASD“
KIRBY: “…the NIEHS had criticized CDC for failing to account for other mercury exposures, including maternal sources from flu shots and immune globulin, as well as mercury in food and the environment. ‘CDC acknowledges this concern and recognizes this limitation,’ the Gerberding reply says.”
ACTUAL QUOTE FROM CDC REPORT: “NIEHS Finding: Difficulty in estimating cumulative exposure of child to organic mercury: The panel expressed concern that VSD adminstrative data or medical charts would not be accurate in recording or estimating a child’s total mercury exposure from sources other than vaccines, such as diet, air and water. CDC Response: CDC acknowledges this concern and recognizes this limitation. In addition to administrative data and medical chart review, CDC has employed parent interviews to identify total cumulative mercury exposure from sources other than vaccines, such as diet. Often, however, parent recall, for events several years in the past, poses limitations as well.”
KIRBY: “The NIEHS also took CDC to task for eliminating 25% of the study population for a variety of reasons, even though this represented, ‘a susceptible population whose removal from the analysis might unintentionally reduce the ability to detect an effect of thimerosal.’ This strict entry criteria likely led to an ‘under-ascertainment’ of autism cases, the NIEHS reported. ‘CDC concurs,’ Gerberding wrote, again noting that its study design was ‘not appropriate for studying this vaccine safety topic. The data are intended for administrative purposes and may not be predictive of the outcomes studied.’
FACT: These four sentences are outright lies. The NIEHS Expert Panel never “took the CDC to task for eliminating 25% of the study population…” On the contrary, when discussing potential alternative designs (other than ecological studies), “another possibility that generated support by the panel was an expansion of the VSD study published by Verstraten et al. The availability of several additional years of VSD data was seen as an opportunity to provide a more powerful test of any potential association between thimerosal and AD/ASD and would enable reconsideration of some aspects of the original study design (e.g., exclusion criteria)” It was unclear to the panel what effect exclusion of low birth weight infants and those with congenital or severe perinatal disorders or born to mothers with serious medical problems of pregnancy had on the results of the Verstraeten et al. study; an expanded future study in which sensitivity analyses both including and excluding children with perinatal problems was recommended. The quote that begins with “CDC concurs” has no bearing on the Verstraeten et al. study, as implied by Kirby. Gerberding is responding to an NIEHS Expert Panel point about case ascertainment. Here is the entire quote from the CDC report: “CDC responds: ‘CDC concurs with the recommendation that broader ICD-9 codes should be considered. The weakness further emphasizes why an ecological design is not appropriate for studying this vaccine safety topic using the VSD. The VSD data are intended adminstrative purposes and may not be predictive of the outcome studied. Because the outcomes have not been validated and considering the sensitivity of this issue, any VSD study of vaccines and autism, including a broader list of ICD-9 codes, would require chart review.’”
KIRBY: “Another serious problem was that the HMOs changed the way they tracked and recorded autism diagnoses over time, including during the period when vaccine mercury levels were in decline. Such changes could ‘affect the observed rate of autism and could confound or distort trends in autism rates,’ the NIEHS warned. ‘CDC concurs,’ Dr. Gerberding wrote again, ‘that conducting an ecologic analysis using VSD administrative data to address potential associations between thimerosal exposure and risk of ASD is not useful.’
FACT: This is correct. Believe it or not, Mr. Kirby has got it right this one time. The charge of the NIEHS Expert Panel was to determine whether the VSD should be used to to do ecological studies. The expert panel concluded, “No.” The CDC concurs.
I’ll leave you with the most important summarizing quote of the CDC report:
NIEHS Finding: Strengths: The panel identified several major strengths of the VSD to be: its ability to detect infrequent, vaccine-related adverse events of modest size; the possibility to supplement the MCO administrative data with reviews of medical records, interviews with parents and children, and additional diagnostic assessments; and the availability of demographic information about the MCO members.
CDC Response: CDC agrees with the panel’s assessment of the strengths of the VSD Project to evaluate vaccine safety concerns. The VSD is a unique public-private collaboration that provides a model for the study of patient safety concerns by using individual-level data. In addition, CDC recognizes the tremendous value of the VSD as a national resource of expertise in vaccine safety research.
I can’t help but agree with Kirby’s recommendation, “I hope everyone will read these documents, including the recommendations to make the VSD better, and the CDC’s agreement with all of the suggestions.” As the waning weeks of Omnibus Autism testimony get underway, I can’t help but wonder if a little housecleaning might be going on at Huffington Post and other news outlets looking for real medical reporters, rather than outright liars.