“Medical reporter” David Kirby has delivered a potentially explosive report to his unfortunate and misinformed minions at the Huffington Post, in which he shows a startling string of misunderstandings and complete lack of knowledge of basic epidemiologic design and methods. Furthermore, he writes that Dr. Julie Gerberding “admits to a startling string of errors in the design and methods used in the CDC’s landmark 2003 study that found no link between mercury in vaccines and autism, ADHD, speech delay or tics,” when, in fact, the CDC report admitted no such thing about the 2003 study.

Gerberding was responding to a 2006 Report of the Expert Panel on Thimerosal Exposure in Pediatric Vaccines: Feasibility of Studies Using the Vaccine Safety Datalink to the National Institute of Environmental Health Sciences (NIEHS). Nowhere in the 2006 report, however, did the NIEHS panel conclude that the CDC’s 2003 thimerosal safety study was riddled with “several areas of weaknesses” that combined to “reduce the usefulness” of the study. In fact, in the NIEHS panel meeting that generated the 2006 report, the quality of the CDC’s 2003 thimerosal safety study was not even discussed. This can be seen clearly if you carefully read the NIEHS Report of the Expert Panel.

In addition, earlier this week Epi Wonk had a long discussion with one of the Expert Panel Members (who adamantly insisted that he/she remain nameless), who confirmed three things for me:
(1) The purpose of the NIEHS Expert Panel was exactly as stated in the report:

“It has been proposed that the Vaccine Safety Datalink could be used to look at the association between autistic disorder (AD) or autism spectrum disorders (ASD) by means of an ECOLOGIC ANALYSIS (emphasis mine) that would compare rates before and after the removal of thimerosal from most childhood vaccinations. To determine the feasibility and potential contribution and/or drawbacks of such a study, and to consider alternative study designs that could be conducted using the VSD database, the NIEHS convened a panel of experts…

(2) The quality of previous epidemiological studies of the association between thimerosal and autism was not discussed.
(3) The overall quality of the 2003 Verstraeten et al. study was not discussed. Indeed, in the section of the report in which the expert panel considered research panels other than ecologic analyses, which they did dismiss as riddled with “several areas of weaknesses” that combined to “reduce the usefulness” of ecologic studies, the expert panel “…recommended that further consideration be given to conducting an extension of the Verstraten study that would include additional years for follow up, would add more managed care organizations and reexamine the criteria for exclusion of births and/or take a sensitivity analyses approach to examining the impact of various exclusion criteria.”

In the HuffPost story, David Kirby quotes Julie Gerberding as writing that her agency “does not plan to use” the Vaccine Data Safetylink (VSD) for any future “ecological studies” of autism. “In fact”, Kirby continues, “Gerberding’s report said, any continued use of the VSD for continued ecological studies of vaccines and autism ‘would be uninformative and completely misleading.’”

Well, yes, that’s what the CDC thinks about using the VSD for ecologic analyses. I couldn’t agree more. At this point I obviously need to step back and explain about ecologic analyses. Fortunately, I taught epidemiologic design and methods for about 35 years, I had some students almost as clueless as David Kirby, but I’m a patient teacher. Another interesting fact is that there has only been one ecologic study published using the VSD, and I’ve written extensively about the study on this blog. Guess what? It wasn’t done by the CDC, who knew better long before the 2006 NIEHS Expert Panel. I’m speaking of the infamous Young-Geier Autism Study. So let me paraphrase from my explanation of “ecologic” in my previous critique of that paper:

It seems that much of the confusion and difficulty in understanding the Young-Geier Autism paper comes from the use of the term ecological study or ecological design. In order to understand the concept of an ecological-level study, it’s best to first think of what is meant by an individual-level study. In an individual-level study the investigator has data on every variable for every participant in the study. This may sound ridiculously simple, but it needs careful explication here, because an ecological study is quite different. In an individual-level study of thimerosal containing vaccines (TCV) and neurodevelopmental disorders (ND), for each child in the study we would have vaccination history for that child, ND diagnosis or diagnoses (if diagnosed), exact age of diagnosis or diagnoses (if diagnosed), date of birth, gender, age when follow-up ended, and information on as many potential confounders as possible for that child: birth weight, gestational age at birth, socioeconomic status, etc., etc. and all of that data would be linked together for that individual. Thus, in an individual-level study, the individual is the unit of analysis. Both the 2003 Verstraeten et al study using the VSD and the Thompson et al VSD special study of Early thimerosal exposure and neuropsychological outcomes at 7 to 10 years were individual level studies.
In an ecological study, data is collected at the group-level, as opposed to the individual level. The group is the unit of analysis. In fact, it would probably be easier to think of the Young-Geier Autism study as a group-level study with a group-level design and a group-level analysis, rather than using the confusing term ecological. 
 
This post is getting a bit long, and I don’t want to bore you by dissecting every single one of Kirby’s false statements. But let’s pick out a few.
KIRBY: “The final NIEHS report was a serious and thoughtful critique of where the CDC went wrong in its design, conduct and analysis of the study. The NIEHS panel “identified several serious problems” with the CDC’s effort.
FACT: The final NIEHS report was a serious and thoughtful critique of “using the VSD to look at the association between autistic disorder (AD) or autism spectrum disorders (ASD) by means of an ecologic analysis that would compare rates before and after the removal of thimerosal from most childhood vaccinations, to determine the feasibility and potential contribution and/or drawbacks of such a study, and to consider alternative study designs that could be conducted using the VSD database.” The NIEHS panel “identified several serious problems that were judged to reduce the usefulness of an ecologic study design using the VSD to address the potential association between thimerosal and the risk of AD/ASD

 

KIRBY: “…the NIEHS had criticized CDC for failing to account for other mercury exposures, including maternal sources from flu shots and immune globulin, as well as mercury in food and the environment. ‘CDC acknowledges this concern and recognizes this limitation,’ the Gerberding reply says.”
ACTUAL QUOTE FROM CDC REPORT: “NIEHS Finding: Difficulty in estimating cumulative exposure of child to organic mercury: The panel expressed concern that VSD adminstrative data or medical charts would not be accurate in recording or estimating a child’s total mercury exposure from sources other than vaccines, such as diet, air and water. CDC Response: CDC acknowledges this concern and recognizes this limitation. In addition to administrative data and medical chart review, CDC has employed parent interviews to identify total cumulative mercury exposure from sources other than vaccines, such as diet. Often, however, parent recall, for events several years in the past, poses limitations as well.”

KIRBY: “The NIEHS also took CDC to task for eliminating 25% of the study population for a variety of reasons, even though this represented, ‘a susceptible population whose removal from the analysis might unintentionally reduce the ability to detect an effect of thimerosal.’ This strict entry criteria likely led to an ‘under-ascertainment’ of autism cases, the NIEHS reported. ‘CDC concurs,’ Gerberding wrote, again noting that its study design was ‘not appropriate for studying this vaccine safety topic. The data are intended for administrative purposes and may not be predictive of the outcomes studied.’
FACT: These four sentences are outright lies. The NIEHS Expert Panel never “took the CDC to task for eliminating 25% of the study population…” On the contrary, when discussing potential alternative designs (other than ecological studies), “another possibility that generated support by the panel was an expansion of the VSD study published by Verstraten et al. The availability of several additional years of VSD data was seen as an opportunity to provide a more powerful test of any potential association between thimerosal and AD/ASD and would enable reconsideration of some aspects of the original study design (e.g., exclusion criteria)” It was unclear to the panel what effect exclusion of low birth weight infants and those with congenital or severe perinatal disorders or born to mothers with serious medical problems of pregnancy had on the results of the Verstraeten et al. study; an expanded future study in which sensitivity analyses both including and excluding children with perinatal problems was recommended. The quote that begins with “CDC concurs” has no bearing on the Verstraeten et al. study, as implied by Kirby. Gerberding is responding to an NIEHS Expert Panel point about case ascertainment. Here is the entire quote from the CDC report: “CDC responds: ‘CDC concurs with the recommendation that broader ICD-9 codes should be considered. The weakness further emphasizes why an ecological design is not appropriate for studying this vaccine safety topic using the VSD. The VSD data are intended adminstrative purposes and may not be predictive of the outcome studied. Because the outcomes have not been validated and considering the sensitivity of this issue, any VSD study of vaccines and autism, including a broader list of ICD-9 codes, would require chart review.’”

KIRBY: “Another serious problem was that the HMOs changed the way they tracked and recorded autism diagnoses over time, including during the period when vaccine mercury levels were in decline. Such changes could ‘affect the observed rate of autism and could confound or distort trends in autism rates,’ the NIEHS warned. ‘CDC concurs,’ Dr. Gerberding wrote again, ‘that conducting an ecologic analysis using VSD administrative data to address potential associations between thimerosal exposure and risk of ASD is not useful.’
FACT: This is correct. Believe it or not, Mr. Kirby has got it right this one time. The charge of the NIEHS Expert Panel was to determine whether the VSD should be used to to do ecological studies. The expert panel concluded, “No.” The CDC concurs.

I’ll leave you with the most important summarizing quote of the CDC report:

NIEHS Finding: Strengths: The panel identified several major strengths of the VSD to be: its ability to detect infrequent, vaccine-related adverse events of modest size; the possibility to supplement the MCO administrative data with reviews of medical records, interviews with parents and children, and additional diagnostic assessments; and the availability of demographic information about the MCO members.

CDC Response: CDC agrees with the panel’s assessment of the strengths of the VSD Project to evaluate vaccine safety concerns. The VSD is a unique public-private collaboration that provides a model for the study of patient safety concerns by using individual-level data. In addition, CDC recognizes the tremendous value of the VSD as a national resource of expertise in vaccine safety research.

I can’t help but agree with Kirby’s recommendation, “I hope everyone will read these documents, including the recommendations to make the VSD better, and the CDC’s agreement with all of the suggestions.” As the waning weeks of Omnibus Autism testimony get underway, I can’t help but wonder if a little housecleaning might be going on at Huffington Post and other news outlets looking for real medical reporters, rather than outright liars.

 

Sphere: Related Content


32 Responses to “David Kirby: HuffPost Report on CDC’s Vaccine Safety Datalink Uninformative and Completely Misleading”  

  1. 1 Ken

    I think aggregated data is the best description, as grouped data may be confused with data sampled by groups but then collected individually within each group.

    I’ve come to this late, but i agree. Verstraeten does it correctly, you must condition on the time of birth. Young-Geier can’t. It’s one of those pointless papers that don’t tell you anything you can trust. End of story. If journals stopped publishing these it would make the world a better place.

  2. 2 EpiWonk

    @Ken: Truthfully, I agree with you about the use of the word “aggregated:” I prefer it.

  3. 3 Schwartz

    EpiWonk,

    It certainly appears that Kirby got a number of the details quite wrong. My take from the report, is that it points out quite clearly the piss poor state of vaccine safety data collection that is going on. Although some useful data can be gleaned from the available system, there are a lot of problems and its usefulness is limited.

    I wonder what a detailed analysis of the Denmark and UK databases that the IOM used would point out.

  4. 4 DLC

    a couple of quick comments:
    If individual-level studies are more reliable, then why didn’t Young &Geier use that method ?
    (is that a book I smell being cooked in Geier’s kitchen?)
    And: Has anyone tested Kirby for reading comprehension and retention ?
    Or is he just quote-mining and lying ?

    Excellent article, Wonk. keep up the good work.

  5. 5 TheProbe

    DLC asked: If individual-level studies are more reliable, then why didn’t Young &Geier use that method ?
    (is that a book I smell being cooked in Geier’s kitchen?)

    The correct answer is that Geier (rhymes with liar) is cooking up more “studies” to promote his expert witness fees in the Omnibus, and other cases.

    As for a book, Stephen King has that market locked up.

  6. 6 Uncle Dave

    Kirby;
    “Another serious problem was that the HMOs changed the way they tracked and recorded autism diagnoses over time, including during the period when vaccine mercury levels were in decline. Such changes could ‘affect the observed rate of autism and could confound or distort trends in autism rates,’

    Yes, Autism rates would likely show a decline not an increase if there is a connection?

    Have we or the CDC or others gained any additional information as to other contributing variables within the data, not just this fixation with vaccine safety? I realize it is a vaccine safety datalink study however, have we learned anything more as to the propensity or strong inclination toward more tangible factors other than vaccine?

    I am astounded as to why this vaccine twist of facts continues to get traction in the public? They almost fired Dan Rather for the Bush National Guard story and this goes unabated?

    Keep up the good work, even though this is not my area of knowledge I find your discussions very digestible. Indeed you do appear to be a patient teacher.

  7. 7 varkam

    Thanks for that post!

    @TheProbe:

    Absolutely agree with you. The thing that I find funny is that antivaxxers have zero problem with that conflict of interest. I guess that they are willing to look past a whole lot of problems so long as they are able to get data that supports their beliefs.

  8. 8 Joseph

    The VSD data are intended adminstrative purposes and may not be predictive of the outcome studied. Because the outcomes have not been validated and considering the sensitivity of this issue, any VSD study of vaccines and autism, including a broader list of ICD-9 codes, would require chart review.

    That’s why Thompson et al. (2007) was key. I assume there’s probably a similar study underway that looks at autism specifically too.

    Verstraeten et al. did tend to find what appeared to be false positives. David Kirby probably thinks it found too many false negatives.

    After Thompson et al. (2007) it makes no sense to go back and consider a much weaker study methodologically. Not to mention that Young-Geier, even though a much newer study, is itself much weaker than Verstraeten et al. in may respects. Young-Geier is technically a VSD study, but it didn’t really take advantage of the sort of data VSD is equipped to provide.

  9. 9 EpiWonk

    @Joseph: “I assume there’s probably a similar study underway that looks at autism specifically too.”

    Yes. My understanding is that the CDC is doing a large case-control study of autism using the VSD and chart review and perhaps questionnaires for parents.

  10. 10 EpiWonk

    @Uncle Dave: The CDC began collecting the data (the Vaccine Safety Datalink) from the HMO’s in the early 1990’s solely as a means of monitoring vaccine safety. There had originally been no intention to do studies of autism or neurodevelopment. I’m not sure if it’s an ideal data set for that. You might be interested in reading about the CHARGE study, which is based at UC-Davis.
    See http://www.ehponline.org/members/2006/8483/8483.html.

  11. 11 EpiWonk

    @DLC: “If individual-level studies are more reliable, then why didn’t Young &Geier use that method ?”

    TheProbe is at least half correct about this. The Institutional Review Boards (IRBs) at the HMOs refused to allow the Geiers access to individual-level data, because of concerns about patient confidentiality. Their concerns turned out to be warranted, as you can see at Casewatch at
    http://www.casewatch.org/fdawarning/rsch/geierk.shtml and
    http://www.casewatch.org/fdawarning/rsch/geier.shtml.
    My guess is that the Geiers went ahead with an ecological study anyway, because they knew that the relative risks would be probably grossly inflated, especially with a little data fudging.

  12. 12 Ken

    If it didn’t get the answers they wanted they wouldn’t have published it. “We realised that the study design was wrong” is always a good explanation.

  13. 13 kristina

    If Kirby is not exactly lying in his HuffPo post, he’s presented quite a display of his inability to (1) read a study/report and (2) paraphrase its findings, in non-judgmental language. What he’s done is (as he often does) pull out certain phrases, quote them out of context and without explaining what the phrases mean in the original document, and then surround them with his own rhetorical questions. The result is a pastiche and does less than justice to the original document, to understate the matter.

    I have to wonder how many of his quite faithful readership themselves read the actual report.

  14. 14 Matt

    Well, it looks like you had an effect. The blog post was taken down, rewritten and reposted with an acknowledgement that a mistake was made.

    That’s something new. He didn’t acknowledge his factor of 10 mistake in the “how many
    autistic adults are there in Scotland” post.

  15. 15 Sullivan

    I would contend that the CDC/HHS document that Mr. Kirby posted tells us the outcome of the sister study to the Thompson study that Joseph mentions.

    Based on the fact that the CDC (a) already knows the results of the study (it appears to be already in peer review/publication) and (b) still supports the 2004 IOM conclusions and (c) is not starting new thimerosal/autism studies, it would seem safe to say that the in-press sister study to Thompson et al. shows no link between thimerosal and autism.

    I gave away the punchline, but here is the discussion:

    http://leftbrainrightbrain.co.uk/?p=891

  16. 16 EpiWonk

    @Sullivan: Wow. Interesting. I think you’re right. I think the CDC still doesn’t quite “get” how important this is. They should go to one of the “Big Four” journals (New England Journal/JAMA/Lancet/British Medical Journal) and insist on expedited review and publication. It can be done.

  17. 17 Do'C

    (c) is not starting new thimerosal/autism studies

    Actually, The Study to Explore Early Development (SEED) fact sheet says it will look at select mercury exposures (via maternal and childhood vaccines).

  18. 18 daedalus2u

    Something I don’t understand is why there has been no explicit linkage of the Faroe Islands studies on mercury with the Faroe Islands study on autism? The cohorts overlapped. The ~1,000 consecutive births that had cord blood mercury measured (and then had later testing both mercury and neurological) were in a 2 year cohort of 1,404 children that had 5 cases of ASDs, 2 of autism and 3 of Asperger’s. 743 of those children had cord blood mercury levels above 60 nM/L. 249 of them had cord blood mercury above 200 nM/L.

    http://www.springerlink.com/content/b758767u364g5255/

    These mercury exposures are gigantic compared to any from vaccines.

  19. 19 Uncle Dave

    EpiWonk
    Thank you very much for providing the link to the CHARGE study at UC Davis.

  20. 20 Sullivan

    epiwonk–

    as it turns out, Mr. Kirby doesn’t learn. He misrepresented something at a briefing hosted by representative Maloney. I think it was the information above about the limitations of the VSD, but I am waiting to see if a recording will be posted by one of the vaccine/autism advocacy groups.

  21. 21 Zashkaser

    Great post Jon! I have been following the #p2 effort since you started it, and although I have understood its purpose this post does a really great job solidifying the full rationale.

  22. 22 stepanida

    How do you say

  23. 23 dashulya

    Twisted Pair Wiring vs. Coax

  24. 24 vika

    PRAIRIE EDITOR: The

  25. 25 Chia Andronis

    I am glad for writing to make you know of the remarkable experience my girl obtained reading your web site. She noticed a wide variety of pieces, with the inclusion of how it is like to possess an incredible coaching character to get other people without problems grasp specified grueling matters. You actually exceeded readers’ expected results. Thanks for delivering those useful, safe, educational and also fun thoughts on that topic to Lizeth.

  26. 26 british citizenship applications

    I am no longer sure where you are getting your info, but good topic. I must spend some time studying much more or working out more. Thanks for fantastic info I was looking for this info for my mission.

  27. 27 source

    I Am Going To have to visit again when my course load lets up - nonetheless I am taking your Rss feed so i could go through your blog offline. Thanks.

  1. 1 Wakefield, Healy and the danger of letting misinformation go unchallenged | Autism Library
  2. 2 Autism Blog - Kirby launches torpedo at Verstraeten, sinks Geier | Left Brain/Right Brain
  3. 3 Autism Blog - Congress says no to misleading autism information « Left Brain/Right Brain
  4. 4 Autism Blog - Huffington Post uses erroneous data to promote autism epidemic « Left Brain/Right Brain
  5. 5 Autism Blog - Another example of irresponsible blogging by David Kirby « Left Brain/Right Brain

Leave a Reply