Last week I blogged about the 2007 DeSoto and Hitlan study, Blood levels of mercury are related to diagnosis of autism: a reanalysis of an important data set (Journal of Child Neurology 2007;22:1308-11), in a post entitled Epi Wonk’s Intro to Data Analysis.
Dr. DeSoto has posted a reply at her University of Northern Iowa web site, Frequently Asked Questions about DeSoto and Hitlan (2007), more specifically at http://www.uni.edu/desoto/epiwonk%20query.htm. In response, there are a few issues I should clarify.
1. In my post I stated: “In May 2007 Dr. Catherine DeSoto wrote to the Editorial Office of the Journal of Child Neurology expressing concern about what appeared to be obvious inconsistencies in the data analysis of the results section of the Ip st al article. Dr. DeSotos specific concern related to the statistical interpretation of the data.” As Dr. DeSoto states in her reply: “This is not accurate at all.” Her concern was that the statistical results reported by Ip et al. were just plain wrong. Ip et al. reported that the difference in the means between autistic cases and controls was not statistically signficant, a report that was clearly in error. DeSoto and Hitlan discovered that the difference in means was statisistically significant. The reason this is important is that a disagreement about statistical “interpretation” is (as Dr. DeSoto says) “something about which learned persons may disagree” — not an obvious error of the sort Ip et al. made.
2. In my original post I stated: “According to the abstract, DeSoto & Hitlan ‘found that the original p value was in error and that a significant relation does exist between the blood levels of mercury and diagnosis of an autism spectrum disorder.’” As written in my post , this might be seen as being taken out of context. This conclusion was actually qualified — the full sentence makes clear DeSoto & Hitland are speaking about within this data set: “We have reanalyzed the data set originally reported by Ip et al. in 2004 and have found that the original p value was in error and that a significant relation does exist between the blood levels of mercury and diagnosis of an autism spectrum disorder.” In other words, DeSoto & Hitlan did not draw conclusions beyond the data set.
3. Although I don’t think that I’m guilty of ad hominem attacks on Dr. DeSoto or Dr. Hitlan, it seems that there have been a number of such ad hominem attacks floating around the blogosphere. I think the following statement from Catherine DeSoto may be helpful to some readers in clearing up some unkind speculations: “I was invited by an attorney to testify/get involved in the vaccine court proceedings, but declined…”
4. Dr. DeSoto states: “I I try to be as clear as humanly possible, and hope that you yourself will revise your point three to avoid giving readers on either side of the issue from the impression that we feel we have proven a relationship between mercury and autism with this one data set.” I’m not sure what Dr. DeSoto is referring to here, but I never meant to say, or even imply, that DeSoto and Hitlan felt that they had proven a relationship between mercury and autism with this one data set.
There are a few issues upon which DeSoto & Hitlan and I continue to disagree:
1. In her her response, Dr. DeSoto states, “…I think your website, among other valuable things, serves to make it clear that the difference between autistic and control subjects shows up using a variety of statistical techniques, yes?” My answer to this is: yes and no. I invite readers to to check out my entire analysis, including the graphical results, since in my opinion the findings cannot be summarized in one statistical test. The entire difference between 81 cases and 54 controls is due to an excess of ASD cases in the upper part of the blood mercury distribution, i.e., greater than 25 nmol/L. After I posted my article I received e-mails from two toxicologists telling me that this finding was fascinating, was completely new to them, and needed to be replicated in other samples of autistic children. In other words, nobody would have hypothesized a priori the pattern of differences between ASD cases and controls that I reported. This reinforces my inclination to view my analysis of this data set as an exploratory analysis that needs to be replicated. This — along with the quantitative findings shown in my original post — leans me towards the view that either (1) statistical significance is meaningless in the context of this type of exploratory research; (2) if you pushed me, I’d say my results weren’t significant because they’re totally due to the post hoc discovery of a difference at greaster than 25 nmol/L.
On the other hand, I can’t seem to get away from those who insist on using one statistical test for the entire data set. I received several e-mails and comments arguing that the nonparametric Mann-Whitney Rank Sum Test (AKA the 2-sample Wilcoxon Rank Sum Test) would be appropriate for this data set. My results are shown in Comment section of the original post, but here they are again:
Mann-Whitney’s Statistic = 1730.0
Z statistic = 2.06
2 tailed p = 0.0395
Median difference = 3.00
95% Confidence Interval: 0.00 to 7.00
Okay, from a strictly formal statistical point of view, there is a statistically significant difference (p < 0.05) between the blood mercury distributions of the ASD case group and the control group. I think it’s also worth noting in this context that the geometric means were 11.1 for the control group and 14.4 for the ASD cases, a difference of 3.3 nmol/L. So if I close my eyes, pretend the two distributions aren’t bi- or tri-modal, and try to make a comparison of “measures of central tendency,” both the medians and the geometric means differ by about 3 nmol/L. Is a difference of 3 nmol/L in blood mercury levels between ASD children and controls clinically significant? I’ll let you be the judge.
2. In any event, I still stand by my conclusion: “We can conclude absolutely nothing about the association of ethylmercury [thimerosal] in vaccines to autism from these data“. These data should never have been used in any way in statements about the association between vaccines and autism. Shattuck should never have cited Ip et al. as a recent study that “failed to establish a connection between measles-mumps-rubella vaccination or the use of mercury-based vaccine preservative and autism.” Fombonne et al. should not have cited Ip et al. as a biological studiy of ethylmercury exposure that “failed to support the thimerosal hypothesis.” Why? Because most of the blood mercury measured in these Hong Kong children was almost certainly due to methymercury exposure, not thimerosal.
Nevertheless, I think it is still important to try to carry out data collection efforts in which mercury levels are measured and compared in autistic children and controls. This at least involves observational data collected at the individual level. Otherwise, we end up in a situation where people are making causal inferences about the association between mercury exposure and autism from truly awful ecological studies.