6. Exchange with Jim Burridge, co-author of Hirst&al. with a comment by J.Benveniste

From: I.Vecchi

To: john.foreman@ucl.ac.uk; f.l.pearce@ucl.ac.uk; n.hayes@ucl.ac.uk;
j.burridge-1@plymouth.ac.uk

Subject: I: Evidence of bias?

Date: 17 July 2001 07:06

I enclose a letter to Nature concerning your work. Comments are welcome.

Regards,

Italo Vecchi

----- Original Message -----

From: I.Vecchi

To: corres@nature.com

Sent: Saturday, July 14, 2001 3:19 PM

Subject: Evidence of bias?



SIR - The recently renewed interest for experiments1 on high dilutions suggests that a reappraisal of the avaliable evidence may be appropriate
.The current interest centers mainly on the work of Belon et al.2 and on the controversial Davenas et al.3, but a close inspection of the unsuccessful attempt to detect a difference between control treatments and highly diluted
probes described in Hirst et al.4 reveals interesting features, whosesignificance appears to have been neglected. The combined p-value of 0.0027 in Table 2 of Hirst et al. sums up the probability of obtaining experimental results such us those actually obtained , under the assumption that the null-hypothesis holds, i.e. that "the treatment applied to the cells
produces a response which is not different from the response in theabsence of treatment"4. A p-value of 0.0027 constitutes , by any standard, very strong evidonce that the null-hypothesis should be rejected.

Data from Hirst et al. (Table 2)

Fischer p-value

Succussed high dilution

0.0027

Unsuccussed high dilution

0.086

Control

0.85

Moreover one may question whether the dismissal of the results above the significance threshold in Table 1 of Hirst et al. is justified, also taking into account that the Bonferroni procedure used is extremely conservative.
While it is certainly true that " for any given set of random numbers it isexpected that there will occasions when the examination of randomness by statistical tests fails"4, the purpose of any statististical analysis is precisely to quantify the likelihood of such occurences. From a methodological point of view the dismissal of data above the level of
statistical significance as "chance results"4 may be construed as an instance of subjective bias.

Regards,

Italo Vecchi

1. Milgrom P. The Guardian "Thanks for the memory " March 15, 2001, also available at http://www.guardian.co.uk/Archive/Article/0,4273,4152521,00.html.htm

2. Belon P. et al. Inflammation Research 48, 17 (1999).

3. Davenas et al. Nature 333, 816 (1988).

4. Hirst S. et al. Nature 366 , 525 (1993).

From: "Jim Burridge" <jim.burridge@btinternet.com>

To: "I.Vecchi"

CC: <john.foreman@ucl.ac.uk>

Subject: Re: Evidence of bias?

Date: Fri, 20 Jul 2001 13:25:16 +0100

Italo,

I have read your comments with interest and have re-read the work we reported in Hirst et al. I think you have misread the paper. The summary at the start of Hirst et al clearly says "Our results contain a source of variation for which we cannot account" and that point is discussed at length on page 527 in connection with the results shown in Table 2.

Furthermore, the footnote to Table 2 makes clear that it is only the high dilution mean cell counts that are being analysed in that table - i.e. it is only the high dilution "treatments" that are being compared. We stand by our original statement that "some unidentified part of our experimental procedure might account for them" (i.e. the effects noted in Table 2).

The claim that we "overlooked" or "dismissed" these results is quite misplaced.

Regards,

Jim Burridge.

From: I.Vecchi

To: jim.burridge@btinternet.com

Cc: john.foreman@ucl.ac.uk

Subject: Re: Evidence of bias?

Date: 22 July 2001 21:23

Jim,

Thank you for your reply and for your willingness to discuss the issue.

In your paper you are testing a null-hypothesis.

Both in Table 1 and in Table 2 you exhibit very strong evidence against the null-hypothesis. Then, instead of rejecting the null-hypothesis, you claim that your findings, which make the null-hypothesis untenable, depend on an unknown source of variation in the highly diluted treatments. Yours is a very rare example of a paper attributing its results to unidentified flaws in its own experimental procedure.

Let me ask you a straight question. Do you agree that Hirst et al. provides strong evidence that "the treatment applied to the cells produces a response which IS different from the response in the absence of treatment"?

Regards,

Italo

From : "Jim Burridge" <jim.burridge@btinternet.com>

To : I.Vecchi

CC : <john.foreman@ucl.ac.uk>

Subject : Re: Evidence of bias?

Date : Wed, 25 Jul 2001 15:38:59 +0100

Italo,

The problem is that, within each session, the high dilution treatments are confounded with triplicates. Therefore it is impossible to tell whether the detected effects are a result of the triplicates (by being prepared at the
same time for example) or the treatments. The simplest interpretation of the results is that some feature of the experimental procedure caused each triple to differ in some way from other triples. This is probably an example
of the "plot", "block" or "batch" effect which is a standard feature of most experimental observations and is discussed at length in the experimental design literature. Depending on context, such effects can be large or small.
In agricultural studies, for example, the effects can be very large indeed. In industry it is sufficiently important for paint and wallpaper manufacturers to advise customers to use materials with the same batch number when doing a particular job. In the more precise chemical and physical sciences such effects can be small. The present study probably
falls somewhere in between. We think, but cannot prove, that the results we reported are the result of such an effect. The fact that the "effects" do not seem to be consistent across sessions points to this interpretation. I assume your direct question refers to the comparison of the high dilution results with the controls without treatment. This is the subject of Table 1 which fails to provide evidence of a difference. Regards, Jim.


Jim,

thank you again for your reply.

Your argument is irrelevant. The point is not that the triplicates differ from one another, but that the basophil counts for high dilution triplicates consistently differ from those for control, as revealed by the fact that their p-values are too low. What is being revealed is a non-random effect, otherwise the p-values for high dilution triplicates would be uniformly
scattered.

<<I assume your direct question refers to the comparison of the high dilution results with the controls without treatment. This is the subject of Table 1 which fails to provide evidence of a difference.>>

In Table 1 you use an ultra-conservative Bonferroni procedure, which however yields clear evidence that the null-hypothesis should be rejected for succussed anti-IgE in the higher dilution range. You dismiss such evidence
as a "chance result".

A look at Fig. 3a should make you think. Actually in Table 1 you are not testing the null-hypothesis A that "the treatment applied to the cells produces a response which is not different from the response in the absence
of treatment", but another null-hypothesis B that " the treatment applied to the cells produces a MEAN response which is not different from the MEAN response in the absence of treatment". What is happening is that succussed anti-IgE (and to a lesser extent unsuccussed anti-IgE) strongly enhance the variation in basophil counts, while affecting the mean counts only moderately. The variations cancel out when you take the average, so that the data in Table 1
capture only the lesser effect, which however remains significant for higly diluted anti-IgE.

In Table 2 on the other hand you are testing the null-hypothesis A, which the data clearly show to be untenable.

Regards,

Italo Vecchi

From: I.Vecchi

To: jim.burridge@btinternet.com

CC: john.foreman@ucl.ac.uk; f.l.pearce@ucl.ac.uk; n.hayes@ucl.ac.uk;
j.burridge-1@plymouth.ac.uk

Subject: I: Evidence of bias?

Date: 7 September 2001



Hi Jim,

I asked a friend , who works as a biostatistician at a pharmaceutical product development company, to have a look at your paper. She prefers to stay anonymous, but here is her slightly edited response, which I find quite
interesting.

<<I found it. I read it. And did not like it .

The triplicates were done specifically to reduce the variability. If this variability thought to be a problem, the difference between the treatments within triplicates could be tested, succeeded IgE versus succeeded buffer,
for example. According to Fig.2 this would be highly significant (in the 2 highest ranges of dilutions all means of succeeded IgE are higher than of buffer, so it is unlikely to be insignificant, no matter how high variability is - unfortunately there is no individual data there).

As to Table 1, I do not see the reason to perform an individual test for each dilution separately (I think the t-test was for zero difference between mean cell count for a specific treatment and dilution versus mean cell count
of controls, is that true? From your response it seems that you understand t-tests as testing differences between different dilutions (within same range) on the same treatment) , thus increasing the number of tests and decreasing degrees of freedom. Of course, there is a little power to detect anything with such a procedure. Also, because of high variability of the assay, control was performed every time. The reason to do it - to specifically account for session variation. To do this not the differences of the means should be compared, but mean of the differences (treatment cell
count - control cell count from the same session) should be compared with zero. I can't get through the second part of Table 1 (F-tests) to understand what was tested and how.>>

I think that the content of our exchange is quite significant and I would like to make it available, as pasted below, to other people interested in this issue, also by posting it at my site http:\\www.weirdtech.com\sci\memoryofwater.html ). I am sure that you, as any good scientist, are happy to have your arguments open to public scrutiny.

Thank you again for your replies.

Italo Vecchi