[Eeglablist] basic statistic question bootstrap

[Arnaud Delorme]

Hi David,

I concur with Allan comments. 

However, it is hard to believe that bootstrap and parametric statistics would give you such different results. Ramon would you mind to double check with David.

In paired statistics cases, there is warning message that you should use permutation instead of bootstrap (as bootstrap would only be valid for a large number of conditions). The paired values in the condition should not be bootstrapted as bootstrap is not valid for n<5 (if two conditions then n=2). Instead we should perform sign inversions on the difference between the two paired values, or if more than 2 conditions, look for the overlap of confidence intervals of a given estimator between all pairs of conditions.

Best,

Arno

On Sep 1, 2014, at 7:58 PM, Allan Campopiano <alcampopiano at gmail.com> wrote:

> Hi David,
> 
> I have not used the native bootstrapped ANOVA function in EEGLAB so I can't comment on its output. However, based on your questions, I will comment on a few things that I hope will help. Most of what I know about these methods comes from Rand Wilcox's writings, especially "Introduction to Robust Estimation and Hypothesis Testing".
> 
> Of all the bootstrapped methods, two are talked about most frequently: bootstrap-t and the percentile bootstrap. Both methods can be extended to compare multiple groups, as you would in an ANOVA. I assume that EEGLAB uses the bootstrap-t method when dealing with ANOVAs (just a guess), so I'll describe what I think is going on under the hood.
> 
> The bootstrapped-t method uses the data at hand to estimate an appropriate critical value. This is accomplished by (1) centering the group means so that the null hypothesis is true (i.e.,  Yij = Xij - µj), (2) resampling with replacement from the centered values, and then (3) calculating your F statistic (see Wilcox's book for details on this). This is repeated thousands of times yielding an empirically derived null distribution. If the F statistic based on the original (non-centered) data exceeds the .95 quantile of this distribution, reject the null (publish, get famous, etc…)
> 
> For the write-up, I would describe the method and report the F value, degrees of freedom, number of bootstrap samples, and the p value. See Dien, Michelson, and Franklin (2010) for an example.
> 
> You are probably correct in that EEGLAB gives you a parametric F value in the output somewhere. That is the test statistic based on the original data that is being compared to the critical value of the null distribution.
> 
> Lastly, it does not surprise me that the p values are different when comparing the bootstrap test to the traditional ANOVA. The two methods rely on different sampling distributions (empirical vs. theoretical). Furthermore, EEGLAB's bootstrapped ANOVA might implement some kind of outlier control (trimmed means), or protection against violations of normality or heteroscedasticity (Welch-type method), which all affect the p value.
> 
> I hope this helps in some way,
> 
> Al
> 
> 
> 
> Allan Campopiano | MA Candidate
> Laboratory of Cognitive and Affective Neuroscience
> Brock University | Psychology Department | 500 Glenridge Ave.
> St. Catharines, ON Canada L2S 3A1
> T 905-688-5550 x3451 F 905-688-6922
> 
> 
> On Thu, Aug 28, 2014 at 6:58 PM, david grahms <david.grahms at gmail.com> wrote:
> Hi,
> 
> I would appreciate some help in understanding the output the bootstrap statistic implemented in eeglab and how to report it. Should I report the df and F-value when I perform bootstrap analysis? Am I correct in that I get the parametric F-value when I run the bootstrap analysis?
> 
> Also
> I find it hard to understand that the bootstrap analysis gives me a p=0.00024988 and parametric gives me F(3,30)=0.96, p=0.42629.
> 
> Regards,
> David
> 
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu
> 
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20140909/21b1848d/attachment.html>