[Eeglablist] Bootstrap significance probability level

[Arnaud Delorme]

Dear Adrian,

>I wonder, whether anyone has some practical considerations about which level
>of significance probability should be chosen for bootstrap statistics. 
>
>Since we have to deal with multiple comparisons, the significance threshold
>should be decreased, but the question is how much. As much as I understand,
>the Bonferroni correction for multiple testing would correspond to a level
>of p < 0.05/(number of time points * number of frequency bins). However,
>this would require a very high amount of surrogate data.
>  
>
The problem is that in any time-frequency decomposition, neighboring 
time-frequency points are highly correlated so that standard bonferoni 
correction (that assumes independent observations) are overly 
conservative. In the following paper, a method is decribed to calculated 
the minimum time and frequency resolution (number of independent 
time-frequency estimates - about 200 in our case per plot):

High-frequency gamma-band activity in the basal temporal cortex during 
picture-naming and lexical-decision tasks.
J Neurosci. 2005 Mar 30;25(13):3287-93.
Tanji K, Suzuki K, Delorme A, Shamoto H, Nakasato N. 
<http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=15800183&query_hl=3>

Better than the bonferoni methos is also the Holm's method. First choose 
a significance level p=  (e.g., p=0.05). Then compute the exact p-value 
for each test (which is usually possible using modern computerized 
approaches). Rank the collection of p-values from smallest to largest. 
The smallest p-value is tested against  /N, where N is the number of 
tests. If the smallest p-value is not less than  /N, stop the procedure. 
However, if it is less than  /N, proceed to test the second smallest 
p-value against  /(N-1), etc...

>Most of the studies seem to use p-values < 0.01. What's the argumentation
>for this?
>
>In their J Neurosci Meth; 2004:9-21 Paper, Arnaud Delorme and Scott Makeig
>write that they have 
>"implemented a method to fit the observed data distribution using a forth
>order distribution fit. This feature will be available in a near-term release
>of EEGLAB."
>Does this feature allow a significance level of 0.05? Is there a way to obtain it?
>  
>
Based on discussions with R Oostenveld, we actually changed our mind 
about this idea and now prefer to accumulate more values. The reason is 
that, using these functions, we were fitting a distribution and trying 
to make inference about the tails of this distribution using the fitted 
curve. However the fitting might be especially innacurate at the tails 
(where there is few data points) so to be conservative it seems more 
appropriate to accumulate more values than to try to fit the distribution.

Best,

Arno


>Thank you very much for your help.
>
>
>Adrian Guggisberg, MD
>Neurologische Klinik und Poliklinik
>Inselspital
>CH-3010 Bern
>Switzerland
>
>_______________________________________________________________________
>Your Site for Swiss Maps: http://www.swissinfo-geo.org/
>
>
>
>
>
>_______________________________________________
>eeglablist mailing list eeglablist at sccn.ucsd.edu
>http://sccn.ucsd.edu/mailman/listinfo/eeglablist
>Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
>
>
>  
>


-- 

*Arnaud Delorme, Ph.D.*
Swartz Center for Computational Neuroscience, INC, University of San 
Diego California
La Jolla, CA92093-0961, USA

*Tel* :/(+1)-858-458-1927 ext 15/
*Fax* :/(+1)-858-458-1847/
*Web page*: sccn.ucsd.edu/~arno <http://www.sccn.ucsd.edu/%7Earno>
*To think upon*:

    In spite of all our wanderings HAPPINESS is always found within a
    narrow compass and among objects which lie within our immediate reach.

        /Bulwer/  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20051107/6042b019/attachment.html>