[Eeglablist] Cluster-based permutation tests for 3 conditions

[Stephen Politzer-Ahles]

As far as I understand, double-dipping (in the form of non-independent
analysis, e.g. Baker, Hutchison, & Kanwisher 2007) is when you use the same
test statistic to identify an analysis region and to test it. e.g., if you
do a cluster-based permutation t-test, find a p=.2 cluster, and then do a
regular t-test on the average activation within that cluster and get
p=.00001, that's double-dipping. Using a different analysis to identify a
cluster than what you look at within the cluster is not double-dipping ---
e.g., you can use some control manipulation to identify a cluster, and then
look at the manipulation of interest just within this cluster. The issue
described above is I guess kind of halfway in between, so I'm not totally
sure. Using an omnibus F-test to find a cluster and then testing the
pairwise comparisons in it is not using the exact same test twice; on the
other hand, though, the F-test is not totally independent of the pairwise
comparisons within it. My intuition is that it should be ok to look at
these pairwise comparisons as long as your interpretation of them is "there
was a significant omnibus ANOVA effect *and it was driven by a difference
between these conditions*", rather than saying directly "there was a
difference between these conditions" (because the latter is technically not
what was tested in the cluster-based permutation test). I have done this
before and gotten away with it (although that doesn't mean it's right; all
kinds of bad stats stuff gets published, maybe my analysis was more of
that). I'd be interested learn what others think.


---
Stephen Politzer-Ahles
The Hong Kong Polytechnic University
Department of Chinese and Bilingual Studies
http://www.mypolyuweb.hk/~sjpolit/
<http://www.nyu.edu/projects/politzer-ahles/>

On Tue, Sep 12, 2017 at 2:56 AM, Angel Caputi <caputiangel at gmail.com> wrote:

> I wonder What kind of post-hoc test can be used for testing that the
> cluster site (i.e. combination of times and channels) we need to use for
> spatio-temporal localization.  Would not be the use a sencond test on the
> cluster region a kind of double dipping? What would the correct procedure
> for testing where the spatio-temporal pattern differ?
> Sincerely
> Angel
>
>
> 2017-09-09 9:49 GMT-03:00 Stephen Politzer-Ahles <politzerahless at gmail.com
> >:
>
>> Cluster-based permutation tests are not necessarily between two
>> conditions. They operate on any test statistic; this test statistic can be
>> an F test from an ANOVA with three conditions. Just like with a
>> two-condition permutation test, if the test comes out significant this will
>> tell you that the three conditions differ in this dataset (whatever
>> selection of channels, times, and frequency bands you looked at), and the
>> cluster extent can give you an idea of what channel/time/frequency cluster
>> is driving that difference. With an F-test, you would need to do follow-up
>> comparisons to see which particular conditions are driving the difference.
>>
>>
>> ---
>> Stephen Politzer-Ahles
>> The Hong Kong Polytechnic University
>> Department of Chinese and Bilingual Studies
>> http://www.mypolyuweb.hk/~sjpolit/
>> <http://www.nyu.edu/projects/politzer-ahles/>
>>
>> On Thu, Sep 7, 2017 at 9:43 AM, 時本真吾 <tokimoto at mejiro.ac.jp> wrote:
>>
>>> Dear EEGLAB users,
>>>
>>> I usually perform cluster-based permutation tests for my EEG analyses. I
>>> understand permutation tests are tests between two conditions. However, I
>>> have realized that the test results can be presented for the comparison of
>>> three conditions, as is shown by the image file below. I usually perform
>>> the test from the GUI of EEGLAB. Could anyone tell me how I should
>>> understand the test results? Thank you in advance.
>>>
>>> http://tokimoto.o.oo7.jp/ERSP_sample.jpg
>>>
>>> ******************************************
>>> Shingo Tokimoto, Ph.D.
>>> in Linguistics and Psychology
>>> Department of Foreign Languages
>>> Mejiro University
>>> 4-31-1, Naka-Ochiai, Shinjuku, Tokyo,
>>> 161-8539, Japan
>>> tokimoto at mejiro.ac.jp
>>> ******************************************
>>>
>>> _______________________________________________
>>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>>> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.uc
>>> sd.edu
>>> For digest mode, send an email with the subject "set digest mime" to
>>> eeglablist-request at sccn.ucsd.edu
>>>
>>
>>
>> _______________________________________________
>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.uc
>> sd.edu
>> For digest mode, send an email with the subject "set digest mime" to
>> eeglablist-request at sccn.ucsd.edu
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20170912/75407b63/attachment.html>