[Eeglablist] permutation and FDR

[Ben Files]

FDR and Bonferroni are both possibilities, although if one is doing
permutation testing, it is worth considering some other options that
are only really available for permutation tests. Nichols and Holmes
(2001; https://doi.org/10.1002/hbm.1058) have a nice paper describing
permutation testing for functional brain imaging. They describe
corrections for multiple comparisons that correct at the level of
single-voxels or based on cluster statistics.

I'll provide bumper-sticker versions of these corrections here, but
the paper really does a good job explaining them. Also, I believe both
of these methods are implemented in the Mass Univariate Toolbox
https://openwetware.org/wiki/Mass_Univariate_ERP_Toolbox

The first method computes the permutation distribution of your test
statistic (t) at each of your N channels by M timepoints. These
distributions are combined (taking the absolute value), and the
100-alpha percentile of that pooled distribution becomes your
corrected-for-multiple-comparisons threshold for statistical
significance.

In the second method, you compute your statistic for each NxM element
for a single permutation and apply some
uncorrected-for-multiple-comparisons threshold. From this, you build
clusters of adjacent elements that all exceed that threshold. Find the
biggest cluster in the map, and put that cluster size into your
cluster permutation distribution. Repeat many times to build a null
distribution of cluster sizes; any clusters in your true-label
statistical map that exceed that threshold may then be identified as
statistically significant.

The first method is more conservative. The second method has more
experimenter degrees of freedom (choosing the threshold to use, the
definition of adjacency, and the cluster size measure) and is less
conservative.
Best regards,
Ben

On Mon, Jun 11, 2018 at 7:24 AM, Ramtin Mehraram (Student)
<R.Mehraram2 at newcastle.ac.uk> wrote:
> Dear Szilvia,
>
> To my knowledge, the permutation-based statistics "corrects" for non-normal sample distribution. This allows you to use parametric tests (such as t-test) with any sample distribution. It does not correct for multiple comparisons.
>
> The false discovery rate (FDR) is not very conservative as Bonferroni is. FDR is usually recommended whether you have huge amount of data (like genetic data) or in exploratory stage of your study. A more "strict" correction is recommended when finalizing the obtained results.
>
> A good variant of the Bonferroni correction is the Holm-Bonferroni, which is slightly more powerful.
>
> I hope this helps.
>
> B.
>
> Ramtin Mehraram
> PhD Student @ramtinTVT
> Biomedical Research Building 3rd floor
> Institute of Neuroscience
> Newcastle University
> NE4 5PL, United Kingdom
> www.lewybodylab.org
> https://www.newcastlebrc.nihr.ac.uk/research-themes/dementia/
>
> -----Original Message-----
> From: eeglablist [mailto:eeglablist-bounces at sccn.ucsd.edu] On Behalf Of Szilvia Linnert
> Sent: 08 June 2018 09:37
> To: EEGLABLIST <eeglablist at sccn.ucsd.edu>
> Subject: [Eeglablist] permutation and FDR
>
> Hi everyone,
>
> I am planning to use permutation-based statistics using the study function in EEGLAB.
>
> I am confused whether I need to correct for multiple comparisons (using
> FDR) additionally, or the permutation method itself also corrects for multiple comparisons. I've got contradictory suggestions about this.
>
> Many thanks ,
>
> Szilvia
>
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu



-- 
Benjamin T. Files