[Eeglablist] Inconsistent statistical output from cluster-based permutation

[Makoto Miyakoshi]

Hi Noelle,

You have two solutions. One is impractical, the other one is superficial.

1. You use full combinations of permutations. This is impractical.
2. You use rng() to set a random seed before using functions that involve
random number generation.

Usually, we give up exact reproducibility when using permutation and
bootstrap tests. We use a large number of iterations such as 10,000 times
to make it acceptably stable.

Makoto

On Mon, Mar 4, 2024 at 11:50 AM Jacobsen,Noelle A via eeglablist <
eeglablist at sccn.ucsd.edu> wrote:

> Hello,
>
> I am trying to use cluster-based permutation in Fieldtrip (called by
> EEGLAB's std_stat) to test for differences between two event-related
> spectral perturbation plots (ERSPs). I find that when I re-run std_stat, I
> don't get the same result every time. Sometimes a pixel cluster is
> significant, and sometimes it is not. For example, when running std_stat 10
> times, it found a significant cluster 80% of the time.
>
> I'm guessing the inconsistency in the output is related to the
> randomization involved in calculating the parametric threshold for
> clustering. What's the best method to achieve reproducible results? I could
> repeat std_stat a fixed number of times and only consider a pixel cluster
> significant if it exceeds the significance threshold a majority of the
> time, but then how would I calculate the p-values and corresponding effect
> sizes? Am I doing something wrong? Please advise.
>
> I am using Matlab 2022a, EEGLAB 2022.0 and the following Fieldtrip plugin:
> Fieldtrip-lite20230716. My statistical parameters are as follows:
>
>
>   *   stats.paired: 'on'
>   *   stats.effect: 'marginal'
>   *   stats.mode: 'fieldtrip'
>   *   stats.fieldtrip:
>      *   naccu: 10000
>      *   method: 'montecarlo'
>      *   alpha: 0.0500
>      *   mcorrect: 'cluster'
>      *   clusterparam: ''clusterstatistic','maxsum''
>      *   channelneighbor: [0×0 struct]
>      *   channelneighborparam: ''method','triangulation''
>
> Thanks,
>
> Noelle Jacobsen
> University of Florida / Imperial College London
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to
> eeglablist-request at sccn.ucsd.edu
>