[Eeglablist] How to interpret ERSP statistics

[Makoto Miyakoshi]

Dear Jinwon,

Welcome to the EEGLAB mailing list Jinwon!
This month, I will answer the questions on the list.

For the time-frequency domain correction, there are largely two types of
corrections often used.
1. Strong family-wise error rate (FWER) control
2. Weak FWER
You might think strong sounds better because it is strong. But the strong
correction means you pay too much money for practically unnecessary levels
of time-frequency details. You can choose to trade it off with much more
statistical power. Also, from the spatial smoothness (i.e correlation to
neighbors) of the time-frequ data (Morlet you use is designed to have
Gaussian smoothness in both time and freq axes), applying the strong FWER
means you are prone to Type II error i.e. missing the true positive.

Ok, thus far is the short review of the time-freq stats.
Now, EEGLAB's default is probably the strong FWER. This means that even if
you have only a single time-freq cell (like a pixel) significant, you can
report it with 100% confidence (hence 'strong'). But practically, if you
report it and I were to review it, I would reject it ha ha. Why? Because we
are trained/biased to see more nicely distributing broad significant blobs
in the time-frequency domain. Remember, even if you apply the 'strong'
correction for every single pixel, about 5% of error is still expected. In
the time-freq domain, you typically have something like 50x200=10,000
pixels. 5% of them is 500. So you want to show more convincing results than
randomly sprinkled 500 significant pixels.

The weak FWER control is also known as a cluster-level correction. I would
say more than 90% of fMRI researchers use it because it is much more
powerful for detecting significant regions, contrary to what the name
indicates. Cyril Pernet's solution supports this cluster-level correction,
I believe. So check out his solution.

To sum up, to answer your questions,

> For example, sometimes there are very short and small dots with p<0.05 in
ERSP image. Does it also
represent significant difference?

Yes, as long as you applied the strong FWER control i.e. FDR or
Bonferroni(-Holm) correction etc..

>What are the criteria to insist there is actual difference?

See the distinction between the strong and weak FWER controls above.

> How long range of difference is minimally required?  Or is it totally
arbitrary depending on the author?

It is in fact pretty arbitrary I would say. It also depends on how the
reviewers are educated.
As a rule of thumb, if you get a big stable blob no matter how you change
the trivial details in the parameters, that is 'convincing'.

Makoto


On Thu, Jan 5, 2023 at 11:17 PM 장진원 <jinwon06292 at gmail.com> wrote:

> Dear all,
>
> I'm a beginner at EEGLAB. While analyzing ERSP statistics with a paired
> t-test, I have a problem determining whether there is a statistically
> significant difference along a certain timeline. For example, sometimes
> there are very short and small dots with p<0.05 in ERSP image. Does it also
> represent significant difference? What are the criteria to insist there is
> actual difference? How long range of difference is minimally required?  Or
> is it totally arbitrary depending on the author?
>
> Best wishes,
> Jinwon Chang
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to
> eeglablist-request at sccn.ucsd.edu
>