[Eeglablist] Inconsistent results using clean_artifacts

[Makoto Miyakoshi]

Dear Cristina,

> Do you think that this channel discrepancy will impact later results or
do you think that it will be diluted when looking into large datasets?

Frankly speaking, I would not use tools that produce non-replicable results
if I have other choices. These days, I only use the ASR part in
clean_rawdata() after taking care of highpass filtering and 'bad' channel
rejection.

How much impact it gives to the downstream processes and final results are
empirical questions which are (probably) not fully investigated yet.
You can easily write a technical paper discussing this issue.

Sorry for the negative answer!

Makoto

On Tue, Mar 8, 2022 at 7:49 AM Gil Avila, Cristina <cristina.gil at tum.de>
wrote:

> Dear Prabhakar and Makoto,
>
> Thank you for your responses. I have tried both
> 1) Uncomment the lines in clean_channels() that fix the seed for random
> stream and
> 2) Keep the clean_channels() code as it as and fix in my code the seed of
> the random number generator: rng('default');
>
> In both cases I get a stable set of bad channels although the set of bad
> channels in 1) is different than the set of bad channels in 2). These
> 'tricks' make stable the bad channel detection. Nevertheless, I am
> concerned about the RANSAC algorithm itself. I believe it is suboptimal
> that, depending on how the channels were initially chosen,  different sets
> of bad channels are given as an output. I don't know if this will have an
> impact on later results such as power effects between groups. Do you think
> that this channel discrepancy will impact later results or do you think
> that it will be diluted when looking into large datasets?
>
> Best,
> Cristina
>
> -----Original Message-----
> From: Makoto Miyakoshi <mmiyakoshi at ucsd.edu>
> Sent: Thursday, 3 March 2022 18:40
> To: eeglablist at sccn.ucsd.edu
> Subject: Re: [Eeglablist] Inconsistent results using clean_artifacts
>
> Dear Cristina,
>
> Sorry for the inconvenience.
>
> > I am using the last version of clean_rawdata plugin and every time
> > that I
> run clean_rawdata on the same dataset I get a different set of 'bad
> channels'. Is this problem fixed?
>
> According to this page, clean_channels() was updated 2 years ago. Without
> testing, I guess it is probably not fixed.
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sccn_clean-5Frawdata&d=DwIFaQ&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm1p6x6e36h3EglE&m=8vkQdVejMdf_IHSTIf0e5McIIPD3TbqIxIXVpdRjCZxECTENaMdkB4EoFHSDinQX&s=yz8W2rwlcw_xyQhxNND6QqqHF0ahjP6l7vk_BsP5sgw&e=
>
> If I remember correctly, it uses RANSAC which uses 'repeated random
> subsampling'. The unstable output is a nature of the algorithm and it
> cannot be 'fixed'.
> I know this is an annoying issue. We always prefer to use deterministic
> approaches for the purpose of reproducibility.
>
> Here are my suggested work arounds.
> 1. This is a little bit technical, but you can fix the seed of a random
> number generator by using rng(). Theoretically, this guarantees the
> complete reproducibility from the algorithm. In clean_channels() line 171,
> you can see the following line commented out. I don't know why it is
> commented out, but if you can fix the seed for the random number generator
> there, it may work out. I would try rng() instead of what is there now.
> %stream = RandStream('mt19937ar','Seed',435656);
>
> 2. Alternatively, you disable the electrode removal function in
> clean_rawdata() and perform your own electrode rejection process.
> Sometimes even a simple max value approach like the one used in
> trimOutlier() is sufficient, so you may try it out.
>
> Makoto
>
>
> On Wed, Mar 2, 2022 at 9:01 AM Gil Avila, Cristina via eeglablist <
> eeglablist at sccn.ucsd.edu> wrote:
>
> > I would like to reopen a topic from 2020 regarding the consistency of
> > clean_artifacts. I am using the last version of clean_rawdata plugin
> > and every time that I run clean_rawdata on the same dataset I get a
> > different set of 'bad channels'. Is this problem fixed? Specifically I
> am running:
> >
> > EEG =
> > pop_clean_rawdata(EEG,'FlatlineCriterion',5,'ChannelCriterion',0.8,'Li
> > neNoiseCriterion',4,'Highpass',[0.25
> > 0.75],'BurstCriterion','off','WindowCriterion','off','BurstRejection',
> > 'off','Distance','Euclidian','WindowCriterionTolerances','off');
> >
> > Best,
> > Cristina Gil
> > --
> > Cristina Gil Ávila - PhD candidate
> > Department of Neurology
> > Technische Universität München
> > Munich, Germany
> > cristina.gil at tum.de<mailto:cristina.gil at tum.de>
> > painlabmunich.de<
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.painlabmunich
> > .de_&d=DwIFAw&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm
> > 1p6x6e36h3EglE&m=c_kRevMMCP43N5DUC96kTPQlI81ezyGf6D0jCBCbtNsjpI0SNRrFx
> > tNyOi0J1P05&s=nZ9fhNHv4Zt2tPe1W18k_LDsgbW7YJcgl9PBRpNInAk&e=
> > >
> >
> > _______________________________________________
> > Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > To unsubscribe, send an empty email to
> > eeglablist-unsubscribe at sccn.ucsd.edu
> > For digest mode, send an email with the subject "set digest mime" to
> > eeglablist-request at sccn.ucsd.edu
> >
>
>