[Eeglablist] Inconsistent results using clean_artifacts

Tue Mar 8 12:17:36 PST 2022

I have added 6 test cases to check the reproducibility of clean_rawdata/clean_artifacts and, at least on the sample EEGLAB dataset, the results are reproducible.

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sccn_eeglab-5Ftests_tree_master_unittesting-5Fclean-5Frawdata&d=DwIGaQ&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm1p6x6e36h3EglE&m=iKNSivKdsf0I-8JNOFeRjFkORB_kXkzMVVRHWKZ5-Z_sU6xFXyeS1vB6T0Yywswj&s=zT8NS9cZQ9wIfpUuV08XZF99cmYEY40eT5ZoXoELDs0&e= 

Totals:
   6 Passed, 0 Failed, 0 Incomplete.
   12.1697 seconds testing time.

If there is some lack of reproducibility, please submit an issue at https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sccn_clean-5Frawdata&d=DwIGaQ&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm1p6x6e36h3EglE&m=iKNSivKdsf0I-8JNOFeRjFkORB_kXkzMVVRHWKZ5-Z_sU6xFXyeS1vB6T0Yywswj&s=Bs18yPXwce5Hx3yLtsQSJjkHP6DYftyGRgvdr9wPHwE&e=  , add details and, if possible, provide a link to your data.
We are still working on adding more test cases and working on GitHub automated reporting
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sccn_eeglab-5Ftests_runs_5441687584-3Fcheck-5Fsuite-5Ffocus-3Dtrue&d=DwIGaQ&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm1p6x6e36h3EglE&m=iKNSivKdsf0I-8JNOFeRjFkORB_kXkzMVVRHWKZ5-Z_sU6xFXyeS1vB6T0Yywswj&s=VgpGvRy2lj1vc4l_syqzfOTmm8CAjYSCJ5a_S2a36Po&e= 

Cheers,.

Arno

> On Mar 8, 2022, at 9:02 AM, Makoto Miyakoshi via eeglablist <eeglablist at sccn.ucsd.edu> wrote:
> 
> Dear Cristina,
> 
>> Do you think that this channel discrepancy will impact later results or
> do you think that it will be diluted when looking into large datasets?
> 
> Frankly speaking, I would not use tools that produce non-replicable results
> if I have other choices. These days, I only use the ASR part in
> clean_rawdata() after taking care of highpass filtering and 'bad' channel
> rejection.
> 
> How much impact it gives to the downstream processes and final results are
> empirical questions which are (probably) not fully investigated yet.
> You can easily write a technical paper discussing this issue.
> 
> Sorry for the negative answer!
> 
> Makoto
> 
> On Tue, Mar 8, 2022 at 7:49 AM Gil Avila, Cristina <cristina.gil at tum.de>
> wrote:
> 
>> Dear Prabhakar and Makoto,
>> 
>> Thank you for your responses. I have tried both
>> 1) Uncomment the lines in clean_channels() that fix the seed for random
>> stream and
>> 2) Keep the clean_channels() code as it as and fix in my code the seed of
>> the random number generator: rng('default');
>> 
>> In both cases I get a stable set of bad channels although the set of bad
>> channels in 1) is different than the set of bad channels in 2). These
>> 'tricks' make stable the bad channel detection. Nevertheless, I am
>> concerned about the RANSAC algorithm itself. I believe it is suboptimal
>> that, depending on how the channels were initially chosen,  different sets
>> of bad channels are given as an output. I don't know if this will have an
>> impact on later results such as power effects between groups. Do you think
>> that this channel discrepancy will impact later results or do you think
>> that it will be diluted when looking into large datasets?
>> 
>> Best,
>> Cristina
>> 
>> -----Original Message-----
>> From: Makoto Miyakoshi <mmiyakoshi at ucsd.edu>
>> Sent: Thursday, 3 March 2022 18:40
>> To: eeglablist at sccn.ucsd.edu
>> Subject: Re: [Eeglablist] Inconsistent results using clean_artifacts
>> 
>> Dear Cristina,
>> 
>> Sorry for the inconvenience.
>> 
>>> I am using the last version of clean_rawdata plugin and every time
>>> that I
>> run clean_rawdata on the same dataset I get a different set of 'bad
>> channels'. Is this problem fixed?
>> 
>> According to this page, clean_channels() was updated 2 years ago. Without
>> testing, I guess it is probably not fixed.
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sccn_clean-5Frawdata&d=DwIFaQ&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm1p6x6e36h3EglE&m=8vkQdVejMdf_IHSTIf0e5McIIPD3TbqIxIXVpdRjCZxECTENaMdkB4EoFHSDinQX&s=yz8W2rwlcw_xyQhxNND6QqqHF0ahjP6l7vk_BsP5sgw&e=
>> 
>> If I remember correctly, it uses RANSAC which uses 'repeated random
>> subsampling'. The unstable output is a nature of the algorithm and it
>> cannot be 'fixed'.
>> I know this is an annoying issue. We always prefer to use deterministic
>> approaches for the purpose of reproducibility.
>> 
>> Here are my suggested work arounds.
>> 1. This is a little bit technical, but you can fix the seed of a random
>> number generator by using rng(). Theoretically, this guarantees the
>> complete reproducibility from the algorithm. In clean_channels() line 171,
>> you can see the following line commented out. I don't know why it is
>> commented out, but if you can fix the seed for the random number generator
>> there, it may work out. I would try rng() instead of what is there now.
>> %stream = RandStream('mt19937ar','Seed',435656);
>> 
>> 2. Alternatively, you disable the electrode removal function in
>> clean_rawdata() and perform your own electrode rejection process.
>> Sometimes even a simple max value approach like the one used in
>> trimOutlier() is sufficient, so you may try it out.
>> 
>> Makoto
>> 
>> 
>> On Wed, Mar 2, 2022 at 9:01 AM Gil Avila, Cristina via eeglablist <
>> eeglablist at sccn.ucsd.edu> wrote:
>> 
>>> I would like to reopen a topic from 2020 regarding the consistency of
>>> clean_artifacts. I am using the last version of clean_rawdata plugin
>>> and every time that I run clean_rawdata on the same dataset I get a
>>> different set of 'bad channels'. Is this problem fixed? Specifically I
>> am running:
>>> 
>>> EEG =
>>> pop_clean_rawdata(EEG,'FlatlineCriterion',5,'ChannelCriterion',0.8,'Li
>>> neNoiseCriterion',4,'Highpass',[0.25
>>> 0.75],'BurstCriterion','off','WindowCriterion','off','BurstRejection',
>>> 'off','Distance','Euclidian','WindowCriterionTolerances','off');
>>> 
>>> Best,
>>> Cristina Gil
>>> --
>>> Cristina Gil Ávila - PhD candidate
>>> Department of Neurology
>>> Technische Universität München
>>> Munich, Germany
>>> cristina.gil at tum.de<mailto:cristina.gil at tum.de>
>>> painlabmunich.de<
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.painlabmunich
>>> .de_&d=DwIFAw&c=-35OiAkTchMrZOngvJPOeA&r=kB5f6DjXkuOQpM1bq5OFA9kKiQyNm
>>> 1p6x6e36h3EglE&m=c_kRevMMCP43N5DUC96kTPQlI81ezyGf6D0jCBCbtNsjpI0SNRrFx
>>> tNyOi0J1P05&s=nZ9fhNHv4Zt2tPe1W18k_LDsgbW7YJcgl9PBRpNInAk&e=
>>>> 
>>> 
>>> _______________________________________________
>>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>>> To unsubscribe, send an empty email to
>>> eeglablist-unsubscribe at sccn.ucsd.edu
>>> For digest mode, send an email with the subject "set digest mime" to
>>> eeglablist-request at sccn.ucsd.edu
>>> 
>> 
>> 
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu