[Eeglablist] ICA Misinformation

Robert Thatcher rwthatcher2 at yahoo.com
Tue Jun 20 07:31:07 PDT 2017


Dear Stefan,    Thank you for your thoughtful comments.  First, I am not blaming EEGlabs, quite the opposite because it has helped educate many young people and professionals.  My goal is to open a discussion to recognize the limitations of ICA reconstruction where there is a price to pay when a smaller number of components is used to "reconstruct" a larger dataset.  There are several mathematical theorums (Takens and in differential geometry) where there always is a loss of information.   ICA for feature detection and general decomposition is very useful but also has some limitations such as the assumption of independence when the underlying physiology of reenterant loops and phase delays that determine the probability of action potentials, etc. is critical for how the brain works (see Buzsaki or Walter Freeman, etc.).   My concern is about the ICA reconstruction of creating something out of nothing where the time between electrical potentials between channels and phase differences between channels and other spectra features in the entire EEG record can be altered at all frequencies and all channel combinations when ICA reconstruction is used for a few brief eye movements.  
As far as I know only WinEEG a Russian commercial EEG company uses ICA reconstruction and I have no idea who the programmer is.  I agree with Arnaud that it is important to view the ICA components themselves but at this time I do not have access to the WinEEG software itself.   Perhaps I can gain access in the future?  The problems with ICA reconstruction were 1st brought to my attention in 2010 when I testified as an expert witness in a court case where there was high cross-validation of a person's coherence and phase differences by NxLink, NeuroRep, SKIL and NeuroGuide but a large difference with a ICA reconstruction dataset.  The court disallowed admission of the outlier once it was not possible to establish the validity of the analyses.  Then around 2012 several clinicians and scientists discovered alteration in phase differences in all channel combinations and in all frequency bands when using the WinEEG ICA reconstruction.   
It is not necessary to monitor all eye-related movements using eye tracking measures to prove that ICA "adulterates" the EEG signal in-between blinks. For example, one can simply compute the instantaneous cross-spectrum using the Hilbert transform at each data sample in-between eye related artifacts.  Also, one can demonstrate changes in phase differences between P3 and P4 or P3/4 and C3/4 or other distant locations in all frequency bands in-between eye blinks or movements.   
No one ignors artifact but over the last 50 years of QEEG and the over 100,000 publications in the National Library of Medicine (Pubmed) the standard has been to delete the eye movement and other artifact when it occurs and not use ICA.   There have been a variety of very good artifact detection methods followed by and deletion and used by all of the major medical centers and Universities around the world without the need to do ICA reconstruction to allegedly "correct the record".   The scientific literature of QEEG since the late 1950s with Ross Adey and Donald Walter at UCLA in their NASA studies have demonstrated cross-validation accuracies greater than 0.95 and this level of discriminant accuracy continued in the 1970s and beyond by E. Roy John and Frank Duffy and many others.  All of these hundreds of studies used artifact deletion methods.
Georges and Arnaud and others have generally agreed that we should consider standards to test whether or not there is any alteration in phase differences between channels in parts of the EEG record that have no artifact. A simple suggestion is to compare the phase differences in the artifact free sections of the original recording to the ICA reconstructed new time series.   If there are significant changes in the phase differences then report these changes in a publication.  Editors of Journals can insist on this comparison before a ICA reconstruction study is published which will help safe guard against unknown adulteration and thereby increase the reliability and validity of EEG analyses.
Georges mentioned NeuroGuide that does not use ICA recontruction and also has a built-in Signal Generator that allows one to test any and all analytics to determine accuracy.  Also, the FDA audits NeuroGuide and requires that strict validation and reliability tests are performed on any updates of the software.  We are required to document this before we release new updates.  Thus, there are some standards that are in place that unfortunately not all commercial EEG companies follow but nonetheless many do.  I also agree that more and better standards are needed in the commercial realm, from amplifiers to data analyses to training of those using this technology.   
Thank you again and best wishes,
Robert 

On Tuesday, June 20, 2017, 6:56:56 AM EDT, Georges Otte <georges.otte at telenet.be> wrote:

Good morning Stefan

i accept your point of view that the open source community should not be held responsible for correct or crooked implementations of their software in commercial packages. As a clinician i am, logically i guess,  concerned with the output quality of the software. As i discussed  with Arnaud some mails ago, there is a contrast between rigourous standards that are set for hardware ( amplifiersl) where there are no obligations to do so for the totality of the commercial systems ( including the software) Even if quality differences between several brands of amplifiers are clearly present (at least a user knows when he byes a Porche or a Lada what to expect) this is not the case in closed ( most commercial) environments. In open source this assurance is indirectly present  as there is continuous multiuser and multi-expert feedback, bug resolution and improvement. However i feel that as far as commercial software is concerned we do not know. Like Arnaud i have seen flawed commercial software more then enough.. As such i have always admired the strategy in Neuroguide to provide user with signal generator and acces to all algorithms ( even in the free demo version) in order to check reliability for themselves. Your suggestion  to delegate that quality assurance task to the manufacturer ( but probably you were referring to more general responsability) is perfect provided they would be obliged to get approval from some official quality assurance organisme ( fi  CE label). It is of course not stricktly the mission of research and open source communities to take up that  burden,  but signals in this direction ( such as Arnaud did for EEGLab use) are IMHO very usefull. It would also be usefull that the research community would point out the conditions and constraints that are linked to using ICA in the field. This is not a hidden critic on the method but is a general remark that seems  valid for all algorithms FFT, Hilbert and other transforms etc....

It is obvious that many clinicians " blindly" :-) apply methods that are push button implemented on their EEG machines without clear awareness of the algorithm  constraints. The seem to have fundamental confidence in the commercial providers. For instance in ICA the sources must be independent. How many clinicians are aware what that means? Stationarity is another one of those definitions. 

As a clinician it seems normal that we turn to the research community for help as many new algorithms are being sprouted by them. Thanks for thta! Do they then carry some responsibility or none at all ?  It is an " open" ethical question ( not a criticism) but a feel that  some move to common grounds would be helpfull for the whole field of qEEG. Cooperation beats competition. 

The problem of blink artefacts is indeed very fascinating and i approve completely with your suggestion on the need of further exploration. Theta - delta in prefrontal and frontal regions are often cloacked by ocular artefacts and although some are visible and can be cleaned other can be much more difficult to detect. The danger of confusing them with " excess of theta delta" is big but also the danger of missing the theta delta caused by a slowly developing astrocytoma or glioblastoma. So the problem and question you raise are crucially important and maybe we should link to our collegues neurosurgeons who have done  eeg on these patients and found those tumors ( in MRI) and operated upon. Would ICA have been helpfull in improving the signal to noise ratio and pathology detection rate? Geeting preop EEG traces could be very instructive . Maybe this is not an important problem for pure research as such but as a clinician research without clinical purpose is " intellectual spielerei"  of which i do not disapprove on the contrary but in the end there is a common goal as in the clinic is where the rubber meets the road. 

As far as the EEG model is concerned, i had the impression ( but of course my perception can be wrong) that the ICA community relies quite heavily on the chaotic brain model where independent sources are projected simultaneously on all sensors and where one cannot come to meaningfull conclusions about underlying brain dynamics without some source detection algorithm in case  ICA or another member of the BSs family) As said this personal perception can be wrong ( mine often are) as it would lead to a tautology and hence a cult. 
As we rely on differential  amplifiers to reject the "common mode" we do can do so in software ( cf Guido Nolte et al) to look at signals with non zero phase differences to learn about information transfert in brain networks. This model is what i would prefer to be more " optimistic" 

About " my website". It obviously has been hacked ! Thanks for warning me about this. I feel that i can maybe sue Tommy Hilfiger and use the money to support the EEG rsearch community. '-) ? i

Unfortunately the website was only an announcement for the z score nfb workshops that  were held in Gent and in no way were supposed to provide deep  scoentific insight in that technique. 

Thanks however for pointing this out. Being a hacker myself ( a good white hacker) this is a nice attack and an open invitation for a counter attack challenge. '-) . I will keep you posted....

Sincerely

Georges




Verstuurd door Dr. Georges Otte
Neuro-psychiatrie
1/42/652/36/760
georges.otte at pandora.be
Consultatie: Bijlokehof 27 9000 Gent (Be).
GSM: 0032(0)478 205 202
www.zscoreloretanfb.be


> Op 20 jun. 2017 om 10:44 heeft Stefan Debener <stefan.debener at uni-oldenburg.de> het volgende geschreven:
> 
> Hi there,
> 
> It feels odd to me that an open source community spends time and effort into dealing with commercial software packages others apparently make good money with - by selling it to clinicians acquiring it in good faith. It should be the responsibility of commercial software developers to make sure their implementations are accurate, not the responsibility of this forum. Matter of fact, ICA has been so successful that many providers included it into their products. The few developers I know have done this with care, by validating their implementations against those distributed with EEGLAB. Does this hold for the developers of WinEEG as well? Who are the developers of WinEEG? How do they advise their "thousands of clinicians" on how to use ICA? Do they just offer a button to click or do they really care about their customers? Why are we, the experienced ICA users in this forum, blamed here?
> 
> Again, we should keep in mind the complexity of artifacts. Arno's reply (https://sccn.ucsd.edu/~arno/download/clean_edf_file_analysis2.pdf) nicely shows that there was a strong lateral eye movement artifact present at sec 3.25 Robert apparently ignored. When it comes to eye related artifacts, several sources may contribute even to a simple eye blink artifact as recorded with scalp EEG. While most of the blink may come from movement of the eyelids (causing the large monophasic deflection), some eye muscles need to contract as well to make the eyelids move, no doubt about it (all human behaviour requires muscle contraction, hence is generated by an electrial signature or "EEG artifact"). Sometimes the EMG initiation of a blink is visible after good ICA decompositions as a short, low amplitude high-frequency burst at the onset of the eye blink artifact (a hint of this "EMG" initiation of the blink is evident for instance in figure 5A of the CORRMAP reference paper (Fig 5A, black trace; in Viola et al., 2009, Clin Neurophysiology). At the same time the eyeballs rotate during a blink (interestingly in different direction, depending on whether the blink is voluntary or spontaneous! See https://www.ncbi.nlm.nih.gov/pubmed/15792897), and because the eyeballs are electrically charged as well this contribution can be expected to leave a trace in the EEG. Then of course we have lateral eye movements as in Roberts recording, and microsaccades and maybe some other processes like eyelid fluttering I am not well aware of... To avoid the "mess", some people may prefer an eyes closed EEG recording condition, aiming for artifact-free EEG. However, in many subjects one can clearly observe a slow rotation of the eye balls, which again will contribute an artifact in the EEG. I don't want to know how many "abnormal delta" diagnoses this may have caused...
> 
> More likely brain processes are more complex than EEG artifacts, but the complexity of artifacts should not be disregarded, as artifacts can easily result in misinterpretation. So, ignoring artifact is NOT the solution. I hope there is agreement here, at least.
> 
> In the hand of experienced users, ICA has been shown to be super useful in artifact correction, but far from perfect! Experienced users usually do not advertise ICA as the magic solution to every problem EEG comes with! Limitations are usually mentioned very clearly in discussion sections of methods papers, but of course authors cannot be blamed if readers do not read their whole paper. Robert's line of reasoning seems to be that because ICA "adulterates" the EEG signal in between blinks it does not work properly. In order to test this bold assumption it would be necessary to monitor all eye-related movements carefully (eye tracking & magnetic search coil) and simultaneously with EEG. Only then we would know what the eye blink generators are actually doing in between blinks, and how well, or how poorly, ICA captures and disentangles the different contributors. It seems unlikely to me that eye artifact generators are perfectly electrically silent in between blinks! In any case, such a multi-modal study should be done for MORE than 19 channels and for MORE than 1 min 41 secs per subject! It has been discussed frequently here and in several papers that ICA benefits from more channels and longer recording durations (and adequate preprocessing, etc...).
> 
> In my view, reasons for the confusion may be a poor software implementation, the insufficient consideration of EEG artifact complexity, and the relying on far from perfect recording conditions. Therefore it is not justified to make any bold statements on "ICA misinformation". Better would be to make proposals on how to adress the issue of ICA sensitivity/specificity in an open-minded way. This would require data to determine the sensitivty and specificity of ICA for removing eye artifact along with concurrent eye tracking and magnetic search coil recordings. As I mentioned before, and others have agreed, relying on the phase of the raw EEG trace is NOT a good gold standard. Even for high-SNR brain signals such as alpha activity some form of spatial filtering is necessary to optimize it (even a bipolar channel is a spatial filter!). The best approach would be to consult with SSVEP experts like Ramesh's group, and discuss an experiment where phase is known (or can be reasonably well estimated by the experimental manipulations). Maybe the company behind WinEEG, or the company seeling plenty of EEG software on appliedneuroscience.com, wants to sponsor such a multi-modal study?
> 
> Georges, I don't think we have a pessimistic view on the EEG, we enjoy the challenge that comes with a scientific view. Maybe good part of this discussion comes down to a scientific versus clinical view on EEG. If that's true, this would be sad news (for patients, actually!), but explain why a good part of the applied neurofeedback community seems so disconnected from the scientific EEG community. BTW, Georges, can you explain why your link www.zscoreloretanfb.be is a Belgian website selling Tommy Hilfiger outfit? I am not interested in Tommy Hilfiger outfit, I only want to know what zscoreloretanfb is.
> 
> Arno, I truly admire your patience!
> 
> Best,
> Stefan
> 
> 
> 
>> Am 20.06.17 um 07:12 schrieb Arnaud Delorme:
>> Dear Robert,
>> 
>> 1) *On my ICA decomposition analysis on your data.* You have selected a subset of the file where there is 1 minute and 41 second data of eye free data. I was only able to select 40 seconds in the same file, and I also showed that even in this short file, there was some residual eye movements. Jason and Stefan agreed with me. This is the reason why ICA components power spectrum over frontal channels (and frontal channels only) was affected below 10 Hz frequency band in my data analysis. So on my ICA decomposition, our disagreement comes from the interpretation. You feel that the power we remove at low frequency in frontal channel is not eye movement. In an attempt to convince you, I have picked up a clean region from your EDF dataset, and did some dipole localization at this latency. We see that in the clean data, the best dipolar fit (with 2 symmetrical dipoles) ends up near the eye balls with a residual variance of 6.9%. Hopefully this convinces you that your data is not free of eye movement artifacts. If you are willing to take a step further you might contemplate the idea that ICA can remove this residual spurious activity.
>> 
>> 2) *On the WinEEG ICA decomposition analysis.* It is critical for us to see the scalp topography (and if possible continuous activity) of the components the people at the Australia workshop selected. Without this, it is not possible for us to comment on the cleaned data. I agree with you that there was some phase distortion in alpha (visible directly in the raw data in the first email you sent) and that this should not be the case. However, without seing the ICA decomposition, it is not possible for us to conclude as to wether people selected the wrong ICA components or if the ICA decomposition implemented in this software is buggy (ICA is not a simple algorithm and it is sensitive to numerical imprecision and a lot of other parameters - a suboptimal implementation could easily explain the WinEEG results). Also, you seem to imply that the WinEEG people were running ICA on their data then throwing away the raw data (which is why their ICA biased neurofeedback database is useless for practical purposes). Is that correct? One should never throw away the raw data. If they did throw away the raw data, it is an indication that the WinEEG are not rigorous in their approach and therefore might not have implemented ICA in an optimal way. If it is not the case, one may easily reconstruct the database of measures with or without ICA decomposition (assuming ICA is done right which does not seem to be the case) then assess data measure distoritions (power, phase index, etc…) in a statistical fashion.
>> 
>> Best wishes,
>> 
>> Arno
>> 
>> http://sccn.ucsd.edu/~arno/download/clean_edf_file_analysis2.pdf <http://sccn.ucsd.edu/%7Earno/download/clean_edf_file_analysis2.pdf>
>> 
>>> On Jun 18, 2017, at 11:44 AM, Robert Thatcher <rwthatcher2 at yahoo.com <mailto:rwthatcher2 at yahoo.com>> wrote:
>>> 
>>> <Pre-ICA-Hand Artifact free selections.edf>
>> 
>> 
>> 
>> _______________________________________________
>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
>> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20170620/8decce4b/attachment-0001.html>


More information about the eeglablist mailing list