[Eeglablist] ICA Misinformation

Fri Jun 23 15:51:11 PDT 2017

Robert et al.,
I've been following the discussion with interest, and beg everyone's pardon
for pointing out a few matters of context that might be obvious to some.
I'm not an expert in the mathematical details of EEG analysis. I'm a
consumer - but one with a number of years of experience grappling with a
variety of methods for analyzing multimodal data from very different
sources. From that perspective, I'd like to point out:

1) Neither ICA nor any other method (regression/GLM, Bayesian modeling,
meta-analysis, etc.) is magic. For any such method it's trivially easy to
find data sets of poor quality that yielded results of questionable
reliability, validity, and generalizability. Many such examples have been
published.

A brief, artifact-rich dataset including 19 channels is the sort of toy
dataset that might yield a poor deconstruction, especially if
pre-processing is not done carefully. Also, if the exact ICA parameters and
algorithm are not specified, it's hard to know why the deconstruction is
distorted. But to nonetheless impugn *all* ICA analysis, including
potentially more sophisticated algorithms used on better datasets, seems a
bit bold.

For all sorts of data and all sorts of output: GIGO. If that had been the
original point (e.g., "Hey everyone, be aware that in *this* situation,
here's a specific problem that can arise from ICA, and (either) here's how
to avoid it (or) let's figure out a solution") - THAT approach would have
avoided this long debate (which has been somewhat interesting, but mostly
from a sociological perspective, IMO).

Illustration by analogy: a non-trivial proportion of manuscripts have
inappropriate uses of all sorts of quantitative procedures, and flawed data
sets. The procedures include t-tests, F tests, correlational and regression
analyses, nonparametric statistics, factor analysis, various flavors of
modeling, etc. In many cases, the inappropriate use of quant methods yields
questionable or invalid interpretations. Of course, with a t or F test of a
small data set, a content expert can often see the poor decisions and why
the result is invalid. With more complex datasets and modeling algorithms
(like ICA), this is much harder, even for an expert.

But my question for Robert is: Does that fact that I can easily find any
number of these examples for every kind of analysis, mean that we should
scrap F tests, correlations, Bayesian modeling, neural networks, and all?
Or does it instead mean that people should be more scrupulous in designing
their studies, in collecting and cleaning their datasets, and in following
best practices for analyses of all types?

I do believe one thing users of ICA can and should do is run multiple ICAs
and show that their IC deconstructions are stable. I also think researchers
should more often make their code and, if possible, their datasets
available so others can check the results. More transparency in science is
always better.

2) Robert made the point that 19 channels are common in medical EEG. Many
of us consider that a pretty sparse array. So the assumption that any
shortcomings lie in the *analytic* approach is logically invalid (Arnaud
made this point in one his early turns). There are many common practices in
medicine that involve very flawed, out-of-date measurement techniques. So
it is equally valid, given Robert's example, to pin the problem on the
measurement practices of medical electrophysiology, as on analytic measures
that have been optimized for research-grade data. If I might hazard some
rank speculation: maybe neurologists who use these spare arrays could
eventually know more about their patients if they started using more
sensitive and denser EEG arrays, and learned newer analytic methods. At the
same time, perhaps there are ways to achieve more robust and high-fidelity
decompositions of sparser EEG arrays. Research should continue on that
front. Individuals who communicate with both practitioners and researchers,
like Robert, might find it a most rewarding use of their time to help those
communities collaborate to figure out how to best advance medical
practices, rather than hastily assuming that one side or the other is
simply 'wrong'.

3) Robert's assertion that results from (properly applied) ICA are less
biologically veridical than channel-level averaging is inconsistent with my
understanding of volume conductance and other fundamental physiological
properties of the EEG signal. However, I am not a physiologist. So I am
wondering if Robert or anyone else can point to a reference that would
correct my imperfect understanding? That is, is there any evidence - even a
single data set - showing that channel-level data are better reflections of
sources of cortical EEG patterns than sensibly unmixed estimates of those
sources? (Assuming reasonable head models & etc.) If there is evidence of
that sort we should all know about it, so please share.

So my conclusion from this thread is: Robert showed that a small and
mediocre or worse data set, subjected to an unknown ICA algorithm, yielded
results that are somewhat distorted in some regards. So ICA is susceptible
to GIGO, just like all other analytic methods ever tried. But if such
existence proofs invalidated a methodological approach, we'd have no
applied mathematics whatsoever.
    Instead, as examples like these accumulate, fortunately many
researchers are not immediately compelled to give up: rather, they militate
for better data sets and more judicious analysis choices, and devise even
cleverer and more nuanced analytic procedures. In fact that's been
happening in the EEG community for years. So, I conclude that Robert's
original post and video have done very little good except to reveal some
misconceptions and possibly weak practices, and to reinforce some caveats
about data quality and use of ICA that EEGLab workshop presenters and
authors have for years regularly disclaimed and warned users about.
     In short, I see no compelling evidence that the sky has fallen.

If any of this interpretation is unfair or misguided, I'm happy to be
corrected.

On Fri, Jun 23, 2017 at 6:52 AM, Stefan Debener <
stefan.debener at uni-oldenburg.de> wrote:

> Dear Robert,
>
> I have expanded my illustration and now consider the phase differences
> between two channels, slides 13 to 16 of the updated pdf:
> https://www.dropbox.com/s/e70qhf91dgc5anu/Thatcher_summary_2.pdf?dl=0
>
> Note that phase values were derived by the Hilbert transform of the
> bandpass filtered signal, as explained by W Freeman here:
> http://www.scholarpedia.org/article/Hilbert_transform_for_brain_waves
>
> More details on the particular implementation I used are here:
> https://de.mathworks.com/help/signal/ref/hilbert.html
>
> If you measure phase differences between two channels, consider the result
> as your gold standard, and then apply a spatial filter operation such as
> ICA or other, the phase differences may indeed be different. I assume any
> spatial filter (that effectively spatially filters the data) changes phase
> values and phase difference values. As a toy example I include the common
> average reference. If you apply a common average reference to the raw data,
> then bandpass filter as before, and compare the phase difference values to
> your "gold standard", then the phase differences will change as well. Now
> which phase values are valid, those obtained by one particular reference
> scheme or those by another? In my view they are both arbitraty, since
> recording settings as well as preprocessing steps may have a strong impact
> on the actually measured phase. There is no reason to assume that a change
> in phase, or in phase differences, "adulterates" a magically clean phase
> signal obtained from the raw data - simply because there is no such
> magically clean raw brain signal available in the first place!
>
> Your claim that ICA has somehow corrupted the data such that previously
> super reliable clinical effects all over a sudden vanished is not
> convincing either. Artifacts such as eye blinks and lateral eye movements
> are very common, I hope you can agree at least here. Now, keep in mind that
> they contribute fixed spatial patterns  - as long as the electrodes cap
> does not shift during acquisition the projections of the sources of those
> artifacts do not change. My illustrations above show very clearly how
> artifacs indeed adulterate phase values, just as Arnos illustrations do!
> Now, if you disregard artifactual influences you may end up with highly
> reliable connectivity effects - but they tell you very little about brain
> function! Even more troubling, if you compare two individuals EEGs (say,
> one "healthy", one "abnormal"), then a different amount of artifacts in the
> data, if not carefully taken care of during preprocessing, will produce
> spurious results that are falsely attributed to differences in brain
> function. Actually, given that many artifacts often contribute much more
> variance to that raw signals than (reasonably well validated) brain
> signals, such as fronto-midline theta, this is actually very likely! So,
> what we learn is that:
>
> Artifacts not accounted for adulterate EEG phase values
>
> Best,
>
> Stefan
>
>
>
> Am 22.06.17 um 20:30 schrieb Robert Thatcher:
>
>> Dear Stefan,
>>     The attachment did not contain any measures of phase differences
>> between channels.   It is very difficult to visually see differences in
>> phase differences.  One must use the cross-spectrum to calculate phase
>> differences and compare phase differences in degrees.   Phase difference
>> varies from -180 to 180 degrees and one must look at the numbers.   Below
>> is a url to the two power points that also show visually similar EEG
>> tracings but also computed the instantaneous phase differences using the
>> Hilbert transform (complex demodulation).  Four identical time points were
>> selected and they demonstrated totally different phase differences with
>> respect to the O1 channel and the other 18 channels.  No matter what
>> reference channel one selects and no matter what identical time points one
>> selects there are always large differences in the phase difference between
>> channels in all frequency bands.   I also computed the average phase
>> difference in the artifact free parts of the record and the averages were
>> statistically significantly different at P < 0.0001 and the same for the
>> FFT.
>>
>> Proof of phase difference adulteration is in the power points.   I am
>> again copying the hyperlink here:
>>
>>
>> http://www.appliedneuroscience.com/Phase_Diff-Original_&_Del
>> orme-Post-ICA-4_time_points.zip
>>
>>
>> This cannot be explained by a low quality ICA reconstruction because the
>> ICA reconstruction was conducted by Arnu using EEGLab software.
>>
>> Robert
>> On Thursday, June 22, 2017, 2:00:19 PM EDT, Stefan Debener <
>> stefan.debener at uni-oldenburg.de> wrote:
>>
>>
>> Dear Robert,
>>
>> I looked up some own data and find absolutely no evidence in favour of
>> your ICA phase adulteration claim, see the attached pdf report. I guess
>> you simply used a poor ICA implementation, and/or a poor component
>> selection. The attached example is in full accordance with Arnos reply,
>> with the difference that I zoom into a clearly visibile alpha
>> oscillation, to have a reference brain signal. The example shows no
>> evidence that occipital alpha phase is biased by ICA eye blink
>> correction. This is a very typical example and based on a quick and
>> dirty ICA decomposition, nothing fancy, to keep this demo simple. Better
>> preprocessing and component selection would easily further improve the
>> signal quality.
>>
>> Best,
>>
>> Stefan
>>
>>
>>
>> Am 20.06.17 um 19:53 schrieb Robert Thatcher:
>> >
>> > Dear Arno,
>> >
>> > 1)*On Phase Differences in the Original vs the Delorme ICA
>> > Reconstruction: *We can agree or disagree about whether or not some
>> > small eye movement artifact was in the hand selection that I did.  But
>> > that misses the main point here.  That is the ICA reconstruction
>> > alters each and every data point in the entire record including all
>> > artifact free portions no matter what one selects. For example, the
>> > record is 6 minutes and 51 seconds = 411 seconds. The Mitsar sample
>> > rate was 250 samples per second = 102,750 data samples. Phase
>> > difference for each frequency band for each and every one of the
>> > 102,750 data samples has been altered by your own ICA reconstruction
>> > in the EDF file that you emailed to me. Unless you were to sit next to
>> > me or if we do a Team Viewer it is not possible for me to demonstrate
>> > this for all of the data points and then create a power point for all
>> > of these data samples.  However, I can show some exemplars, for
>> > example, I have created two figures at 4 different time points (1 sec;
>> > 2:27 sec; 42 sec & 5:49 sec) that you can download. You can extract
>> > each screen capture and expand them so that you can see that the exact
>> > same time points were selected and the Hilbert transform JTFA for the
>> > 4 time points resulted in different phase differences in all channel
>> > combinations with respect to O1 for all frequencies.  The same is true
>> > no matter which channel is selected to compute the phase differences
>> > in degrees.  The same is true also if one computes averages of the
>> > instantaneous phase differences or if one uses the FFT.  Here is the
>> > download URL:
>> >
>> > http://www.appliedneuroscience.com/Phase_Diff-Original_&_Del
>> orme-Post-ICA-4_time_points.zip
>> >
>> >
>> > 2)*On the WinEEG ICA Reconstruction: *I agree that having access to
>> > ICA components themselves and the topography is critical in
>> > understanding exactly what the WinEEG software did. Unfortunately, I
>> > personally do not have access to the WinEEG software.
>> > Clinician/Scientists in Australia use the WinEEG software and they
>> > were the ones that expressed concern about phase difference distortion
>> > at a workshop in Adelaide and gave me the original and the WinEEG ICA
>> > eye movement corrected files in EDF format.  They explained that they
>> > removed only one ICA component for eye movement before they
>> > reconstructed a new time series.  At first, I was impressed because
>> > the eye movements were absent in the reconstructed time series.  I
>> > then was able to use JTFA (Hilbert transform) to compare the two edf
>> > files and discovered that all of the phase differences for all
>> > channels for all frequencies had been altered by the ICA
>> > reconstruction including artifact free periods.  I could demonstrate
>> > this by individual time comparisons or averages of instantaneous phase
>> > differences or by the FFT.  A user of WinEEG explained that they do
>> > not throw away the original raw digital data, however I was told that
>> > they believe that the ICA reconstructed times series is artifact free
>> > and therefore they compute means and standard deviations for their
>> > normative database using the ICA reconstructed data and not the hand
>> > edited or artifact deleted original data samples like other commercial
>> > companies do.  Your ICA reconstructed time series is actually less
>> > different than the original phase difference in comparison to the
>> > WinEEG ICA.  Nonetheless, both your ICA reconstruction and the WinEEG
>> > reconstructions are significantly different than the original recording.
>> >
>> > Best regards,
>> >
>> > Robert
>> >
>> > Cp���
>> >
>> >
>> > On Tuesday, June 20, 2017, 1:12:41 AM EDT, Arnaud Delorme
>> > <arno at ucsd.edu <mailto:arno at ucsd.edu>> wrote:
>> >
>> >
>> > Dear Robert,
>> >
>> > 1) *On my ICA decomposition analysis on your data.* You have selected
>> > a subset of the file where there is 1 minute and 41 second data of eye
>> > free data. I was only able to select 40 seconds in the same file, and
>> > I also showed that even in this short file, there was some residual
>> > eye movements. Jason and Stefan agreed with me. This is the reason why
>> > ICA components power spectrum over frontal channels (and frontal
>> > channels only) was affected below 10 Hz frequency band in my data
>> > analysis. So on my ICA decomposition, our disagreement comes from the
>> > interpretation. You feel that the power we remove at low frequency in
>> > frontal channel is not eye movement. In an attempt to convince you, I
>> > have picked up a clean region from your EDF dataset, and did some
>> > dipole localization at this latency. We see that in the clean data,
>> > the best dipolar fit (with 2 symmetrical dipoles) ends up near the eye
>> > balls with a residual variance of 6.9%. Hopefully this convinces you
>> > that your data is not free of eye movement artifacts. If you are
>> > willing to take a step further you might contemplate the idea that ICA
>> > can remove this residual spurious activity.
>> >
>> > 2) *On the WinEEG ICA decomposition analysis.* It is critical for us
>> > to see the scalp topography (and if possible continuous activity) of
>> > the components the people at the Australia workshop selected. Without
>> > this, it is not possible for us to comment on the cleaned data. I
>> > agree with you that there was some phase distortion in alpha (visible
>> > directly in the raw data in the first email you sent) and that this
>> > should not be the case. However, without seing the ICA decomposition,
>> > it is not possible for us to conclude as to wether people selected the
>> > wrong ICA components or if the ICA decomposition implemented in this
>> > software is buggy (ICA is not a simple algorithm and it is sensitive
>> > to numerical imprecision and a lot of other parameters - a suboptimal
>> > implementation could easily explain the WinEEG results). Also, you
>> > seem to imply that the WinEEG people were running ICA on their data
>> > then throwing away the raw data (which is why their ICA biased
>> > neurofeedback database is useless for practical purposes). Is that
>> > correct? One should never throw away the raw data. If they did throw
>> > away the raw data, it is an indication that the WinEEG are not
>> > rigorous in their approach and therefore might not have implemented
>> > ICA in an optimal way. If it is not the case, one may easily
>> > reconstruct the database of measures with or without ICA decomposition
>> > (assuming ICA is done right which does not seem to be the case) then
>> > assess data measure distoritions (power, phase index, etc…) in a
>> > statistical fashion.
>> >
>> > Best wishes,
>> >
>> > Arno
>> >
>> > http://sccn.ucsd.edu/~arno/download/clean_edf_file_analysis2.pdf <
>> http://sccn.ucsd.edu/%7Earno/download/clean_edf_file_analysis2.pdf%20>
>> > <http://sccn.ucsd.edu/%7Earno/download/clean_edf_file_analysis2.pdf>
>> >
>> >> On Jun 18, 2017, at 11:44 AM, Robert Thatcher <rwthatcher2 at yahoo.com
>> <mailto:rwthatcher2 at yahoo.com>
>> >> <mailto:rwthatcher2 at yahoo.com <mailto:rwthatcher2 at yahoo.com>>> wrote:
>>
>> >>
>> >> <Pre-ICA-Hand Artifact free selections.edf>
>>
>> >
>> >
>> >
>> > Dieser Nachrichteninhalt wird auf Anfrage komplett heruntergeladen.
>>
>>
>>
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.uc
> sd.edu
> For digest mode, send an email with the subject "set digest mime" to
> eeglablist-request at sccn.ucsd.edu
>

-- 

Gedeon O. Deák, Ph.D.
Department of Cognitive Science
9500 Gilman Dr.
Univ. CA, San Diego
La Jolla, CA 92093-0515

cogdevlab.ucsd.edu
http://www.gedeondeak.com/
http://ucsd.academia.edu/GedeonDeák
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20170623/e85f1511/attachment.html>