[Eeglablist] AMICA lrate gets stuck

[Makoto Miyakoshi]

Dear Jason,

If you have time could you please let us know about the update schedule?

Makoto

On Tue, Nov 10, 2015 at 1:02 PM, Norman Forschack <forschack at cbs.mpg.de>
wrote:

> Hi Jason and Makoto,
>
> in this thread Jason announced a debugged version of amica coming soon. I
> just wanted to kindly ask whether this is already done or if we should
> apply the mentioned workaround?
>
> Thanks a lot!
> Norman
>
> ----- On Aug 25, 2015, at 6:28 AM, Jason Palmer japalmer29 at gmail.com
> wrote:
>
> > Sorry this line should be:
> >
> >
> >
> >>> [W,S,mods] = runamica12(newdat, ....);
> >
> >
> >
> >
> >
> >
> > From: Jason Palmer [mailto:japalmer at ucsd.edu]
> > Sent: Monday, August 24, 2015 9:17 PM
> > To: 'Kevin Tan'; 'mmiyakoshi at ucsd.edu'; 'EEGLAB List'
> > Subject: RE: [Eeglablist] AMICA lrate gets stuck
> >
> >
> >
> >
> > Hi Kevin,
> >
> >
> >
> > Sorry, there is a bug in the code tries to take the sqrt of the negative
> > eigenvalue (even though that dimension is being removed) making the
> LL=NaN and
> > aborting. The eigenvalue is actually essentially zero, resulting from
> rank
> > deficiency likely due to re-referencing, so more cleaning won’t
> necessarily
> > change the arbitrarily small value to positive, unless you increase the
> > dimensionality of the data. I’m currently fixing this bug along with
> some other
> > sphering and PCA issues and will release debugged versions soon.
> >
> >
> >
> > For now you can do the work around of sphering before Amica, e.g.
> >
> >
> >
> >>> [U,D] = eig(cov(EEG.data(:,:)’));
> >
> >>> U = fliplr(U); D = fliplr(flipud(D)); % make descending order
> >
> >>> dd = diag(D); numeig = sum(dd > 1e-9);
> >
> >>> Sph = diag(sqrt(1./dd(1:numeig))) * U(:,1:numeig)‘;
> >
> >>> newdat = Sph * EEG.data(:,:); % reduce to numeigs dimensions
> >
> >>> [W,S,mods] = runamica12(EEG.data(:,:), ....);
> >
> >>> EEG.icasphere = S*Sph;
> >
> >>> EEG.icaweights = W;
> >
> >
> >
> > This should run amica on full rank data an avoid the negative near zero
> > eigenvalue problem until it is fixed in the Amica code.
> >
> >
> >
> > Best,
> >
> > Jason
> >
> >
> >
> >
> >
> >
> >
> > From: Kevin Tan [ mailto:kevintan at cmu.edu ]
> > Sent: Monday, August 24, 2015 6:39 PM
> > To: Jason Palmer; mmiyakoshi at ucsd.edu ; EEGLAB List
> > Subject: Re: [Eeglablist] AMICA lrate gets stuck
> >
> >
> >
> >
> > Hi Jason,
> >
> >
> >
> >
> >
> > I'm running into a negative min eigenvalue issue for ~25% of my
> subjects. This
> > results in the binary not exporting anything to the amica output dir,
> stopping
> > the main loop prematurely.
> >
> >
> >
> >
> >
> > Before running AMICA, the data is fairly aggressively cleaned:
> >
> >
> > 1) PREP pipeline
> >
> >
> > 2) remove mastoids & PREP-interpolated chans for rank reduction
> >
> >
> > 3) 1hz hi-pass
> >
> >
> > 4) epoch no baseline correction
> >
> >
> > 5) epoch rejection (ch means deviation, variance, max amplitude dif >
> 2.5 SDs)
> >
> >
> >
> >
> >
> > Not sure what else I can do to clean the data to make the eigenvalues
> positive.
> >
> >
> >
> >
> >
> > I'm using Biosemi 128ch which is known for dynamic range issues, but I
> run
> > everything in double. Not sure if demeaning each channel would help
> since it's
> > already hi-passed.
> >
> >
> >
> >
> >
> > Also, not sure if it matters, but AMICA seems to do dimension reduction
> despite
> > me removing channels to make up for 'robust' reference rank reduction.
> >
> >
> >
> >
> >
> > For the subjects that do run on AMICA, the ICs seem a lot cleaner than
> Infomax,
> > which makes me want to stick to AMICA.
> >
> >
> >
> >
> >
> > Bad subject log example:
> >
> >
> >
> >
> > 1 : data = -2.3969683647155762 -2.910743236541748
> > getting the mean ...
> > mean = -7.73083349593588626E-2 -8.98852135101791128E-2
> -0.17064473790401868
> > subtracting the mean ...
> > getting the sphering matrix ...
> > cnt = 706560
> > doing eig nx = 128 lwork = 163840
> > minimum eigenvalues = -4.02618476752492072E-14 0.59534647773064309
> > 0.66105027982216646
> > maximum eigenvalues = 3718.0696499000956 2980.9762500746847
> 1012.6027880321443
> > num eigs kept = 127
> > numeigs = 127
> >
> >
> >
> >
> >
> > Good subject log example:
> >
> >
> >
> >
> > 1 : data = 3.1855385303497314 5.7855358123779297
> > getting the mean ...
> > mean = -0.38155908557715745 -0.27761248863920301 -0.3608881566308772
> > subtracting the mean ...
> > getting the sphering matrix ...
> > cnt = 703488
> > doing eig nx = 130 lwork = 169000
> > minimum eigenvalues = 1.35676859295476523E-13 0.80288149429025613
> > 1.1256218532296671
> > maximum eigenvalues = 9749.2425686202987 1277.5793884179475
> 700.98046655297128
> > num eigs kept = 129
> > numeigs = 129
> >
> >
> >
> >
> >
> > Many many thanks for the continued help!
> >
> >
> > –Kevin
> >
> >
> >
> >
> >
> > --
> >
> >
> > Kevin Alastair M. Tan
> >
> >
> > Lab Manager/Research Assistant
> >
> >
> > Department of Psychology & Center for the Neural Basis of Cognition
> >
> >
> > Carnegie Mellon University
> >
> >
> >
> >
> >
> > Baker Hall 434 | kevintan at cmu.edu | tarrlab.org/kevintan
> >
> >
> >
> >
> >
> > On Fri, Aug 21, 2015 at 7:36 PM, Makoto Miyakoshi < mmiyakoshi at ucsd.edu
> > wrote:
> >
> >
> > Dear Kevin and Jason,
> >
> > In the Figure 1 of the following paper, you can see an example of the
> shift of
> > log likelihood of AMICA model along with iteration.
> >
> >
> >
> >
> >
> > Rissling AJ, Miyakoshi M, Sugar CA, Braff DL, Makeig S, Light GA. (2014).
> > Cortical substrates and functional correlates of auditory deviance
> processing
> > deficits in schizophrenia. Neuroimage Clin. Oct 01; 6 424-437
> >
> >
> >
> >
> >
> > You can see that after 700 iterations there is no 'jump' any more, which
> may
> > correspond to what Jason says reaching to the 'noise floor'. In the beta
> > version (?) of AMICA we use here in SCCN, it has a convergence criterion
> and
> > usually stops at around 1000 iterations (smallest I saw was around 700,
> maximum
> > 1500).
> >
> >
> >
> >
> >
> > Kevin, your questions are always very interesting and I learn a lot from
> them.
> > Thank you Jason for your answers and sharing knowledge.
> >
> >
> >
> >
> >
> > Makoto
> >
> >
> >
> >
> >
> > On Mon, Aug 17, 2015 at 4:35 PM, Jason Palmer < japalmer29 at gmail.com >
> wrote:
> >
> >
> >
> >
> >
> > Hi Kevin,
> >
> >
> >
> > The Infomax wchange is actually the weight change TIMES the lrate, which
> is
> > going to 1e-7. So the actual final wchange for extended infomax is 1e7 *
> > wchange.
> >
> >
> >
> > For Amica, if the nd weight change gets down to the 10^-5 magnitude,
> that is
> > usually about the best you can expect with the large number of
> parameters being
> > estimated and the finite computer precision. How small it can get
> depends on
> > the number of samples you have compared to the number of channels. More
> > channels = more parameters (nchan^2) = relatively little data = “noisier”
> > convergence. More data = better determined optimum = less noisy
> convergence =
> > smaller nd. For 64 channels with 100,000 samples, an nd of 10^-5 sounds
> > appropriate.
> >
> >
> >
> > However you can change “maxiter” from the default 2000, using the
> ‘maxiter’
> > keyword. This LL should continue to increase and the nd decrease (or at
> least
> > not increase) beyond 2000 iterations, but not significantly. There
> should be a
> > weight change “noise floor” reached, where the LL continues to increase
> by less
> > and less, with possible reductions in lrate, and the nd hovers around the
> > “noise floor”.
> >
> >
> >
> > Best,
> >
> > Jason
> >
> >
> >
> > From: Kevin Tan [mailto: kevintan at cmu.edu ]
> > Sent: Monday, August 17, 2015 4:21 PM
> > To: japalmer at ucsd.edu ; EEGLAB List
> > Subject: Re: AMICA lrate gets stuck
> >
> >
> >
> >
> >
> > Hi Jason,
> >
> >
> >
> >
> >
> > Thanks so much for the detailed response, really helps clarify what
> drives the
> > lrate changes between the two implementations.
> >
> >
> >
> >
> >
> > However, for the same dataset processed the same way, AMICA yields higher
> > wchange at last iteration (0.0000464763) versus extended Infomax
> (0.000000).
> >
> >
> >
> >
> >
> > What are some reasons for this discrepancy, and what can I do improve
> it? Or is
> > weight change between the two implementations not comparable? The entire
> AMICA
> > log is linked in original post if that helps.
> >
> >
> >
> >
> >
> > Thanks again,
> >
> >
> > Kevin
> >
> >
> >
> >
> >
> > --
> >
> >
> > Kevin Alastair M. Tan
> >
> >
> > Lab Manager/Research Assistant
> >
> >
> > Department of Psychology & Center for the Neural Basis of Cognition
> >
> >
> > Carnegie Mellon University
> >
> >
> >
> >
> >
> > Baker Hall 434 | kevintan at cmu.edu | tarrlab.org/kevintan
> >
> >
> >
> >
> >
> > On Mon, Aug 17, 2015 at 7:06 PM, Jason Palmer < japalmer29 at gmail.com >
> wrote:
> >
> >
> > Hi Kevin,
> >
> >
> >
> > The Amica lrate is not supposed to decrease. The algorithm is a more
> typical
> > gradient descent / Newton optimization algorithm, as opposed to the
> Infomax
> > implementation in runica.m, which uses a type of simulated annealing,
> deciding
> > whether to reduce the learning rate based on the angle between recent
> update
> > directions. The idea there is that this angle will be small when the
> algorithm
> > is near an optimum, as though it is “heading right for it”, so the lrate
> gets
> > reduced if the algorithm is moving “erratically” with a large angle
> between
> > consecutive directions, and doesn’t get reduced if the estimate is
> “moving
> > smoothly”. In practice, this annealing method usually ends up in fact
> reducing
> > the learning rate continuously until it reaches the preset minimum, which
> > usually happens at around 500 iterations (500 reductions). I.e. the
> angle is
> > never actually small, so the stopping condition is essentially a maximum
> number
> > of iterations, with the updates being of smaller and smaller magnitude
> due to
> > the lrate decreasing.
> >
> >
> >
> > Amica only reduces the lrate if the likelihood decreases. In theory,
> with a
> > reasonable optimum, an optimization algorithm should be able to converge
> > without reducing the learning rate. The convergence is measured by the
> weight
> > change (the nd in the amica output) independently of the lrate. That is,
> the
> > weight change should theoretically decrease to zero with a constant
> > (sufficiently small) lrate—the higher the better since higher lrate means
> > faster convergence. A potential issue with the runica Infomax is early
> > convergence if you are starting far from the optimum. Fortunately the
> optimum
> > is usually not far from the “sphering” starting point, so 500 iterations
> is
> > usually enough to converge (even with decreasing lrate).
> >
> >
> >
> > So in Amica, the convergence is judged by the “nd”, not the lrate. The
> lrate
> > should be ideally be 0.5 or 1.0, and the LL should be increasing, and
> the nd
> > should be decreasing to zero.
> >
> >
> >
> > Hope that is helpful.
> >
> >
> >
> > Best,
> >
> > Jason
> >
> >
> >
> >
> >
> > From: Kevin Tan [mailto: kevintan at cmu.edu ]
> > Sent: Monday, August 17, 2015 2:31 PM
> > To: jason at sccn.ucsd.edu ; EEGLAB List
> > Subject: AMICA lrate gets stuck
> >
> >
> >
> >
> >
> > Hi Dr. Palmer & EEGLAB list,
> >
> >
> >
> >
> >
> > I'm trying out AMICA for artifact rejection and DIPFIT. In my tests, the
> lrate
> > consistently gets stuck at 0.5, stopping only due to max iteration
> limit. This
> > does not happen with extended Infomax.
> >
> >
> >
> >
> >
> > This happens whether I use the cluster (128 threads) or a normal PC (4
> threads).
> > I run AMICA 'locally' as it's called within a matlab script already run
> via
> > PBS, not sure if that makes a difference.
> >
> >
> >
> >
> >
> > Here's the AMICA test stream:
> >
> >
> > - PREP pipeline
> >
> >
> > - Remove PREP-interpolated channels
> >
> >
> > - Remove 1 additional channel for rank consistency
> >
> >
> > - 1hz FIR hi-pass
> >
> >
> > - Epoch -500 to 1000ms no baseline correction
> >
> >
> > - Epoch rejection
> >
> >
> > - AMICA (using EEG(:,:) -- is it ok to concatenate epochs like this?)
> >
> >
> >
> >
> >
> > Here's the output log (using the cluster):
> >
> >
> > https://cmu.box.com/s/t7j3shmwjj1wj8to80au8mdm6b5676rh
> >
> >
> >
> >
> >
> > Many thanks,
> >
> >
> > Kevin
> >
> >
> > --
> >
> >
> > Kevin Alastair M. Tan
> >
> >
> > Lab Manager/Research Assistant
> >
> >
> > Department of Psychology & Center for the Neural Basis of Cognition
> >
> >
> > Carnegie Mellon University
> >
> >
> >
> >
> >
> > Baker Hall 434 | kevintan at cmu.edu | tarrlab.org/kevintan
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> > For digest mode, send an email with the subject "set digest mime" to
> > eeglablist-request at sccn.ucsd.edu
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> >
> > Makoto Miyakoshi
> > Swartz Center for Computational Neuroscience
> > Institute for Neural Computation, University of California San Diego
> >
> >
> >
> >
> > _______________________________________________
> > Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> > For digest mode, send an email with the subject "set digest mime" to
> > eeglablist-request at sccn.ucsd.edu
>



-- 
Makoto Miyakoshi
Swartz Center for Computational Neuroscience
Institute for Neural Computation, University of California San Diego
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20151127/9e77afa7/attachment.html>