<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:black;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>Hi Kevin,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>The Amica lrate is not supposed to decrease. The algorithm is a more typical gradient descent / Newton optimization algorithm, as opposed to the Infomax implementation in runica.m, which uses a type of simulated annealing, deciding whether to reduce the learning rate based on the angle between recent update directions. The idea there is that this angle will be small when the algorithm is near an optimum, as though it is “heading right for it”, so the lrate gets reduced if the algorithm is moving “erratically” with a large angle between consecutive directions, and doesn’t get reduced if the estimate is “moving smoothly”. In practice, this annealing method usually ends up in fact reducing the learning rate continuously until it reaches the preset minimum, which usually happens at around 500 iterations (500 reductions). I.e. the angle is never actually small, so the stopping condition is essentially a maximum number of iterations, with the updates being of smaller and smaller magnitude due to the lrate decreasing.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>Amica only reduces the lrate if the likelihood decreases. In theory, with a reasonable optimum, an optimization algorithm should be able to converge without reducing the learning rate. The convergence is measured by the weight change (the nd in the amica output) independently of the lrate. That is, the weight change should theoretically decrease to zero with a constant (sufficiently small) lrate—the higher the better since higher lrate means faster convergence. A potential issue with the runica Infomax is early convergence if you are starting far from the optimum. Fortunately the optimum is usually not far from the “sphering” starting point, so 500 iterations is usually enough to converge (even with decreasing lrate).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>So in Amica, the convergence is judged by the “nd”, not the lrate. The lrate should be ideally be 0.5 or 1.0, and the LL should be increasing, and the nd should be decreasing to zero.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>Hope that is helpful.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>Best,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>Jason<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Kevin Tan [mailto:kevintan@cmu.edu] <br><b>Sent:</b> Monday, August 17, 2015 2:31 PM<br><b>To:</b> jason@sccn.ucsd.edu; EEGLAB List<br><b>Subject:</b> AMICA lrate gets stuck<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>Hi Dr. Palmer & EEGLAB list, <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I'm trying out AMICA for artifact rejection and DIPFIT. In my tests, the lrate consistently gets stuck at 0.5, stopping only due to max iteration limit. This does not happen with extended Infomax. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>This happens whether I use the cluster (128 threads) or a normal PC (4 threads). I run AMICA 'locally' as it's called within a matlab script already run via PBS, not sure if that makes a difference. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Here's the AMICA test stream:<o:p></o:p></p></div><div><p class=MsoNormal>- PREP pipeline<o:p></o:p></p></div><div><p class=MsoNormal>- Remove PREP-interpolated channels<o:p></o:p></p></div><div><p class=MsoNormal>- Remove 1 additional channel for rank consistency<o:p></o:p></p></div><div><p class=MsoNormal>- 1hz FIR hi-pass<o:p></o:p></p></div><div><p class=MsoNormal>- Epoch -500 to 1000ms no baseline correction<o:p></o:p></p></div><div><p class=MsoNormal>- Epoch rejection<o:p></o:p></p></div><div><p class=MsoNormal>- AMICA (using EEG(:,:) -- is it ok to concatenate epochs like this?)<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Here's the output log (using the cluster):<o:p></o:p></p></div><p class=MsoNormal><a href="https://cmu.box.com/s/t7j3shmwjj1wj8to80au8mdm6b5676rh" target="_blank">https://cmu.box.com/s/t7j3shmwjj1wj8to80au8mdm6b5676rh</a><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Many thanks, <o:p></o:p></p></div><div><p class=MsoNormal>Kevin<o:p></o:p></p></div><div><div><div><div><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>--</span><o:p></o:p></p></div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>Kevin Alastair M. Tan</span><o:p></o:p></p><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>Lab Manager/Research Assistant</span><o:p></o:p></p><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>Department of Psychology & Center for the Neural Basis of Cognition</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>Carnegie Mellon University</span><o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'><a href="https://www.google.com/maps/place/40%C2%B026%2729.5%22N+79%C2%B056%2744.0%22W/@40.4414869,-79.9455701,61m/data=!3m1!1e3!4m2!3m1!1s0x0:0x0" target="_blank">Baker Hall 434</a> | <a href="mailto:kevintan@cmu.edu" target="_blank">kevintan@cmu.edu</a> | <a href="http://tarrlabwiki.cnbc.cmu.edu/index.php/KevinTan" target="_blank">tarrlab.org/kevintan</a></span><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></body></html>