[Eeglablist] Running AMICA in a super-computer

[Jason Palmer]

Hi Tyler,

 

Are you running this on the SCCN cluster or your own cluster? Could you tell
me what command you are using to run amica in matlab, and if it is your own
cluster setup, what mpirun or mpiexec command is running, whether it is sge
or torque and what if any qsub script is being run?

 

Also the first lines of the stdout generated by Amica where it identifies
the processes and their respective nodes .

 

I wonder if you are actually running multiple (overlapping) copies of the
same run on several nodes, as can happen with the mpirun commands in some
environment setups, instead of having the processes run in concert and
communicating.

 

Best,

Jason

 

From: eeglablist [mailto:eeglablist-bounces at sccn.ucsd.edu] On Behalf Of
Tyler Grummett
Sent: Monday, July 24, 2017 9:00 AM
To: EEGLABLIST
Subject: [Eeglablist] Running AMICA in a super-computer

 

Dear eeglab,

 

Recently Im facing a problem when running multiple AMICAs in a
super-computer at once, where I get the following error:

...

 iter   258 lrate =  1.0000000000 LL =  -0.0238247322 nd =  0.0001777274, D
=   0.89129E-01  0.89129E-01  ( 33.34 s, 182.8 h)

 iter   259 lrate =  1.0000000000 LL =  -0.0238213053 nd =  0.0001768634, D
=   0.89103E-01  0.89103E-01  ( 34.18 s, 187.4 h)

 iter   260 lrate =  1.0000000000 LL =  -0.0238178712 nd =  0.0001762253, D
=   0.89076E-01  0.89076E-01  ( 33.08 s, 181.4 h)

forrtl: Permission denied

forrtl: severe (28): CLOSE error, unit 19, file "Unknown"

Image              PC                Routine            Line        Source


amica15ub          00000000010D4003  Unknown               Unknown  Unknown

amica15ub          00000000010D1A6D  Unknown               Unknown  Unknown

amica15ub          0000000000441780  Unknown               Unknown  Unknown

amica15ub          0000000000419E08  Unknown               Unknown  Unknown

amica15ub          00000000004021DE  Unknown               Unknown  Unknown

amica15ub          000000000118C1A4  Unknown               Unknown  Unknown

amica15ub          00000000004020C1  Unknown               Unknown  Unknown

 

However, this error doesn the error doesnt generate an error in matlab so it
tries to proceed forward and uses the ICA weights from the most recent
iteration. Obviously this isnt ideal, as AMICA requires a lot of iterations
to get good ICs.

 

Does anyone have any idea how to fix this or find a work around? I can
confirm that there are no jobs using the same temporary file, temporary
output folder, or any of the functions that are run in AMICA. They are all
copied to their own folder and run independently. At least thats what I hope
is happening.

 

Im running out of things to try.

 

Regards,

Tyler

 

************************* 

 

Tyler Grummett ( BBSc, BSc(Hons I))

PhD Candidate

Brain Signals Laboratory

Flinders University

Rm 5A301

Ext 66125

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20170726/fc83b291/attachment.html>