[Eeglablist] distributed processing

[Arnaud Delorme]

Dear Joseph, Thomas, Robert and other interested in parallel computing,

we have developed a function called qsubfunc 
(http://sccn.ucsd.edu/~arno/qsubfunc/qsubfunc.php) that take as input 
cell arrays of argument { input_function_name { argin1 argin2 argin3 ... 
} { 'argout1' 'argout2' ... } }. The function uses the Sun Grid Engine 
queue system for Linux that is freely available at 
http://gridengine.sunsource.net/ (there is no need to recompile the 
Linux Kernel: this program runs as a daemon on all machines of your 
cluster). Below is an example to compute the average ERSP for the first 
16 components of the tutorial dataset (in approximately the time it 
takes to compute one). The first advantage of this function is that it 
runs from Matlab and return results in the global Matlab workspace. The 
second advantage of this function is that you can run jobs creating 
Matlab figures. Finally, the function pops up a window that allows you 
to continue to use your current Matlab session. Note that this 
implementation runs into the problem mentioned by Robert that every 
process requires a Matlab license (in the example below, 16 licenses).

Arno

eeglab;
EEG = pop_loadset( 'ee114squaresepochsICA.set', '/home/arno/ee114/');
argtf = { EEG.pnts, [EEG.xmin EEG.xmax]*1000, EEG.srate, 3, 'plotersp', 
'off', 'plotitc', 'off' };

ersp = zeros(15, 200, 16);
qsubfunc({{ 'timef', { EEG.icaact(1,:), argtf{:} }, { 'ersp(:,:,1)', 
'itc', 'times', 'freq' }}, ...
{ 'timef', { EEG.icaact(2,:), argtf{:} }, { 'ersp(:,:,2)' }}, ...
{ 'timef', { EEG.icaact(3,:), argtf{:} }, { 'ersp(:,:,3)' }}, ...
{ 'timef', { EEG.icaact(4,:), argtf{:} }, { 'ersp(:,:,4)' }}, ...
{ 'timef', { EEG.icaact(5,:), argtf{:} }, { 'ersp(:,:,5)' }}, ...
{ 'timef', { EEG.icaact(6,:), argtf{:} }, { 'ersp(:,:,6)' }}, ...
{ 'timef', { EEG.icaact(7,:), argtf{:} }, { 'ersp(:,:,7)' }}, ...
{ 'timef', { EEG.icaact(8,:), argtf{:} }, { 'ersp(:,:,8)' }}, ...
{ 'timef', { EEG.icaact(9,:), argtf{:} }, { 'ersp(:,:,9)' }}, ...
{ 'timef', { EEG.icaact(10,:), argtf{:} }, { 'ersp(:,:,10)' }}, ...
{ 'timef', { EEG.icaact(11,:), argtf{:} }, { 'ersp(:,:,11)' }}, ...
{ 'timef', { EEG.icaact(12,:), argtf{:} }, { 'ersp(:,:,12)' }}, ...
{ 'timef', { EEG.icaact(13,:), argtf{:} }, { 'ersp(:,:,13)' }}, ...
{ 'timef', { EEG.icaact(14,:), argtf{:} }, { 'ersp(:,:,14)' }}, ...
{ 'timef', { EEG.icaact(15,:), argtf{:} }, { 'ersp(:,:,15)' }}, ...
{ 'timef', { EEG.icaact(16,:), argtf{:} }, { 'ersp(:,:,16)' }}});

figure; imagesc(mean(ersp(:,:),3));

Joseph Brooks wrote:

> Hello All,
>
> I planning on writing some code to distribute processing of timef and 
> crossf for many subjects across various computers on our network. This 
> will help to reduce the overall time it takes to process a group of 
> subjects with these functions. Basically, I just want to do as many 
> subjects as possible in parallel. Has anyone done something like this 
> already? If so, I'd appreciate any advice you have on doing it or code 
> you would care to share. I will, of course, share any code that I 
> develop for this purpose. We have a linux network of computers as well 
> as networked windows machines. So, anything on either of those 
> platforms would be helpful.
>
> I'm looking into the recently-released Distributed Computing Toolbox 
> for Matlab, but I haven't gotten my hands on it yet. Does anyone have 
> any experience with this?
>
> Thanks!
>
> Joseph Brooks