[Eeglablist] Loading EDF files quickly for parallel processing

Arnaud Delorme arno at ucsd.edu
Wed Sep 14 01:00:54 PDT 2011


Dear Wesley,

you might be using an old version of EEGLAB. Like a lot of users, I process most of my data without a graphical interface and typing "eeglab" works for me. I type "eeglab" and it detect that I do not have a graphical interface and simply adds the relevant paths. Typing "eeglab" also works under Octave. Also, regarding your other question, EEGLAB does not compile any mex files at startup (in fact, it never compiles any mex files).

Best,

Arno

On Aug 12, 2011, at 12:59 AM, Wesley Kerr wrote:

> Hi all,
> 
>      I'm wondering if there's a way to speed up the loading of an edf file into eeglab using pop_biosig without loading a GUI. 
> 
>      When I use the MATLAB GUI, I can do:
> eeglab;
> clear;
>      If I save the path, then loading all subsequent edf files goes a lot faster (10 seconds vs 10 minutes).
> 
>      My issue is that I'm trying to process my data in parallel, so I'm using a non-GUI based MATLAB where I need to reload the path on each processing node that I'm using. When I try to make the processing node do "eeglab;", since I can't use a GUI (and other reasons), MATLAB errors out.
>      I feel like eeglab compiles mex files or does something to speed the process up, but I can't figure out what from reading the code.
> 
> 
>      Does anyone have a fix?
> 
> 
> 
> 
> Details, if it helps:
>      I'm currently working with a database of 100 GB (and growing) of .edf files, each with 26 channels. Subjects have between 10MB and 800MB of data in one file. Of course, I load one hour of one channel at a time and loop through. If I don't do that, I get out of memory errors. (Loading more than an hour also gives memory problems.)
>      My home machine is an iMac OSX 10.5.8 with 4GB of RAM and I'm working on a linux based cluster with up to 16GB of RAM per computing node. Cluster MATLAB is 2010b, local MATLAB is 2009a.
> 
> Actual call:
> EEG=pop_biosig(fname,'channels',i,'blockrange',[(j-1)*window_length j*window_length],'blockepoch','off');
> 
> fname is filename
> i is the channel index (1:26)
> j is the block index (1:header.records/window_length)
> window_length= 60*60 (seconds) = 1 hour
> 
> 
> Thanks so much,
> 
> Wesley
> 
> -- 
> Wesley Kerr
> UCLA MD-PhD Program
> Biomathematics
> Cohen Lab
> wesleytk at ucla.edu
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20110914/60030f50/attachment.html>


More information about the eeglablist mailing list