[Eeglablist] Loading EDF files quickly for parallel processing

[Wesley Kerr]

Hi all,

     I'm wondering if there's a way to speed up the loading of an edf file
into eeglab using pop_biosig without loading a GUI.

     When I use the MATLAB GUI, I can do:
eeglab;
clear;
     If I save the path, then loading all subsequent edf files goes a lot
faster (10 seconds vs 10 minutes).

     My issue is that I'm trying to process my data in parallel, so I'm
using a non-GUI based MATLAB where I need to reload the path on each
processing node that I'm using. When I try to make the processing node do
"eeglab;", since I can't use a GUI (and other reasons), MATLAB errors out.
     I feel like eeglab compiles mex files or does something to speed the
process up, but I can't figure out what from reading the code.


     Does anyone have a fix?




Details, if it helps:
     I'm currently working with a database of 100 GB (and growing) of .edf
files, each with 26 channels. Subjects have between 10MB and 800MB of data
in one file. Of course, I load one hour of one channel at a time and loop
through. If I don't do that, I get out of memory errors. (Loading more than
an hour also gives memory problems.)
     My home machine is an iMac OSX 10.5.8 with 4GB of RAM and I'm working
on a linux based cluster with up to 16GB of RAM per computing node. Cluster
MATLAB is 2010b, local MATLAB is 2009a.

Actual call:
EEG=pop_biosig(fname,'channels',i,'blockrange',[(j-1)*window_length
j*window_length],'blockepoch','off');

fname is filename
i is the channel index (1:26)
j is the block index (1:header.records/window_length)
window_length= 60*60 (seconds) = 1 hour


Thanks so much,

Wesley

-- 
Wesley Kerr
UCLA MD-PhD Program
Biomathematics
Cohen Lab
wesleytk at ucla.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20110811/3fa603bc/attachment.html>