[Eeglablist] general solution for memory problems?

[Brian Murphy]

Hi

I was wondering if people wanted to brianstorm on a general solution for 
the memory problems when loading large files. Personally I have been 
working with Biosemi acquired BDF files, and importing them via BIOSIG, 
but it appears that the problem is not specific to that particular pairing.

I have used one of the manual approaches suggested by arno at salk.edu 
(http://sccn.ucsd.edu/pipermail/eeglablist/2007/001776.html), of 
splitting the foreign format file in in smaller chunks, and then 
concatenating them with the append function. The "peak" memory required 
for the conversion does seem rather gargantuan - for example with 1GB of 
free physical RAM, I could not read in a 110 MB BDF file, even though it 
only required 300 MB once imported to EEGLAB.

But, presumably this could be automated. So, perhaps something like this 
would work:
 - determine the largest contiguous piece of RAM available to Matlab (I 
understand from previous posts that it is contiguous RAM rather than 
total RAM that matters), lets call this A(vailable)
 - estimate the total memory needed for the final imported data (in my 
experience, it seems to be the raw filesize by a factor of three) - F(inal)
 - then the temporary space available for conversion T(emp) is A-F
 - then determine how big a fraction of the input file can be dealt with 
at a time - divide the Temporary space by the peak conversion factor C 
(again, in my experience, more than 10 times the original filesize of 
RAM is required - what is other people's experience?): T/C
 - then split the time-course of the original file (of size O) into O/C 
pieces.
 - load these pieces in
 - merge them
 - discard the pieces

Any comments? I am new to Matlab and EEGLAB, so much or all of what I've 
just said might be rubbish! If not, does anyone have suggestions on how 
to implement any of the steps (e.g. I don't know how to ask Matlab how 
much memory is available) as part of a script? Or would it be just as 
easy to rewrite the original reading routines?

Regards,

Brian
Centre for Mind/Brain Sciences
University of Trento