[Eeglablist] importing edf, bdf - memory and speed issues

[Andreas Widmann]

Dear Robert, dear list

just my 2 cents on this thread, which is an interesting one.

Sorry Robert, I'm unable to compile your read_24bit.c code. Mex fails with
Error READ_24BIT.C 93 undefined reference to _NOT
with MATLAB 7.1.0.246 on 32bit WinXP SP2 and
read_24bit.c: In function `mexFunction':
read_24bit.c:91: error: parse error before '/' token
read_24bit.c: At top level:
read_24bit.c:102: error: `dat' undeclared here (not in a function)
read_24bit.c:102: warning: data definition has no type or storage class
read_24bit.c:103: error: parse error before "return"
on MATLAB 7.1.0.183 on 32bit SuSE Linux kernel 2.6.8.
Any idea?

> That is because the fread function always  
> returns doubles.
This is of course correct for this code snippet but might be
misunderstood: One has to explicitly define what data type fread should
return, as e.g. with fread(fhandle, n, 'bit24=>float32') or
fread(fhandle, n, '*char').

On speed:
fread's implicit conversion from 24bit integer to 32bit integer/float is
indeed rather slow.

On memory overhead:
Most approaches to reading bdf files I know do read blockwise. This is
slower than reading the file at once, however needs less memory since
one does not need to permute the resulting matrix.
However, when reading blockwise the maximum memory overhead needed for
conversion from 24 to 32bit is only one block, i.e. 256 channels by 2048
samples by 4 byte (with MATLAB > R14) ~ 2MB (max on most setups). Thus,
the conversion should not be the bottleneck.

I was so curious to understand this issue, that I programmed a short and
very crude proof of concept (I hope it does not contain a major bug).
With the function attached I can read a 680 MB bdf file on my machine
(WinXP, 2GB working memory) in 64 seconds without out of memory error
(scaling takes another 20 secs but can for sure be optimized). Pure file
I/O takes approximately 20 of the 64 secs, the remaining 44 secs are 24
to 32 bit conversion and assignment. So, one can read rather large bdf
files at reasonable speed with a memory overhead as small as one block.
Using the fread implicit conversion the function takes about 3 mins to
read the same file.

Under Windows 2k and XP MATLAB's usage of memory can be evaluated using
Process Explorer from
http://www.sysinternals.com/Utilities/ProcessExplorer.html. Double
clicking on the MATLAB process, then on the Performance Graph tab one
can follow online how much working memory MATLAB currently uses. This
helped me a lot programming more memory efficient functions with large
matrices. Using breakpoints in functions one can see immediately how
much memory a processing step consumes (in MB range).

Finally, using Process Explorer I got aware that MATLAB sometimes does
not free all (except the always needed ~100 MB) memory when using clear
all (which also should clear the globals, or am I wrong here?). I did
not yet find a clear system, most times this happened when a function
was interrupted by some error. I can systematically replicate this bug
(?) when opening the array editor on a structure with a large matrix in
one field. After closing the array editor clear all does remove the
structure from workspace but does no longer free any memory allocated to
the structure before. This bug, makes the out of memory problem even
worse since sometimes unused memory is not given back to the operating
system after an out of memory error appeared. The only cure I found was
to restart MATLAB. Does one of you have an idea where this might come
from or how it could be solved?

Best,
Andreas

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: simplebdfreader.m
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20060203/c4523538/attachment.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3190 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20060203/c4523538/attachment.bin>