[Eeglablist] importing edf, bdf - memory and speed issues

[Andre Achim]

Could it be that part of the memory problem in reading .BDF files comes from
the conversion that would be applied to very large portions of the file?
Reading the three bytes of all sample values (often a very large number) in
separate arrays would require 3 (or perhaps 4, with the final array of
doubles) times as much RAM space as necessary.  My own routines incorporate
something like the following, embedded in higher level commands that return
any required segment for all channels.  I do not see that this should be a
relatively slow process, especially when compared to the disk access time.


André Achim


function DATA=read_Biosemi(fid,start,numValues,scale, zero)
% fid is identifier for an already opened file
% start is offset in file where next reading must start
% numValues is number of consecutive values to read. This command would be
embedded into another one
%    that sets ,start' appropriately, taking into account the segment
structure and makes sure to read
%    only consecutive values belonging to the same channel
% scale and zero are absent to indicate that the channel read is the event
channel (of type
% Boolean).  In this case DATA is (2,numValues) containing 'event' and
'control' codes respectively
% if 'scale' is present, it is the conversion factor from digitized units to
uV and data will be (1,numValues)
% zero, if present is the channel offest
fseek(fid,start,'bof');
DATA=fread(fid,[3,numValues],'uint8');
if nargin>3
    DATA=[1 256 65536]*DATA;
    DATA(DATA>=8388608)=DATA(DATA>=8388608)-16777216; % if >=2^23, subtract
2^24
    DATA=DATA*scale
    if nargin==5, DATA=DATA+zero;
else
    DATA(1,:)=[1 256]*DATA(1:2,:);
    DATA(2,:)=[];
End


-----Message d'origine-----
De : eeglablist-bounces at sccn.ucsd.edu
[mailto:eeglablist-bounces at sccn.ucsd.edu] De la part de Robert Oostenveld
Envoyé : 1 février 2006 03:59
À : eeglablist
Cc : eeglab at sccn.ucsd.edu; Jim Kroger
Objet : [Eeglablist] importing edf, bdf - memory and speed issues

Dear EEGLAB users and colleagues,

For our F.C. Donders FieldTrip toolbox I wrote a mex file for reading  
in Biosemi BDF data (together with Peter Desain of the NICI in  
Nijmegen). The BDF format is very similar to EDF: it has the same  
header, only the data itself is stored in 24 bits (3 bytes) instead  
of 16 bits (2 bytes). Reading in 24 bit per value is possible in  
matlab, but converting the 24 bits into a floating point value number  
makes it very slow. In plain C, this conversion can be done very  
fast. Therefore I thought that the slowness that I observed for  
reading the BDF file was due to the bit-conversion. However, now I  
also noticed recently that other people have problems with the EDF  
format (slow and especially consuming larger amounts of memory than  
expected).

So perhaps it is an idea to adapt the mex file to suit 16 bit EDF  
data. I already have made a quick attempt, but I would appreciate it  
if one of you could pursue this. Attached you can find 6 files:
  * read_edf.m and read_bdf.m, which read in the header and data from  
an EDF/BDF file respectively
  * read_16bit.m and read_24bit.m, which contain the help for the low- 
level mex files
  * read_16bit.c and read_24bit.c, which are the actual low-level mex  
files

Note: I have not tested the 16 bit code and I am not even 100% sure  
whether the 24 bit code works completely correct on any platform  
(Peter Desain told me that it worked for him). I am at least aware  
that the calibration (converting the bit values to physical "uV"  
values) is not yet implemented. I would appreciate it if you could  
compare either the BDF or EDF implementation (depending on your  
dataformat) to the traditional pure-matlab implementations included  
in EEGLAB. Please report back on whether they work (or do not work).  
If they require changes in order for them to work correctly, please  
tell me where in the code and why the changes are needed. I will then  
incorporate your suggestions in my "official" version, which will  
benefit all users.

The main matlab functions will not work directly with EEGLAB, since  
they are designed for Fieldtrip and have a sligtly different API. But  
that should not be a major issue, it is straightforward to convert  
them to EEGLAB-like functions. Testing the functionality should be  
done on the Matlab command line without the EEGLAB gui anyway. The  
mex files require compilation before they work: on the matlab command  
line you should enter "mex read_16bit.c" and/or "mex read_24bit.c"

I am looking forward to your reactions,
Robert

PS I have already discussed with Scott, Arno and colleagues that we  
want to share more file-input routines between EEGLAB and FieldTrip  
(and BIOSIG). We still have to work out the details for organising  
this, but in the future efforts like these will be more smoothly  
integrated in our respective open source projects.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Robert Oostenveld, PhD
F.C. Donders Centre for Cognitive Neuroimaging
Radboud University Nijmegen
phone: +31-24-3619695
http://www.ru.nl/fcdonders/
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~