[Eeglablist] readbdf - big data set - convert to single?

[Arnaud Delorme]

We have a in-house beta version of EEGLAB (EEGLAB 12) that is fully memory mapped meaning that you can process very large files with little memory. If you use BIOSIG for example, the function will read the data file one channel at a time (as Alois mentioned) and resave the data on disk. It is slower but at least it should allow to remove most memory error problems.

Beta release (EEGLAB 12) probably this month.

Arno

On Sep 4, 2012, at 4:41 AM, Yuan-fang Chou wrote:

> When you change data from double to single precision before loading,it seems better for you to re-change inversely before running ICA,as ICA can do its best under double precision condition.
> 
> 2012/8/30 Alois Schloegl <alois.schloegl at ist.ac.at>
> The biosig interface has multiple option to address this problem.
> You can read only specified channels, and/or segments, and output the
> data as single type.
> 
> 
> % select limited list of channels with CHANLIST,
> % return data as single
> HDR = sopen(filename,CHANLIST,'OUTPUT:SINGLE');
> 
> % read segment of length duration
> duration = 1;   % segment lenght
> for k = 0:duration:NRrec*HDR.SPR/HDR.SampleRate,
>         [dat, HDR] = sread(HDR, duration, start);
> end;
> 
> % do not forget to close the file
> HDR = sclose(HDR);
> 
> 
> 
> On 08/29/12 20:09, ANDREW HILL wrote:
> > I would suggest downsampling first - 512 Hz (2 ms resolution in an ERP) should be fine, if that's sufficient for your needs.
> >
> > That'll speed things up remarkably :)  You can use "Decimator" from Biosemi to do this before you import, if you want.
> >
> > Also, I tend to use "Converter" to move BDF files into EDF+ format before importing into Matlab.  This essentially applies a 0.016 Hz highpass filter, discarding the DC offset and making much smaller files.
> >
> > Lastly, your CPU and operating system both need to be 64-bit versions (as does your Matlab) if you want to ever be able to use more than 2GB of RAM - I'm assuming you are running out of RAM even with a lot of it and 64-bit, but just in case that's another gotcha.
> >
> >
> >
> > Best,
> >
> > andrew
> >
> >
> >
> >
> > On Today 12:49 AM, davidebaldo84 at gmail.com wrote:
> >
> > Dear all,
> > Lately I have been working with an EEG dataset containing 64 channels (sampling rate: 2048Hz, Biosemi data).When importing the data into Matlab (via pop_readbdf function) I experience two problems:
> > 1. It takes a lot of time to load and afterwards the pc became extremely slow (I assume that is because of the huge amount of  RAM needed)2. Sometimes the pc run out of memory
> >
> > Thus I have modified the readbdf function, converting the EEG data from double to single precision (each value occupies 4 bytes instead of 8 bytes):
> >     ... (line 100)   catch,           warning backtrace off;           warning('Warning: file might be incomplete');           Records(nrec:end) = [];           DAT.Record(RecLen*length(Records)+1:end,:) = [];           S(nrec:end,:) = [];           break;       end;
> >   (line 109)  end;
> > %%%%%%% START DAVIDE MODIFICATION %%%%%%%%
> >   >>>>>>>      DAT.Record = single(DAT.Record);<<<<<<<  CONVERTING THE EEG DATA FROM DOUBLE TO SINGLE
> > %%%%%%% END DAVIDE  MODIFICATION  %%%%%%%%
> >
> > if rem(Mode,2)==0% AutocalibDAT.Record=[ones(RecLen*length(Records),1) DAT.Record]*EDF.Calib;end;
> > DAT.Record=DAT.Record';
> > ..
> >
> > This way the data importing is much faster. The question is: Do you think I can have any problem because of converting EEG data from double to single precision?
> >
> > Thanks a lot,
> >
> > Davide.
> >
> >
> >
> >
> > _______________________________________________
> > Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> > For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu
> 
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu
> 
> 
> 
> -- 
> Yuan-Fang Chao 
> School of Psychology
> SouthWest University
> Beibei,Chongqing,China
> 
> 
> 
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to eeglablist-request at sccn.ucsd.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20120906/e2678955/attachment.html>