[Eeglablist] script example, to extract epochs?

[Søren Andersen]

Dear Hamish,

the problem you describe often occurs when Matlab 
has multiple set files in memory at the same 
time. Preventing this is fairly simple, but not 
as straightforward as one might expect.

I'll use an example to explain this:

EEG = pop_loadset( 'filename1', files(i).name, 
'filepath', rawpath); %Load first file
% ...
EEG = pop_loadset( 'filename2', files(i).name, 
'filepath', rawpath); %Load second file

When the first line is executed, Matlab only 
needs memory to keep this set file in memory. 
When the second line is executed, the first 
data-set is still in memory, while the second 
dataset is being loaded. The first set-file is 
only overwritten AFTER the second set-file is 
fully loaded, i.e. Matlab requires memory enough to store both set-files.

The following script is functionally the same, but does not have this probem:

EEG = pop_loadset( 'filename1', files(i).name, 
'filepath', rawpath); %Load first file
% ...
clear('EEG')
EEG = pop_loadset( 'filename2', files(i).name, 
'filepath', rawpath); %Load second file

Furthermore, when executing EEGLab in a Script, 
there is often no need to keep a separate ALLEEG 
structure. I've edited your script to minimize 
memory usage. This way, it should be able to run 
through all at once without restarting:

>% Matlab script for epoching all the set files
>
>% 1) get list of set files in raw data directory
>
>rawpath =  'D:\_PROJECTS\_FB\_ANALYSIS\__EEGLAB\SET\01raw\';
>savepath = 'D:\_PROJECTS\_FB\_ANALYSIS\__EEGLAB\SET\02epochs\';
>triggers = {'111' '122' '131' '141' '142' '151' '152' '162'};
>
>cd(rawpath);
>files = dir('*.set');
>nfiles = size(files,1);
>
>
>for i = 1:nfiles
>     EEG = pop_loadset( 'filename', files(i).name, 'filepath', rawpath);
>     EEG = pop_epoch( EEG, triggers, [-.8 1], 
> 'newname', [files(i).name'epochs'], 'epochinfo', 'yes');
>     EEG = pop_rmbase( EEG, [-200    0]);
       EEG.setname='epochs';
>     EEG = pop_saveset( EEG,  'filename', 
> strcat(files(i).name,'_epochs'), 'filepath', savepath);
>     clear('EEG')
>  end


I also removed the checkset commands to make 
things simpler. Those or usually not necessary, 
unless there is some error in the set-structure.

best regards.
Søren Andersen





>% Matlab script for epoching all the set files
>
>% 1) get list of set files in raw data directory
>
>rawpath =  'D:\_PROJECTS\_FB\_ANALYSIS\__EEGLAB\SET\01raw\';
>savepath = 'D:\_PROJECTS\_FB\_ANALYSIS\__EEGLAB\SET\02epochs\';
>triggers = {'111' '122' '131' '141' '142' '151' '152' '162'};
>
>cd(rawpath);
>files = dir('*.set');
>nfiles = size(files,1);
>
>
>for i = 1:nfiles
>     %[ALLEEG EEG CURRENTSET ALLCOM] = eeglab;
>
>     EEG = pop_loadset( 'filename', files(i).name, 'filepath', rawpath);
>     EEG = eeg_checkset( EEG );
>     [ALLEEG EEG CURRENTSET ] = eeg_store(ALLEEG, EEG);
>     EEG = pop_epoch( EEG, triggers, [-.8 1], 'newname', [files(i).name
>'epochs'], 'epochinfo', 'yes');
>     EEG = eeg_checkset( EEG );
>     EEG = pop_rmbase( EEG, [-200    0]);
>     EEG = eeg_checkset( EEG );
>     %EEG = pop_saveset( EEG,  'filename', strcat(files(i).name,
>'_epochs'), 'filepath', savepath);
>     [ALLEEG EEG CURRENTSET] = pop_newset(ALLEEG, EEG, CURRENTSET,
>'setname', 'epochs');
>     EEG = eeg_checkset( EEG );
>
>     EEG = pop_saveset( EEG,  'filename', strcat(files(i).name,
>'_epochs'), 'filepath', savepath);
>     ALLEEG = pop_delset( ALLEEG, [1 2] );
>  end





At 00:39 26.06.2009, you wrote:
>Memory always seems to be an issue.  I've heard using 64-bit matlab will
>help, but that might not be possible (it's not for me at least), so here
>is my solution (it's not pretty I admit): I try and do everything in
>small steps, but all the subjects at once, restarting matlab when it
>crashes and picking up where it left off... ie I do all the epoching in
>one script (for all the subjects), then all the re-referencing in
>another, and so on.
>
>Matlab is bound to crash (memory), and the only solution seems to be to
>actually restart matlab (the pack command doesn't do much, clear and
>restart eeglab doesn't work either). So I have the "input files" in one
>directory ('rawpath' in the script below), then the output files go into
>a separate directory ('savepath'). When matlab inevitabley runs out of
>memory on a bigger-than-usual .set file, it crashes, and I just look in
>the 'savepath' directory and see where it got up to.  I then have a 3rd
>directoy called "finished" - I move the sucessfully processed .set files
>into the "finished" directory (usually just by hand in windows), then
>run the script again. The script works by finding a list of files in the
>raw directory, so once the processed files are moved out of there, it no
>longer sees them, and starts at the next un-processed file.
>
>So far I've been lucky, my files are just small enough that restarting
>matlab freshly will allow them to be epoched, but some of them wont
>epoch if anything else has been opened first. Seems the .set file can be
>up to around 150MB for me.
>
>Like I said, it's not an elegant solution, but as you mentioned the
>memory issue only appeared on the 2nd time around the loop, I thought
>this might be the same problem.
>
>(this is for epoching continuous data, but I guess the principle is the
>same)
>
>
>If anyone has a more elegant solution I'd also be glad to hear it.
>
>
>
>
>% Matlab script for epoching all the set files
>
>% 1) get list of set files in raw data directory
>
>rawpath =  'D:\_PROJECTS\_FB\_ANALYSIS\__EEGLAB\SET\01raw\';
>savepath = 'D:\_PROJECTS\_FB\_ANALYSIS\__EEGLAB\SET\02epochs\';
>triggers = {'111' '122' '131' '141' '142' '151' '152' '162'};
>
>cd(rawpath);
>files = dir('*.set');
>nfiles = size(files,1);
>
>
>for i = 1:nfiles
>     %[ALLEEG EEG CURRENTSET ALLCOM] = eeglab;
>
>     EEG = pop_loadset( 'filename', files(i).name, 'filepath', rawpath);
>     EEG = eeg_checkset( EEG );
>     [ALLEEG EEG CURRENTSET ] = eeg_store(ALLEEG, EEG);
>     EEG = pop_epoch( EEG, triggers, [-.8 1], 'newname', [files(i).name
>'epochs'], 'epochinfo', 'yes');
>     EEG = eeg_checkset( EEG );
>     EEG = pop_rmbase( EEG, [-200    0]);
>     EEG = eeg_checkset( EEG );
>     %EEG = pop_saveset( EEG,  'filename', strcat(files(i).name,
>'_epochs'), 'filepath', savepath);
>     [ALLEEG EEG CURRENTSET] = pop_newset(ALLEEG, EEG, CURRENTSET,
>'setname', 'epochs');
>     EEG = eeg_checkset( EEG );
>
>     EEG = pop_saveset( EEG,  'filename', strcat(files(i).name,
>'_epochs'), 'filepath', savepath);
>     ALLEEG = pop_delset( ALLEEG, [1 2] );
>  end
>
>
>Hamish Innes-Brown
>Research Assistant, Music and Pitch Project
>
>The Bionic Ear Institute
>384-388 Albert Street
>East Melbourne, VIC 3002
>Tel: +61 3 9667 7529
>Fax: +61 3 6997 7518
>
>
>
>
>________________________________
>
>From: eeglablist-bounces at sccn.ucsd.edu
>[mailto:eeglablist-bounces at sccn.ucsd.edu] On Behalf Of Andrew Hill
>Sent: Thursday, 25 June 2009 5:50 PM
>To: eeglablist
>Subject: Re: [Eeglablist] script example, to extract epochs?
>
>
>several people have suggested solutions for this - thanks to everyone!
>i'm still having issues understanding a few things in EEGLab scripting,
>and especially in memory management.
>
>here is my script:
>
>
>
>
>
>for S = [108 109];
>     eeglab
>     Cond =[ 'LVF_ConTarg_AllCue'; 'LVF_IncTarg_AllCue';
>'RVF_ConTarg_AllCue'; 'RVF_IncTarg_AllCue' ];
>     trigs=[110 111 112 113; 120 121 122 123; 210 211 212 213; 220 221
>222 223 ];
>
>
>
>
>    for c = 1:4;
>
>
>
>
>     %load a file, for example 108.set, which is not filtered and uses
>common vertex reference.
>     EEG = pop_loadset( 'filename', [int2str(S),'.set'], 'filepath',
>'/Users/andrew/Documents/Research/DATA/stage/');
>     [ALLEEG, EEG, CURRENTSET] = eeg_store( ALLEEG, EEG, 0 );
>     EEG = eeg_checkset( EEG );
>
>
>
>
>     % re-reference to common average reference, excluding VEOU and VEOL
>     EEG = pop_reref( EEG, [], 'refstate',0, 'exclude',[65 66] )
>
>
>
>
>     % lookup Channel Locations
>     EEG=pop_chanedit(EEG,  'lookup',
>'/Users/andrew/Documents/Research/eeglab2008October01_beta/plugins/dipfi
>t2.2/standard_BESA/standard-10-5-cap385.elp');
>
>
>
>
>     % EXTRACT EPOCHS, for trigs relevant to Cond
>     EEG = pop_selectevent( EEG, 'event',trigs(c,:) , 'deleteepochs',
>'on');
>     [ALLEEG EEG CURRENTSET] = pop_newset(ALLEEG, EEG, 1, 'setname',
>Cond(c,:), 'savenew', [int2str(S),'_AVGREREF_',Cond(c,:)], 'overwrite',
>'on', 'gui', 'off');
>
>
>
>     %baseline filter using pre-stimulus interval.
>     %NOT working ->"pop_rmbase(): Wrong point range"
>     %EEG = pop_rmbase( EEG, [-1000  0]);
>
>
>
>
>     %maybe CLEAR some RAM? NOT working.
>     %ALLEEG = pop_delset( ALLEEG, [1] );
>     %CURRENTSET = pop_delset()
>     end;
>end;
>
>so some strangeness:
>
>1) when i save a new .set named per the S(ubject) and Cond(ition), the
>'deleteepochs', 'on'doesn't seem to work - the file contains ALL the
>epochs, not just 'trigs' ones specified./
>
>
>2) the pop_rmbase syntax seems to be correct, from my eegh, but when put
>in a script it throws "pop_rmbase(): Wrong point range" . and including
>'timerange' doesn't help.
>
>
>3) the biggest problem is this script the second time around the loop,
>e.g. it only produces files for the first condition, first subject.
>
>
>108_AVGREREF_LVF_ConTarg_AllCue.set
>108_AVGREREF_LVF_ConTarg_AllCue.dat
>
>and then fails with out of memory errors:
>??? Error using ==> ctranspose
>Out of memory. Type HELP MEMORY for your options.
>
>i tried ALLEEG = pop_delset( ALLEEG, [1] ); to clear memory before
>running through the Condition loop a 2nd time, but this doesn't work.
>what am i doing wrong? is there any better way to clear the memory
>before the next loop through?
>
>
>thanks,
>andrew
>
>On Jun 18, 2009, at 4:56 PM, Andrew Hill wrote:
>
>
>
>
>         does anyone have a simple example script, to open a .set file
>and
>
>
>         extract a series of epochs by event types into different files?
>
>
>
>
>
>_______________________________________________
>Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>To unsubscribe, send an empty email to eeglablist-unsubscribe at sccn.ucsd.edu
>For digest mode, send an email with the subject 
>"set digest mime" to eeglablist-request at sccn.ucsd.edu

Dr. Søren K. Andersen

Universität Leipzig
Institut Psychologie I
Allgemeine Psychologie & Methodenlehre
Seeburgstr. 14-20
D-04103 Leipzig
Tel.: +49 - 341 - 97 39 54 4
Fax: +49 - 341 - 97 35 96 9

E-Mail: <mailto:andersen at rz.uni-leipzig.de>andersen at rz.uni-leipzig.de
http://www.uni-leipzig.de/~psyall2/andersen/index.html  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20090629/87916405/attachment.html>