[Eeglablist] Filter distortion around boundary events

Wed Jul 18 06:17:06 PDT 2018

> I never use epoched data for filtering, precisely because of the reasons mentioned (though I would have never been able to express them so clearly). And as expected, filtering the original continuous data works fine.
Ah, ok. Indeed we had a misunderstanding then.

> To me, this suggests something like Makoto's first suggestion: eeglab internally retains information about previous event boundaries, and does not really treat the data as continuous.
Not actually tested myself but from help eeg_epoch2continuous:

  eeg_epoch2continuous() - convert epoched dataset to continuous dataset
                           with data epochs separated by boundary events.

Thus, eeg_epoch2continuous separates former epochs by boundary events (even if they were adjacent segments before). The EEGLAB filter will never filter across any boundary event but filter each segment separated by boundary events separately. Therefore, a discontinuity/step at a boundary event after filtering is expected.

You will have to manually or automagically remove the boundary event markers between segments which where adjacent epochs before. Presumably easiest would be writing your own eeg_epoch2continuous function checking whether data were rejected between two epochs and setting boundary events accordingly.

Best,
Andreas

> Am 18.07.2018 um 14:04 schrieb Roy Cox <roycox.roycox at gmail.com>:
> 
> Hi Andreas, Makoto, and others,
> 
> I may have not have explained myself clearly: let me try again :)
> 
> I never use epoched data for filtering, precisely because of the reasons mentioned (though I would have never been able to express them so clearly). And as expected, filtering the original continuous data works fine.
> 
> However, when I epoch data, remove some epochs, transform the data back to continuous format, and filter this (pseudo-continuous) data, I see filter artifacts at timepoints corresponding to my previous epoch boundaries (that no longer exist).
> 
> To be clear, this is to be expected at boundaries where an intervening epoch has been removed, because the unfiltered data already contains jumps when concatenating non-contiguous epochs. But critically, I also see these post-filtering jumps at boundaries where absolutely no data has been removed (and no such jumps exist in the unfiltered data: double and triple checked).
> 
> To me, this suggests something like Makoto's first suggestion: eeglab internally retains information about previous event boundaries, and does not really treat the data as continuous.
> 
> Roy
> 
> 
> 
> On Wed, Jul 18, 2018 at 12:29 PM Andreas Widmann <widmann at uni-leipzig.de> wrote:
> Hi Roy,
> 
> > When I then apply a highpass filter [pop_eegfiltnew(EEG,0.3,0)], I often see subtle but clear jumps right at the edge of where my epoch boundaries used to be. Critically, these jumps were there even if no data had been removed in between. In contrast, just filtering the original continuous data works fine.
> 
> This is the expected behavior of a high-pass filter (a filter effect rather than a filter artifact). Besides removing the global DC potential a high-pass filter will also compensate local DC shifts (rate depending on cutoff and order). If you have a particular feature (a drift, a large unipolar ERP, etc.) in the data close to the edge of an epoch, the filter will compensate for the shift of the DC potential until the end of the epoch but not beyond even if necessary. When filtering the next epoch, the filter will not compensate for the DC shift introduced by the particular feature at the end of the previous epoch (actually temporally close but now invisible to the filter). In general, I would consider this case as rule rather than as exception. In some cases the discontinuity/step might just be so small that it cannot be identified by eye but it is still there.
> 
> I will add some MATLAB code illustrating the problem below. In summary, filtering should be done on the continuous data whenever possible. Data discontinuities should be marked by boundary events when stitching back together data filtered in epochs.
> 
> Best,
> Andreas
> 
> fs = 500;
> sig = zeros( fs * 60, 1 ); % 60 sec
> sig( 14251:14750, 1 ) = exp( -( -250:249 ) .^ 2 / ( 2 * 62.5 ^ 2 ) ); % test signal: Gaussian close to end of first epoch
> 
> plot( ( 0:length( sig ) - 1 ) / fs, sig )
> xlim( [ 25 35 ] )
> hold all
> 
> b = firws( 5500, 0.15 / ( fs / 2 ), 'high', windows('hamming', 5501 ) ); % filter coefficients
> 
> fsig_cnt = filter( b, 1, [ zeros( 2750, 1 ); sig; zeros( 2750, 1 ) ] ); % filter continuous
> fsig_cnt = fsig_cnt( 5501:end, 1); % remove padding
> 
> plot( ( 0:length( fsig_cnt ) - 1 ) / fs, fsig_cnt )
> 
> tmp = filter( b, 1, [ zeros( 2750, 1 ); sig( 1:15000, 1 ); zeros( 2750, 1 ) ] ); % filter first epoch
> fsig_epo( 1:15000, 1 ) = tmp( 5501:end, 1); % remove padding
> tmp = filter( b, 1, [ zeros( 2750, 1 ); sig( 15001:30000, 1 ); zeros( 2750, 1 ) ] ); % filter second epoch
> fsig_epo( 15001:30000, 1 ) = tmp( 5501:end, 1); % remove padding
> 
> plot( ( 0:length( fsig_epo ) - 1 ) / fs, fsig_epo )
> legend( 'raw', 'filtered continuous', 'filtered epoched and stitched' )
> 
> > Am 17.07.2018 um 18:20 schrieb Roy Cox <roycox.roycox at gmail.com>:
> > 
> > hi all,
> > 
> > Just wanted to chime in that I've noticed this (or something similar), too.
> > 
> > I typically epoch continuous sleep data into 30 s "trials" to exclude bad epochs, before stitching everything back together using eeg_epoch2continuous.
> > 
> > When I then apply a highpass filter [pop_eegfiltnew(EEG,0.3,0)], I often see subtle but clear jumps right at the edge of where my epoch boundaries used to be. Critically, these jumps were there even if no data had been removed in between. In contrast, just filtering the original continuous data works fine.
> > 
> > I checked to see if the continuous->epoched and epoched->continuous transformations somehow resulted in skipping a sample, or some other change of amplitude, but amplitude values were identical across boundaries in all cases. It's as if the pop_eegfiltnew function somehow notices when continuous data previously existed in an epoched form, and alters its behavior based on that...
> > 
> > My ugly workaround has been to do all my filtering on continuous data (even if it includes lots that I know will be rejected anyway), but would be good if this issue were remedied.
> > 
> > Roy