[Eeglablist] Filter distortion around boundary events

[Roy Cox]

hi Andreas,

ah, that has to be it... I guess it makes sense for filters to not cross
boundary events, but I never realized this was the case, or that
eeg_epoch2continuous
could trigger this behavior.

Thanks for shining a light on this, much appreciated.

Roy


On Wed, Jul 18, 2018 at 3:17 PM Andreas Widmann <widmann at uni-leipzig.de>
wrote:

> > I never use epoched data for filtering, precisely because of the reasons
> mentioned (though I would have never been able to express them so clearly).
> And as expected, filtering the original continuous data works fine.
> Ah, ok. Indeed we had a misunderstanding then.
>
> > To me, this suggests something like Makoto's first suggestion: eeglab
> internally retains information about previous event boundaries, and does
> not really treat the data as continuous.
> Not actually tested myself but from help eeg_epoch2continuous:
>
>   eeg_epoch2continuous() - convert epoched dataset to continuous dataset
>                            with data epochs separated by boundary events.
>
> Thus, eeg_epoch2continuous separates former epochs by boundary events
> (even if they were adjacent segments before). The EEGLAB filter will never
> filter across any boundary event but filter each segment separated by
> boundary events separately. Therefore, a discontinuity/step at a boundary
> event after filtering is expected.
>
> You will have to manually or automagically remove the boundary event
> markers between segments which where adjacent epochs before. Presumably
> easiest would be writing your own eeg_epoch2continuous function checking
> whether data were rejected between two epochs and setting boundary events
> accordingly.
>
> Best,
> Andreas
>
> > Am 18.07.2018 um 14:04 schrieb Roy Cox <roycox.roycox at gmail.com>:
> >
> > Hi Andreas, Makoto, and others,
> >
> > I may have not have explained myself clearly: let me try again :)
> >
> > I never use epoched data for filtering, precisely because of the reasons
> mentioned (though I would have never been able to express them so clearly).
> And as expected, filtering the original continuous data works fine.
> >
> > However, when I epoch data, remove some epochs, transform the data back
> to continuous format, and filter this (pseudo-continuous) data, I see
> filter artifacts at timepoints corresponding to my previous epoch
> boundaries (that no longer exist).
> >
> > To be clear, this is to be expected at boundaries where an intervening
> epoch has been removed, because the unfiltered data already contains jumps
> when concatenating non-contiguous epochs. But critically, I also see these
> post-filtering jumps at boundaries where absolutely no data has been
> removed (and no such jumps exist in the unfiltered data: double and triple
> checked).
> >
> > To me, this suggests something like Makoto's first suggestion: eeglab
> internally retains information about previous event boundaries, and does
> not really treat the data as continuous.
> >
> > Roy
> >
> >
> >
> > On Wed, Jul 18, 2018 at 12:29 PM Andreas Widmann <widmann at uni-leipzig.de>
> wrote:
> > Hi Roy,
> >
> > > When I then apply a highpass filter [pop_eegfiltnew(EEG,0.3,0)], I
> often see subtle but clear jumps right at the edge of where my epoch
> boundaries used to be. Critically, these jumps were there even if no data
> had been removed in between. In contrast, just filtering the original
> continuous data works fine.
> >
> > This is the expected behavior of a high-pass filter (a filter effect
> rather than a filter artifact). Besides removing the global DC potential a
> high-pass filter will also compensate local DC shifts (rate depending on
> cutoff and order). If you have a particular feature (a drift, a large
> unipolar ERP, etc.) in the data close to the edge of an epoch, the filter
> will compensate for the shift of the DC potential until the end of the
> epoch but not beyond even if necessary. When filtering the next epoch, the
> filter will not compensate for the DC shift introduced by the particular
> feature at the end of the previous epoch (actually temporally close but now
> invisible to the filter). In general, I would consider this case as rule
> rather than as exception. In some cases the discontinuity/step might just
> be so small that it cannot be identified by eye but it is still there.
> >
> > I will add some MATLAB code illustrating the problem below. In summary,
> filtering should be done on the continuous data whenever possible. Data
> discontinuities should be marked by boundary events when stitching back
> together data filtered in epochs.
> >
> > Best,
> > Andreas
> >
> > fs = 500;
> > sig = zeros( fs * 60, 1 ); % 60 sec
> > sig( 14251:14750, 1 ) = exp( -( -250:249 ) .^ 2 / ( 2 * 62.5 ^ 2 ) ); %
> test signal: Gaussian close to end of first epoch
> >
> > plot( ( 0:length( sig ) - 1 ) / fs, sig )
> > xlim( [ 25 35 ] )
> > hold all
> >
> > b = firws( 5500, 0.15 / ( fs / 2 ), 'high', windows('hamming', 5501 ) );
> % filter coefficients
> >
> > fsig_cnt = filter( b, 1, [ zeros( 2750, 1 ); sig; zeros( 2750, 1 ) ] );
> % filter continuous
> > fsig_cnt = fsig_cnt( 5501:end, 1); % remove padding
> >
> > plot( ( 0:length( fsig_cnt ) - 1 ) / fs, fsig_cnt )
> >
> > tmp = filter( b, 1, [ zeros( 2750, 1 ); sig( 1:15000, 1 ); zeros( 2750,
> 1 ) ] ); % filter first epoch
> > fsig_epo( 1:15000, 1 ) = tmp( 5501:end, 1); % remove padding
> > tmp = filter( b, 1, [ zeros( 2750, 1 ); sig( 15001:30000, 1 ); zeros(
> 2750, 1 ) ] ); % filter second epoch
> > fsig_epo( 15001:30000, 1 ) = tmp( 5501:end, 1); % remove padding
> >
> > plot( ( 0:length( fsig_epo ) - 1 ) / fs, fsig_epo )
> > legend( 'raw', 'filtered continuous', 'filtered epoched and stitched' )
> >
> > > Am 17.07.2018 um 18:20 schrieb Roy Cox <roycox.roycox at gmail.com>:
> > >
> > > hi all,
> > >
> > > Just wanted to chime in that I've noticed this (or something similar),
> too.
> > >
> > > I typically epoch continuous sleep data into 30 s "trials" to exclude
> bad epochs, before stitching everything back together using
> eeg_epoch2continuous.
> > >
> > > When I then apply a highpass filter [pop_eegfiltnew(EEG,0.3,0)], I
> often see subtle but clear jumps right at the edge of where my epoch
> boundaries used to be. Critically, these jumps were there even if no data
> had been removed in between. In contrast, just filtering the original
> continuous data works fine.
> > >
> > > I checked to see if the continuous->epoched and epoched->continuous
> transformations somehow resulted in skipping a sample, or some other change
> of amplitude, but amplitude values were identical across boundaries in all
> cases. It's as if the pop_eegfiltnew function somehow notices when
> continuous data previously existed in an epoched form, and alters its
> behavior based on that...
> > >
> > > My ugly workaround has been to do all my filtering on continuous data
> (even if it includes lots that I know will be rejected anyway), but would
> be good if this issue were remedied.
> > >
> > > Roy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20180718/74d809ae/attachment.html>