[Eeglablist] Filter distortion around boundary events

[Roy Cox]

Hi Andreas, Makoto, and others,

I may have not have explained myself clearly: let me try again :)

I never use epoched data for filtering, precisely because of the reasons
mentioned (though I would have never been able to express them so clearly).
And as expected, filtering the original continuous data works fine.

However, when I epoch data, remove some epochs, transform the data back to
continuous format, and filter this (pseudo-continuous) data, I see filter
artifacts at timepoints corresponding to my previous epoch boundaries (that
no longer exist).

To be clear, this is to be expected at boundaries where an intervening
epoch has been removed, because the unfiltered data already contains jumps
when concatenating non-contiguous epochs. But critically, I also see these
post-filtering jumps at boundaries where absolutely no data has been
removed (and no such jumps exist in the unfiltered data: double and triple
checked).

To me, this suggests something like Makoto's first suggestion: eeglab
internally retains information about previous event boundaries, and does
not really treat the data as continuous.

Roy



On Wed, Jul 18, 2018 at 12:29 PM Andreas Widmann <widmann at uni-leipzig.de>
wrote:

> Hi Roy,
>
> > When I then apply a highpass filter [pop_eegfiltnew(EEG,0.3,0)], I often
> see subtle but clear jumps right at the edge of where my epoch boundaries
> used to be. Critically, these jumps were there even if no data had been
> removed in between. In contrast, just filtering the original continuous
> data works fine.
>
> This is the expected behavior of a high-pass filter (a filter effect
> rather than a filter artifact). Besides removing the global DC potential a
> high-pass filter will also compensate local DC shifts (rate depending on
> cutoff and order). If you have a particular feature (a drift, a large
> unipolar ERP, etc.) in the data close to the edge of an epoch, the filter
> will compensate for the shift of the DC potential until the end of the
> epoch but not beyond even if necessary. When filtering the next epoch, the
> filter will not compensate for the DC shift introduced by the particular
> feature at the end of the previous epoch (actually temporally close but now
> invisible to the filter). In general, I would consider this case as rule
> rather than as exception. In some cases the discontinuity/step might just
> be so small that it cannot be identified by eye but it is still there.
>
> I will add some MATLAB code illustrating the problem below. In summary,
> filtering should be done on the continuous data whenever possible. Data
> discontinuities should be marked by boundary events when stitching back
> together data filtered in epochs.
>
> Best,
> Andreas
>
> fs = 500;
> sig = zeros( fs * 60, 1 ); % 60 sec
> sig( 14251:14750, 1 ) = exp( -( -250:249 ) .^ 2 / ( 2 * 62.5 ^ 2 ) ); %
> test signal: Gaussian close to end of first epoch
>
> plot( ( 0:length( sig ) - 1 ) / fs, sig )
> xlim( [ 25 35 ] )
> hold all
>
> b = firws( 5500, 0.15 / ( fs / 2 ), 'high', windows('hamming', 5501 ) ); %
> filter coefficients
>
> fsig_cnt = filter( b, 1, [ zeros( 2750, 1 ); sig; zeros( 2750, 1 ) ] ); %
> filter continuous
> fsig_cnt = fsig_cnt( 5501:end, 1); % remove padding
>
> plot( ( 0:length( fsig_cnt ) - 1 ) / fs, fsig_cnt )
>
> tmp = filter( b, 1, [ zeros( 2750, 1 ); sig( 1:15000, 1 ); zeros( 2750, 1
> ) ] ); % filter first epoch
> fsig_epo( 1:15000, 1 ) = tmp( 5501:end, 1); % remove padding
> tmp = filter( b, 1, [ zeros( 2750, 1 ); sig( 15001:30000, 1 ); zeros(
> 2750, 1 ) ] ); % filter second epoch
> fsig_epo( 15001:30000, 1 ) = tmp( 5501:end, 1); % remove padding
>
> plot( ( 0:length( fsig_epo ) - 1 ) / fs, fsig_epo )
> legend( 'raw', 'filtered continuous', 'filtered epoched and stitched' )
>
> > Am 17.07.2018 um 18:20 schrieb Roy Cox <roycox.roycox at gmail.com>:
> >
> > hi all,
> >
> > Just wanted to chime in that I've noticed this (or something similar),
> too.
> >
> > I typically epoch continuous sleep data into 30 s "trials" to exclude
> bad epochs, before stitching everything back together using
> eeg_epoch2continuous.
> >
> > When I then apply a highpass filter [pop_eegfiltnew(EEG,0.3,0)], I often
> see subtle but clear jumps right at the edge of where my epoch boundaries
> used to be. Critically, these jumps were there even if no data had been
> removed in between. In contrast, just filtering the original continuous
> data works fine.
> >
> > I checked to see if the continuous->epoched and epoched->continuous
> transformations somehow resulted in skipping a sample, or some other change
> of amplitude, but amplitude values were identical across boundaries in all
> cases. It's as if the pop_eegfiltnew function somehow notices when
> continuous data previously existed in an epoched form, and alters its
> behavior based on that...
> >
> > My ugly workaround has been to do all my filtering on continuous data
> (even if it includes lots that I know will be rejected anyway), but would
> be good if this issue were remedied.
> >
> > Roy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20180718/72550f0a/attachment.html>