[Eeglablist] 0.01 Hz highpass filter isn't running

[Makoto Miyakoshi]

Hi Andreas, Metin, and Lei,

Andreas, you might have missed to include Metin in your previous reply. I
included him here to save your time.

I confirmed that you edited the github document firfilt_FAQ. Thank you for
taking care of it.
https://urldefense.com/v3/__https://github.com/sccn/sccn.github.io/commits/27b93f5cc729a0267f49c28baff2c7ffa889e935/others/Firfilt_FAQ.md__;!!Mih3wA!BYhUnJrikvVBWNQeJJqTkajkI8LADqN4N_ULv1d8sbeCb5mn8qQdBTloPSk5XVjT_TaiwVARdyY30r6FQhmw6Q_WAKA$ 

Arno wrote in his reply that it is pretty common to find EEG recorders that
have 0.01Hz or around high-pass filter. I agree with him. I mean, unless an
amp explicitly says 'DC recording capable', a very moderate hardware
high-pass filter (which is by definition IIR) is applied. I think designing
a 0.01-Hz FIR high-pass filter is practically a challenge due to its
required filter order (> 5 min long). But if your data is 24 hours
recording or something, it's not an issue. After all, it's a matter of
relative time scale differences between your filter and your data.

For 2-s epoch ERP analysis, 0.01-Hz FIR seems unnecessary. Epoch-wise
linear trend removal sounds more reasonable to me in terms of signal
processing.

Makoto


On Mon, Feb 12, 2024 at 11:11 AM Andreas Widmann via eeglablist <
eeglablist at sccn.ucsd.edu> wrote:

> Hi Metin,
>
> > To answer @Andreas' question: I read it here (
> https://urldefense.com/v3/__https://eeglab.org/others/Firfilt_FAQ*q-what-is-the-difference-between-the-basic-fir-filter-legacy-and-the-basic-fir-filter-new-which-should-i-use__;Iw!!Mih3wA!CzyMfeMfP6W1qbLLr_RRn0yns2BG0AHT5nKXYeh2pW_cBv5714VCF5eAu7669RejMhkVVc4o-x32tgDFqEPIBPIXL7Lk$
> ; under the question "Q. What is the lower limit for cutoff frequency for
> high-pass filter?") that "a lower limit of about 0.1 Hz is recommendable
> for FIR filters.
> The first parts of the FAQ were written by Makoto and me about 12 years
> ago. So it is most likely me to blame. Thanks for reporting this!
>
> I have asked to change in the Firfilt FAQ from "To our experience […]" to
> "The computation time for very long FIR filters becomes enormous. You can
> try the 'usefftfilt' option in "Basic FIR Filter (new, default)"
> (pop_eegfiltnew) or "Windowed sinc FIR Filter" (pop_firws) to speed up
> filtering for FIR filter orders higher than 2000 to 3000. Note that
> lowering the sampling rate also reduces the required filter orders. If you
> are experiencing stability problems with IIR filters with lower cutoff
> frequencies, also consider reducing the sampling frequency of the signal."
>
> I have also added a GUI checkbox for the usefftfilt option in the "Basic
> FIR Filter (new, default)" (pop_eegfiltnew.m) in the latest firfilt version
> 2.8.
>
> Both changes may take some time to show up in EEGLAB.
>
> Hope this helps! Best,
> Andreas
>
> > Am 12.02.2024 um 08:57 schrieb Metin Özyağcılar <
> metinozyagcilar at gmail.com>:
> >
> > Dear all,
> >
> > thank you all so very much for all the valuable input and introducing me
> other ways of applying these filters (e.g. ERPLAB, usefftfilt  ... ) that I
> was not aware of.
> >
> > I have also tested myself (just to test the speed) and they run indeed
> very fast, as compared to the pop_eegfiltnew with its default parameters.
> >
> > To answer @Andreas' question: I read it here (
> https://urldefense.com/v3/__https://eeglab.org/others/Firfilt_FAQ*q-what-is-the-difference-between-the-basic-fir-filter-legacy-and-the-basic-fir-filter-new-which-should-i-use__;Iw!!Mih3wA!CzyMfeMfP6W1qbLLr_RRn0yns2BG0AHT5nKXYeh2pW_cBv5714VCF5eAu7669RejMhkVVc4o-x32tgDFqEPIBPIXL7Lk$
> ; under the question "Q. What is the lower limit for cutoff frequency for
> high-pass filter?") that "a lower limit of about 0.1 Hz is recommendable
> for FIR filters. For lower cutoff frequencies consider IIR filters combined
> with a reduced sampling frequency of the signal." But, on that page, the
> stability issue of IIR is also mentioned somewhere.
> >
> > Thanks!
> >
> > All the best,
> > Metin
> >
> > On Thu, Feb 8, 2024 at 1:58 PM Andreas Widmann <widmann at uni-leipzig.de>
> wrote:
> > Dear all,
> >
> > To make sure that no problem has been introduced in recent versions of
> firfilt, MATLAB, or EEGLAB, I double-checked. Filtering an EEG file with
> >
> > * 64 channels
> > * 1000 Hz sampling rate and
> > * 3.600.000 samples (1h recording time)
> >
> > filtered with
> >
> > [EEG, com, b] = pop_eegfiltnew(EEG, 'locutoff', 0.01, 'usefftfilt', 1,
> 'plotfreqz', 0);
> >
> > takes 18 seconds on a 5-year-old Ryzen desktop PC (and 10 seconds on an
> entry-level M2 MacBook).
> >
> > There is actually a small problem with plotting the frequency response.
> The default FFT length for plotting is too short. Evaluate manually with a
> longer FFT instead:
> > freqz(b, 1, 2^16, EEG.srate)
> >
> > > Am 07.02.2024 um 22:18 schrieb Lena Marreel via eeglablist <
> eeglablist at sccn.ucsd.edu>:
> > > EEG = pop_basicfilter( EEG, 1:64, 'Filter', 'highpass', 'Design',
> 'butter',
> > > 'Cutoff', 0.01, 'Order', 4, 'RemoveDC', 'on' );
> > Took 9 seconds. BUT ERPLAB cheated a little on you. It did not compute
> an order 4 Buttwerworth filter but filtered the data twice (forward and
> backward) with an order 2 Butterworth filter giving different results. An
> order 4 0.01 Hz Butterworth filter would not be stable with double
> precision. Lack of stability is a persistent problem of more extrem IIR
> filters.
> >
> > EEG.srate = 1000;
> > [b, a] = butter(4, 0.01 / (EEG.srate / 2), 'high');
> > isstable(b,a)
> >
> > Best,
> > Andreas
> >
> > > Am 08.02.2024 um 00:52 schrieb Cedric Cannard via eeglablist <
> eeglablist at sccn.ucsd.edu>:
> > >
> > > Hi,
> > >
> > > I'll let Andreas provide a real answer to this.
> > >
> > > To spare him time, this is described in his paper(
> https://urldefense.com/v3/__https://ul.qucosa.de/api/qucosa*3A32715/attachment/ATT-0/__;JQ!!Mih3wA!EweSRN-X3Cvkq_vmhzvZy78NtIjVMTFyJJJbWzFtYZ5gFxA_sAAGBpI_Rrj1aEQ0wniFd4UuYTHCIseNTnVYXs-Lrg$
> ): "the passband edges are to be specified rather than the cutoff
> frequencies. If filter order is not specified, a default for filter order
> is implemented, providing a transition bandwidth of 25% of the lower
> passband edge but, where possible, not lower than 2 Hz and otherwise the
> distance from the passband edge to the critical frequency (DC, Nyquist)."
> > >
> > > As he reminded me recently, pop_eegfiltnew() doesn't take the center
> of the transition bandwidth as cutoff like it is normal done, simply to
> maintain backward compatibility with old EEGLAB versions. However, rest
> assured that this is taken care of by pop_eegfiltnew().
> > >
> > > Regarding speed, I guess that it is working in the background but the
> filter order is so high that it takes forever (depending on your computer
> capacities). To address this, you could:
> > > - downsample your data if the sample rate is unnecessarily high
> > > - use the FFT version of the algorithm (much much faster for high
> filter orders)
> > > - with such a low cutoff, you could almost simply apply a baseline
> removal (subtract the signal mean), which is instantaneous and doesn't
> introduce potential filter artifacts.
> > >
> > > You can design your own FIR filter parameters (cutoff, transition
> bandwidth, order, etc.) in EEGLAB using Andreas Widmann's plugin:
> > > Tools > Filter data > Windowed sinc FIR filter
> > >
> > > If want to work with command line, after having done this manually in
> the GUi, you can type 'eegh' in command window to get the commands.
> > >
> > > And my understanding is that IIR filters are not recommended for the
> required orders as they will introduce massive, undesired ripples.
> > >
> > > Hope this helps,
> > >
> > > Cedric Cannard
> > >
> > >
> > >
> > > On Wednesday, February 7th, 2024 at 1:18 PM, Lena Marreel via
> eeglablist <eeglablist at sccn.ucsd.edu> wrote:
> > >
> > >> Hi Metin,
> > >>
> > >> I recently went down a similar route on highpass filters.
> > >> Some practical points I found out, EEGlab FIR filters (the new
> default)
> > >> don't offer a filter order high enough to be able to handle those low
> > >> frequency cut offs. An alternative is to use IIR butterworth filters
> > >> instead.
> > >> I tend to just report the filter order, which informs about the role
> off of
> > >> your filter in the frequency domain. I recommend ERPlab (
> > >>
> https://urldefense.com/v3/__https://erpinfo.org/erplab__;!!Mih3wA!AVaLZlRIT0nEkiq_O4CriQT7S1krYWhGT8mL_WWktZN5mdeaVnC6QAePhj76drsQnSyZZUJh5uZxGVdyYyLvt_ExzQ0$
> ) as a useful tool to play around and see how
> > >> different parameters and filters interact with your data. The ERPlab
> > >> interface has some useful plots to show you this. ERPlab becomes
> > >> an extension of your EEGlab GUI and has functions that build on EEGlab
> > >> functions.
> > >> I don't know if pop_eegfiltnew() offers an IIR setting, but here is an
> > >> alternative ERPlab filter I use
> > >> EEG = pop_basicfilter( EEG, 1:64, 'Filter', 'highpass', 'Design',
> 'butter',
> > >> 'Cutoff', 0.01, 'Order', 4, 'RemoveDC', 'on' );
> > >>
> > >> I too noticed the filters running slow. Though, admitingly, we (lab
> > >> colleagues and I) consider it something we cannot fix ourselves and
> just
> > >> accepted it as 'the way things are'. The best cause we could guess at
> was
> > >> related to how EEGlab calls in the function over and over taking a
> lot of
> > >> memory.
> > >>
> > >> Hope this helps!
> > >>
> > >> Best,
> > >> Lena
> > >>
> > >> On Wed, Feb 7, 2024 at 8:26 PM Metin Özyağcılar via eeglablist <
> > >> eeglablist at sccn.ucsd.edu> wrote:
> > >>
> > >>> Dear EEGLAB community,
> > >>>
> > >>> I have recently tried to change the settings for my highpass filter
> that I
> > >>> used to pre-process my data. I was using a 0.05Hz highpass filter,
> now
> > >>> trying out 0.01 Hz, and I am using the function pop_eegfiltnew().
> However,
> > >>> when I tried to run the function, even after 10-20minutes I see no
> > >>> progress.
> > >>>
> > >>> After doing some research and reading the manuals online, I’ve seen a
> > >>> recommendation that for FIR filters a lower limit of about 0.1 is
> > >>> recommendable and limits lower than that are not. I’ve additionally
> learned
> > >>> that the lower and higher limits used in this function do not
> directly
> > >>> correspond to the “cut-off” (e.g using 0.05, the “cut-off frequency”
> is set
> > >>> to 0.025). All of these led me to have some questions:
> > >>>
> > >>> 1) Is there any way to still run 0.01 highpass filter with
> pop_eegfiltnew()
> > >>> and if so how?
> > >>> 2) If I chose to run a 0.05 Hz highpass filter instead, would it be
> > >>> appropriate to still set the lower limit as 0.05 but also report the
> > >>> outcome of the function when writing the paper (i.e. transition band
> width:
> > >>> 0.05 Hz / passband edge(s): 0.05 Hz / cutoff frequency(ies) (-6 dB):
> 0.025
> > >>> Hz / filtering the data; zero-phase, non-causal)? I feel not so
> confident
> > >>> playing around with the parameters and the default settings of the
> > >>> function.
> > >>> 3) On an unrelated note: the filters are run very slow, even the
> 0.05Hz.
> > >>> Does anyone else maybe have the same issue and if so how are you
> dealing
> > >>> with it? I updated MATLAB, EEGLAB, even my operating system to the
> latest
> > >>> version + freed up my storge but nothing has changed.
> > >>>
> > >>> Any feedback would be highly appreciated. Thank you!
> > >>>
> > >>> Best,
> > >>> Metin
> > >>> _______________________________________________
> > >>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > >>> To unsubscribe, send an empty email to
> > >>> eeglablist-unsubscribe at sccn.ucsd.edu
> > >>> For digest mode, send an email with the subject "set digest mime" to
> > >>> eeglablist-request at sccn.ucsd.edu
> > >>
> > >> _______________________________________________
> > >> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > >> To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> > >> For digest mode, send an email with the subject "set digest mime" to
> eeglablist-request at sccn.ucsd.edu
> > > _______________________________________________
> > > Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> > > To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> > > For digest mode, send an email with the subject "set digest mime" to
> eeglablist-request at sccn.ucsd.edu
> >
> >
>
> _______________________________________________
> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
> To unsubscribe, send an empty email to
> eeglablist-unsubscribe at sccn.ucsd.edu
> For digest mode, send an email with the subject "set digest mime" to
> eeglablist-request at sccn.ucsd.edu