<div dir="ltr"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">I think you should be very careful handling NaNs:</span>
<div class="gmail_quote"><div link="blue" vlink="purple" lang="NO-BOK"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Substitution by zero gives that time point a value that e.g. will add frequency content to the signal. Would you think about substituting any other
value with zero?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">You may remove the NaNs, but then you end up with a varying sampling interval and have to use math feasible for that (and the index as a time measure).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Pål<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d">Pål G. Larsson M.D., PhD.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Head of Clinical Neurophysiology<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Department of Neurosurgery<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Division of Surgery and Clinical Neuroscience<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Oslo University Hospital<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Po.box 4950 Nydalen<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">0424 Oslo<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Norway<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d">Tel: <a href="tel:%28%2B47%29%2023074407" value="+4723074407" target="_blank">(+47) 23074407</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d">Mobile: <a href="tel:%28%2B47%29%2093429791" value="+4793429791" target="_blank">(+47) 93429791</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d">E-mail:
<a href="mailto:pall@ous-hf.no" target="_blank">pall@ous-hf.no</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d">not sensitive<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Fra:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:eeglablist-bounces@sccn.ucsd.edu" target="_blank">eeglablist-bounces@sccn.ucsd.edu</a> [mailto:<a href="mailto:eeglablist-bounces@sccn.ucsd.edu" target="_blank">eeglablist-bounces@sccn.ucsd.edu</a>]
<b>På vegne av</b> leo budinich<br>
<b>Sendt:</b> 13. mai 2016 14:33<br>
<b>Til:</b> <a href="mailto:eeglablist@sccn.ucsd.edu" target="_blank">eeglablist@sccn.ucsd.edu</a><br>
<b>Emne:</b> [Eeglablist] Fwd: How to high-pass filter eeg data containing NaNs?<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">---------- Forwarded message ----------<br>
From: <b>leo budinich</b> <<a href="mailto:leo.budinich@edu.unito.it" target="_blank">leo.budinich@edu.unito.it</a>><br>
Date: 2016-05-13 10:56 GMT+02:00<br>
Subject: Re: [Eeglablist] How to high-pass filter eeg data containing NaNs?<br>
To: Andreas Widmann <<a href="mailto:widmann@uni-leipzig.de" target="_blank">widmann@uni-leipzig.de</a>><br>
<br>
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Thanks Andreas! Your answer has been really helpful! <br>
Now I understand a little better how filtering works...<u></u><u></u></p>
</div>
<p class="MsoNormal">I'll make a try with cutting away the NaN parts and then applying the high-pass filter, keeping in mind that it's highly possible to incur in edge-effects.
<br>
I think also that I am going make a try with a more traditional approach to baseline correction (subtracting the mean amplitude of the 100ms preceding the target stimulus).<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
I would like also to try what Carolina proposed (transforming NaNs in zeros and then filtering the set): do you think it could work?<u></u><u></u></p>
</div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<p class="MsoNormal">Leo <u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2016-05-12 16:25 GMT+02:00 Andreas Widmann <<a href="mailto:widmann@uni-leipzig.de" target="_blank">widmann@uni-leipzig.de</a>>:<u></u><u></u></p>
<p class="MsoNormal">Hi Leo,<br>
<br>
filtering is the convolution of two signals-the data and the filter’s impulse response. The impulse response has a duration, 1651 samples or 6.6 seconds in your case. If any sample of both signals is NaN the output is also NaN. That’s why your filter output
mainly consists of NaNs. Unfortunately, there’s is not so much one could do. Thus, I would consider the decision NaNing large parts of the continuous data as, hm, suboptimal.<br>
<br>
Pragmatically, the only thing you could do is cutting away the NaN parts (the data are effectively kind of „epoched“ already anyway).<br>
<br>
tmp = any( isnan( EEG.data ), 1 );<br>
onsetArray = find( diff( [ 0 tmp ] ) == 1 );<br>
offsetArray = find( diff( [ tmp 0 ] ) == -1 );<br>
EEG = eeg_eegrej( EEG, [ onsetArray; offsetArray ]' );<br>
[ ALLEEG, EEG, CURRENTSET ] = pop_newset( ALLEEG, EEG, CURRENTSET );<br>
eeglab redraw<br>
<br>
Now, at least there is signal in the filter output where there was signal in the filter input.<br>
<br>
I would, however, recommend being careful with the further analysis of the data and interpretation of the results. High-pass filtering should always be done on the continuous data. You have to expect edge-effects up to the duration of the impulse response at
the beginning and end of each epoch (actually all data regions which were signal in the filter input and NaN in the filter output before; essentially the observed NaNs reflect some kind of edge-effect themselves). Type, size, and duration of the edge-effects
depend on impulse response length and whether and how data are padded during filtering but there is nothing you could do against edge-effects in general. Given the minimal required durations of the impulse responses for the filter cutoffs suggested in the
paper I would consider the final sentence of footnote 3 as incorrect. This does also not depend on filter type (unfortunately not described in detail in the paper).<br>
<br>
Finally, you have to be careful with high-pass filtering with higher cutoffs in ERP analysis in general. Recent relevant literature on this issue you may find at the end of this page:<br>
<a href="http://sccn.ucsd.edu/wiki/Firfilt_FAQ" target="_blank">http://sccn.ucsd.edu/wiki/Firfilt_FAQ</a><br>
<br>
Hope this helps!<br>
Andreas<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
> Am 11.05.2016 um 15:38 schrieb leo budinich <<a href="mailto:leo.budinich@edu.unito.it" target="_blank">leo.budinich@edu.unito.it</a>>:<br>
><br>
> Hi Eeglablist,<br>
><br>
> I'm writing here because I'm a new user of EEGLAB and I am facing a problem with high-pass filtering some data.<br>
><br>
> I am trying to apply on a freely downloadable dataset some of the same processing steps that have been used by the researchers that provided the data (see
<a href="http://www.stefanfrank.info/pubs/BL2015.pdf" target="_blank">http://www.stefanfrank.info/pubs/BL2015.pdf</a>).<br>
> The set I want to filter has 32 channels and has already been been band-pass filtered at 0.05Hz - 25Hz, recalibrated and re-referenced to the mastoids.<br>
><br>
> An important detail is that the EEG recordings concern the reading of two hundred english sentences; the data recorded between the presentations of two sentences have been "set to NaN" by the researchers who provided the dataset, so the channel data of, e.g.
subject01, presents this aspect when you inspect it:<br>
><br>
> <-------------sentence_1-------------><---------wait----------><---------sentence_2---------><br>
><br>
> signal-signal-signal-signal-signal-NaN-NaN-NaN-NaN-signal-signal-signal-signal<br>
><br>
><br>
> What I would like to do is high-pass filter the entire set to 0.50Hz, as a way to "mitigate the baseline problem by reducing the correlation between the baselines and amplitudes by applying an additional high-pass filter with a sufficiently high cut-off frequency"
(see the reference, pp.4).<br>
><br>
> Unfortunately, when I apply a FIR filter with a 0.50Hz lower-edge I obtain a lot of NaNs on the areas that previously were 'signal', but not everywhere (and, curiously, the processing is really fast), so that it gets this kind of aspect:<br>
><br>
><br>
> <-------------sentence_1-------------><---------wait----------><---------sentence_2---------><br>
><br>
> NaN-NaN-NaN-NaN-NaN-NaN-NaN-NaN-NaN-NaN-NaN-NaN-NaN-signal-NaN-NaN<br>
><br>
><br>
> Here's the eeglab output:<br>
><br>
> pop_eegfiltnew() - performing 1651 point highpass filtering.<br>
> pop_eegfiltnew() - transition band width: 0.5 Hz<br>
> pop_eegfiltnew() - passband edge(s): 0.5 Hz<br>
> pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.25 Hz<br>
> pop_eegfiltnew() - filtering the data (zero-phase)<br>
> firfilt(): |====================| 100%, ETE 00:00<br>
> Done.<br>
><br>
><br>
> I imagined that the filter function doesn't produce the right output because of the NaNs present between the sentences, but as my comprehension of the functioning of filters and of eeglab in general is extremely limited at the moment, it turns out to be just
a speculation.<br>
><br>
> Could you help me understanding what's wrong with my filtering?<br>
><br>
> And, if my hypothesis is correct, i.e. the filter cannot be applied to data containing NaNs, how would you apply a high-pass filter to data structured as I indicated above (i.e., containing NaNs in some parts)?<br>
><br>
> Thank you!<br>
> Leo<br>
><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">> ------------------------<br>
><br>
><br>
> Indirizzo istituzionale di posta elettronica degli studenti e dei laureati dell'Università degli Studi di Torino<br>
> Official University of Turin email address for students and graduates<br>
> _______________________________________________<br>
> Eeglablist page: <a href="http://sccn.ucsd.edu/eeglab/eeglabmail.html" target="_blank">
http://sccn.ucsd.edu/eeglab/eeglabmail.html</a><br>
> To unsubscribe, send an empty email to <a href="mailto:eeglablist-unsubscribe@sccn.ucsd.edu" target="_blank">
eeglablist-unsubscribe@sccn.ucsd.edu</a><br>
> For digest mode, send an email with the subject "set digest mime" to <a href="mailto:eeglablist-request@sccn.ucsd.edu" target="_blank">
eeglablist-request@sccn.ucsd.edu</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div><div>
<p class="MsoNormal"><span style="font-size:15.5pt">------------------------<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="border:solid windowtext 1.0pt;padding:0cm"><img src="cid:~WRD000.jpg" alt="Bilde er fjernet av sender." border="0" height="100" width="100"></span><u></u><u></u></p>
</div><span class="">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Indirizzo istituzionale di posta elettronica degli studenti e dei laureati dell'Università degli Studi di Torino<u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">Official University of Turin email address for students and graduates </span><u></u><u></u></p>
</div>
</span></div>
</div>
</div><br></div>
<br>
<div style="font-size:1.3em">------------------------</div><span style="font-size:small"><div><img src="http://www.mail.unito.it/img/logounito.png"></div><div><span style="font-size:small"><br></span></div>Indirizzo istituzionale di posta elettronica degli studenti e dei laureati dell'Università degli Studi di Torino</span><div><font size="2">Official University of Turin email address for students and graduates </font></div>