[Eeglablist] Fwd: SIFT toolbox

[Matt Gerhold]

Thanks Makoto for you’re keen powers of observation!

My primary reference has been Helmut Lütkepohl’s text, New Introduction to
Multiple Time Series Analysis
<https://www.springer.com/gp/book/9783540401728>, which I have found to be
excellent in terms of understanding the structure of the models.

As Makoto points out, there are metrics that bring into consideration the
window length, model order and number of trials. Too little data, and one
is really just getting a biased model that doesn’t necessarily represent
the cortical process, but eccentricities in the data. So, it’s important to
use this to validate the preprocessing stage.

The bug Makoto points out is deceptive and can lead one down the garden
path, if one is not rooted in the theory behind VAR models, that’s why I
would suggest having the above text on hand. It’s really well-written and
well-presented.

Determining model order is normally an information criterion-based
approach: Akaike and Swarts/Bayesian are popular; however, the former has a
tendency to choose too high a model order, leading to an over-parametrised
model. These criteria all will give you varying results, so one has to
iterate through a number of models, varying the parameters and other
preprocessing choices and then choose the best model, via an objective
function(s).

Rgds,

Matt

On Fri, Jul 12, 2019 at 2:01 AM Makoto Miyakoshi <mmiyakoshi at ucsd.edu>
wrote:

> Dear Akiterini and Matt,
>
> I found an error in SIFT model order validation part. The SIFT function
> and the SIFT manual uses [number of channels/component]^2 in the numerator,
> but according to the original references it is NOT square. I mentioned it
> in my wiki page. See step 18. I have already talked to Tim, and he said he
> would do something for this soon.
>
> https://sccn.ucsd.edu/wiki/Makoto's_preprocessing_pipeline#Dependency_across_the_preprocessing_stages_.2807.2F05.2F2019_updated.29
>
>
> Makoto
>
> On Mon, Jul 8, 2019 at 6:59 AM AIKATERINI LYMPERIDOU <
> med1p1040133 at med.uoc.gr> wrote:
>
>> Thank you very much for your time and your answer!
>> That was really helpful for me!
>>
>>
>> Quoting Matt Gerhold <matt.gerhold at gmail.com>:
>>
>> > If the residuals aren't white, that means not all the information in the
>> > timeseries has been absorbed into the coefficient matrix.
>> >
>> >
>> >
>> > In terms of the math, if you look at the equations for the VAR model,
>> the
>> > modelling procedure should extract the coefficient matrices upto a given
>> > lag/model order, added to this is a noise component. The noise
>> component is
>> > by definition, gaussian and white: all the observations are independent
>> > from each other, across time and across channels. So, if you have
>> > successfully modelled the data, then the residuals should be white. If
>> you
>> > have one or two models across time that don't fit this, i.e. 96.5% of
>> > models are white, then that is alright as well.
>> >
>> >
>> >
>> > So, you have to go back to the pre-processing stage and tend to your
>> > datasets to ensure you get all the information out of the data and into
>> the
>> > model coefficients. It all has to do with careful pre-processing. They
>> are
>> > not the easiest models to fit, so you have to proceed with a certain
>> level
>> > of determination.
>> >
>> >
>> > On Mon, Jul 8, 2019 at 1:52 PM AIKATERINI LYMPERIDOU <
>> > med1p1040133 at med.uoc.gr> wrote:
>> >
>> >> Thank you for the answer Matthew.
>> >>
>> >> The thing is that my model pass the "percent consistency test" and the
>> >> "stability index test" but did not pass the "Residual whiteness test".
>> >> Is that an important issue as well?
>> >>
>> >> Quoting Matt Gerhold <matt.gerhold at gmail.com>:
>> >>
>> >> > Aikaterini:
>> >> >
>> >> > You do need to validate your model(s). Reviewers and examiners will
>> >> request
>> >> > validation statistics. This should include:
>> >> >
>> >> >    - Whiteness of residuals
>> >> >    - Test for model stability
>> >> >    - A check if the residuals are Gaussian (optional for some)
>> >> >
>> >> > This tells us whether the modelling procedure has been successfully
>> >> applied
>> >> > to the data. Window length in relation to model order can contribute
>> to
>> >> > estimation bias, so these parameters have to be chosen carefully.
>> Often,
>> >> > one has to pre-process cautiously and iterate through a few models in
>> >> order
>> >> > to get a good final model that satisfies the criteria.
>> >> >
>> >> > Rgds,
>> >> >
>> >> > Matthew
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Jul 8, 2019 at 12:51 PM AIKATERINI LYMPERIDOU <
>> >> > med1p1040133 at med.uoc.gr> wrote:
>> >> >
>> >> >>
>> >> >> I just noticed that I can compute the connectivity measures
>> >> >> successfully even my model does not pass the Validation Tests. So if
>> >> >> you are facing the same problem, just go the next step without
>> >> >> bothering a lot. The most important things are Model Order
>> selection,
>> >> >> the selection of the window step and the window length.
>> >> >>
>> >> >> Hope you the very best!
>> >> >>
>> >> >>
>> >> >>
>> >> >> Quoting AIKATERINI LYMPERIDOU <med1p1040133 at med.uoc.gr>:
>> >> >>
>> >> >> > Please note that the window for the "Model Validation Results"
>> >> >> > appears but none of the windows pass the "Residual Whiteness Test"
>> >> >> > and the graph for the "Whiteness Significance" does not appear.
>> >> >> >
>> >> >> > I cannot understand if this is a problem of preprocessing or I am
>> >> >> > missing something.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > Quoting AIKATERINI LYMPERIDOU <med1p1040133 at med.uoc.gr>:
>> >> >> >
>> >> >> >> Hello to everyone!
>> >> >> >>
>> >> >> >> I am using the SIFT toolbox (EEGLAB-compatible toolbox for
>> analysis
>> >> >> >> and visualization of multivariate causality).
>> >> >> >>
>> >> >> >> My data are task-related (button pushed when the subject realize
>> if
>> >> >> >> he see a figure inside the context of the whole pictue ). I used
>> >> >> >> filtering (bandpass filter (2:65), hamming window) and
>> >> >> >> preprocessing and the other steps the group of SIFT recommends
>> for
>> >> >> >> analysis in their "SIFT_Practicum"
>> >> >> >> (https://sccn.ucsd.edu/wiki/SIFT) for a sample dataset.
>> >> >> >>
>> >> >> >>
>> >> >> >> Unfortunately, in the step of "Validate Model" I get this error.
>> >> >> >>
>> >> >> >> Warning: defaultParallelConfig will be removed in a future
>> release.
>> >> Use
>> >> >> >> parallel.defaultClusterProfile instead.
>> >> >> >> WARNING: The MVAR algorithm 'BSBL L1' depends on BSBL_L1_noise.m,
>> >> >> >> which cannot be located on the path. This algorithm will not be
>> >> >> >> available.
>> >> >> >> Constant detrending each window...
>> >> >> >> done.
>> >> >> >> Done.
>> >> >> >> WARNING: The MVAR algorithm 'BSBL L1' depends on BSBL_L1_noise.m,
>> >> >> >> which cannot be located on the path. This algorithm will not be
>> >> >> >> available.
>> >> >> >> Warning: defaultParallelConfig will be removed in a future
>> release.
>> >> Use
>> >> >> >> parallel.defaultClusterProfile instead.
>> >> >> >> Constant detrending each window...
>> >> >> >> done.
>> >> >> >> Done.
>> >> >> >> Undefined function or variable "f".
>> >> >> >>
>> >> >> >> Error in findobjuser (line 45)
>> >> >> >> h = h(f);
>> >> >> >> Error in PropertyGrid/FindPropertyGrid (line 409)
>> >> >> >>            h = findobjuser(@(userdata) userdata.(member) == obj,
>> >> >> >> '__PropertyGrid__');
>> >> >> >>
>> >> >> >> Error in PropertyGrid.OnPropertyChange (line 428)
>> >> >> >>            self = PropertyGrid.FindPropertyGrid(obj, 'Model');
>> >> >> >>
>> >> >> >> Done.
>> >> >> >>
>> >> >> >>
>> >> >> >> Could anyone face the same error as me?
>> >> >> >> I would really appreciate it if you have any idea why this is
>> >> happening.
>> >> >> >>
>> >> >> >> Thank you,
>> >> >> >> Katerina
>> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> >> >> >> To unsubscribe, send an empty email to
>> >> >> eeglablist-unsubscribe at sccn.ucsd.edu
>> >> >> >> For digest mode, send an email with the subject "set digest mime"
>> >> >> >> to eeglablist-request at sccn.ucsd.edu
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> >> >> > To unsubscribe, send an empty email to
>> >> >> eeglablist-unsubscribe at sccn.ucsd.edu
>> >> >> > For digest mode, send an email with the subject "set digest mime"
>> to
>> >> >> > eeglablist-request at sccn.ucsd.edu
>> >> >>
>> >> >>
>> >> >>
>> >> >> _______________________________________________
>> >> >> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> >> >> To unsubscribe, send an empty email to
>> >> >> eeglablist-unsubscribe at sccn.ucsd.edu
>> >> >> For digest mode, send an email with the subject "set digest mime" to
>> >> >> eeglablist-request at sccn.ucsd.edu
>> >> >>
>> >>
>> >>
>> >>
>> >>
>>
>>
>>
>> _______________________________________________
>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> To unsubscribe, send an empty email to
>> eeglablist-unsubscribe at sccn.ucsd.edu
>> For digest mode, send an email with the subject "set digest mime" to
>> eeglablist-request at sccn.ucsd.edu
>>
>
>
> --
> Makoto Miyakoshi
> Assistant Project Scientist, Swartz Center for Computational Neuroscience
> Institute for Neural Computation, University of California San Diego
>