[Eeglablist] Permutation Tests for Agreement of Three or More Confusion Matrices

[Aleksandra Vuckovic]

Hi,
Can somebody explain me how Permutation test works for ANOVA (and 2 way ANOVA).
This is what I found in Blair R, Karniski W.. Psychophysiology. 1993;30:518-524 article, we normally reference for their paper for Permutation statistic, but they do no explain this.



A Brief Note on Permutation Tests for Agreement of Three or More Confusion Matrices

To this point, the matrix permutation test has been described within the context of comparing the agreement between two proximity matrices. Hubert (1979a, 1979b) extended this permutation test to three or more proximity matrices. The generation of the complete distribution for the [(n!).sup.Q] possible realizations of the index value is impractical for most n and Q, where Q is the number of matrices. This limitation necessitates the reliance on Monte Carlo sampling methods (i.e., using random number generators) to evaluate the significance of the index. If the statistic is sufficiently extreme with respect to the simulated agreement, then the null hypothesis of no agreement among the matrices is rejected.

My question is:
a.     do we check all possible options (e.g. if I have three groups then I'll swap values between 1 and 2, leave 3 unchanged, then between 1 and 3 leave 2 unchanged etc) or we do just a certain percentage of it, and what percentage?
b.     Would bootstrapping for the same analysis be any more reliable?

Many thanks,
Aleksnadra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20130327/bceabc07/attachment.html>