<div dir="ltr"><div>Thank you very much for your response Luca! I'm copying the listserve here to spread the awareness and transparency. <br><br></div>James<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 3:03 PM, Luca Pion-Tonachini <span dir="ltr"><<a href="mailto:lpionton@ucsd.edu" target="_blank">lpionton@ucsd.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi James, Tarik, Makoto, and all,<br>
<br>
Thank you all for your interest in the project. For anyone who has
not seen this before, you can read about it (and even participate)
at <a>reaching.ucsd.edu:8000/tutorial</a>.<br>
<br>
James - I appreciate your concern about bias from the . More
labels and more labels would indeed be helpful. To mitigate bias
in the labels collected, I'm using an algorithm which learns which
labels to trust on which component types. This type of problem is
generally called "Crowd Labeling" and there are a host of
algorithms which have been shown to provide more accurate label
estimates than simply taking the majority vote on each component.
Nevertheless, such an algorithm alone cannot overcome a bias with
is prevalent within the community of labelers. To counteract such
a bias, we have singled out certain labelers who are generally
bias free and "communicated" that belief to the algorithm to put
more "validity" on their votes. Without going too much into
details, this is done through Bayesian priors in the model so that
nothing is hard coded and unchangeable should they go against the
data. <br>
<br>
Tarik - See above for how I process the labels. As for what will
happen next: the labels estimated from the website results will be
used to make an IC classifier that will be released for public use
as part of EEGLAB (and possibly as a standalone as well). As for
the website, I do plan to expand it as a teaching aid in the
future. Unfortunately I don't have time to do that right now, but
I hope to get to it in the coming months. The idea is to show
users their estimated abilities and to provide a labeling format
which does not collect labels, but instead shows the opinions of
others on a given component.<br>
As to pruning the ICs shown, that is already happening.<br>
<br>
Makoto - I agree, I hope the above is clear enough to give an idea
of the process.<br>
<br>
Thank you again for all your help and concerns.<span class="HOEnZb"><font color="#888888"><br>
Luca</font></span><div><div class="h5"><br>
<br>
<br>
<br>
<br>
On 06/29/2016 06:26 PM, Tarik S Bel-Bahar wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:#333399">Greetings James
and Luca,</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">Hoping all is
well, nice to know about this. I've directed at least 4 RAs to
each contribute at least 400 each or so, and will also have a
few more go through the process, albeit newbies.</div>
<div class="gmail_default" style="color:#333399">In the
meantime, can either of you share a bit more about current
findings, ways to "look inside" the rater data, or some
existing IC classes with stable classification metrics thus
far ?</div>
<div class="gmail_default" style="color:#333399">Will the
outcome be a public library for IC classification routines, or
will you try to monetize/weaponize the resource ? :) </div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">Overall, as
I've shared a little with Luca, as far as I'm concerned this
is a great teaching tool for anyone with students working with
ICA and/or eeglab, and would also be an interesting project
for whole classes to work on for a part of a term. Perhaps the
critical thing that's lacking from my perspective is some kind
of feedback and review mechanism, as for now it's quite
opaque, especially for newbies. If a feedback function of any
kind after making a response (like the likely categories based
on existing ratings by experts) it might make the experience
better for more users to use.</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">That being
said, I think it might also benefit the effort to 1)prune down
the ICs to hi/middle/low quality ICs (rather than the whole
available zoo), and then invite the 100+ or so authors who've
used ICA to get at least one lab-member to do several hundred,
and to do some themselves. I think it would take a personal
email to each of those researchers, and even then the response
rate won't be awesome. Another thing to consider is as
something for EEG-related conference attendees to do or try.</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">But more about
pruning the universe of ICs: can't there be a "smarter" subset
of all ICs that are presented to users, rather than kind of
asking users to brute force through seemingly all of them.
Maybe there could be something like adaptive testing, so that
once there's a clear score on a class of ICs, they are not
shown again. Another option is to give users the option to
sample from the "whole universe of ICs" you have, or just from
the "subset" where it will be really useful to get more
classifications?</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">Good energy for
your current work, and looking forward to seeing the long-term
results come to fruition.</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">Best wishes, T</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399">ps: getting
some info on the raters by emailing them with a short survey
might be a good idea if you haven't as they might be ICA
classification robots.</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
<div class="gmail_default" style="color:#333399"><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jun 27, 2016 at 3:10 PM, James
Jones-Rounds <span dir="ltr"><<a href="mailto:jj324@cornell.edu" target="_blank">jj324@cornell.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Luca and all,
<div><br>
</div>
<div>I just want to encourage more folks to contribute
labels to the IC Label project (where anyone can help
annotate and label ICA components, in order to improve
future classification efforts). I'm a little concerned
that after over 13,000 components have labeled, over
half were done by only three people. I think we need
greater representation from other reviewers before we
can draw conclusions from the results that might be
culled from the database of ICs as it currently stands.</div>
<div><br>
</div>
<div>Thank you, and keep it up everyone!</div>
<span><font color="#888888">
<div><br>
James<br clear="all">
<div><br>
</div>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">
<div>James Jones-Rounds</div>
Laboratory Manager<br>
Human Development EEG and Psychophysiology (HEP)
Laboratory,
<div>Department of Human Development,<br>
--------------------------------------------<br>
Cornell University | Ithaca, NY<br>
</div>
<div><a href="tel:607-255-9883" value="+16072559883" target="_blank">607-255-9883</a></div>
<div><a href="mailto:eeg@cornell.edu" target="_blank">eeg@cornell.edu</a></div>
</div>
</div>
</div>
</font></span></div>
<br>
_______________________________________________<br>
Eeglablist page: <a href="http://sccn.ucsd.edu/eeglab/eeglabmail.html" rel="noreferrer" target="_blank">http://sccn.ucsd.edu/eeglab/eeglabmail.html</a><br>
To unsubscribe, send an empty email to <a href="mailto:eeglablist-unsubscribe@sccn.ucsd.edu" target="_blank">eeglablist-unsubscribe@sccn.ucsd.edu</a><br>
For digest mode, send an email with the subject "set digest
mime" to <a href="mailto:eeglablist-request@sccn.ucsd.edu" target="_blank">eeglablist-request@sccn.ucsd.edu</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>James Jones-Rounds</div>Laboratory Manager<br>Human Development EEG and Psychophysiology (HEP) Laboratory,<div>Department of Human Development,<br>--------------------------------------------<br>Cornell University | Ithaca, NY<br></div><div>607-255-9883</div><div><a href="mailto:eeg@cornell.edu" target="_blank">eeg@cornell.edu</a></div></div></div>
</div>