[Eeglablist] Need for more labelers on the IC Label website

Tue Jul 5 12:06:19 PDT 2016

Thank you very much for your response Luca! I'm copying the listserve here
to spread the awareness and transparency.

James

On Tue, Jul 5, 2016 at 3:03 PM, Luca Pion-Tonachini <lpionton at ucsd.edu>
wrote:

> Hi James, Tarik, Makoto, and all,
>
> Thank you all for your interest in the project. For anyone who has not
> seen this before, you can read about it (and even participate) at
> reaching.ucsd.edu:8000/tutorial.
>
> James - I appreciate your concern about bias from the . More labels and
> more labels would indeed be helpful. To mitigate bias in the labels
> collected, I'm using an algorithm which learns which labels to trust on
> which component types. This type of problem is generally called "Crowd
> Labeling" and there are a host of algorithms which have been shown to
> provide more accurate label estimates than simply taking the majority vote
> on each component. Nevertheless, such an algorithm alone cannot overcome a
> bias with is prevalent within the community of labelers. To counteract such
> a bias, we have singled out certain labelers who are generally bias free
> and "communicated" that belief to the algorithm to put more "validity" on
> their votes. Without going too much into details, this is done through
> Bayesian priors in the model so that nothing is hard coded and unchangeable
> should they go against the data.
>
> Tarik -  See above for how I process the labels. As for what will happen
> next: the labels estimated from the website results will be used to make an
> IC classifier that will be released for public use as part of EEGLAB (and
> possibly as a standalone as well). As for the website, I do plan to expand
> it as a teaching aid in the future. Unfortunately I don't have time to do
> that right now, but I hope to get to it in the coming months. The idea is
> to show users their estimated abilities and to provide a labeling format
> which does not collect labels, but instead shows the opinions of others on
> a given component.
> As to pruning the ICs shown, that is already happening.
>
> Makoto - I agree, I hope the above is clear enough to give an idea of the
> process.
>
> Thank you again for all your help and concerns.
> Luca
>
>
>
>
>
> On 06/29/2016 06:26 PM, Tarik S Bel-Bahar wrote:
>
> Greetings James and Luca,
>
> Hoping all is well, nice to know about this. I've directed at least 4 RAs
> to each contribute at least 400 each or so, and will also have a few more
> go through the process, albeit newbies.
> In the meantime, can either of you share a bit more about current
> findings, ways to "look inside" the rater data, or some existing IC classes
> with stable classification metrics thus far ?
> Will the outcome be a public library for IC classification routines, or
> will you try to monetize/weaponize the resource ? :)
>
> Overall, as I've shared a little with Luca, as far as I'm concerned this
> is a great teaching tool for anyone with students working with ICA and/or
> eeglab, and would also be an interesting project for whole classes to work
> on for a part of a term. Perhaps the critical thing that's lacking from my
> perspective is some kind of feedback and review mechanism, as for now it's
> quite opaque, especially for newbies. If a feedback function of any kind
> after making a response (like the likely categories based on existing
> ratings by experts) it might make the experience better for more users to
> use.
>
> That being said, I think it might also benefit the effort to 1)prune down
> the ICs to hi/middle/low quality ICs (rather than the whole available zoo),
> and then invite the 100+ or so authors who've used ICA to get at least one
> lab-member to do several hundred, and to do some themselves. I think it
> would take a personal email to each of those researchers, and even then the
> response rate won't be awesome. Another thing to consider is as something
> for EEG-related conference attendees to do or try.
>
> But more about pruning the universe of ICs: can't there be a "smarter"
> subset of all ICs that are presented to users, rather than kind of asking
> users to brute force through seemingly all of them. Maybe there could be
> something like adaptive testing, so that once there's a clear score on a
> class of ICs, they are not shown again. Another option is to give users the
> option to sample from the "whole universe of ICs" you have, or just from
> the "subset" where it will be really useful to get more classifications?
>
> Good energy for your current work, and looking forward to seeing the
> long-term results come to fruition.
>
> Best wishes, T
>
> ps: getting some info on the raters by emailing them with a short survey
> might be a good idea if you haven't as they might be ICA classification
> robots.
>
>
>
>
>
> On Mon, Jun 27, 2016 at 3:10 PM, James Jones-Rounds <jj324 at cornell.edu>
> wrote:
>
>> Hi Luca and all,
>>
>> I just want to encourage more folks to contribute labels to the IC Label
>> project (where anyone can help annotate and label ICA components, in order
>> to improve future classification efforts). I'm a little concerned that
>> after over 13,000 components have labeled, over half were done by only
>> three people. I think we need greater representation from other reviewers
>> before we can draw conclusions from the results that might be culled from
>> the database of ICs as it currently stands.
>>
>> Thank you, and keep it up everyone!
>>
>> James
>>
>> --
>> James Jones-Rounds
>> Laboratory Manager
>> Human Development EEG and Psychophysiology (HEP) Laboratory,
>> Department of Human Development,
>> --------------------------------------------
>> Cornell University | Ithaca, NY
>> 607-255-9883
>> eeg at cornell.edu
>>
>> _______________________________________________
>> Eeglablist page: http://sccn.ucsd.edu/eeglab/eeglabmail.html
>> To unsubscribe, send an empty email to
>> eeglablist-unsubscribe at sccn.ucsd.edu
>> For digest mode, send an email with the subject "set digest mime" to
>> eeglablist-request at sccn.ucsd.edu
>>
>
>
>

-- 
James Jones-Rounds
Laboratory Manager
Human Development EEG and Psychophysiology (HEP) Laboratory,
Department of Human Development,
--------------------------------------------
Cornell University | Ithaca, NY
607-255-9883
eeg at cornell.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://sccn.ucsd.edu/pipermail/eeglablist/attachments/20160705/a09f8266/attachment.html>