The EEGLAB News #12
Tzyy-Ping Jung, Ph.D.
Co-Director, Center for Advanced Neurological Engineering, Institute for Neural Computation (INC) and Institute of Engineering in Medicine (IEM), and Associate Director, Swartz Center for Computational Neuroscience (SCCN), INC, both at UC San Diego
Adjunct Professor, Department of Bioengineering, UC San Diego
Adjunct Professor, Department of Computer Science and Department of Electrical Engineering, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Adjunct Chair Professor, College of Education, National Tsing Hua University, Hsinchu, Taiwan
Adjunct Professor, Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan, Taiwan
Adjunct Professor, School of Precision Instrument and Opto-electronic Engineering, Tianjin University, Tianjin, China
Few people know the full story of the origin of SCCN and EEGLAB. Fortunately, Dr. Tzyy-Ping Jung (Associate Director of the Swartz Center for Computational Neuroscience (SCCN) and Co-Director of the Center for Advanced Neurological Engineering of the Institute for Neural Computation (INC) and Institute of Engineering in Medicine (IEM), UC San Diego) was one of the key players. And he is eager to tell the story.
In 1993, as a newly minted Ph.D. in electrical engineering from Ohio State University, Dr. Jung was recruited by Dr. Scott Makeig (Director of SCCN) to help him analyze electroencephalographic (EEG) data. Soon after, he began working with Scott at the laboratory of collaborator Dr. Terrence Sejnowski (Co-Director of the INC and Director of the Computational Neurobiology Laboratory (CNL) at the Salk Institute).
Dr. Jung’s move to San Diego didn’t start as smoothly as he would have hoped. “That is a very interesting story that very few people know,” he relates. “In October 1993, I just started my position with Scott as a postdoc at the Naval Health Research Center (NHRC) in Point Loma, San Diego. After working there for one week, the Center bureaucracy realized that I was not a U.S. citizen and had no permission to work on a Navy base! Scott approached Terry Sejnowski, and he allowed me to work temporarily at the Salk Institute in his CNL Lab. So, a week after beginning my work at NHRC, I was ordered out of the Navy base, and arrived in Terry’s lab. I was very appreciative of Terry’s gesture, and when the Navy Center obtained clearance for me to work on the Navy base about six months later, I decided to stay at CNL. It was an unusual situation: My funding came directly from the National Academy of Sciences and my official office was at NHRC, but physically I was working in Terry’s lab, CNL. Soon Scott began joining me there after lunch each day, until he and Terry won funding for us both to join Salk officially.” The later opportunity to meet Dr. Jerry Swartz and, as our work grew, to propose to him the concept of the UC San Diego Swartz Center grew out of that. (Photo: Dr. Jung and Dr. Makeig in Sandai, Japan, 2002)
“In 1995,” Dr. Jung shares, “while working with Scott and Terry, I was fascinated by a presentation given by Tony Bell, a postdoc at the CNL at the time. He described something called independent component analysis (ICA), a computational method for analyzing data from multiple channels. It gave me an idea. A channel is just like a microphone. Maybe we could apply ICA to EEG, to separate a multivariate signal into subcomponents. Perhaps we could use ICA to identify brain sources measured by EEG when a person is drowsy vs. when they are alert. I was so excited, that I immediately sent the preprint to Scott, and he shared my excitement.”
“Early the next morning,” Dr. Jung continues, “Scott and I began discussing this technology in the conference room of Terry’s lab. Tony Bell happened to pass by the door of the conference room, and we grabbed him into the conference room and told him, ‘We think that your independent component analysis could be ideal for biomedical signal processing because we are using multi channels. And when we collect data, each channel is a mixture of all the underlying sources.’ Although Tony didn’t have experience with EEG, he thought the idea had great potential and he immediately sent me the source code. We began experimenting –and when we analyzed the first 14-channel EEG dataset, we immediately saw very interesting results.”
The team found that ICA was an effective method for removing artifacts and separating sources of the brain signals from these recordings. They had less than two weeks to prepare a submission to the Neural Information Processing Systems (NIPS, now NeurIPS) Conference (which was accepted and presented at its next meeting). But the team took a few months until they completely figured out how to interpret the signal--unmixing matrix, now known as the matrix of component scalp maps. The team also submitted a short abstract to the Society for Neuroscience (SfN) Conference in 1995, which was disclosed a few months before the NIPS Proceedings (on June 11, 1996).
The ICA EEG Toolbox and EEGLAB. Dr. Makeig and Dr. Jung soon began developing MATLAB functions for ICA decomposition, visualization, and evaluation. In 1997, they released a precursor to EEGLAB on the Salk Institute website that they called the ICA EEG Toolbox. “These functions were the backbone of the ICA EEG Toolbox, and now, of EEGLAB,” Dr. Jung explains, “There was no graphical user interface (GUI) associated with it. If you knew how to use the Matlab command line, you could enter your data and parameters into these functions to select, preprocess and decompose your data and then visualize the results.”
That changed when Dr. Arnaud Delorme (then a postdoc in Terry’s lab) learned of the ICA EEG Toolbox and was immediately captivated. “Scott encouraged Arno to build a GUI around the toolbox functions,” Dr. Jung explains. “And this became EEGLAB. Since then, Arno and many, many others have created many more visualization tools and statistical and spectral analysis methods." EEGLAB is now receiving on average over 15 new citations on Google Scholar each day.
Today, Dr. Jung uses EEGLAB often in his research: “75% of my papers involve using at least part of EEGLAB,” he clarifies. “And my international collaborators are all big fans of EEGLAB!” He has also taught EEGLAB many times worldwide – in Taiwan, Australia, and the United States – and helped Dr. Makeig organize the first few EEGLAB Workshops in San Diego as well as later those given in Taiwan and China. (Photo above: 15th EEGLAB Workshop in Beijing)
Background. Born and raised in Taiwan, Dr. Jung received his B.S. degree in electronics engineering from National Chiao Tung University in Taiwan in 1984, and moved to the states to pursue his Ph.D. in 1987. In 1993 the newly graduated Dr. Jung was recruited by Dr. Makeig as a Postdoctoral fellow of the National Academy of Science to work with him on EEG analysis in San Diego. Little did he know this position would turn into his career base. “At first, I thought I would be there for two weeks,” he chuckles, “but two weeks became five years – and now almost 30!” (Photo: Early SCCN teatime. Closest to the table Drs. Onton, Makeig, and Jung).
Real-World Neuroimaging, Brain Computer Interfaces (BCI), and Mobile Brain Body Imaging (MoBI). Dr. Dr. Jung was immediately captivated by the research in both Dr. Makeig’s and Dr. Sejnowski’s labs. “Scott brought me into EEG data analysis and Terry brought me into computational neuroscience,” he reflects. As well as Associate Director at SCCN, T-P is now Co-Director of the Center for Advanced Neurological Engineering of the INC and the IEM at UC San Diego. His overarching goal and passion is to advance basic neuroscience research in real-world environments. “In the past thirty years, we have witnessed a lot of progress in neuroscience research and neuroengineering. But most of those studies have been performed in a well-controlled laboratory setting. Not in real life. That is the biggest problem holding back development of real-world applications.”
“The paradigms used in the lab are so simple,” he continues. “Lab research paradigms use what we call stimulus-response contingency. Basically, you see something displayed on the computer screen and make some simple response to that (usually a simple button press). You think that when you present the stimuli people simply respond to them. But this is not how it really is in the real world. For instance, say I had only three hours of sleep last night -- that will actually impact my cognition. Lack of sleep will even impact my temperament and my mood, and those things will in turn affect my cognitive functioning. If my mood is really bad, I will get mad easily, and I may actually use different task strategies. This model of how the brain functions is called embodied cognition – the concept that activities of our sensory and motor systems are tightly integrated with those supporting our cognitive processing. That is why in studying how our brain supports our behavior and experience we need to consider how our physical, mental, and psychological states impact our cognition.”
Dr. Jung’s research covers a head-spinning array of topics (read about his Demos and Current projects). He describes one of his recent real-world neuroimaging studies: “We track subjects over five months, measuring their physical, mental, and psychological stress by tracking their sleep quality using a wristband, their galvanic skin response (GSR) through skin conductance, and daily morning self-reports of stress, drowsiness, sleepiness. Then they come into the lab for four hours of EEG studies (following naturally varying sleep amounts and quality). We ask them to perform a reaction time test, a simulated driving test, etc., and then try to see how that impacts our neuroengineering models.” He resumes, “What we find is very intriguing. There is huge intra-subject variability. There may be a different relationship between EEG spectra and task performance for the same subject on different days or different kinds of days, depending on their physical, mental, and psychological state. There is also huge inter-subject variability – differences across subjects.”
Dr. Jung explains further: “That’s what nobody talks about. Because most of the time in neuroscience research, we just put up a flier to recruit subjects. When subjects come in, we’re so happy and feel privileged to get the subjects we need for our experiments. And we then just let them go. We don’t really know their background, their sleep history, or how stressed they are during the experiment. When researchers analyze the data and their model looks like it works well, they are excited and publish papers. But when they then test this model in the real world, it often doesn’t work nearly as well. Then the industry says, ‘BCI and EEG are not ready for prime time. Go back to the lab and conduct more experiments.’ But if we just go back to the well-controlled lab, it’s a ‘Catch 22’,” he declares. “In a lab, you never capture the complications of life in the real world.”
That is why Dr. Jung and collaborators have proposed the concept – or paradigm shift – of real-world neuroimaging. “We want to track a person’s physical, mental, and psychological states, while allowing people to move around naturally in their environment. To do this we have developed a multi-modal imaging approach, recording high-density EEG and combining eye tracking, motion capture, and tracking of physiological data – such as heart rate, ECG, GSR, skin conductance. We can also present audio-visual stimulation and measure neural and physiological data simultaneously. We also want to know what subjects experience and do - including their gaze pattern, so we record video from the first-person viewpoint, for instance, through a camera mounted on their glasses. We also want to know what they hear – the acoustics in the room.” This approach to neuroimaging is called Mobile Brain Body Imaging (MoBI), a vision (and acronym) proposed by SCCN Director Dr. Scott Makeig and colleagues in 2009. (Photo: Dr. Jung and colleagues demonstrate a real-world neuroimaging experiment at SCCN).
Challenges.
While MoBI is a very powerful tool, developing a MoBI system can be extremely expensive. “When the Office of Naval Research gave us funding to set up a first MoBI lab at SCCN,” Dr. Jung recalls, “It cost us more than a million dollars. That is not very affordable for other labs.” He continues, “One of my goals is to create low-cost MoBI instrumentation and software that everybody can use, so that other researchers in the world can study more realistic cognition in real-world environments.” An example of this is the open-source Lab Streaming Layer (LSL) software framework developed by Christian Kothe at SCCN, which now forms the backbone of multimodal MoBI data collection in laboratories around the world.
Real-world neuroimaging also requires portability, user convenience and long-term wearability. In the late 1990’s, Dr. Jung began his quest to develop more portable and convenient EEG and now MoBI systems. Together with colleagues in Taiwan and UC San Diego, he started working on developing and using dry scalp sensors to reduce subject EEG preparation time. More and more scientists are developing cost-effective equipment. For example, Mike Chi, a UC San Diego Ph.D. graduate, developed Cognionics, Inc., in 2011, a company that specializes in producing state-of-the-art dry electrode headsets. And more recently, a UC San Diego Ph.D. student of T-P, Siddharth Siddharth, developed head-mounted eye-tracking, mounted on a glasses frame, combined with a dry-electrode EEG system and other data streams. “Nowadays,” Dr. Jung shares, “you canalso use single head-facing video camera combined with computer vision technology based on artificial intelligence (AI) to track a subject’s body and eye movements without attaching any sensors to the subject.” For example, MediaPipe from Google does this. While cost-effective systems may not offer the same resolution or fidelity as elaborate and expensive laboratory systems, they make real-world neuroimaging possible for many more scientists and students. “Now people can actually perform and replicate studies using much less expensive tools,” Dr. Jung relates. (Photo above: left, Trevor Kerth and Dr. Mike Chi of Cognionics Inc. sit with a model of wireless sensors; right, Siddharth Siddharth wearing his head mounted eye-tracking system.)
Clinical Applications. Dr. Jung is also excited about the potential impact this work can have on future clinical research and practice – using real-world MoBI methods and tools to enhance human performance and resilience to physical and mental stress (e.g., using cognitive monitoring, emotion detection, education/training), and to help create novel strategies to benefit health. “Our portable Brain Computer Interface (BCI) headset has a lot of potential for evaluating neurological and psychiatric disease,” he shares enthusiastically. His team began using multifocal steady-state visual evoked potentials (mfSSVEPs) to diagnose several diseases. SSVEPs are the EEG responses to visual stimulation delivered rhythmically at specific frequencies. When a user looks at a flashing stimulus, we can measure the portion of their brain EEG that has the same frequency as the stimulus train. By analyzing the EEG data, researchers can identify the stimulus frequency and diagnose damage to the visual field when the brain response is lacking. Using this approach, Dr. Jung’s team can create many clinical applications (View demo for details).
“For example, we first applied this BCI device to non-invasively assess glaucoma,” Dr. Jung shares. Glaucoma, an optic disease that causes visual field deficits, is the leading cause of blindness. “Using our device,” he continues, “if we stimulate different regions of the retina and find abnormal SSVEPs, we can then identify the visual field and brain areas affected by the glaucoma. (Nakanishi M, Wang YT, Jung TP, et. al., 2017) This device shows very reliable performance in diagnosing glaucoma and replaces a much older and less exact method.” Dr. Jung recently expanded visual field evaluation to assess the progression of multiple sclerosis (an autoimmune disease). “This approach can be easily applied to traumatic brain injury detection, as well,” he adds eagerly. “Recent papers discuss afferent and efferent visual pathway abnormalities as biomarkers of the early stage of dementia. So, our next step is to try to use this approach to diagnose dementia in its earliest stages."
But commercializing BCI devices for clinical application can be tricky, Dr. Jung admits: “Oftentimes, the healthcare business is slow to adapt to new ideas. It is very difficult for new entrants to gain a foothold in the market. So, we have to let them know the benefits of these applications.” One recent commercial endeavor is to produce the Ngoggle headset -- designed as wearable “virtual reality” (VR) goggles with embedded EEG sensors, custom optics, and BCI technologies to remotely assess visual function. Dr. Jung clarifies, “We founded a mobile digital-health company called Ngoggle Diagnostics that uses a neuro-diagnostic headset to sense, track, and manage diseases like glaucoma, dementia, and sleep disorders. We have received ~$2.2 million dollars from the National Institutes of Health (NIH) SBIR/STTR program (Small Business Program) to build this device and conduct a clinical trial."
Funding.
When asked about the biggest challenge he faces in his research, Dr. Jung is quick to answer: Funding. He explains, “Nowadays, we spend most of our time raising funds to support our work and lab. With Scott Makeig, John Iversen, Terry Sejnowski, we have a lot of brilliant ideas. But because the ideas are often judged to be radical, even though they are groundbreaking, oftentimes it is difficult to get funding to explore these ideas, even when they could revolutionize science and/or clinical practice. We have to come up with effective strategies to convince reviewers to fund our projects. From conception and formulation of the idea to actually getting the funding for a new idea usually takes one to three years, maybe longer…”
“I don’t know the optimal solution,” Dr. Jung admits, although he believes that collaboration with other scientists is crucial. “International and interdisciplinary collaborations are a big part of my career. Easily, I have over 100 papers from international collaborations. Another example: In 2018, before the pandemic hit, SCCN hosted 47 visiting students and scholars in one year. I think that’s a record. Forty-seven! Still, to this day, I receive requests to become a visiting scholar/student almost every week. They come here as non-salaried volunteers – and stay anywhere from two weeks to two years.” Dr. Jung pauses, recalling an important memory that has affected him to this day: “Way back when I first came as a postdoc and was not allowed on the Navy base, Terry allowed me to sit in his lab and participate almost like a permanent employee. I was very appreciative of that, so I adopted this approach from Terry. My lab agrees that interdisciplinary research and International collaboration is a big component of scientific research.”
The Future. Dr. Jung is energized when talking about his research and the future: “I really enjoy better understanding how the different parts of the brain interact with each other to achieve human perception, awareness, and cognition in real world environments. This new approach is very exciting. I feel passionate about the field – I should say fields, because working on these ideas covers many different fields!”
Dr. Jung works long hours, usually rising at 4:00 a.m. and when needed working around the clock, except for daily exercise (tennis is a favorite). He is deeply appreciative of the encouragement he receives from his wife and daughter. (Photo: Dr. Jung with his wife and daughter in Taiwan 2020). “My family is very supportive. They know research is my passion. This is my love. Some people talk about work-life balance. For me, this is my hobby!,” he jokes, adding, “To be able to develop and utilize these devices, data analysis methods, machine learning and artificial intelligence techniques, to decode our brain activity for real-time applications – that is really, really fruitful and fulfills my dream.”
R. Weistrop, May 2022
_______________________
Most cited papers:
Jung TP, Makeig S, Humphries C, Lee TW, McKeown MJ, Iragui V, Sejnowski TJ. Removing electroencephalographic artifacts by blind source separation. Psychophysiology. 2000 Mar;37(2):163-78. PMID: 10731767.
Makeig S, Bell A, Jung TP, Sejnowski T. Independent Component Analysis of Electroencephalographic Data. Advances in Neural Information Processing Systems 8 (NIPS 1995).
McKeown MJ, Makeig S, Brown GG, Jung TP, Kindermann SS, Bell AJ, Sejnowski TJ. Analysis of fMRI data by blind separation into independent spatial components. Hum Brain Mapp. 1998;6(3):160-88. doi: 10.1002/(SICI)1097-0193(1998)6:3<160::AID-HBM5>3.0.CO;2-1. PMID: 9673671; PMCID: PMC6873377.
Makeig S, Westerfield M, Jung TP, Enghoff S, Townsend J, Courchesne E, Sejnowski TJ. Dynamic brain sources of visual evoked responses. Science. 2002 Jan 25;295(5555):690-4. doi: 10.1126/science.1066168. Erratum in: Science 2002 Feb 22;295(5559):1466. PMID: 11809976.