Developing an Online Steady-State Visual Evoked Potential-Based Brain-Computer Interface System Using EarEEG

Yu-Te Wang, Student Member, IEEE, Masaki Nakanishi, Member, IEEE, Simon Lind Kappel, Student Member, IEEE, Preben Kidmose, Senior Member, IEEE, Danilo P. Mandic, Fellow, IEEE,

Yijun Wang, Member, IEEE, Chung-Kuan Cheng, and Tzyy-Ping Jung, Fellow, IEEE

Abstract— The purpose of this study is to demonstrate an online steady-state visual evoked potential (SSVEP)-based BCI system using EarEEG. EarEEG is a novel recording concept where electrodes are embedded on the surface of earpieces customized to the individual anatomical shape of users’ ear. It has been shown that the EarEEG is capable to collect evoked brain activities such as SSVEP in previous studies. However, a long distance between the visual cortex and the ear makes the signal-to-noise ratio (SNR) of SSVEPs acquired by earpieces relatively low. Recently, filter bank- and training databased canonical correlation analysis algorithms have shown significant performance improvement in terms of accuracy of target detection and information transfer rate (ITR). This study implemented an online four-class SSVEP-based BCI system using EarEEG. Two subjects participated in offline and online BCI experiments. The offline classification results obtained average accuracy of 78.75±1.18 % using 4 sec-long SSVEPs acquired from earpieces. In the online experiment, both subjects successfully completed the tasks with average accuracy of 87.44±8.62 %, leading to an average ITR of 15.71±5.1 bits/min. The results suggested the ability of using EarEEG to perform practical BCI applications. This study might lead to an alternative path of EEG recording to implement real-world BCI applications.

I. INTRODUCTION

Steady-state visual evoked potential (SSVEP)-based brain computer interface (BCI) systems have gained a lot of attentions due to the high information transfer rate (ITR) and little user training [1]. In most of applications and fundamental studies, placing electrodes on the top of the occipital site seems reasonable since it’s the closest spot to the visual cortex and its signal to noise ratio (SNR) is relatively high. However, constrains such as gel usage, long time skin preparation, uncomfortable to wear for long time

*This work was in part by the Army Research Laboratory under Cooperative Agreement Number W911NF-10-2-0022.

Y. -T. Wang and C. -K. Cheng are with Computer Science and Engineering Department, University of California San Diego, La Jolla, CA92093 USA.

M. Nakanishi is with Swartz Center for Computational Neuroscience, Institute for Neural Computation, University of California, San Diego, La Jolla, CA92093 USA.

S. L. Kappel and P. Kidmose are with Department of Engineering, Aarhus University, DK-8200 Aarhus N, Denmark.

D. P. Mandic is with Department of Electrical and Electronics Engineering, Imperial College London, UK.

Yijun Wang and Tzyy-Ping Jung are with Swartz Center for Computational Neuroscience, Institute for Neural Computation, and Center for Advanced Neurological Engineering, Institute of Engineering in Medicine, University of California San Diego, La Jolla, CA92093 USA. (e-mail: jung@sccn.ucsd.edu; phone: 858-822-7555; fax: 828-822-7556).

and the fact that the recording devices are clearly visible make scalp based acquisition of SSVEPs impractical in most real-world applications.

Many efforts have been made in order to overcome those constrains. Chi et al. [2] proposed dry and noncontact electroencephalogram (EEG) sensors to acquire SSVEPs without gel usage from hair-covered areas on scalp. The electrodes featuring custom integrated, high-impedance analog frontend, fingered contact posts, and active buffering circuitry have been successfully demonstrated in real-world applications. Lin et al. [3] developed dry polymer form electrodes fabricated by electrically conductive polymer covered with a conductive fabric as an alternative approach for long-term EEG acquisitions from non-hair-covered area. Similar efforts were also made by Huang et al. [4], as they proposed an active comb-shaped dry electrode, which can avoid signal attenuation and phase distortion. In sum, dry or noncontact sensors introduced a good solution to overcome the challenges including gel usage and long-time preparation. However, considering minimally intrusive or truly wearable device for a long time and continuously EEG recording, these types of dry/noncontact electrode systems seem not meet the requirements. An alternative approach to perform the data collection therefore is crucial. A recent innovation in wearable EEG device is so-called EarEEG where the EEG signals are recorded from devices placed in the ear [5]. The EarEEG device comprises electrodes embedded on the surface of an earpiece customized to the anatomical shape of user’s ear. Recordings of both auditory and visual evoked potentials have been performed with the EarEEG methodology, and the recordings showed comparable performance to scalp EEG near the ear [6]. Several types of physiological artifacts have also been characterized for EarEEG, and for most artifact conditions, the performance of EarEEG is comparable to that of the scalp EEG [7]. However, currently it remains unknown if the SNR of single-trial SSVEPs recorded using EarEEG can achieve good performance in an online SSVEP BCI. The low SNR, which is caused by the distance between the visual cortex and ears, challenges the online BCI implementation.

Performance of SSVEP-based BCIs depends on three major factors: stimulus presentation, multiple target coding, and target identification algorithm [8]. In previous studies, frequency or phase information has been generally used to code multiple visual stimuli [9]. Our recent studies demonstrated the efficiency of the hybrid frequency and phase coding methods to maximize the differentiation of visual stimuli [8], [10]. To achieve high performance, the target identification method also plays an important role. The canonical correlation analysis (CCA)-based method has been widely used to detect SSVEPs tagged with frequency coding [11], [12]. However, misclassification can be caused by the interference from the spontaneous EEG activities [13]. To reduce the effect of the spontaneous EEG signals, an extended CCA-based method, which incorporates prerecorded individual calibration data, has been proposed [13]. This method showed significantly improved performance over the standard CCA-based method [8], [10]. To optimize the performance of an online SSVEP-based BCI, these aforementioned issues should be jointly considered.

This study aims to explore the feasibility of an online SSVEP-based BCI using EarEEG recordings. To facilitate target identification, this study employed the training databased CCA method for target detection [12] and the joint frequency-phase approach for target coding [9]. The fourclass BCI system included four visual stimuli coded with different frequencies (8 Hz, 9 Hz, 10 Hz, and 11 Hz) and phases (0, 0.5 π, π, and 1.5 π).

II. METHODS

A. EEG Recording

The visual stimulator consisted of four flickers rendered on a 27-inch monitor with a resolution of 1280×800 (Fig. 1(a)). Each target was a 7×7 cm square flickering at white and black contrast. This study used the joint frequencyphase coding method [9] to design the stimuli. The details of stimulus presentation and target coding approaches can be found in [8], [10], [14]. Subjects were seated in a comfortable chair 60 cm in front of the monitor in a dim room.

Two subjects (one 29 years old male and one 28 years old female) with normal eyesight participated in offline and online experiments. Note that, the earpiece has to be customized for individual use so currently only two subjects were recruited for this pilot study. They both signed a consent form approved by the Human Research Protections Program of the University of California San Diego before running the experiment.

The EarEEG earpieces used for recordings in this study were personalized to the individual anatomical shape of the subjects’ ear and were 3D printed in hard acrylic plastic. The earpieces had six passive silver electrodes embedded on the surface. The electrodes were positioned according to the scheme described by Kidmose et al. [6]: four electrodes in the ear canal and two electrodes in the concha part of the outer ear. Measurements were collected from both ears simultaneously, leading to a total of 12 EarEEG channels. The earpieces were connected to a Biosemi ActiveTwo EEG system (Biosemi, Inc.) using in-line buffer amplifiers. The subjects had their ears cleaned with alcohol and abrasive gel prior to measurement to remove dead skin cells in the ear canal and outer ear. Conductive gel was applied to the surface of the EarEEG electrodes in order to obtain impedances below 10 KΩ. In addition, eight scalp EEG

(a)

(b)

Fig. 1. (a) Visual stimulus layout of the proposed system. Top blank space with two forward symbols was used to show subject’s input. When the training starts, each target was flickering at coded frequency and phase. In the offline training, a red square is the cue that appears for 1 second to assist subjects to gaze on it. While in online experiment, the red square appears for 1 second to indicate the decision by the system. (b) An earpiece with 6 electrodes (left) and a subject wearing an EarEEG earpiece in left ear.

electrodes were placed over the occipital lobe, and recordings from these electrodes were used for comparison. All EarEEG and scalp recordings were referenced to an electrode located at forehead area. The recordings were performed with a sampling rate at 2048 Hz but down-sampled to 256 Hz for analysis.

B. Offline experiment

For each subject, the offline experiment consisted of 20 sessions. Each session had 12 trials lasting 60 seconds. Each trial included 4 seconds for target gazing and 1 second for gaze shifting. Each of the four stimuli was gazed three times in a random order in one session. At the beginning of each trial, a target was rendered as red for 1 second as a cue (see Fig. 1(a)). Subjects were asked to switch their gaze to the target stimulus as soon as possible. A few minutes break was scheduled between two consecutive sessions.

The recorded EEG data were first analyzed to compare the amplitude and SNR of SSVEPs collected from the occipital area and EarEEG electrodes. The SNR was calculated as follows:

SNR= \frac{K × F (f)}{∑ _{k=1}^{K/2}(F (f + kΔf) + F (f − kΔf))}  (1)

(a)

(b)

(c)

Fig. 2. (a) The average SSVEPs of raw data, (b) power spectrum, and (c) SNR from occipital site (black), left ear (blue), and right ear (right) when a subject was gazing at the 9 Hz stimulus.

where F (f) is the amplitude value at a frequency f, Δf is the frequency resolution in the amplitude spectrum [15]. In this study, Δf was 0.25 Hz and K was set to 12. This study also performed an offline classification of SSVEPs. 4s-long epochs were extracted according to event triggers generated by the stimulus program. In this study, the extended CCAbased method, that combines filter bank approach and individual template-based approach, was employed to identify a target stimulus [8], [13], [16]. The procedures were as follows: (1) decompose individual template signals and test data into multiple sub-band signals, (2) calculate correlation coefficients between decomposed template signals and test data after spatial filtering as feature values, (3) calculate a weighted sum of squares of the correlation values as final feature values, and (4) identify target based on the feature values. The accuracy and simulated ITR were estimated by ten-fold cross validation.

C. Online Experiment

The online task was to input 12 targets without visual cues. Subjects were asked to gaze at a target until the target was rendered to red, which indicated that the target was identified by the online program, and then they had one second to switch to the next target. If the system did not identify the target correctly, subjects had to keep gazing on the target until it was correctly identified. The subjects practiced a few minutes to get familiar with the online task before real

(a)  (b)

Fig. 3. Simulated online results of (a) classification accuracy and (b) ITR using EarEEG with different data lengths from two subjects.

experiments. For each subject, three successful sessions were recorded including the total time and input sequences.

III. Results

A. Offline Analysis

Fig. 2 shows the average waveform, power spectrum, and SNR of SSVEPs at 9 Hz collected from the occipital area and both ears of one subject. The amplitude in either time domain or frequency domain from ear SSVEPs were evidently smaller than the SSVEPs acquired from the occipital area. In power spectrum density, the fundamental peaks are all clearly located at 9 Hz, and second harmonics appear in occipital channels while left/right ears have small peaks. In general, ear SSVEPs for both subjects have small second harmonics and almost no further harmonics, so the optimal parameter for the filter bank was set to 2 harmonics in data analysis. In terms of SNR, both ears’ SSVEPs have slightly lower SNRs, compared the occipital one in fundamental peaks (9 Hz). The small SNR difference might be in part attributed to the fact that the occipital EEG data were collected by active electrodes that were designed to improve the quality of the acquired signals.

B. Simulated Online and Online Analysis

This study first used leave-one-out cross-validation to estimate the BCI performance. As shown in Fig. 3(a), the accuracy reached ∼80 % for both subjects, and the accuracy increased as data length increased, while ITR reached ∼12 bits/min using 4-second data. Note that, the average accuracy using occipital channels was 99.38±0.88 % and the ITR was 23.3±0.99 bits/min using 4 second window for both subjects.

Table I lists the online results for two subjects. Both subjects completed the tasks including three 12-target trials. The average accuracy was 87.44 %, leading to an average ITR of 15.71 bits/min. These results suggested that EarEEG could provide reliable recordings of SSVEPs for an online BCI.

TABLE I

Online free input result

Subject  Input Time Accuracy  ITR
length (sec) (%)  (bits/min)
S1  Trial 1 15  75  80.00  11.53
Trial 2 15  75  80.00  11.53
Trial 3 15  75  80.00  11.53
S2  Trial 1 13  65  92.31  17.84
Trial 2 13  65  92.31  17.84
Trial 3 12  60  100.00  24.00
Mean  87.44±8.62 15.71±5.10

IV. CONCLUSIONS AND DISCUSSIONS

Ease-of-use of EEG acquisition remains a challenging issue in BCIs. Towards practical BCI systems in real-world applications, recent studies have proposed different approaches to improve feasibility and practicality of BCIs. For example, dry electrodes have been demonstrated promising results in practical BCI systems. Alternatively, EarEEG featuring the un-seen, easy-of-use, and continuously monitoring characteristics opens another path to increase the practicality of BCIs for real-life applications. This paper implemented an SSVEPbased BCI system using EarEEG. The offline training analysis was used to find the best parameters (i.e. sub-band and number of harmonics). The online BCI experiments obtained promising results in terms of accuracy (87.44±8.62 %) and ITR (15.71±5.10 bits/min). Although the results were lower than the occipital data, long distance from the occipital site causing lower SNR for the EarEEG recordings made the comparison unfair. However, considering the portability and the unobtrusive characters of the EarEEG, it is more suitable for everyday life applications, when compared to scalp EEG based technologies.

References

[1] Y. Wang, R. Wang, X. Gao, B. Hong, and S. Gao, “A practical VEPbased brain-computer interface,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 234239, 2006.

[2] C. Y. Mike, Y. -T. Wang, Y. Wang, C. Maier, T. -P. Jung, and G. Cauwenberghs, “Dry and noncontact EEG sensors for mobile braincomputer interface,” IEEE Trans. Neural Syst. Rehabil. Eng., vol.20, no.2, pp.228-235, 2012.

[3] C. -T. Lin, L. -D. Liao, Y. -H. Liu, I. -J. Wang, B. -S. Lin, and J. -Y. Chang, “Novel dry polymer foam electrodes for long-term EEG measurement,” IEEE Trans. Biomed. Eng., vol 58, no. 5, pp.1200-1207, 2011.

[4] Y. -J. Huang, C. -Y. Wu, A. M. K. Wong, and B. -S. Lin, “Novel active comb-shaped dry electrode for EEG measurement in hairy site,” IEEE Trans. Biomed. Eng., vol 62, no. 1, pp.256-263, 2015.

[5] D. Looney, P. Kidmose, C. Park, M. Ungstrup, M. L. Rank, K. Rosenkranz, and D. P. Mandic, “The in-the-ear recording concept: User-centered and wearable brain monitoring,” IEEE Pulse Magazine, vol. 3, no. 6, pp. 3242, 2012.

[6] P. Kidmose, L. David, U. Michael, L. R. Mike, and P. M. Danilo, “A study of evoked potentials from ear-EEG,” IEEE Trans. Biomed. Eng., vol. 60, no. 10, pp. 2824-2830, 2013.

[7] S. L. Kappel, D. Loonney, D. P. Mandic, and P. Kidmose, “A study of evoked potentials from ear-EEG,” in 36th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Chicago, IL, 2014, pp. 1686-1690.

[8] M. Nakanishi, Y. Wang, Y. T. Wang, Y. Mitsukura, and T. P. Jung, “A high-speed brain speller using steady-state visual evoked potentials,” Int. J. Neural Syst., vol. 24, no. 6, 1450019, 2014.

[9] Y. Wang, X. Gao, B. Hong, C. Jia, and S. Gao, “Brain-computer interfaces based on visual evoked potentials: feasibility of practical system design,” IEEE Eng. Med. Biol. Mag., vol. 27, no. 5, pp. 6471, 2008.

[10] X. Chen, Y. Wang, M. Nakanishi, T. P. Jung, and X. Gao, “Hybrid frequency and phase coding for a high-speed SSVEP-based BCI speller,” in 36th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Chicago, IL, 2014, pp. 3993-3996.

[11] Z. Lin, C. Zhang, W. Wu, and X. Gao, “Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs,” IEEE Trans. Biomed. Eng., vol.54, no.6, pp.1172-1176, 2007.

[12] G. Bin, X. Gao, Z. Yan, B. Hong, and S. Gao, “An online multichannel SSVEP-based brain-computer interface using a canonical correlation analysis method,” J. Neural Eng., vol. 6, no. 4, 046002, 2009.

[13] Y. Wang, M. Nakanishi, Y. T. Wang, and T. P. Jung, “Enhancing detection of steady-state visual evoked potentials using individual training data,” in 36th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Chicago, IL, 2014, pp. 3037-3040.

[14] Y. Wang, Y. -T. Wang, and T. -P. Jung, “Visual stimulus design for high-rate SSVEP,” Electron. Lett., vol. 46, no. 15, pp. 10571058, 2010.

[15] M. Nakanishi, Y. Wang, Y. -T. Wang, Y. Mitsukura and T. -P. Jung, “Generating visual flickers for eliciting robust steady-state visual evoked potentials at flexible frequencies using monitor refresh rate,” PLoS One, vol. 9, no. 6, e99235, 2014.

[16] X. Chen, Y. Wang, S. Gao, T. -P. Jung, and X. Gao, “ Filter bank canonical correlation analysis for implementing a high-speed SSVEPbased brain-computer interface,” (under review).