Multistream Browser deconstructed
Contents |
Concept of Multistream Browser
The Multistream Browser is one of the key features of Mobilab that allow users to browse and annotate data while examining synced data streams from multiple modalities simultaneously. It is an indispensable functionality to annotate data with events visually identified in video streams by experts, for example.
Solution
The proof of concept was demonstrated by MoBILAB’s Multistream Browser, which is part of the free open-source MoBILAB package developed by Alejandro Ojeda. For reference, see the following paper.
Ojeda A, Bigdely-Shamlo N, Makeig S. (2014). MoBILAB: an open source toolbox for analysis and visualization of mobile brain/body imaging data. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2014.00121
Limitations
While the Multistream Browser is a powerful tool in theory, its effectiveness is limited by the platform. In practice, Matlab’s handling of graphics and objects is not conducive to plotting video alongside time series of EEG activities. For example, when the speed of the video playback is changed or when too many EEG channel time series are plotted, the video frame rate will drop significantly. Furthermore, while object-oriented programming is fitting for processing streams from different modalities, it complicates troubleshooting for the user who is more accustomed to procedural programming and would rather modify the code to fit his or her exact needs.
Deconstructed Solution
Background. The motivation for investigating this issue was to address the limitation for a collaboration project with Sandra Loo (UCLA). They use mobile brain-body imaging (MoBI) environment to perform EEG recording from patients of chronic Tic disorder. In performing this project, experts annotate onsets of tic by watching the video, for which event-related EEG analyses will be performed.
Goal of the study and Method. Our goal was to strip the core functionality of Multistream Brower by building an alternative solution outside of MoBILAB with minimum components to test the limit of its performance, and ultimately to allow the end-user to easily choose the parameters and trade-offs pertaining to speed and smoothness of real-time operation.
Download link for the alternative solution (Clement's EEG-Video Annotation).
Results. We found that
- If playing video alone, 30 frames per second (fps) real-time performance is possible . The playback speed is based on keyboard repeat rate and higher fps is possible; the upper limit should be based on Matlab and computer performance but we did not test this.
- When combined with stream-browsing 128-ch time-series data, the frame rate per second dropped down to 9.
- When the number of channels was reduced to 16, the frame rate increased from 9 to 26 fps.
Conclusions.
- When browsing EEG and video using the generic ‘multistream browser’ solution on Matlab, there is a trade-off between speed/smoothness and the amount of data shown in the time-series browser; even when the number of channels shown is reduced to 16, the fps is less than 30.
- Also, we found that skipping frames (i.e., showing every other frame to obtain x2 speed) is less smooth. The same is true for rewinding the footage by playing the previous frame.
Acknowledgement
This project was supported by grants from the National Institutes of Health NS054124 and NS97484 (PI Sandra Loo, UCLA) and general gift of Swartz Foundation (Oldfield, NY).
This page was written by Makoto Miyakoshi and Clement Lee (04/13/2018)
