BCI technology based on brain waves data has varied potential uses
Original Article (behind paywall) featured in IISE Magazine by the IISE. All photo credits go to IISE.
The human body is an important source of big data. Biometric data can be collected to track health status and actively improve quality of life. Though data analysis has recently been actively used as a tool for industrial innovation, biometric data analysis seems to approach the essence of engineering in terms of developing technologies that benefit human beings.
In April 2022, NeuraLink, a company co-founded by Tesla CEO Elon Musk, released a video of a monkey playing a game with an implanted chip in its brain. In the video, a chip and more than 2,000 electrodes are implanted in a 9-year-old monkey’s brain and a game similar to Pong is played using a joystick. All the while, the monkey was given a banana smoothie as a reward.
During the learning process, brain wave data was wirelessly collected from the chip to train a machine-learning algorithm. Specifically, the monkey was performing a motor imagery task where it imagines moving its hand. Afterward, even if the joystick is disconnected, the game is played only through the monkey’s brain activity as this imaginary motor movement is repeated.
This is a demonstration of brain-computer interface (BCI) technology that allows communication with computers through brain activity. Although NeuraLink’s video is not technically new, the flashy video quickly went viral, sparking the interest of the media and industry alike (see Figure 1).
Our research team, with the support of the South Korean government, recently conducted deep learning-based brain wave data analysis research to investigate how to analyze electroencephalogram (EEG) data for BCI applications. Through this project, we investigated what deep-learning technologies are being used in the field and the explainable artificial intelligence (XAI) technique for EEG analysis.
Brain-computer interface (BCI)
BCI refers to technology that analyzes brain activity to recognize the user’s state, identifies intentions and provides a variety of information. It refers to techniques that connect a human brain and a computer to interpret and utilize brain nerve signals in real time or enhance human capabilities.
The types of brain waves can be classified according to the frequency and amplitude observed. Wavelengths of EEG signals range from 0-50 hertz and are conveniently classified by frequency into delta waves (δ, 0.2~3.99 Hz), theta waves (θ, 4~7.99 Hz), alpha waves (α, 8~12.99 Hz), beta waves (β, 13~29.99 Hz) and gamma waves (γ, 30~50 Hz).
EEG signals can be used in various fields, and the analysis purposes can be divided into clinical applications for pathology (sleep state, etc.), epilepsy, Alzheimer’s disease, dementia, stroke and schizophrenia, etc.; monitoring cognitive load, emotion and physical exercise; BCI applications for operational intentions, mental work and voice interpretation; personal tendencies and identification purposes; musical semantics and more exotic applications.
Modern BCI technology was launched at UCLA in 1973. Since then many studies have explored the relationship between muscle movement in humans and electrical activity in the motor cortex of the brain. Some studies in particular investigated severe physical disabilities like Lou Gehrig’s disease, spinal injuries and cerebral palsy. Recently, BCI research has become more active due to the miniaturization of devices, development of more precise computer and sensor technology, biomaterials suitable for humans and deep learning-based data analysis.
Methods for measuring brain activity of a user can largely be divided into invasive and noninvasive. Invasive methods include spike array methods, where the skull is opened for measuring brain activity directly, electrocardiography (ECoG) to measure brain signals from the cerebral cortex or surgically implanting a chip, like the monkey from NeuraLink. Noninvasive method measures brain nervous system activity outside the scalp and is divided into functional magnetic imaging resonance (fMRI), functional near-infrared spectroscopy (FNIRS), positron emission tomography (PET) and single-photon emission computed tomography (SPEC) to measure hemodynamic signals, and EEG or magnetoencephalography (MEG) for electrical signals. The later two noninvasive methods are used by attaching electrodes directly to the head or wearing of an electrode cap.
EEG, despite its popularity, actually has a relatively low spatial resolution among brain wave analysis techniques, but it is the cheapest and has high temporal resolution without straining the human body, making it the common data analysis tool for BCI studies. Because of its relative simplicity, EEG research has a long history. In the 1870s, British physiologist Richard Caton first demonstrated the electrical currents in brains of mammals such as rabbits and monkeys. In 1924, German psychiatrist Hans Berger first demonstrated the human EEG.
In general, neural activity of the brain is measured through the generated voltage in response to some stimulus. By analyzing voltage patterns, researchers seek to work backward and infer the nature of the stimulus.
EEG is fundamentally analyzed by measuring the potential electrical difference between a particular reference electrode and the remaining electrodes. The electrodes are distributed evenly on the surface of the scalp, usually using the “10-20 Electrode Placement Method” enacted by the International Federation of Societies for Electroencephalography and Clinical Neurophysiology in 1958.
The biggest issue of EEG analysis is that classification accuracy of software is still low, and many advanced data analysis techniques have been applied to overcome this shortcoming. In the past, traditional machine learning techniques such as linear discriminant analysis (LDA), support vector machines (SVM) and elastic net have been used to improve performance.
Convolutional Neural Network (CNN), Autoencoder, Recurrent Neural Network (RNN), Generative Adversarial Networks (GAN) and other deep learning techniques have been introduced to further improve EEG performance. According to the analysis of a review paper (Figure 2) of the deep learning network structures used in EEG analysis, 40% are CNN models, 13% are RNN models, 13% are Autoencoder models and other active models include feedforward neural network (FNN) or CCN+RNN mixed structures.
Deep learning-based BCI analysis
Though there have been many advances recently in EEG classification, deep learning techniques in particular have shown excellent performance in EEG data classification operations such as motor imagery and emotion recognition.
Figure 3 shows the overall framework for developing deep learning technology for BCI classification, based on the process used by our research team. It consists of EEG data collection and preprocessing, building and training deep learning models and BCI classification and interpretation.
EEG data includes more than just brain activity, though, also containing all of the noise from muscle and eye movements (Figure 4A). Therefore, we extracted only the frequency region 0.1 to 40Hz (delta to gamma) to filter and use (Figure 4B). In addition, to increase resolution of low-band frequencies and maintain a uniform sampling rate for each brain wave, all brain waves are down-sampled to 125 Hz. As a result, we obtain “clean” EEG data that can be easily used for machine learning (Figure 4C).
This filtered and down-sampled data may then be used as an input for deep learning models. Various CNN-based deep learning models such as DeepConvNet, ShallowConvNet and EEGNet have large bodies of literature supporting their use. Among those, we will briefly describe the EEGNet model (Figure 5).
EEGNet is a CNN-based, deep learning model developed to analyze EEG data and is designed around minimizing the number of learning parameters. This is because the amount of EEG data available for training is often much smaller than the amount of data traditionally used to train CNNs, such as for image processing. This way, a small number of EEG data samples can still produce significant performance.
As shown in Figure 5, the pattern extraction result passes through the Conv2D layer, the DepthwiseConv2D layer and the SeparableConv2D layer and is transferred to the classification layer, completing the EEG classification. The Conv2D layer applies several two-dimensional convolution filters in parallel to extract a time-series pattern from the EEG signal. The DepthwiseConv2D layer is a two-dimensional convolution layer that extracts a pattern over several channels from the Conv2D results, and the SeparableConv2D layer is a convolution layer that again intersects a feature map between time-series and between channels generated by the two previous Conv2D layers. The last classification layer is a generic, fully connected layer for performing multiclass classification.
The data set of the BCI Competition IV contains four imaginary motor movement tasks: the left hand, right hand, foot and tongue. This was chosen as our input data; like EEGNet, there is a large amount of literature supporting its use.
Our experimental results are shown in Figure 6. The accuracies of the CNN-based deep learning models shallow ConvNet and EEGNet were better than those of the traditional models like LDA, SVM and elastic net. Furthermore, the performance of all the classification models can be improved by performing data augmentation (generating “fake” EEG samples from the real data) using a GAN technique called cWGAN-GP, developed by our team, rather than directly inputting original EEG data alone. When deep learning is not used, elastic net had the best accuracy at 70%, but after data enhancement through cWGAN-GP, EEGNet classification accuracy was improved to 82%. Although the accuracy is still lacking for direct use in products, as this is close to a 1 in 5 error rate, deep learning has still enabled a great improvement in accuracy.
Explainable AI technology for BCI
Deep learning models, often applied in areas such as image classification, voice recognition, failure detection and product recommendation, often show the best performance but are difficult to understand due to their complicated network structures. The reasoning behind the model’s decision is often unclear. To counter this, various XAI techniques have recently emerged for enhanced interpretation of deep learning models. However, EEG data is multichannel, amorphous (nonstationary), nonlinear and has a low signal-to-noise ratio, making analysis of the deep learning model more difficult than others.
In our study, among the many XAI techniques available, the example-based XAI technique was developed to analyze which point of time and EEG channel affects the BCI performance more (or less) than others. A visualization of temporal and spatial significance of EEG experimental data can be created by applying an XAI technique called DeepSHAP to an EEGNet model.
On the other hand, a spatial, temporal and frequency perturbation technique can be applied to the training data to derive an influence score on the test sample by each training sample in EEGNet. For influence evaluation, techniques such as influence function, representer point selection or a TracIn can be applied. By applying this example-based XAI technique, we could confirm that noise at specific time points and electrode locations was negatively impacting the classifier’s accuracy, and that BCI accuracy could then be improved through further experimentation and more informed data handling.
In addition, these techniques function just as well with a feature-based XAI and can identify the location and time of the brain signals that significantly affect the deep learning-based BCI classification performance. The results of training on the dataset IVa of the BCI Competition III show a binary-classification of right hand and the foot motor imagery by EEGNet. When evaluating hand and foot motor imagery, electrode CP3 was shown to be the most important.
As described above, the channel and time points of the brain activity, which play an important role for BCI classification, are presented to the BCI experts by our methodology to be used for the development of AI technology and further improving BCI performance.
BCI creating the ‘Internet of Thinks’
The advancement of brain-computer interface technology has opened up vast potential for how humans can communicate and connect through artificial intelligence technology.
The technology brings together the fields of neuroscience, engineering and data science. One of the main uses being pioneered by numerous researchers would allow people with speech and motor disabilities to communicate and manipulate objects through their brain waves.
In a recent study published by Nature Communications and cited in an article in The New York Times, researchers at the University of Tübingen and the Wyss Center for Bio and Neuroengineering in Geneva, Switzerland, were able to help a man stricken with Lou Gehrig’s disease or ALS to translate his thoughts into letters through a synthetic voice connection.
The three main types of BCI are noninvasive, partially invasive and invasive, categorized by their physical effects on the user’s brain. Noninvasive involves reading brain waves via electroencephalogram (EEG) sensors attached to the skull. Partially invasive involves attaching sensors inside the skull but outside the brain to read electric pulses; invasive includes inserting electrodes fully under the scalp.
For more on North Carolina State’s Brain-Computer Interface and Neuroergonomics Lab, visit www.ise.ncsu.edu/bci
BCI technology based on EEG data can be seen as the next-generation interface in various fields such as neurostimulation therapy, handicapped aid, console games and virtual/augmented reality. Here we describe a BCI technique for analyzing EEG to determine human intention, a process of applying a CNN-based deep learning model for analyzing EEG data and an XAI technique for understanding the resulting model.
Results derived from our XAI technique are difficult for the public to understand. This is because some neuroscience knowledge and expertise in EEG are almost always required to understand such a complex system. Furthermore, BCI experts also need to understand AI technologies such as CNN
and XAI, something that is not a simple task. To support this, experts from various fields must cooperate to create new technology in new industry fields, requiring technology exchange and education. In this regard, domestic and foreign institutions should participate to activate government-supported projects that support this convergence of different human resources and knowledge.
Musk founded the BCI startup NeuraLink to create an augmented human capable of countering the threat of AI. Although the invasive BCI proposed by NeuraLink will surely pioneer early markets with its accuracy, noninvasive BCI offers a safer and more convenient technology in the future, provided researchers continue the development of data analysis technology.
Is a sci-fi era on the horizon in which people play music, games and move only by thinking? The answer rests with academia and industries studying BCI technology and biometrics.
Acknowledgement
This work was supported by the Ministry of Science, ICT & Future Planning, Korea, under the High-Potential Individuals Global Training Program (No. 2020001560) supervised by the Institute for Information & Communications Technology Planning & Evaluation.
Chang S. Nam is a professor at the Edward P. Fitts Department of Industrial and Systems Engineering at North Carolina State University. He is also an associated faculty in the UNC/NCSU Joint Department of Biomedical Engineering, Psychology and UNC-CH Brain Imaging Centner. His research interests center around brain-computer interfaces, social cognitive and affective neuroscience, human-robot interaction and human-centered explainable AI. He is a senior IISE member and serves as editor-in-chief of the journal Brain-Computer Interfaces. Contact him at csnam@ncsu.edu.
Jae-Yoon Jung is a professor in the Department of Industrial and Management Systems Engineering/Department of Big Data Analytics at Kyung Hee University in South Korea (KHU) since 2007. He is also director of the Industrial Artificial Intelligence Laboratory at KHU. His research interests include industrial artificial intelligence, machine learning and process mining.
Sangwon Lee is an associate professor in the Department of Interaction Science/Human-Artificial Intelligence Interaction/Applied Artificial Intelligence at Sungkyunkwan University in South Korea. He earned his Ph.D. in industrial engineering at the Pennsylvania State University in 2010. His research interests include human-computer interaction, user experience, affective computing and XAI interface.
Sanghyun Choo is a Ph.D. candidate in the Edward P. Fitts Department of Industrial and Systems Engineering at North Carolina State University. His research interests include machine learning, deep learning, explainable AI, human-in-the-loop and brain-computer interface.
Zachary Traylor is a Ph.D. student in the Brain-Computer Interface and Neuroergonomics Lab of the Edward P. Fitts Department of Industrial and Systems Engineering at North Carolina State University. His research has focused on the possibilities of brain-to-brain interfacing, developing higher performing brain-computer interfaces and investigating neural signals and their potential neuroergonomic applications.
Hoonseok Park is a Ph.D. candidate in the Department of Big Data Analytics at Kyung Hee University in South Korea. His research interests are explainable artificial intelligence and machine learning applications for spatiotemporal data in manufacturing and healthcare industry.
Donghyun Park is a Ph.D. candidate in the Department of Big Data Analytics in the Kyung Hee University in South Korea. His main areas of research are machine learning, artificial intelligence and big data analytics. He is also studying the explainable artificial intelligence methods for applying them to various research fields.
Sangyeon Kim is a lecturer at Sungkyunkwan University in South Korea. He earned a Ph.D. in interaction science from Sungkyunkwan University in 2022. His research interests include human-computer interaction, human factor, human-centered AI, intelligent user interface and explainable AI interface.