The aim of this study is to propose a method for analyzing measured signal obtained from functional Near-Infrared Spectroscopy (fNIRS), which is applicable for neuroimaging studies for car drivers. We developed a signal processing method by multiresolution analysis (MRA) based on discrete wavelet transform. Statistical group analysis using Z-score is conducted after the extraction of task-related signal using MRA. Brain activities of subjects with different level of mental calculation are measured by fNIRS and fMRI. Results of mental calculation with nine subjects by using fNIRS and fMRI showed that the proposed methods were effective for the evaluation of brain activities due to the task. Finally, the proposed method is applied for evaluating brain function of car driver with and without adaptive cruise control (ACC) system for demonstrating the effectiveness of the proposed method. The results showed that frontal lobe was less active when the subject drove with ACC.
In recent years, various driving assistance systems have been developed to ensure safety by reducing driver workloads. Examples include the Adaptive Cruise Control (ACC) system, which maintains a safe distance between the driver’s vehicle and the vehicle ahead of it and the lane-keeping assistance system, which keeps the car in a lane through steering support.
However, it is also possible that while driver workload is reduced, the driver’s attention is also reduced, resulting in unexpected accidents. Therefore, it is necessary to examine driver workload from the viewpoints of cognitive engineering and human physiology. It is necessary to clarify the relationship between driver workload and brain activity, which includes recognition and judgment. It is then necessary to evaluate the driver’s attention and to clarify the relationship between brain activity and driving performance.
A small number of neuroimaging studies using driving simulator examine brain activity in car driving. In these studies [
Near-Infrared Spectroscopy (NIRS) has gained attention in recent years [
NIRS can detect the hemodynamic of the brain in real time while the subject is moving. Therefore, brain activity can be measured in various environments. Recent research has used functional Near-Infrared Spectroscopy (fNIRS) to measure brain activity of train driver [
Various arguments have focused on interpretation of signals obtained from fNIRS, and no uniform signal-processing method has yet been established. Averaging and base-line correction are conventional signal-processing methods used for the fNIRS signal. These methods require block design, an experimental technique that involves repeating the same stimuli (tasks) and resting multiple times in order to detect brain activation during a task. However, it has been pointed out that brain activation gradually declines when one subject repeats the same task multiple times [
Fourier analysis, which is frequently used for signal analysis, transforms information in the time domain into the frequency domain through the Fourier transform. However, time information is lost in the course of transform. As the fNIRS signal has unsteady nature, time-frequency analysis is suitable for the fNIRS signal.
The wavelet transform is an efficient method of time-frequency analysis [
In addition to this, oxygenated hemoglobin and deoxygenated hemoglobin measured in fNIRS are relative value from the beginning of measurement, which is changeable for subject and part of the brain. Thus, simple averaging of fNIRS signal should not be applied for statistical analysis. To solve this problem, we propose Z-scored fNIRS signal for statistical analysis.
The aim of the study is to propose the signal processing method suitable for fNIRS signal which is applicable for neuroimaging studies for car drivers using fNIRS. In this paper, we first describe the principle of measurement of brain activity with fNIRS. Then, we propose the discrete wavelet-based MRA to extract the task-related signal from the original fNIRS recordings. We conducted simultaneous measurement experiments with fNIRS and fMRI using mental calculation tasks to confirm the validity of the proposed method. The Z-scored fNIRS signal is proposed for statistical analysis. We show the possibility of the proposed method for evaluating driver’s brain activity in realistic driving environment.
Using near-infrared rays, fNIRS noninvasively measures changes in cerebral blood flow. Its principle of measurement, which was developed by Jöbsis [
In uniformly distributed tissue, incident light is attenuated by absorption and scattering. Therefore, the following expression, a modified Lambert-Beer law, was used:
If it is assumed that no scattering changes in brain tissue occur during activation of the brain, the change in absorption across the activation can be expressed by the following expression:
In general, the averaged path length
Mental calculation tasks, low-level task: simple one-digit addition (e.g.,
Figure
Position of optical fibers and channels in recoding fNIRS signal (Mental calculation task:
Time history of fNIRS signal in mental calculation (channel number 20).
In fNIRS analysis, it is necessary to separate noise that is related to a task from that which is not, since fNIRS measures not only the signals of brain activity during a task but also other signals, including measurement noise.
In general, changes in oxygenated hemoglobin and deoxygenated hemoglobin when the brain is activated and restored to the original state exhibit the trend illustrated in Figure
Averaging and base-line correction are conventional signal processing methods. These methods require block design, an experimental technique that involves repeating the same stimuli (tasks) and resting multiple times in order to detect brain activation during a task.
Averaging is the method by which data is averaged for each task. Randomly generated noises approach zero by averaging, and only periodical data is left. Averaging is effective when similar reactions are generated repeatedly. However, for cerebral blood flow that has large scattering of reactions to the same stimuli, the reliability of averaged signals is low, and false signals may be created. Furthermore, it is possible that even significant signals may become undetectable after averaging.
Base-line correction corrects the start point and end point of a block to zero to remove gentle trends, based on the assumption that blood flow is restored to its original state during a task block. However, because blood flow involves irregular fluctuations, the reference points are unstable. Therefore, if the whole block is corrected based on those two points alone, signals may be distorted.
Figure
Fourier analysis, which is frequently used for waveform analysis, transforms information in the time domain into information in the frequency domain through the Fourier transform. However, time information is lost in the course of transform.
Short-time Fourier transform, or windowed Fourier transform, can be used for time-frequency analysis of signals. However, the detecting capacity varies largely, depending on the setting of the window.
In contrast, wavelet transform is an efficient method of time-frequency analysis. It adapts the window width in time and frequency so that the window width in frequency becomes smaller when the window width in time is large, or the window width in frequency becomes larger when the window width in time is small.
Wavelet transform expresses the local shape of the waveform to be analyzed,
Continuous wavelet transform is given by
MRA decomposes signals into a tree structure using the discrete wavelet transform. MRA decomposes the object time-series signals,
Signal
The coefficient of the approximated component is calculated by
In the wavelet transform, the choice of a mother wavelet
Figure
Reconstructed signals are illustrated in Figure
Schematic hemoglobin concentration change due to neural activity.
Results of signal processing with base-line correction and denoising.
Decomposition of fNIRS signal using MRA.
Results of signal processing using MRA.
To confirm the validity of the signal processing method explained in the previous section, we measured brain functions through simultaneous use of fNIRS and fMRI.
To measure brain activity under workload, we used the workload of mental calculation. Mental calculation tasks were set to low, medium, and high levels as follows:
Low-level task: simple one-digit addition (e.g.,
Medium-level task: one-digit addition of three numbers (e.g.,
High-level task: subtraction and division with decimal fraction (e.g.,
The design of the experiment is presented in Figure
Design of experiment.
A 28 seconds-task consisted of 14 questions at 2 seconds-intervals for the low level, 10 questions at 2.8 seconds-intervals for the intermediate level, or two questions at 14 seconds-intervals for the high level. The subject answered the questions displayed on the PC screen without speaking. During the 36 seconds-rest time, the subject rested while steadily gazing at the cross mark displayed on the PC screen.
The brain activity in the prefrontal lobe was measured using fNIRS and fMRI simultaneously. fNIRS data were collected on OMM-3000, Shimadzu Corporation, Japan, in MRI scanner.
fMRI data (3 mm thickness, 40 slices) were collected on Siemens Symphony 1.5 T (T2*-weighted gradient-echo sequence,
fMRI data were preprocessed using Statistical Parametric Mapping (SPM99, Welcome Department of Imaging Neuroscience, UK) Normalized contrast images were smoothed with an isotropic Gaussian kernel (
The subjects were nine healthy men and women. The arrangement of optical fiber units and measurement positions is shown in Figure
Figure
Hemoglobin concentration changes in frontal.
Figure
Decomposition of fNIRS signal using MRA (channel number 26).
Reconstructed signals of channel number 26 are illustrated in Figure
Comparison of original signal and task-related signal (channel number 26).
Original signal
Reconstructed fNIRS signal
Figure
Functional brain imaging by fMRI and fNIRS.
The fNIRS signal expresses the quantity of relative changes using the start point as the reference; however, comparisons of measurements between subjects or statistical processing of measurements of all subjects cannot be implemented using this signal as it is. Therefore, we propose a method for converting data of oxygenated hemoglobin and deoxygenated hemoglobin reconstructed by MRA into Z-scores using the following expression, so that the mean value is 0 and standard deviation is 1:
Figure
Results of group analysis of fNIRS signals for nine subjects.
Averaged reconstructed fNIRS signal (channel number 26)
Brain activity imaged by averaged fNIRS signals
Functional brain imaging by fMRI and fNIRS (Group analysis for nine subjects).
In this experiment, the workload of each subject was measured using the Japanese version of NASA-TLX to evaluate the correlations between the workload of mental calculation tasks and the objective evaluation with fNIRS. NASA-TLX is composed of six measures: mental requirements, physical requirements, temporal demand, work performance, effort, and frustration. Before workload evaluation, the subject performed one-to-one comparisons of the importance of elements of the workload involved in task performance.
The weight of each measure was based on the number of times an element was selected as more important during 15 one-to-one comparisons. When evaluating the workload of each task, the subject placed a mark at the appropriate position on the segment drawn between both extremes for each of the six measures.
A Weighted Workload (WWL) score was obtained by reading the position of each evaluation mark on a scale of 0 to 100 and multiplying it by the weight for each measure determined by one-to-one comparison, and then averaging all the products.
Figure
Workload evaluation by NASA-TLX.
Figure
Maximum gradient of fNIRS signal during the task.
Drivers of motor vehicles obtain visual information on the surrounding environment, recognize and judge that information suitably, and then control their vehicle through steering wheel, accelerator, and brake pedal operations. Human brain activity functions to control all of these processes. In situations where it is necessary to predict unexpected danger, it is thought that a driver's brain activity strengthens the cognition function by spontaneously raising the level of attention. In the course of developing driver support systems, it is important to have a clear understanding of human brain activity in such driving situations.
To verify that the driving workload reduction of Adaptive Cruise Control (ACC) could be evaluated from brain activity, we conducted an experiment that involved the use of a driving simulator to follow a vehicle (Figure
Speed pattern of leading car.
Main specification of the driving simulator is as follows: Dimension: 2440 mm (
Driving tests were conducted under two conditions: one involved following a vehicle by utilizing ACC, and the other involved following a vehicle while driving without ACC. The subject performed practice runs to become somewhat skillful in handling the driving simulator and then drove two times under each condition. Brain activity during one condition was compared with that during the other condition.
Brain activity in the frontal lobe was measured using fNIRS. Figure
Experiment with driving simulator (driver follows the proceeding vehicle with and without ACC).
Position of optical fibers and channels (driving task:
The fNIRS signals include signals that are not related to brain activity (e.g., noise of the measurement instrument, influences of breathing, and changes in blood pressure). It was necessary to remove these unrelated signals to evaluate brain activity in detail. Therefore, the measured fNIRS signals were decomposed through MRA using discrete wavelet transform, and the components related to the driving task were reconstructed. Then, group analysis using Z-score was conducted for all subjects.
Figures
Result of group analysis for four drivers without ACC system.
Averaged reconstructed fNIRS signal (channel number 26)
Functional brain imaging
Result of group analysis for four drivers with ACC system.
Averaged reconstructed fNIRS signal (channel number 26)
Functional brain imaging
Figure
Figure
Signal processing method to extract the task-related components with multiresolution analysis (MRA) based on discrete wavelet transform is proposed for fNIRS. Then the integration of data of multiple subjects using Z-scores is developed for statistical group analysis.
The brain activity of the subject who was given workload by different levels of mental calculation tasks was measured with fNIRS and fMRI. The fNIRS images constructed with the proposed method agree to fMRI images in different workload levels. Those results show that the proposed method is effective for evaluation brain activity measured by fNIRS.
The changes in brain activity in connection with workload were compared with the subjective evaluation of workload by NASA-TLX. Good correlation was observed between the brain activity detected by fNIRS and the workload scores obtained from NASA-TLX. This result indicates that it is possible to evaluate workload from the cerebral blood flow signals obtained from fNIRS.
Whether the reduction of driving workload by ACC can be evaluated from brain activity was evaluated through experiments using a driving simulator. The results revealed that while the outer portions of the frontal lobe were active in connection with driving performance when the subject drove without ACC, it indicated no activity related to driving performance with the use of ACC. These results suggest the possibility of evaluating driving assistance systems through evaluation of the driving workload from measurement of brain activity using fNIRS.
Neuroimaging studies of car drivers using fNIRS should be conducted with increased number of subjects. We cannot conclude that lowering brain activity by reducing driving workload leads to safe driving; thus, in the future, we will design and evaluate driving assistance systems that require an appropriate level of brain activity.
This work was supported by the Nihon University Multidisciplinary Research Grant for 2006 and 2007.