Publications
Speaker separation in realistic noise environments with applications to a cognitively-controlled hearing aid
March 4, 2021
Journal Article
Published in:
Neural Ntw., Vol. 140, 4 March 2021, pp. 136-47.
Summary
Future wearable technology may provide for enhanced communication in noisy environments and for the ability to pick out a single talker of interest in a crowded room simply by the listener shifting their attentional focus. Such a system relies on two components, speaker separation and decoding the listener's attention to acoustic streams in the environment. To address the former, we present a system for joint speaker separation and noise suppression, referred to as the Binaural Enhancement via Attention Masking Network (BEAMNET). The BEAMNET system is an end-to-end neural network architecture based on self-attention. Binaural input waveforms are mapped to a joint embedding space via a learned encoder, and separate multiplicative masking mechanisms are included for noise suppression and speaker separation. Pairs of output binaural waveforms are then synthesized using learned decoders, each capturing a separated speaker while maintaining spatial cues. A key contribution of BEAMNET is that the architecture contains a separation path, an enhancement path, and an autoencoder path. This paper proposes a novel loss function which simultaneously trains these paths, so that disabling the masking mechanisms during inference causes BEAMNET to reconstruct the input speech signals. This allows dynamic control of the level of suppression applied by BEAMNET via a minimum gain level, which is not possible in other state-of-the-art approaches to end-to-end speaker separation. This paper also proposes a perceptually-motivated waveform distance measure. Using objective speech quality metrics, the proposed system is demonstrated to perform well at separating two equal-energy talkers, even in high levels of background noise. Subjective testing shows an improvement in speech intelligibility across a range of noise levels, for signals with artificially added head-related transfer functions and background noise. Finally, when used as part of an auditory attention decoder (AAD) system using existing electroencephalogram (EEG) data, BEAMNET is found to maintain the decoding accuracy achieved with ideal speaker separation, even in severe acoustic conditions. These results suggest that this enhancement system is highly effective at decoding auditory attention in realistic noise environments, and could possibly lead to improved speech perception in a cognitively controlled hearing aid.
Summary
Future wearable technology may provide for enhanced communication in noisy environments and for the ability to pick out a single talker of interest in a crowded room simply by the listener shifting their attentional focus. Such a system relies on two components, speaker separation and decoding the listener's attention to...
READ MORE
Ablation analysis to select wearable sensors for classifying standing, walking, and running
December 30, 2020
Journal Article
Published in:
Sensors, Vol. 21, No. 1, 30 December 2020, 194.
Summary
The field of human activity recognition (HAR) often utilizes wearable sensors and machine learning techniques in order to identify the actions of the subject. This paper considers the activity recognition of walking and running while using a support vector machine (SVM) that was trained on principal components derived from wearable sensor data. An ablation analysis is performed in order to select the subset of sensors that yield the highest classification accuracy. The paper also compares principal components across trials to inform the similarity of the trials. Five subjects were instructed to perform standing, walking, running, and sprinting on a self-paced treadmill, and the data were recorded while using surface electromyography sensors (sEMGs), inertial measurement units (IMUs), and force plates. When all of the sensors were included, the SVM had over 90% classification accuracy using only the first three principal components of the data with the classes of stand, walk, and run/sprint (combined run and sprint class). It was found that sensors that were placed only on the lower leg produce higher accuracies than sensors placed on the upper leg. There was a small decrease in accuracy when the force plates are ablated, but the difference may not be operationally relevant. Using only accelerometers without sEMGs was shown to decrease the accuracy of the SVM.
Summary
The field of human activity recognition (HAR) often utilizes wearable sensors and machine learning techniques in order to identify the actions of the subject. This paper considers the activity recognition of walking and running while using a support vector machine (SVM) that was trained on principal components derived from wearable...
READ MORE
Ankle torque estimation during locomotion from surface electromyography and accelerometry
November 29, 2020
Conference Paper
Published in:
2020 8th IEEE Intl. Conf. on Biomedical Robotics and Biomechatronics, BioRob, 29 November - 1 December 2020.
Summary
Estimations of human joint torques can provide quantitative, clinically valuable information to inform patient care, plan therapy, and assess the design of wearable robotic devices. Standard methods for estimating joint torques are limited to laboratory or clinical settings since they require expensive equipment to measure joint kinematics and ground reaction forces. Wearable sensor data combined with neural networks may offer a less expensive and obtrusive estimation method.We present a method of mapping joint torque estimates obtained from motion capture and ground reaction forces to wearable sensor data. We use several different neural networks to learn the torque mapping for the ankle joints during standing, walking, running, and sprinting. Our results show that neural networks that consider time (recurrent and long short-term memory networks) outperform feedforward network architectures, producing results in the range of 0.005-0.008 N m/kg mean squared error (MSE) when compared to the inverse dynamics model on which it was trained. As a point of reference, the typical measurement errors from inverse dynamics models are in the range of 0.0004-0.0064 N m/kg MSE. Errors tended to increase with locomotion speed, with the highest errors during sprinting and the lowest during standing or walking. Future work may investigate model generalizability across sensor placements, subjects, locomotion variants, and usage duration. The proposed method relies on learning from a motion capture dataset, but once the model is built, the method uses wearable sensors that enable torque estimation without the motion capture data. These methods also have potential uses for the design and testing of wearable robotic systems outside of a laboratory environment.
Summary
Estimations of human joint torques can provide quantitative, clinically valuable information to inform patient care, plan therapy, and assess the design of wearable robotic devices. Standard methods for estimating joint torques are limited to laboratory or clinical settings since they require expensive equipment to measure joint kinematics and ground reaction...
READ MORE
Ultrasound and artificial intelligence
November 20, 2020
Book Chapter
Published in:
Chapter 8 in Machine Learning in Cardiovascular Medicine, 2020, pp. 177-210.
Topic:
R&D area:
R&D group:
Summary
Compared to other major medical imaging modalities such as X-ray, computed tomography (CT), and magnetic resonance imaging, medical ultrasound (US) has unique attributes that make it the preferred modality for many clinical applications. In particular, US is nonionizing, portable, and provides real-time imaging, with adequate spatial and depth resolution to visualize tissue dynamics. The ability to measure Doppler information is also important, particularly for measuring blood flows. The small size of US transducers is a key attribute for intravascular applications. In addition, accessibility has been increased with the use of portable US, which continues to move toward a smaller footprint and lower cost. Nowadays, some US probes can even be directly connected to a phone or tablet. On the other hand, US also has unique challenges, particularly in that image quality is highly dependent on the operator’s skill in acquiring images based on the proper position, orientation, and probe pressure. Additional challenges that further require operator skill include the presence of noise, artifacts, limited field of view, difficulty in imaging structures behind bone and air, and device variability across manufacturers. Sonographers become highly proficient through extensive training and long experience, but high intra- and interobserver variability remains. This skill dependence has limited the wider use of US by healthcare providers who are not US imaging specialists. Recent advances in machine learning (ML) have been increasingly applied to medical US (Brattain, Telfer, Dhyani, Grajo, & Samir, 2018), with a goal of reducing intra- and interobserver variability as well as interpretation time. As progress toward these goals is made, US use by nonspecialists is expected to proliferate, including nurses at the bedside or medics in the field. The acceleration in ML applications for medical US can be seen from the increasing number of publications (Fig. 8.1) and Food and Drug Administration (FDA) approvals (Table 8.1) in the past few years. Fig. 8.1 shows that cardiovascular applications (spanning the heart, brain and vessels) have received the most attention, compared to other organs. Table 8.1 shows that pace of US FDA-cleared artificial intelligence (AI) products that combine AI and ultrasound is accelerating. Of note, many of the products have been approved over the last couple of years. Companies such as Butterfly Network (Guilford, CT) have also demonstrated AI-driven applications for portable ultrasound and more FDA clearances are expected to be published. The goals of this chapter are to highlight the recent progress, as well as the current challenges and future opportunities. Specifically, this chapter addresses topics such as the following: (1) what is the current state of machine learning for medical US application, both in research and commercially; (2) what applications are receiving the most attention and have performance improvements been quantified; (3) how do ML solutions fit in an overall workflow; and (4) what open-source datasets are available for the broader community to contribute to progress in this field. The focus is on cardiovascular applications (Section Cardiovascular/echocardiography), but common themes and differences for other applications for medical US are also summarized (Section Breast, liver, and thyroid ultrasound). A discussion is offered in Discussion and outlook section.
Summary
Compared to other major medical imaging modalities such as X-ray, computed tomography (CT), and magnetic resonance imaging, medical ultrasound (US) has unique attributes that make it the preferred modality for many clinical applications. In particular, US is nonionizing, portable, and provides real-time imaging, with adequate spatial and depth resolution to...
READ MORE
Image processing pipeline for liver fibrosis classification using ultrasound shear wave elastography
October 1, 2020
Journal Article
Published in:
Ultrasound in Med. & Biol., Vol. 46, No. 10, October 2020, pp. 2667-2676.
Topic:
R&D area:
Summary
The purpose of this study was to develop an automated method for classifying liver fibrosis stage >=F2 based on ultrasound shear wave elastography (SWE) and to assess the system's performance in comparison with a reference manual approach. The reference approach consists of manually selecting a region of interest from each of eight or more SWE images, computing the mean tissue stiffness within each of the regions of interest and computing a resulting stiffness value as the median of the means. The 527-subject database consisted of 5526 SWE images and pathologist-scored biopsies, with data collected from a single system at a single site. The automated method integrates three modules that assess SWE image quality, select a region of interest from each SWE measurement and perform machine learning-based, multi-image SWE classification for fibrosis stage >=F2. Several classification methods were developed and tested using fivefold cross-validation with training, validation and test sets partitioned by subject. Performance metrics were area under receiver operating characteristic curve (AUROC), specificity at 95% sensitivity and number of SWE images required. The final automated method yielded an AUROC of 0.93 (95% confidence interval: 0.90-0.94) versus 0.69 (95% confidence interval: 0.65-0.72) for the reference method, 71% specificity with 95% sensitivity versus 5% and four images per decision versus eight or more. In conclusion, the automated method reported in this study significantly improved the accuracy for >=F2 classification of SWE measurements as well as reduced the number of measurements needed, which has the potential to reduce clinical workflow.
Summary
The purpose of this study was to develop an automated method for classifying liver fibrosis stage >=F2 based on ultrasound shear wave elastography (SWE) and to assess the system's performance in comparison with a reference manual approach. The reference approach consists of manually selecting a region of interest from each...
READ MORE
Human balance models optimized using a large-scale, parallel architecture with applications to mild traumatic brain injury
September 22, 2020
Conference Paper
Published in:
2020 IEEE High Performance Extreme Computing Conf., HPEC, 22-24 September 2020.
Topic:
R&D group:
Summary
Static and dynamic balance are frequently disrupted through brain injuries. The impairment can be complex and for mild traumatic brain injury (mTBI) can be undetectable by standard clinical tests. Therefore, neurologically relevant modeling approaches are needed for detection and inference of mechanisms of injury. The current work presents models of static and dynamic balance that have a high degree of correspondence. Emphasizing structural similarity between the domains facilitates development of both. Furthermore, particular attention is paid to components of sensory feedback and sensory integration to ground mechanisms in neurobiology. Models are adapted to fit experimentally collected data from 10 healthy control volunteers and 11 mild traumatic brain injury volunteers. Through an analysis by synthesis approach whose implementation was made possible by a state-of-the-art high performance computing system, we derived an interpretable, model based feature set that could classify mTBI and controls in a static balance task with an ROC AUC of 0.72.
Summary
Static and dynamic balance are frequently disrupted through brain injuries. The impairment can be complex and for mild traumatic brain injury (mTBI) can be undetectable by standard clinical tests. Therefore, neurologically relevant modeling approaches are needed for detection and inference of mechanisms of injury. The current work presents models of...
READ MORE
Sensorimotor conflict tests in an immersive virtual environment reveal subclinical impairments in mild traumatic brain injury
September 8, 2020
Journal Article
Published in:
Sci. Rep., Vol. 10, 2020, Article No. 14773.
Topic:
R&D group:
Summary
Current clinical tests lack the sensitivity needed for detecting subtle balance impairments associated with mild traumatic brain injury (mTBI). Patient-reported symptoms can be significant and have a huge impact on daily life, but impairments may remain undetected or poorly quantified using clinical measures. Our central hypothesis was that provocative sensorimotor perturbations, delivered in a highly instrumented, immersive virtual environment, would challenge sensory subsystems recruited for balance through conflicting multi-sensory evidence, and therefore reveal that not all subsystems are performing optimally. The results show that, as compared to standard clinical tests, the provocative perturbations illuminate balance impairments in subjects who have had mild traumatic brain injuries. Perturbations delivered while subjects were walking provided greater discriminability (average accuracy ≈ 0.90) than those delivered during standing (average accuracy ≈ 0.65) between mTBI subjects and healthy controls. Of the categories of features extracted to characterize balance, the lower limb accelerometry-based metrics proved to be most informative. Further, in response to perturbations, subjects with an mTBI utilized hip strategies more than ankle strategies to prevent loss of balance and also showed less variability in gait patterns. We have shown that sensorimotor conflicts illuminate otherwise-hidden balance impairments, which can be used to increase the sensitivity of current clinical procedures. This augmentation is vital in order to robustly detect the presence of balance impairments after mTBI and potentially define a phenotype of balance dysfunction that enhances risk of injury.
Summary
Current clinical tests lack the sensitivity needed for detecting subtle balance impairments associated with mild traumatic brain injury (mTBI). Patient-reported symptoms can be significant and have a huge impact on daily life, but impairments may remain undetected or poorly quantified using clinical measures. Our central hypothesis was that provocative sensorimotor...
READ MORE
Estimating sedentary breathing rate from chest-worn accelerometry from free-living data
July 20, 2020
Conference Paper
Published in:
42nd Annual Intl. Conf. IEEE Engineering in Medicine and Biology Society, EMBC, 20-24 July 2020.
Topic:
R&D area:
R&D group:
Summary
Breathing rate was estimated from chest-worn accelerometry collected from 1,522 servicemembers during training by a wearable physiological monitor. A total of 29,189 hours of training and sleep data were analyzed. The primary purpose of the monitor was to assess thermal-work strain and avoid heat injuries. The monitor design was thus not optimized to estimate breathing rate. Since breathing rate cannot be accurately estimated during periods of high activity, a qualifier was applied to identify sedentary time periods, totaling 8,867 hours. Breathing rate was estimated for a total of 4,179 hours, or 14% of the total collection and 47% of the sedentary total, primarily during periods of sleep. The breathing rate estimation method was compared to an FDA 510(K)-cleared criterion breathing rate sensor (Zephyr, Annapolis MD, USA) in a controlled laboratory experiment, which showed good agreement between the two techniques. Contributions of this paper are to: 1) provide the first analysis of accelerometry-derived breathing rate on free-living data including periods of high activity as well as sleep, along with a qualifier that effectively identifies sedentary periods appropriate for estimating breathing rate; 2) test breathing rate estimation on a data set with a total duration that is more than 60 times longer than that of the largest previously reported study, 3) test breathing rate estimation on data from a physiological monitor that has not been expressly designed for that purpose.
Summary
Breathing rate was estimated from chest-worn accelerometry collected from 1,522 servicemembers during training by a wearable physiological monitor. A total of 29,189 hours of training and sleep data were analyzed. The primary purpose of the monitor was to assess thermal-work strain and avoid heat injuries. The monitor design was thus...
READ MORE
Predicting cognitive load and operational performance in a simulated marksmanship task
July 3, 2020
Conference Paper
Published in:
Front. Human Neurosci., 3 July 2020.
Summary
Modern operational environments can place significant demands on a service member's cognitive resources, increasing the risk of errors or mishaps due to overburden. The ability to monitor cognitive burden and associated performance within operational environments is critical to improving mission readiness. As a key step toward a field-ready system, we developed a simulated marksmanship scenario with an embedded working memory task in an immersive virtual reality environment. As participants performed the marksmanship task, they were instructed to remember numbered targets and recall the sequence of those targets at the end of the trial. Low and high cognitive load conditions were defined as the recall of three- and six-digit strings, respectively. Physiological and behavioral signals recorded included speech, heart rate, breathing rate, and body movement. These features were input into a random forest classifier that significantly discriminated between the low- and high-cognitive load conditions (AUC=0.94). Behavioral features of gait were the most informative, followed by features of speech. We also showed the capability to predict performance on the digit recall (AUC = 0.71) and marksmanship (AUC = 0.58) tasks. The experimental framework can be leveraged in future studies to quantify the interaction of other types of stressors and their impact on operational cognitive and physical performance.
Summary
Modern operational environments can place significant demands on a service member's cognitive resources, increasing the risk of errors or mishaps due to overburden. The ability to monitor cognitive burden and associated performance within operational environments is critical to improving mission readiness. As a key step toward a field-ready system, we...
READ MORE
Integrative sensor networks, informatics, and modeling for precision and preventative medicine
July 1, 2020
Journal Article
Published in:
IEEE J. Biomed. Health, Vol. 24, No. 7, July 2020, pp. 1858-1859.
Topic:
R&D area:
R&D group:
Summary
The topics of integrative sensor networks, informatics and modeling bring together the tightly coupled and rapidly developing fields of biomedical and health informatics and body sensor networks. Biomedical and health informatics encompasses methods to extract and communicate information from data in order to impact health, healthcare, life sciences and biomedicine. Body sensor networks provide one means to measure the needed data, through continuous monitoring in both clinical and free-living environments. Developments in these areas were highlighted at two co-located conferences: the 2019 IEEE-EMBS International Conferences on Biomedical and Health Informatics (BHI'19) and Wearable and Implantable Body Sensor Networks (BSN'19). BHI and BSN are long-standing IEEE EMBS conferences that provide a forum for researchers and leaders from academia, government and industry to share technical advances and new initiatives in these important areas. Through an open call for this special issue, eleven papers have been included for publication. The majority were presented in an initial form at the 2018 or 2019 BHI and BSN conferences. Nine of the papers were selected through a rigorous peer review. In addition, two keynote speakers from BHI'19 and BSN'19 have provided short position papers.
Summary
The topics of integrative sensor networks, informatics and modeling bring together the tightly coupled and rapidly developing fields of biomedical and health informatics and body sensor networks. Biomedical and health informatics encompasses methods to extract and communicate information from data in order to impact health, healthcare, life sciences and biomedicine...
READ MORE