Publications
ReCANVo: A database of real-world communicative and affective nonverbal vocalizations
August 5, 2023
Journal Article
Published in:
Sci. Data, Vol. 10, No. 1, 5 August 2023, 523.
Topic:
R&D area:
R&D group:
Summary
Nonverbal vocalizations, such as sighs, grunts, and yells, are informative expressions within typical verbal speech. Likewise, individuals who produce 0-10 spoken words or word approximations ("minimally speaking" individuals) convey rich affective and communicative information through nonverbal vocalizations even without verbal speech. Yet, despite their rich content, little to no data exists on the vocal expressions of this population. Here, we present ReCANVo: Real-World Communicative and Affective Nonverbal Vocalizations - a novel dataset of non-speech vocalizations labeled by function from minimally speaking individuals. The ReCANVo database contains over 7000 vocalizations spanning communicative and affective functions from eight minimally speaking individuals, along with communication profiles for each participant. Vocalizations were recorded in real-world settings and labeled in real-time by a close family member who knew the communicator well and had access to contextual information while labeling. ReCANVo is a novel database of nonverbal vocalizations from minimally speaking individuals, the largest available dataset of nonverbal vocalizations, and one of the only affective speech datasets collected amidst daily life across contexts.
Summary
Nonverbal vocalizations, such as sighs, grunts, and yells, are informative expressions within typical verbal speech. Likewise, individuals who produce 0-10 spoken words or word approximations ("minimally speaking" individuals) convey rich affective and communicative information through nonverbal vocalizations even without verbal speech. Yet, despite their rich content, little to no data...
READ MORE
Individualized ultrasound-guided intervention phantom development, fabrication, and proof of concept
July 24, 2023
Conference Paper
Published in:
45th Annual Intl. Conf. of the IEEE Engineering in Medicine and Biology Society, EMBC, 24-27 July 2023.
Topic:
R&D area:
R&D group:
Summary
Commercial ultrasound vascular phantoms lack the anatomic diversity required for robust pre-clinical interventional device testing. We fabricated individualized phantoms to test an artificial intelligence enabled ultrasound-guided surgical robotic system (AI-GUIDE) which allows novices to cannulate deep vessels. After segmenting vessels on computed tomography scans, vessel cores, bony anatomy, and a mold tailored to the skin contour were 3D-printed. Vessel cores were coated in silicone, surrounded in tissue-mimicking gel tailored for ultrasound and needle insertion, and dissolved with water. One upper arm and four inguinal phantoms were constructed. Operators used AI-GUIDE to deploy needles into phantom vessels. Two groin phantoms were tested due to imaging artifacts in the other two phantoms. Six operators (medical experience: none, 3; 1-5 years, 2; 5+ years, 1) inserted 27 inguinal needles with 81% (22/27) success in a median of 48 seconds. Seven operators performed 24 arm injections, without tuning the AI for arm anatomy, with 71% (17/24) success. After excluding failures due to motor malfunction and a defective needle, success rate was 100% (22/22) in the groin and 85% (17/20) in the arm. Individualized 3D-printed phantoms permit testing of surgical robotics across a large number of operators and different anatomic sites. AI-GUIDE operators rapidly and reliably inserted a needle into target vessels in the upper arm and groin, even without prior medical training. Virtual device trials in individualized 3-D printed phantoms may improve rigor of results and expedite translation.
Summary
Commercial ultrasound vascular phantoms lack the anatomic diversity required for robust pre-clinical interventional device testing. We fabricated individualized phantoms to test an artificial intelligence enabled ultrasound-guided surgical robotic system (AI-GUIDE) which allows novices to cannulate deep vessels. After segmenting vessels on computed tomography scans, vessel cores, bony anatomy, and a...
READ MORE
Development of 3D-Printed Individualized Vascular Phantoms for Artificial Intelligence (AI) Enabled Interventional Device Testing
March 1, 2023
Abstract
Published in:
Journal of Vascular and Interventional Radiology 34 (3), S12.
Topic:
R&D area:
R&D group:
Summary
We developed vascular phantoms mapped from human subjects to test AI-enabled ultrasound-guided vascular cannulation. Translational device prototyping necessitates anatomically accurate models. Commercial phantoms fail to address anatomic variability. Uniformity leads to optimistic AI model and operator performance. Individualized 3D-printed vascular phantoms yield anatomically correct models optimized for AI-device testing.
Summary
We developed vascular phantoms mapped from human subjects to test AI-enabled ultrasound-guided vascular cannulation. Translational device prototyping necessitates anatomically accurate models. Commercial phantoms fail to address anatomic variability. Uniformity leads to optimistic AI model and operator performance. Individualized 3D-printed vascular phantoms yield anatomically correct models optimized for AI-device testing.
READ MORE
Dissociating COVID-19 from other respiratory infections based on acoustic, motor coordination, and phonemic patterns
January 28, 2023
Journal Article
Published in:
Sci. Rep., Vol. 13, No. 1, January 2023, 1567.
Topic:
R&D area:
R&D group:
Summary
In the face of the global pandemic caused by the disease COVID-19, researchers have increasingly turned to simple measures to detect and monitor the presence of the disease in individuals at home. We sought to determine if measures of neuromotor coordination, derived from acoustic time series, as well as phoneme-based and standard acoustic features extracted from recordings of simple speech tasks could aid in detecting the presence of COVID-19. We further hypothesized that these features would aid in characterizing the effect of COVID-19 on speech production systems. A protocol, consisting of a variety of speech tasks, was administered to 12 individuals with COVID-19 and 15 individuals with other viral infections at University Hospital Galway. From these recordings, we extracted a set of acoustic time series representative of speech production subsystems, as well as their univariate statistics. The time series were further utilized to derive correlation-based features, a proxy for speech production motor coordination. We additionally extracted phoneme-based features. These features were used to create machine learning models to distinguish between the COVID-19 positive and other viral infection groups, with respiratory- and laryngeal-based features resulting in the highest performance. Coordination-based features derived from harmonic-to-noise ratio time series from read speech discriminated between the two groups with an area under the ROC curve (AUC) of 0.94. A longitudinal case study of two subjects, one from each group, revealed differences in laryngeal based acoustic features, consistent with observed physiological differences between the two groups. The results from this analysis highlight the promise of using nonintrusive sensing through simple speech recordings for early warning and tracking of COVID-19.
Summary
In the face of the global pandemic caused by the disease COVID-19, researchers have increasingly turned to simple measures to detect and monitor the presence of the disease in individuals at home. We sought to determine if measures of neuromotor coordination, derived from acoustic time series, as well as phoneme-based...
READ MORE
An emotion-driven vocal biomarker-based PTSD screening tool
January 1, 2023
Journal Article
Published in:
IEEE Open J. Eng. Med. Biol., early access, 2023.
Topic:
R&D area:
R&D group:
Summary
This paper introduces an automated post-traumatic stress disorder (PTSD) screening tool that could potentially be used as a self-assessment or inserted into routine medical visits for PTSD diagnosis and treatment. Methods: With an emotion estimation algorithm providing arousal (excited to calm) and valence (pleasure to displeasure) levels through discourse, we select regions of the acoustic signal that are most salient for PTSD detection. Our algorithm was tested on a subset of data from the DVBIC-TBICoE TBI Study, which contains PTSD Check List Civilian (PCL-C) assessment scores. Results: Speech from low-arousal and positive-valence regions provide the best discrimination for PTSD. Our model achieved an AUC (area under the curve) equal to 0.80 in detecting PCL-C ratings, outperforming models with no emotion filtering (AUC = 0.68). Conclusions: This result suggests that emotion drives the selection of the most salient temporal regions of an audio recording for PTSD detection.
Summary
This paper introduces an automated post-traumatic stress disorder (PTSD) screening tool that could potentially be used as a self-assessment or inserted into routine medical visits for PTSD diagnosis and treatment. Methods: With an emotion estimation algorithm providing arousal (excited to calm) and valence (pleasure to displeasure) levels through discourse, we...
READ MORE
Noninvasive monitoring of simulated hemorrhage and whole blood resuscitation
December 14, 2022
Journal Article
Published in:
Biosensors, Vol. 12, No. 12, 2022, Art. No. 1168.
Topic:
R&D area:
R&D group:
Summary
Hemorrhage is the leading cause of preventable death from trauma. Accurate monitoring of hemorrhage and resuscitation can significantly reduce mortality and morbidity but remains a challenge due to the low sensitivity of traditional vital signs in detecting blood loss and possible hemorrhagic shock. Vital signs are not reliable early indicators because of physiological mechanisms that compensate for blood loss and thus do not provide an accurate assessment of volume status. As an alternative, machine learning (ML) algorithms that operate on an arterial blood pressure (ABP) waveform have been shown to provide an effective early indicator. However, these ML approaches lack physiological interpretability. In this paper, we evaluate and compare the performance of ML models trained on nine ABP-derived features that provide physiological insight, using a database of 13 human subjects from a lower-body negative pressure (LBNP) model of progressive central hypovolemia and subsequent progressive restoration to normovolemia (i.e., simulated hemorrhage and whole blood resuscitation). Data were acquired at multiple repressurization rates for each subject to simulate varying resuscitation rates, resulting in 52 total LBNP collections. This work is the first to use a single ABP-based algorithm to monitor both simulated hemorrhage and resuscitation. A gradient-boosted regression tree model trained on only the half-rise to dicrotic notch (HRDN) feature achieved a root-mean-square error (RMSE) of 13%, an R2 of 0.82, and area under the receiver operating characteristic curve of 0.97 for detecting decompensation. This single-feature model's performance compares favorably to previously reported results from more-complex black box machine learning models. This model further provides physiological insight because HRDN represents an approximate measure of the delay between the ABP ejected and reflected wave and therefore is an indication of cardiac and peripheral vascular mechanisms that contribute to the compensatory response to blood loss and replacement.
Summary
Hemorrhage is the leading cause of preventable death from trauma. Accurate monitoring of hemorrhage and resuscitation can significantly reduce mortality and morbidity but remains a challenge due to the low sensitivity of traditional vital signs in detecting blood loss and possible hemorrhagic shock. Vital signs are not reliable early indicators...
READ MORE
Predicting ankle moment trajectory with adaptive weighted ensemble of LSTM network
November 1, 2022
Conference Paper
Published in:
2022 IEEE High Perf. Extreme Comp. Conf. (HPEC), 19-23 September 2022, DOI: 10.1109/HPEC55821.2022.9926370.
Summary
Estimations of ankle moments can provide clinically helpful information on the function of lower extremities and further lead to insight on patient rehabilitation and assistive wearable exoskeleton design. Current methods for estimating ankle moments leave room for improvement, with most recent cutting-edge methods relying on machine learning models trained on wearable sEMG and IMU data. While machine learning eliminates many practical challenges that troubled more traditional human body models for this application, we aim to expand on prior work that showed the feasibility of using LSTM models by employing an ensemble of LSTM networks. We present an adaptive weighted LSTM ensemble network and demonstrate its performance during standing, walking, running, and sprinting. Our result show that the LSTM ensemble outperformed every single LSTM model component within the ensemble. Across every activity, the ensemble reduced median root mean squared error (RMSE) by 0.0017-0.0053 N. m/kg, which is 2.7 – 10.3% lower than the best performing single LSTM model. Hypothesis testing revealed that most reductions in RMSE were statistically significant between the ensemble and other single models across all activities and subjects. Future work may analyze different trajectory lengths and different combinations of LSTM submodels within the ensemble.
Summary
Estimations of ankle moments can provide clinically helpful information on the function of lower extremities and further lead to insight on patient rehabilitation and assistive wearable exoskeleton design. Current methods for estimating ankle moments leave room for improvement, with most recent cutting-edge methods relying on machine learning models trained on...
READ MORE
Affective ratings of nonverbal vocalizations produced by minimally-speaking individuals: What do native listeners perceive?
October 18, 2022
Conference Paper
Published in:
10th Intl. Conf. Affective Computing and Intelligent Interaction, ACII, 18-21 October 2022.
Topic:
R&D area:
R&D group:
Summary
Individuals who produce few spoken words ("minimally-speaking" individuals) often convey rich affective and communicative information through nonverbal vocalizations, such as grunts, yells, babbles, and monosyllabic expressions. Yet, little data exists on the affective content of the vocal expressions of this population. Here, we present 78,624 arousal and valence ratings of nonverbal vocalizations from the online ReCANVo (Real-World Communicative and Affective Nonverbal Vocalizations) database. This dataset contains over 7,000 vocalizations that have been labeled with their expressive functions (delight, frustration, etc.) from eight minimally-speaking individuals. Our results suggest that raters who have no knowledge of the context or meaning of a nonverbal vocalization are still able to detect arousal and valence differences between different types of vocalizations based on Likert-scale ratings. Moreover, these ratings are consistent with hypothesized arousal and valence rankings for the different vocalization types. Raters are also able to detect arousal and valence differences between different vocalization types within individual speakers. To our knowledge, this is the first large-scale analysis of affective content within nonverbal vocalizations from minimally verbal individuals. These results complement affective computing research of nonverbal vocalizations that occur within typical verbal speech (e.g., grunts, sighs) and serve as a foundation for further understanding of how humans perceive emotions in sounds.
Summary
Individuals who produce few spoken words ("minimally-speaking" individuals) often convey rich affective and communicative information through nonverbal vocalizations, such as grunts, yells, babbles, and monosyllabic expressions. Yet, little data exists on the affective content of the vocal expressions of this population. Here, we present 78,624 arousal and valence ratings of...
READ MORE
Modeling real-world affective and communicative nonverbal vocalizations from minimally speaking individuals
October 1, 2022
Conference Paper
Published in:
IEEE Trans. on Affect. Comput., Vol. 13, No. 4, October 2022, pp. 2238-53.
Topic:
R&D area:
R&D group:
Summary
Nonverbal vocalizations from non- and minimally speaking individuals (mv*) convey important communicative and affective information. While nonverbal vocalizations that occur amidst typical speech and infant vocalizations have been studied extensively in the literature, there is limited prior work on vocalizations by mv* individuals. Our work is among the first studies of the communicative and affective information expressed in nonverbal vocalizations by mv* children and adults. We collected labeled vocalizations in real-world settings with eight mv* communicators, with communicative and affective labels provided in-the-moment by a close family member. Using evaluation strategies suitable for messy, real-world data, we show that nonverbal vocalizations can be classified by function (with 4- and 5-way classifications) with F1 scores above chance for all participants. We analyze labeling and data collection practices for each participating family, and discuss the classification results in the context of our novel real-world data collection protocol. The presented work includes results from the largest classification experiments with nonverbal vocalizations from mv* communicators to date.
Summary
Nonverbal vocalizations from non- and minimally speaking individuals (mv*) convey important communicative and affective information. While nonverbal vocalizations that occur amidst typical speech and infant vocalizations have been studied extensively in the literature, there is limited prior work on vocalizations by mv* individuals. Our work is among the first studies...
READ MORE
Contrast-enhanced ultrasound to detect active bleeding
October 1, 2022
Abstract
Published in:
J. Acoust. Soc. Am. 152, A280 (2022)
Topic:
R&D area:
R&D group:
Summary
Non-compressible internal hemorrhage (NCIH) is the most common cause of death in acute non-penetrating trauma. NCIH management requires accurate hematoma localization and evaluation for ongoing bleeding for risk stratification. The current standard point-of-care diagnostic tool, the focused assessment with sonography for trauma (FAST), detects free fluid in body cavities with conventional B-mode imaging. The FAST does not assess whether bleeding is ongoing, at which location(s), and to what extent. Here, we propose contrast-enhanced ultrasound (CEUS) techniques to better identify, localize, and quantify hemorrhage. We designed and fabricated a custom hemorrhage-mimicking phantom, comprising a perforated vessel and cavity to simulate active bleeding. Lumason contrast agents (UCAs) were introduced at clinically relevant concentrations (3.5×108 bubbles/ml). Conventional and contrast pulse sequence images were captured, and post-processed with bubble localization techniques (SVD clutter filter and bubble localization). The results showed contrast pulse sequences enabled a 2.2-fold increase in the number of microbubbles detected compared with conventional CEUS imaging, over a range of flow rates, concentrations, and localization processing parameters. Additionally, particle velocimetry enabled mapping of dynamic flow within the simulated bleeding site. Our findings indicate that CEUS combined with advanced image processing may enhance visualization of hemodynamics and improve non-invasive, real-time detection of active bleeding.
Summary
Non-compressible internal hemorrhage (NCIH) is the most common cause of death in acute non-penetrating trauma. NCIH management requires accurate hematoma localization and evaluation for ongoing bleeding for risk stratification. The current standard point-of-care diagnostic tool, the focused assessment with sonography for trauma (FAST), detects free fluid in body cavities with...
READ MORE