Digital Voice-Based Biomarker for Monitoring Respiratory Quality of Life: Findings from the Colive Voice Study

A. Study design

Colive Voice¹ is an international digital health study established and led by the Luxembourg Institute of Health which aims at identifying vocal biomarkers for remote monitoring and screening of various chronic diseases and frequent health symptoms. The multilingual audio databank is collected in four languages (English, French, German and Spanish) and contains recordings of multiple vocal tasks, including sustained vowel phonation, coughing, breathing, reading and counting. Voice recordings are associated with annotated clinical and demographic data, providing an in-depth patient characterization with validated disease-specific questionnaires on symptoms, treatments and quality of life. Colive Voice has been hosted online since June 2021 and is open for participation to anyone, under the condition that: 1) they sign the consent form and 2) they are at least 15 years old.

The study has been approved by the National Research Ethics Committee in Luxembourg (N° 202103/01) in March 2021. Informed written consent was obtained electronically via the Colive Voice application from all participants in the study. The Colive Voice study protocol is also registered on ClinicalTrials.gov (NCT04848623).

Part of the study is dedicated to investigation of RQoL in the general population from voice recordings, accompanied with annotations of RQoL via self-administered VQ11 questionnaire, as well as clinical and demographic data.

Unlike SGRQ and BPQ, which are extensive (76 items in SGRQ and 33 items in BPQ) with complex scoring, making them unsuitable for repeated evaluations in clinical practice as well as a regular use in real life, VQ11 is a brief questionnaire with only 11 items distributed across functional omponents (3 items), psychological components (4 items) and social components (4 items). Although much simpler and faster to record, VQ11 shows high correlation with SGRQ [4]. Each item in VQ11 is represented by five categories (not at all, a little, moderately, much, extremely) which reflect the participant’s feeling about the statement associated with a particular item, and can be represented by a value from 1 to 5. The total score is obtained by summing all individual items, leading to a score between 11 and 55 with lower value indicating better RQoL [4]. We stratify the participants in the study into two categories using the cut-off VQ11 score of 22: 1) Impaired RQoL (V Q11 ≥ 22), and 2) Normal RQoL (V Q11 < 22) [19], [20].

Since the number of participants with impaired RQoL was significantly lower than the normal RQoL, we generate a balanced dataset matched by age and gender composed of 1908 sustained vowel recordings in total, equally distributed between two groups.

A full workflow of RQoL monitoring from data acquisition to the prediction of RQoL is shown in Figure 1.

• Download figure
• Open in new tab

Fig. 1:

Workflow of RQoL monitoring

B. Data acquisition and preprocessing

Participants were recruited via an online crowdsourced campaign or through partnerships with various patient associations, academic institutions, hospitals, or other research initiatives (including Les Sentinelles and the ComPaRe study, AP-HP). The full list of partners is available on the Colive Voice website. Participants were invited to use an app² accessible from participants’ devices equipped with microphones (smartphone, tablet or laptop). Collected information is composed of socio-demographic and clinical data acquired via participants’ self-reported questionnaires and voice recordings.

Socio-demographic data contains information about body mass index (BMI) and smoking habits, while clinical data contains information about day and night coughing, chest pain, sore throat, as well as associated diseases such as asthma and COPD. Categorical variables were encoded as one-hot representations, leading to 23 features in total.

Voice recordings are acquired in the form of sustained vowel phonation (/a/ vowel) produced at a comfortable pitch and loudness as long as possible. Vowel phonation is selected since it provides valuable information about the pulmonary function, and in addition, it is less susceptible to language bias, which may be present in the multi-lingual data collection. Reduced pulmonary function leads to decreased airflow necessary to support phonation [21], which in turn reflects in reduced RQoL.

Participants were advised to make voice recordings in a quiet environment without the external noise in order to preserve high-quality recordings. However, given that data is collected in uncontrolled conditions and to account for the challenges related to the use of different devices, microphones, and recording conditions for data collection, audio preprocessing using a proprietary pipeline was performed to harmonize the recordings and prepare them for the subsequent steps.

C. Statistical analysis

We utilized an independent two-tailed t-test to compare the means of groups with normal and impaired RQoL for continuous variables. For categorical variables, a chi-square test was used. A p-value less than 0.05 indicates a statistically significant difference. Only variables that were statistically significant were used as socio-demographic and clinical features in further processing. Table I provides a summary of the study population characteristics.

D. Feature extraction and fusion

We first extracted a set of 72 handcrafted audio features, that contain time domain, spectral, cepstral, prosodic, and nonlinear dynamics features (Table II). Audio features were extracted using Surfboard [22], a Python library for feature extraction with application to the medical domain, as well as Parselmouth [23], a Python interface to Praat. We selected the audio features that are shown to be relevant for vocal biomarker research across multiple diseases.

In addition to this, we used standard audio feature sets, i.e. extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [24] and ComParE, extracted using the openSMILE [25]. The eGeMAPS is a minimalistic set of acoustic parameters for paralinguistic or clinical speech analysis which is composed of 88 energy/amplitude, frequency, spectral and temporal features, as well as statistical functionals applied to them (arithmetic mean, standard deviation, percentile). ComParE is a brute force audio feature set that contains 65 low-level acoustic descriptors and various statistical functionals applied to them, leading to a total of 6737 audio features.

Finally, we experiment with 4 different types of deep audio embeddings, i.e. VGGish [26], YAMNet, OpenL3 [27], and BYOL-A [28], which are state-of-the-art general audio features pretrained on large audio collections that are successfully used for a number of downstream tasks. Characteristics of different audio embeddings are provided in Table III.

VGGish is a pretrained convolutional neural network (CNN) mostly inspired by the VGG network used in computer vision. The network is adapted to accept 96×64 bin log-mel spectrograms at its input and extracts 128-dimensional embeddings from 960 ms segments of an audio signal. YAMNet employs the Mobilenet v1 depthwise separable convolution architecture used with the same input as VGGish, but outputs 1024-dimensional embeddings for each 960 ms audio segment. Both VGGish and YAMNet are pretrained on the large-scale Audio Set dataset for audio event classification which contains more than 2 million of 10 s YouTube clips of sounds classified into 632 audio events. To summarize features across different audio segments and output the equal size feature vectors from recordings of different lengths average pooling was used.

OpenL3 uses CNN-based L3-Net for self-supervised learning via audio-visual correspondence, to learn whether a particular video frame corresponds to an audio frame; thus, requiring no annotations. The model is pretrained on two subsets of Audio Set, i.e. music and environmental subset, containing 296K and 195K clips respectively, and uses either 128 or 256 band Mel-spectrograms at the input, while the output audio embedding is 512 or 6144-dimensional vector for each 1s audio segment. We use a model pretrained on an environmental subset, with 256 band Mel-spectrograms and 6144-dimensional embeddings.

BYOL-A uses the Bootstrap Your Own Latent (BYOL) method for self-supervised learning of general-purpose image representations, adapted to work with audio. Normalized 96×64 bin log-mel spectrograms are used as an input, and two augmented versions of the input are created by shifting pitch and stretching time, which are further fed into two parallel networks (online and target network). The online network predicts the output representation of the target network, which is then iteratively updated as the exponential moving average of the parameters of the online network. The model is pretrained on the Audio Set dataset and produces 512, 1024 or 2048-dimensional general-purpose audio embeddings. We use 2048-dimensional embeddings.

In addition to voice features, we extracted the demographic/clinical data relevant for RQoL from the subjective self-reports. We used socio-demographic variables that were found statistically significant (BMI, smoking habits), symptoms (day and night coughing, chest pain, sore throat), and associated diseases that can affect RQoL (asthma, COPD), as shown in Table I. All categorical variables were encoded as one-hot representations, except for ensemble-based models (Random Forest, Extreme Gradient Boosting), where single feature representation was kept. One-hot encodings produce sparse feature vectors, which are not suitable for tree-based models, since splitting on such features produces a small gain, and is typically ignored in favor of continuous variables. Features were standardized before feeding them to classification models to put them on the same scale, i.e. all features have zero mean and unit standard deviation.

Given that the size of the audio feature vectors is substantially larger than the size of socio-demographic/clinical features (up to 250 times larger for ComParE features), Principal Component Analysis (PCA) was applied to audio embeddings prior to data fusion, to reduce their dimensionality to the first 23 principal components that explain most of the variance, and to put the features from different modalities to equal dimension.

E. RQoL prediction

Features extracted in the previous section were fed into several classifiers: Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), Extreme Gradient Boosting (XGBoost), and Multilayer Perceptron (MLP).

LR with L2 regularization is used to handle overfitting, as well as SVM with radial basis function kernel, where the model hyperparameters, i.e. the regularization parameter C and the kernel coefficient γ are optimized using a grid search. Two ensemble models include RF and XGBoost. RF was composed of 500 fully grown trees (optimal number of trees was determined after hyperparameter tuning), expanded until all leaves were pure or contained less than 2 samples, with Gini index as the criterion for splitting the node, and the number of features at each split equal to the square root of the total number of features. All models are implemented using the scikit-learn 1.1.3 Python library.

XGBoost is a flexible and distributed gradient boosting algorithm, that allows for custom loss functions, as well as regularization techniques to mitigate the overfitting. We use XGBoost with 500 trees, L2 regularization and log loss objective function. XGBoost is implemented using the xgboost 1.5.0 Python library.

MLP was composed of two hidden layers with 256 neurons each and a ReLU activation function, followed by dropout layers for preventing overfitting with a dropout rate equal to 0.3, and an output layer with a sigmoid activation function is utilized in this paper. We used Adam optimizer, binary cross entropy loss function, batch size equal to 32, while the optimal learning rate (0.0001) and the number of epochs (30) are determined via grid search. Note that Adam has an adaptive per-parameter learning rate, which is computed using the initial learning rate as an upper limit. MLP is implemented using Tensorflow 2.9.1.

F. Evaluation

For evaluation of the model performance we use accuracy, sensitivity, specificity, area under the receiver operating characteristic curve (AUROC), Brier score and net reclassification index (NRI).

Accuracy is the ratio of the number of correctly classified observations and the total number of observations. Sensitivity (true positive rate, recall) is the proportion of participants detected with impaired RQoL (true positives) among those who have impaired RQoL (true positives + false negatives), and shows the model’s ability to correctly identify cases. Specificity (true negative rate) is the proportion of participants detected with normal RQoL (true negatives) among those who have normal RQoL (true negatives + false positives), and refers to the model’s ability to correctly identify healthy controls. ROC curve plots sensitivity against false negative rate (1-specificity) at different classification thresholds, while AUROC is an aggregated performance measure which summarizes ROC curve, with a value of 0.5 denoting random guess, and 1 denoting perfect classification.

To assess model calibration, i.e. the consistency between the predicted probability and the observations, Brier score was used, which is the mean squared deviation of the predicted probability from the actual target. It is a value between 0 and 1, with a lower value indicating a better model.

Given the size of the dataset, to get the reliable and robust performance estimates and preserve the class distribution across folds, we used stratified 5-fold cross-validation [29]. Data is split into five folds, four merged and used for training, and the remaining one for testing. The process is repeated 5 times, so that each fold was used exactly once for testing, and the performance is then averaged over all folds.

Finally, since our aim was to quantify how much voice-related information can improve the reliability of RQoL screening on top of standard clinical features, we used NRI to estimate the improvement in performance due to adding vocal biomarkers to a set of socio-demographic and clinical predictors. The value can range from -2 to 2, with bigger value indicating larger improvement.

A. Evaluation of RQoL from socio-demographic/clinical data

Before evaluating the relevance of vocal biomarkers for estimating RQoL, we set up a baseline experiment where only socio-demographic data (BMI, smoking habits) and clinical data (day and night coughing, chest pain, sore throat, as well as associated diseases such as asthma and COPD) from the participants’ self-reports were used for prediction of RQoL status. Categorical variables were encoded as one-hot representations, leading to 23 features in total. Performance is averaged over 5 folds (Table IV), with the best AUROC of 0.70, and accuracy of 64.1% obtained using LR classifier.

We also presented the feature importance based on the mean impurity decrease for the RF model in Figure 2, revealing that BMI is the most important socio-demographic variable, followed by clinical symptoms related to day and night coughing.

• Download figure
• Open in new tab

Fig. 2:

Feature importance for socio-demographic and clinical features based on the mean impurity decrease

B. Evaluation of RQoL from voice recordings

We investigated whether voice related information could be used as a digital biomarker for RQoL. To that end, we extracted a set of handcrafted audio features (Table II), as well as two widely used general audio feature sets (eGeMAPS and ComParE). In addition to this, four state-of-the-art deep audio embeddings are evaluated (VGGish, YAMNet, OpenL3, BYOL-A) which proved to be highly competitive across multiple audio tasks. The features were either fed directly to the classifier, or in case of deep audio embeddings after applying Principal Component Analysis (PCA) to reduce the dimensionality of feature vectors. The results for assessment of RQoL from voice were provided in Table V, with the best performance reaching AUROC equal to 0.7 and accuracy of 65.57% using BYOL-A deep audio embeddings. BYOL-A substantially outperforms all other feature extraction techniques by over 2%.

To highlight the characteristics of sustained vowel phonation labeled with normal and impaired RQoL, we showed in Figure 3 spectrograms of two participants matched by age and gender (males, 67 years old): one with normal RQoL without the history of pulmonary diseases, but with a diagnosed COVID-19 more than 3 weeks before the recording was made; and one with extremely impaired RQoL (VQ11 score: 46) diagnosed with asthma-COPD overlap syndrome. Even though the normal RQoL example is actually a boundary case (VQ11 score: 21, cut-off value 22), the differences in spectrograms are clearly visible. While the normal RQoL recording is represented by uninterrupted phonation, with clearly distinctive harmonics (Figure 3a), impaired RQoL recording is characterized by strangled voice with multiple stoppages and voice breaks, and increased energy areas in higher frequency bands, which are most likely caused by aperiodic noise produced at a glottal constriction (Figure 3b). Furthermore, the absence of higher harmonics above 1kHz can be observed throughout the spectrogram, and as phonation progresses, even the adjacent lower harmonics become smeared and more difficult to distinguish. However, for the impaired RQoL voice recordings with VQ11 score closer to cut-off value, the differences are not so distinct.

• Download figure
• Open in new tab

Fig. 3:

Spectrograms of sustained vowel phonation of participants matched by age and gender (male, age 67) with a) normal RQoL (VQ11 score: 21); and b) impaired RQoL (VQ11 score: 46). Normal RQoL spectrogram is represented by uninterrupted phonation, with clearly distinctive harmonics. Impaired RQoL spectrogram is characterized by strangled voice with multiple stoppages and voice breaks, and increased energy areas in higher frequency bands. The absence of higher harmonics above 1kHz can be observed, and as phonation progresses, the adjacent lower harmonics become smeared and more difficult to distinguish.

C. Evaluation of RQoL from fused socio-demographic/clinical data and voice recordings

By fusing socio-demographic/clinical with voice features, we can quantify how much voice features can boost the performance of the socio-demographic and clinical data, uncovering the full potential of the multimodal data fusion. The results for the assessment of RQoL from multimodal features are provided in Table VI, whereas the comparison of the best-performing machine learning model for the socio-demographic/clinical features only, voice features only and multimodal features obtained after their fusion is presented in Figure 4. By using intermediate fusion (feature level fusion) we show that clinical data extracted from questionnaires and voice features obtained as the higher-level representations extracted from raw audio signals are complementary, leading to a substantial performance boost (accuracy equal to 70.34% and AUC equal to 0.77 using the combination of BYOL-A audio embeddings and socio-demographic/clinical features). Note that specificity is, in general, higher than sensitivity for all models, i.e. the models are still better at predicting normal than impaired RQoL. This is also visible from the confusion matrix of the best-performing model (fused BYOL-A and socio-demographic/clinical features, trained with LR classifier) shown in Figure 5a, where it is clear that the number of false negatives is substantially larger than the number of false positives. Using the Brier score as a measure of calibration, the same multimodal model achieves the lowest average Brier score over all folds equal to 0.19 with a nearly linear calibration curve, as shown in Figure 5b. Figure 5c displays the ROC curve of the best-performing model.

• Download figure
• Open in new tab

Fig. 4:

Accuracy with the best-performing machine learning model for socio-demographic/clinical features only, voice features only and fused clinical and voice (multimodal) features. Models with both clinical and voice data (“Fused”) systematically outperformed models where clinical variables only or voice features only were used. Error bars represent the standard deviation.

• Download figure
• Open in new tab

Fig. 5:

Performance of the best model (fused BYOL-A deep audio embeddings and socio-demographic/clinical features, trained with logistic regression classifier): a) Confusion matrix; b) Probability calibration curve; and c) ROC curve. Light blue lines denote the ROC curves across 5 cross-validation folds, whereas a thick blue line represents the average ROC curve. Standard deviation is highlighted with the shaded area.

Finally, since our objective was to quantify how much the vocal biomarkers increase the reliability of screening based only on subjective self-reports, NRI was used to estimate the improvement in performance after fusing vocal biomarkers with socio-demographic/clinical predictors. Table VI reveals that vocal biomarkers indeed improve the predictive capability of demographic and clinical variables for all acoustic features, with the biggest improvement measured by NRI of 0.19 for eGeMAPS features modeled with SVM.