ABSTRACT
In this contribution, we present the analyses of vocalisation data recorded in the first observation round of the European Commission’s Erasmus Plus project “EMBOA, Affective loop in Socially Assistive Robotics as an intervention tool for children with autism”. In total, the project partners recorded data in 112 robot-supported intervention sessions for children with autism spectrum disorder. Audio data were recorded using the internal and lapel microphone of the H4n Pro Recorder. To analyse the data, we first utilise a child voice activity detection (VAD) system in order to extract child vocalisations from the raw audio data. For each child, session, and microphone, we provide the total time child vocalisations were detected. Next, we compare the results of two different implementations for valence- and arousal-based speech emotion recognition, thereby processing (1) the child vocalisations detected by the VAD and (2) the total recorded audio material. We provide average valence and arousal values for each session and condition. Finally, we discuss challenges and limitations of child voice detection and audio-based emotion recognition in robot-supported intervention settings.
1. Introduction
Children with autism spectrum disorder (ASD; OMIM 209850) have problems in social interactions and show restricted, repetitive patterns of behaviour, interests, or activities (American Psychiatric Association, 2013). Previous studies revealed that children with ASD usually prefer interactions with human-like looking robots to interactions with humans and therefore benefit from robot-based interventions in order to improve their socio-communicative skills (Bartl-Pokorny et al., 2021). To date, robots used in intervention settings with children with ASD need lots of manual control by therapists as they are not able to automatically and in real time respond to a child’s needs and emotional states. To overcome these limitations of social robots in the future, current research approaches focus on the implementation of automatic emotion recognition systems.
The EU Erasmus+ project “Affective loop in Socially Assistive Robotics as an intervention tool for children with autism” (https://emboa.eu), conducted by an international and multidisciplinary consortium of researchers from Gdansk University of Technology (GUT; Poland), University of Hertfordshire (UH; United Kingdom), Istanbul Teknik Universitesi (ITU; Turkey), Yeditepe University Vakif (YT; Turkey), the Macedonian Association for Applied Psychology (MAAP; North Macedonia), and University of Augsburg (UA; Germany), focuses on the practical evaluation of the use of state-of-the-art emotion recognition technologies in robot-supported interventions and aims to develop guidelines for the application of such technologies in intervention settings.
The project team collects data from multiple observation channels (e.g., vocalisations, eye movements, facial expressions, physiological parameters) during intervention sessions in which children with ASD are interacting with the social robot Kaspar (https://www.herts.ac.uk/kaspar). Based on these data, the researchers aim to evaluate each observation channel in terms of its information gain for automatic emotion recognition. This report focuses on the analyses of vocalisation data recorded in the first observation round.
The report is organised as follows: section 2 describes the available audio data, section 3 provides a description of the utilised child voice activity detection (VAD) system and results on detected vocalisations, section 4 describes the applied speech emotion recognition model (SER) and the detected valence and arousal values. Section 5 sums up the results and discusses revealed challenges and limitations of automatic child voice detection and audio-based emotion recognition in robot-supported intervention settings.
2. Methods
2.1. Audio data
The EMBOA team received ethics approval by Gdansk University of Technology/Poland for conducting the study. The inclusion criteria for the present study were: (1) Children were diagnosed with ASD by a mental health professional in the country of living, (2) children were recruited in institutions in the partner countries North Macedonia, Turkey, Poland, and UK, (3) children had already received treatments for ASD in their caregiving institutions or were newly diagnosed and had not received treatments before inclusion in the study, (4) children were between 2 and 12 years old, (4) the children’s legal guardians signed an informed consent form for participation in the study. As there were no further inclusion criteria defined, the sample was heterogeneous regarding amongst others gender, exact age, language skills, and comorbidities. There were neither clinical assessments nor follow-up investigations done with the children in the framework of the present study.
The participating children with ASD were recorded during intervention sessions that were supported by the social robot Kaspar. During these interventions, at least one researcher and one parent were together with the child and the robot in a room. The audio data were recorded from the partners at GUT/Poland, ITU-YU/Turkey, MAAP/North Macedonia, and UH/United Kingdom using the internal and lapel microphone of the H4n Pro Recorder. For most of the recorded robot-supported intervention sessions, both the recordings from the internal and from the lapel microphone are available. Recordings with the internal microphone of four sessions of MAAP/North Macedonia and recordings with the lapel microphone of all fourteen sessions of ITU-YU/Turkey were excluded from analysis due to insufficient audio quality. Table 1 provides an overview of the final set of analysable audio recordings. The children with ASD were recorded in up to eleven intervention sessions, as listed in Table 2. The minimum number of recorded intervention sessions should be 2 for all children; the actual number of recorded sessions depended on the individual therapeutic needs of the children, the family’s willingness to continue participation, and the achievement of the project goals regarding sample size. The children recorded by GUT/Poland were 6 years old, the children recorded by ITU-YU/Turkey were between 6 and 10 years old, the children recorded by MAAP/North Macedonia were between 2 and 6 years old, and the children recorded by UH/United Kingdom were 11 to 12 years old. All data were recorded in the years 2020 and 2021. The number of included children was in conformity with the EMBOA project protocol approved by the funding agency.
We are aware that certain systematic differences between the data recorded from the different partners such as age, mother tongue, individual language skills of the children, room acoustics, etc. may cause bias. Therefore, we report quantitative results separately for individual children and/or participating sites and discuss qualitative differences in the data.
2.2. Child voice activity detection
Following Milling et al. (2022), we apply a deep learning-based voice activity detection (VAD) system specifically trained for vocalisations of children with ASD to extract child vocalisations, as we expect the child vocalisations to contain the most information about the children’s affective states. The used VAD system has been trained and evaluated on data recorded under similar conditions as the EMBOA data, namely in the framework of the European Commission’s DE-ENIGMA project (Shen et al., 2018; https://cordis.europa.eu/project/id/688835; https://de-enigma.eu). For the intervention sessions in DE-ENIGMA, the robot Zeno was used. The model was trained on audio data from the British study arm of the project containing audio data from 84 robot-supported intervention sessions of 25 English-speaking children with ASD from the United Kingdom, having a total duration of more than 17 hours. For the experiments reported by Milling et al. (2022), the data was partitioned into train, development, and test partitions in a speaker-independent way. In total, this data contains more than 2 hours of annotated child vocalisations (partially overlapping with other vocalisations). The system was trained to predict the probability of a child vocalisation being present in a given 10-millisecond-long audio frame based on so-called low-level descriptors (LLDs) of the said audio frame. LLDs are a common set of characteristics, which can be extracted from an audio signal, including features such as fundamental frequency (F0), variations in the volume (shimmer), and variations in the fundamental frequency (jitter). The deep learning approach consists of a two-layer bi-directional long short-term memory (Bi-LSTM) recurrent neural network (RNN), followed by a dense layer, which is designed to allow information flow for several consecutive audio frames. Different thresholds for the predicted probability of the system allow for the adjustment of trade-off between sensitivity and specificity. For inference, a threshold corresponding to the equal-error-rate (EER) was chosen, i.e., equal values of true positive rate and false positive rate. Finally, the VAD system summarises frame-level predictions with a detection event if the EER threshold is reached for at least 0.25 s of a 1-s-chunk. More details on the VAD are given in Milling et al. (2022). The child VAD system has a reported EER of 0.381 and an area-under-the-curve of 0.662 on the DE-ENIGMA corpus. However, even though the recording conditions of both projects appear rather similar to each other, it is not clear whether this performance translates to the EMBOA data, as a lack of systematic vocalisation annotations prevents a thorough evaluation of the model.
2.3. Speech emotion recognition
In order to estimate emotions from speech we use a deep learning model developed, trained, and evaluated by Milling et al. (2022), which is again based on the same DE-ENIGMA data and partitioning of British children, as described for the VAD component. The models are designed for a continuous speech emotion recognition (SER) task and we apply correspondingly trained models for two types of tasks: 1) each 1 s chunk from the original audio recording is used for the continuous SER task and 2) only 1 s chunks, for which a child vocalisation was detected, are being used for the continuous SER task. Within each task, for each relevant audio chunk, we then extract 88 functional features of the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS; Eyben et al., 2015) from 1 s audio chunks. The features are then fed into a model consisting of two RNN layers with LSTM cells and a hidden layer size of 128 units, a feed-forward layer with 128 neurons, a rectified linear unit (ReLU) activation and a dropout rate of 0.3, and a final feed-forward layer with a single neuron, predicting the valence values (and arousal values, respectively). Reported performances of the models achieve up to 0.168 concordance correlation coefficient (CCC) on the test partition of the DE-ENIGMA corpus. More details on the SER system and performance values can be found in Milling et al. (2022).
3. Results
3.1. Detected child vocalisations
A general overview of detected child vocalisations is provided in Table 3. Table 4 groups child voice activity detections by the session number in a consecutive series of sessions with partly identical children. Detailed session-wise results can be found in Table A.1. of the annexe. The proportion of time with detected child vocalisations per session is similar in the internal and lapel microphone condition (Table 3). The proportion of time with detected child vocalisations is similar for each of the 10 analysed sessions for children recorded by MAAP/North Macedonia (Table 4); we can neither observe an obvious increase nor decrease in terms of detected vocalisations with the children’s increasing experience with robot-supported intervention.
3.2. Predicted valence and arousal values
We apply the SER model with and without previous VAD-based pre-processing, as introduced in section 2.3, on each recorded session. Session-wise averaged emotion predictions are reported in Table A.2 of the annexe and are further discussed below.
4. Discussion
The aim of the present study was to investigate the applicability of existing SER approaches in robot-supported intervention sessions with children with ASD. For this, we deployed a publicly available pipeline of a child VAD and a SER model, both specifically trained for children with ASD. Both the VAD system and the SER system were developed based on data from the DE-ENIGMA project that also focused on robot-supported intervention for children with ASD.
For the present study, the VAD system raises a – partially limited – amount of detection events for the data at hand. The most impactful limitation of our study is that we cannot evaluate our models on the investigated data, collected within the EMBOA project, as annotations for vocalisations and emotions are lacking. This hampers the investigation of potential effects of different languages, different ages, background noise, etc. on voice detection. However, some observations can be noted based on the raw predictions of the network. First, the amount of detected events depends on the utilised microphone. In general, it seems that the internal microphone leads to a larger amount of detection events compared to the lapel microphone (i.e., 3.9 ± 2.6 vs 3.3 ± 2.4; Table 3), which was surprising as we initially hypothesised that the lapel microphone recordings would allow us to detect more vocalisations compared to the internal microphone recordings. For GUT/Poland data, actually the opposite was the case: The VAD system detected longer child vocalisation intervals in the internal microphone recordings compared to the lapel microphone recordings.
In an unstructured, exemplary analysis of the recording based on manual listening, we noticed that in the internal microphone recordings the voices of the other people in the room were considerably softer than the child’s voice. In comparison, the voices of the robot and the researcher as well as background noise in general was much louder in the lapel microphone recordings which is a potential reason for fewer detected child vocalisation intervals. When listening to parts of the audio recordings and comparing them with the automatically detected child vocalisation times, we noticed a number of further challenges for the VAD system: In some of the recordings, the audio recording levels of the microphone were too low which seems to correlate with no or only very few detected child vocalisations. As automatic emotion recognition cannot benefit from such recordings, future studies need to carefully check the microphone settings before recording robot-supported intervention sessions. In some cases, the microphone position resulted in further difficulties; e.g., some of the children did not accept the lapel microphone on their clothes and sometimes the clothes or table where the microphones were positioned on produced noise, which was partly louder than the child’s voice and seems to lower the detected child vocalisation time. In some of the recordings, potentially confounding factors related to room acoustics (e.g., ITU-YU/Turkey data were partly recorded in a room with long reverberation time) and background noise (e.g., in some of the MAAP/North Macedonia recordings, sound originating from chairs, tables, and doors is hearable) occurred. Overall, detected child vocalisation times seemed to be shorter in recordings with more people in a room compared to recordings with fewer overlay of voices (e.g., MAAP/North Macedonia data compared to GUT/Poland data). We also noticed that an apparent challenge for the VAD system is the detection of child vocalisation in the form of crying, which might be related to the limited number of crying instances in the original data. Similar observations can be reported for imitation of animal sounds and singing songs. Surprisingly, the VAD system detected relatively few child vocalisation events in the UH/United Kingdom recordings despite the children’s seemingly advanced language skills. It needs to be mentioned here that the children recorded by UH/United Kingdom were older than most of the children included in the DE-ENIGMA training material, which might have a negative effect on the child-specific VAD system.
As suggested by Milling et al. (2022), the performance of VAD systems can have an impact on the estimation of affective states based on audio recordings. This hypothesis is supported by the clear differences in valence and arousal predictions in Table 5, when comparing the average absolute difference between VAD-based SER predictions and the non-VAD-based SER predictions, in particular for the arousal case. It can further be reported that similarly to the VAD task, the choice of microphones seems to have an additional impact on the SER predictions, even though the observed effect is smaller compared to the differences caused by the VAD-based pre-processing.
However, the expressiveness of the SER models for the investigated data set has to be interpreted with caution as only some children in the EMBOA project have the same language and cultural background as the children based on which the model was trained, which potentially hinders the generalisation capabilities of said models. Additionally, the children in the EMBOA project were on average older than the children who provided the training data. To improve the reliability of the VAD and SER systems, the existing models, which were trained on the DE-ENIGMA data set, may be adapted to the EMBOA recording settings (e.g., acoustic room conditions, microphone type, specific age and language of children, etc.) based on annotated parts of the EMBOA data. Note that all reported observations on data quality and corresponding impact on model performance in this section rely on preliminary analyses and have not been quantitatively evaluated due to a lack of systematic annotations. Still, we gained a number of important insights with regard to VAD and SER that will help to formulate guidelines for the use of audio-based emotion recognition techniques in robot-supported intervention settings. Such guidelines will allow future studies to optimise their methodologies with regard to an improved audio-based automatic recognition of children’s emotional states in intervention settings. This will in turn help the development towards a social robot, which is capable of automatically reacting to a child’s current needs and could therefore enhance the child’s motivation to interact with the robot, potentially resulting in positive influences on the intervention success for children with ASD. A prerequisite for this will further be more robust emotion recognition models, being available for different language, culture, gender, and age groups.
Data Availability
Data used for the present study are speech data, i.e. personal data. The participants did not agree to share these data with people outside the EMBOA project team.
Acknowledgements
We are very grateful to all children, families, and healthcare professionals for their participation in our study. We want to thank all project partners for data acquisition. Furthermore, we thank Alice Baird and Shuo Liu for consultancy in data analysis.
This publication was supported in part by the European Commission’s Erasmus Plus project “EMBOA, Affective loop in Socially Assistive Robotics as an intervention tool for children with autism”, contract no 2019-1-PL01-KA203-065096.
The European Commission’s support for the production of this publication does not constitute an endorsement of the contents which reflect the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
This report is distributed free of charge under Creative Commons License CC BY.
Footnotes
This publication was supported in part by the Erasmus Plus project of European Commission: EMBOA, Affective loop in Socially Assistive Robotics as an intervention tool for children with autism, contract no 2019-1-PL01-KA203-065096. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. This publication is distributed free of charge.