Abstract
Background Research into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools.
Methods To address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis.
Results We identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias.
Limitations Our sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals.
Conclusions We demonstrate that DISC-based facial analysis can be used to reliably identify an individual’s emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.
Highlights
We detected the emotional state of participants in the absence of visible changes in facial expressions
We identified distinct facial movement maps for happy and sad emotions
We developed a highly sensitive, noninvasive, noncontact emotional state assessment tool
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This work was supported by the National Science Foundation through the Growing Convergence Research Program Award #2021002 (to C.B.M. and S.M.), a Targeted Research Opportunity Program FUSION Award #63845 (to S.M.) from the Renaissance School of Medicine at Stony Brook University, as well as SEED grant funding (to S.M.) from the Office of the Vice President for Research at Stony Brook University. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study was conducted under the supervision of the Stony Brook University Committee on Research in Human Subjects (IRB2019-0199). All volunteers gave their written informed consent before participation in the study.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
The data and code used for this manuscript are available upon request from the corresponding author. To protect participant privacy, raw videos of our participants cannot be shared publicly. Modified videos are available upon request from the corresponding author.
