PT  - JOURNAL ARTICLE
AU  - Saadon, Jordan R.
AU  - Yang, Fan
AU  - Burgert, Ryan
AU  - Mohammad, Selma
AU  - Gammel, Theresa
AU  - Sepe, Michael
AU  - Rafailovich, Miriam
AU  - Mikell, Charles B.
AU  - Polak, Pawel
AU  - Mofakham, Sima
TI  - Real-time Emotion Detection by Quantitative Facial Motion Analysis
AID  - 10.1101/2022.10.28.22276059
DP  - 2022 Jan 01
TA  - medRxiv
PG  - 2022.10.28.22276059
4099  - http://medrxiv.org/content/early/2022/10/30/2022.10.28.22276059.short
4100  - http://medrxiv.org/content/early/2022/10/30/2022.10.28.22276059.full
AB  - Background Research into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools.Methods To address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis.Results We identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias.Limitations Our sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals.Conclusions We demonstrate that DISC-based facial analysis can be used to reliably identify an individual’s emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.HighlightsWe detected the emotional state of participants in the absence of visible changes in facial expressionsWe identified distinct facial movement maps for happy and sad emotionsWe developed a highly sensitive, noninvasive, noncontact emotional state assessment toolCompeting Interest StatementThe authors have declared no competing interest.Funding StatementThis work was supported by the National Science Foundation through the Growing Convergence Research Program Award #2021002 (to C.B.M. and S.M.), a Targeted Research Opportunity Program FUSION Award #63845 (to S.M.) from the Renaissance School of Medicine at Stony Brook University, as well as SEED grant funding (to S.M.) from the Office of the Vice President for Research at Stony Brook University. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This study was conducted under the supervision of the Stony Brook University Committee on Research in Human Subjects (IRB2019-0199). All volunteers gave their written informed consent before participation in the study.I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe data and code used for this manuscript are available upon request from the corresponding author. To protect participant privacy, raw videos of our participants cannot be shared publicly. Modified videos are available upon request from the corresponding author.