An App-Based Ecological Momentary Assessment of Undergraduate Mental Health During the COVID-19 pandemic in Canada (Smart Healthy Campus Version 2.0): Longitudinal Study ========================================================================================================================================================================= * Chris Brogly * Daniel J Lizotte * Marc Mitchell * Mark Speechley * Arlene MacDougall * Erin Huner * Kelly K Anderson * Michael A Bauer ## Abstract This paper presents results from the Smart Healthy Campus 2.0 study/smartphone app, developed and used to collect mental health-related lifestyle data from 86 Canadian undergraduates January – August 2021. This was a longitudinal repeat measures study conducted over 40 weeks. A 9-item mental health questionnaire was accessible once daily in the app. Two variants of this mental health questionnaire existed; the first was a weekly variant, available each Monday or until a participant responded during the week. The second was a daily variant available after the weekly variant. Mixed models were fit for responses to the two variants and 12 phone digital measures (e.g. GPS, step counts). A second round of models was fit based on backward elimination to determine associations between the 12 digital measures and the variants. 6518 digital measure samples and 1722 questionnaire responses were collected. The daily questionnaire had positive associations with floors walked, installed apps, and campus proximity, while having negative associations with uptime, and daily calendar events. Daily depression had a positive association with uptime. Daily resilience appeared to have a slight positive association with campus proximity. The weekly questionnaire variant had positive associations with device idling and installed apps, and negative associations with floors walked, calendar events, and campus proximity. Physical activity, weekly, had a negative association with uptime, and a positive association with calendar events and device idling. SHC 2.0, via phone digital measures, identified indicators of lifestyle that appeared to be associated with measures of mental health in undergraduates during COVID-19. **Author Summary** This paper analyzes the associations between digital measures from smartphones (such as GPS and step counts) and a broad mental health questionnaire (covering items like depression and anxiety). This data was collected from students at a relatively large, urban university in Canada during the COVID-19 pandemic. We conducted this study because smartphone-based studies observing aspects of mental health in students as they go about their daily lives are uncommon in Canada. Additionally, mental health concerns, such as depression and anxiety, can be common on university campuses, although it isn’t always clear what is associated with those concerns, especially when only looking at them with smartphone data. We were also interested in how an overview of student mental health would relate to digital measures coming from smartphones during the pandemic, as this information would be relevant to inform future pandemics. In general, these relationships might potentially inform ways to improve student mental health, or potentially predict aspects of it, based on data coming in from smartphones. ## Introduction Mental health issues on university campuses are common during periods of traditional in-person study 1. Universities in Canada 2 have recognized that undergraduate students are faced with a number of stresses that challenge mental health. Undergraduate studies are demanding, and students often feel overwhelmed by their obligations 1. Additionally, the COVID-19 pandemic had an unprecedented effect on post-secondary students all over Canada, who experienced disruptive 3–7 changes to their studies in response to necessary public health measures regarding COVID-19. Canadian undergraduates have been surveyed regarding aspects of mental health using traditional sampling techniques such as questionnaires and interviews on several occasions 8–10. Another sampling technique particularly useful to this area is the Ecological Momentary Assessment (EMA) which allows for students to be sampled regarding aspects of their mental health as they go about their days; EMA studies are generally facilitated via mobile apps that run on Android and iOS. In addition to questionnaire responses, the apps can collect a wealth of data related to mental health from device hardware, such as GPS location, step counts, and indicators of life activity. For instance, the StudentLife study from Dartmouth College was one of the first in this area to assess student mental health using an app11. We were interested in studying aspects of student mental health in general via mobile apps as, to the best of the author’s knowledge, little work has been done in this area at Canadian universities. We also had a particular interest in how mental health measures might be associated with various digital measures coming from smartphone hardware. As a result, we developed and launched the Smart Healthy Campus 2.0 app on Android and iOS. This ran on a research platform we called EMAX, which has been outlined in a previous paper 12. SHC 2.0 is the successor to a limited, pilot SHC 1.0 study, also detailed in a previous paper 13. The SHC 2.0 app asks a small number of mental health-related questions on a daily basis and collects data from phone hardware as questionnaires are submitted and also occasionally in the background. It was designed to be run during traditional on-campus study for an extended period, such as several months or even semesters. However, the SHC 2.0 study ended up being launched and run during a significant portion of the COVID-19 pandemic, as it was still relevant. We also launched a separate, strongly-pandemic focused study/app called Student Pandemic Experience (SPE) 14, which collected more data than the SHC 2.0 study, but was not suitable for use past the pandemic due to the specialized questionnaires used. For SHC 2.0, objectives of the study were to 1) address the lack of longitudinal data in Canada on overviews of student mental health and any potentially related measures of lifestyle, and 2) to identify associations between these self-reported overviews of mental health (from questionnaires) and lifestyle-related measures (from smartphone digital measures). In this study, we focused on providing a more complete analysis of any and all device digital measures that might relate to student mental health rather than focusing heavily on specific ones. Available smartphone digital measures (such as GPS, step counts) were still categorized into related groups, informed by previous work in mobile sensing15, including 1) Movement and physical activity, 2) Device usage, and 3) Social and life activity indicators. As this was exploratory observational research, we did not make specific hypotheses about how these various phone sensors might explain aspects of participant mental health – we were only interested in the prospect of whether any associations between the two might exist. We consequently decided not to elaborate on the meaning or function of any associations. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F1) Figure 1. Image of the Smart Healthy Campus 2.0 app during a questionnaire. ## Methods ### Recruitment Mass email recruitment was conducted. All Western University undergraduate students were invited over email to download the SHC 2.0 app in January 2021 and in March 2021 in order to participate in the study. The app was available on Google Play (Android) and the Apple App store. Sign-up information, including demographic information, was collected entirely in the app. To sign up for the SHC 2.0 app, participants had to view and accept the terms of the letter of information. At sign-up, they were also allowed to consent to providing their Twitter username (for tweet analysis) and to linking their SHC 2.0 app data with health records and psychological services records from campus health services. However, due to low consent rates, these sources of additional data were not used for SHC 2.0. 94 participants signed up for the app, and 86 participants both signed up and did at least one questionnaire in the app. ### Inclusion and Exclusion Criteria The only inclusion criterion was that a participant was a full-time undergraduate student at Western University. Exclusion criteria were students enrolled in graduate or professional programs. Participants had to select their undergraduate program from a list in the SHC 2.0 app during the sign-up process. ### Study Design SHC 2.0 was an Ecological Momentary Assessment-type app-based longitudinal cohort study with a repeated measures design. The SHC 2.0 app collected responses to a short questionnaire with a range of data from phone hardware potentially relevant to student mental health; responses and data were sent encrypted to our server when a participant submitted a questionnaire. The SHC 2.0 app also collected this same phone data in the background automatically on an hourly basis; participants could set this collection event to 1, 2, or 3 hours. This paper reports on the first 40 weeks (out of 52 weeks) of data collected for SHC 2.0; the majority of the study data was collected during this time. Participants could download and use the SHC 2.0 app and complete surveys as often or as little as they wanted, as there were no specific requirements on participation. As a result, compliance rates and missingness vary widely each day/week. ### Participant Compensation The SHC 2.0 app included a points-based incentive system. Participants could accumulate points through app use and redeem 30000 points for a $5 Amazon gift card. Rewards were offered as the study progressed, and there were no limits on the total amount of gift cards redeemed. The first survey of the week, the weekly survey, would yield 3000 points. Daily surveys would provide 500 points, and background data collection events would provide 100. These point values were selected to prevent too much incentivization during a single week. Figure 2 shows an example of using the incentive system. It also allowed for the potential to add new rewards as the study progressed, although only the $5 Amazon gift cards were used during the study. ![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F2) Figure 2. SHC 2.0 point-based incentive system screen (left image). The app has to be scrolled to see the entire screen (right image). ### Measures – Smart Healthy Campus 2.0 Questionnaire – Details The SHC 2.0 app always makes a 9-item questionnaire available once per day. However, there are two variants to this questionnaire; the first is the weekly variant, which available during a new week starting Monday. The weekly variant was designed to capture the experience of the entire past week in case participants were infrequent responders. It would remain open as the first questionnaire to be completed for that week. As an example, if a participant waited until Friday or even Saturday or Sunday to provide a questionnaire response, the weekly variant would still be available then, since that would be the first questionnaire response of that week. If the weekly variant was completed, the daily variant was then available every day until the next Monday, including on Saturday and Sunday. These variants are very similar – only 2 of the 9 questions change and the text of the questions were edited to reflect either the daily or weekly focus. If a participant did not complete at least the weekly variant, then they did not provide any responses at all for that given week, although hourly background data may still have been collected by the SHC 2.0 app. Reminders to complete questionnaires were sent to participant devices via standard push notification manually by research personnel, usually at the beginning of the week, and occasionally throughout the week, but a set schedule for reminders was not followed. These notifications were generic reminders for all participants and did not change if a participant had completed a survey for that day. Question topics from both variants cover a range of mental health-related measures relevant to students. For instance, two questions on resilience (weekly/daily) were included. Resilience is important for mental health, and is the ability to bounce back from, and adapt to difficult situations 16. One question on community connectedness was included (weekly/daily) and was of general interest to this study during typical in-person undergraduate studies, but the shift to online learning during the pandemic likely resulted in this having different but potentially interesting connotations. ### Measures – Smart Healthy Campus 2.0 Questionnaire, Weekly Variant The weekly variant is almost the same as our pilot SHC 1.0 study questionnaire (which was also distributed weekly) and this was shown in the pilot SHC 1.0 study to significantly correlate with mental health domains based on validated full-length questionnaires 13. The first 7 questions are the same as the SHC 1.0 questionnaire. The last two new questions were considered for but disused in the original study; these were added here to increase data collection. ### Measures – Smart Healthy Campus 2.0 Questionnaire, Daily Variant The daily variant was again based on the pilot SHC 1.0 questionnaire, and the content of the first 7 questions was the same, but they had more edits to clarify the questions were reflective of the day. Like the weekly variant, the last two new questions were considered for, but disused, in the initial SHC 1.0 pilot study and were again added here to increase data collection. View this table: [Table 1.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T1) Table 1. Mental health domain coverage of the SHC 2.0 weekly and daily questionnaire variants. This table lists the domain of each of the 9 questions and its source. The two questionnaires are available in Multimedia Appendix 1. ### Digital Measures – Overview From the SHC 2.0 app on the participant devices, 12 digital measures were selected for analysis in this paper (these data came almost exclusively from smartphones but tablets still were occasionally used). There were more than 12 measures collected via the SHC 2.0 app, but these 12 selected for analysis here had the greatest amount of data available across both Android and iOS platforms. A full list of digital measures that the SHC 2.0 app collected can be found in our paper on the EMAX platform which powered the SHC 2.0 app 12. These digital measures were collected during two types of data collection “events”: Response and Background. Response events occurred whenever a participant responded to a survey. Background events occurred as a result of ongoing data collection, which happened on a 1, 2, or 3 hour basis (approximately). Participants could use a slider in the app to choose whether background data collection should occur approximately every 1, 2, or 3 hours. For both the Response and Background events, digital measure data were uploaded to our server. The Response event, unlike the Background event, also contained the response data to a weekly survey. Since Background events did not have this data, but it was needed for mixed model fits, an algorithm was run over all the Background event data to connect it to the most recent Response questionnaire result found before the Background event occurred. ### Digital Measures - Movement and Physical Activity To determine associations between mental health questionnaire results and movement and physical activity, we examined 4 sources of data. First, we looked at total weekly step count. Step counts are commonly analyzed in mobile sensing studies 26. Second, we examined GPS locations visited. We define GPS locations visited as a sum of any significant changes in geographic location. Our SHC 2.0 app only detected GPS coordinates to two decimal places, so a “significant change” of at least 0.01 in the latitude/longitude of a coordinate is required to increase this sum. Third, we define floors walked as floors gone up + floors gone down over the past week. Fourth, a location “flag” was used (called “Campus Dist.” in the results section), and it could have three values: (1) on-campus, (2) in London, Canada and (3) outside of London, Canada. ### Digital Measures - Device usage To determine associations between mental health questionnaire results and device usage, we examined 5 sources of data. First, we collected system uptime in milliseconds. We also collected system uptime (including deep sleep), in milliseconds (uptime + time the device was sleeping). Second, we obtained available RAM (available phone Random Access Memory) in bytes. Unlike CPU usage, available RAM (in bytes) is straightforward to collect across iOS/Android platforms. Third, we obtained a percentage of internal free space. Finally, we timed how long users had run the SHC 2.0 app on their device, in milliseconds. ### Digital Measures – Social/life activity indicators Contacts and calendars are data sources that may be indicative of social or life activity and they were straightforward to access across platforms. The sum of calendar events were collected. These events may include items like reminders for physical activity, events, and appointments. A sum of all contact counts at each sample was also obtained. Additionally, a basic algorithm to check for family-related names (or containing family-related names) in contacts was run to collect a sum of these. ### Digital Measures - Additional item: Number of apps installed One additional data source was included in this analysis and considered a digital measure. This was a percentage of popular installed apps from a known list of 40. However, we did not split these by category due to the small list of 40. ### Statistical Analysis The focus of the statistical analysis in this paper is on the relationship between phone digital measures and the SHC 2.0 questionnaire variants. The total score of the SHC 2.0 questionnaire variants are examined (some items were recoded to make them suitable for a total sum), along with scores from individual questions, which are representative of various mental health domains 13 (see Methods). SHC 2.0 required participants to answer all questions since the questionnaires were short. R 4.1.1 “Kick Things” and RStudio were used for all statistics. All mixed linear models shown were constructed using the lmerTest library 27 which provides p-values for effects using the Satterthwaite degrees of freedom method 27. sjPlot was used for coefficient graphs and ggplot2 was used for line graphs. This approach to mixed linear models with EMA data and the same R packages were previously used by Huckins et al. 28 and Mack et al. 29 in their study on student mental health and the impact of COVID-19 at Dartmouth College. With regards to building the mixed linear models, z-score standardization was used on digital measure items with R’s scale(…) function as values could vary considerably. For each mental health questionnaire (DV), a first model was fit with all 12 previously discussed digital measure items as fixed-effects. The participants were the random-effect. Then, all the digital measure items that lmerTest did not report significance for were removed, and a second, final model was fit with only the significant digital measure items (backward elimination). The reason for fitting and only using a second model with backward elimination was that removing those insignificant terms sometimes greatly increased the amount of complete case digital measure samples available for use in the lmerTest models. It also removed digital measure items with weaker (although in the first model, initially significant) p-values. #### Data Inclusion Highly correlated mixed model parameters can make each other non-significant even if they are predictive. There were many significant correlations between the parameters used for these models, however, the majority were very weak or weak based on Spearman coefficients (0.0-0.20). Some coefficients did enter moderate territory in the range of 0.4 to 0.6. Additionally, there was a strong correlation between system uptime, and system uptime with deep sleep, although we argue this was acceptable as those likely better highlighted differences between device use and potential idle or sleep time for participants; additionally, more than one model resulted in significance for only one of these parameters, while others produced significance for both, suggesting the correlation was not important. We argue these correlations in our 12 selected digital measure items, most of them being very weak or weak, were not detrimental to the mixed model fits as the 12 selected items were the most available across participants and platforms (iOS, Android), and thus best representative of lifestyle and behaviors during this time. Since backward elimination was applied to remove insignificant digital measure items from final models, sample sizes and the associated number of complete case digital measure sample observations used for the model fits may vary, so they are shown with the associated model in the results section. ## Results ### Overview Our main results consist of 17 mixed linear models. Note that the mixed model “Measures” (fixed-effects) such as “RAM” or “GPS Locations Visited” are described in the “Digital Measures” sections under Methods. There are two rounds of fits for the 17 mixed models. The second, revised models are based on backward elimination of the first in order to produce only the significant fixed-effects (device digital measures) that might predict scores for SHC 2.0 weekly/daily questionnaires and also for individual questions. This backward elimination process was used to increase the complete case count - samples that contained data for all 12 digital measures selected. Individual questions are examined here in addition to the total questionnaire score, as on their own, they were designed to capture various domains of mental health (e.g. depression, anxiety) 13 (discussed in Methods). The goal of the mixed linear model analysis is that by knowing the measurable behaviors/lifestyle associated with positive mental health, it should become easier to design traditional or computer-related interventions to direct participants to those associated behaviors/lifestyle. Another goal is that these models might be integrated into future apps that directly provide interventions or predictions, that might be triggered by these models or similar ones. First, the demographics of the 94 participants in SHC 2.0 are presented. Then, we present plots of selected mean scores from our SHC 2.0 weekly questionnaire with key pandemic event markers. Lastly, the results of the 17 mixed linear model fits are shown between the 12 selected phone digital measures and A) the SHC 2.0 daily questionnaire total score and individual questions, and B) the SHC 2.0 weekly questionnaire total score and individual questions. The results of the daily questionnaire model fits are presented to start as more data was collected daily. ### Demographics Table 2 below presents the demographic information of the 94 participants in SHC 2.0. Participants primarily identified as male and living in off-campus housing. View this table: [Table 2.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T2) Table 2. Participant demographics at baseline. As questions can be skipped, sometimes the cells for each item may not necessarily add to 100%. ### Pandemic events and selected mean scores of SHC 2.0 weekly questionnaire Although SHC 2.0 was not specifically designed to run during the pandemic, it ended up being launched during COVID-19 and ran for a significant amount of time during the pandemic, as the app’s data collection and questionnaire remained relevant. We provide visualizations of select mental health measures overlayed by key provincial lockdown measures in Ontario below. The mean scores of the sum of the SHC 2.0 weekly questionnaire, along with means of selected individual questions from the weekly variant, are shown in Figures 3 and 4. Important dates related to the COVID-19 pandemic are shown in Table 3. ![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F3.medium.gif) [Figure 3.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F3) Figure 3. SHC 2.0 mean weekly questionnaire score over study weeks during COVID-19. The weekly score is representative of a broad overview of participant mental health (Life satisfaction, psychological well-being, resilience, anxiety, depression, and community connectedness). Higher scores are representative of a more positive mean overview (ie. better). ![Figure 4.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F4.medium.gif) [Figure 4.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F4) Figure 4. Selected measures from SHC 2.0 over study weeks during the COVID-19 pandemic: community connectedness, psychological well-being, and resilience. Higher scores can be considered positive (ie. better). View this table: [Table 3.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T3) Table 3. Key events related to the SHC 2.0 study and COVID-19 during January 2021-June 2021. These correspond to the event lines (E1-E6) in Figure 3 and 4. #### Interpretation of coefficient plots This section explains how to interpret the coefficient plots of the mixed models below. Every plot shows the amount of change associated with the dependent variable (questionnaire measure) as there is a +1 unit change in the digital measure item (independent variable, y-axis). Note that digital measure items were z-score standardized due to different very scales, so a +1 unit change is a +1 standard deviation from the mean. Dependent variables were not standardized for future prediction purposes. Essentially, an association can be positive or negative, and the value on the x-axis “Estimates” for any predictor on the y-axis would be added to the dependent variable as there is a +1 unit change for that y-axis predictor. ### Mixed linear models of SHC 2.0 daily questionnaire variant In this section, the mixed linear model fits for the measures of the SHC 2.0 daily questionnaire are presented. In the second round of model fits, significant fixed effects were found for all the daily variant’s individual questions and for the total questionnaire sum. #### Regressing daily questions 1-3 on device digital measures View this table: [Table 4.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T4) Table 4. First round of mixed linear models for daily life satisfaction, daily psychological well-being, and daily resilience. #### Revised Models #1-3 using significant parameters only, with coefficient plots View this table: [Table 5.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T5) Table 5. Revised second round (using only significant digital measure items) of mixed linear models for daily life satisfaction, daily psychological well-being, and daily resilience. ![Figure 5.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F5.medium.gif) [Figure 5.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F5) Figure 5. Coefficient plot for parameters of Model #1 – Daily Life Satisfaction. System uptime had a negative association with life satisfaction. There was also a negative association with free space. This suggests less device usage may be associated with increased life satisfaction. There also appeared to be a positive association with floors walked. Based on this model, physical activity did not appear to be associated with increased life satisfaction. ![Figure 6.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F6.medium.gif) [Figure 6.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F6) Figure 6. Coefficient plot for parameters of Model #2 – Daily Psychological Well-being. Surprisingly, app steps had a negative association with daily psychological well-being. There were positive associations with locations visited (to an extent) and daily calendar counts, suggesting more life activity was related to psychological well-being. ![Figure 7.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F7.medium.gif) [Figure 7.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F7) Figure 7. Coefficient plot for parameters of Model #3 – Daily Resilience Measure #1. Floors walked, device usage via SHC 2.0 total app use time, and installed apps all had negative associations with the first daily measure of resilience. There also was a positive association with proximity to campus. #### Regressing daily questions 4-6 on device digital measures Table 6 shows the results of three mixed linear models relating device digital measures to a first daily measure of anxiety, a daily measure of depression, and a daily measure of community connectedness. View this table: [Table 6.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T6) Table 6. First round of mixed linear models for daily anxiety, daily depression, and daily community connectedness. #### Revised Models #4-6 using significant parameters only, with coefficient plots View this table: [Table 7.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T7) Table 7. Revised second round (using only significant digital measure items) of mixed linear models for daily anxiety, daily depression, and daily community connectedness. ![Figure 8.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F8.medium.gif) [Figure 8.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F8) Figure 8. Coefficient plot for parameters of Model #4 – daily anxiety measure #1. Uptime had a small negative association with the first measure of daily anxiety. There was a more significant negative association with the number of installed apps. ![Figure 9.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F9.medium.gif) [Figure 9.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F9) Figure 9. Coefficient plot for parameters of Model #5 – daily depression. App steps and locations visited had minor positive associations with daily depression. There was a positive association with system uptime and daily depression, and a negative association with system sleep time and depression. ![Figure 10.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F10.medium.gif) [Figure 10.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F10) Figure 10. Coefficient plot for parameters of Model #6 – daily community connectedness. App steps, locations visited, and proximity to campus had positive associations with daily community connectedness. There was a negative association with the number of installed apps and system sleep time with community connectedness. The number of weekly calendar events also had a positive association with community connectedness. #### Regressing daily questions 7-9 on device digital measures Table 8 shows the results of three mixed linear models relating device digital measures to a second daily measure of anxiety, a daily measure of distress, and a second daily measure of resilience. View this table: [Table 8.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T8) Table 8. First round of mixed linear models for daily anxiety, distress, and resilience. #### Revised Models #7-9 using significant parameters only, with coefficient plots View this table: [Table 9.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T9) Table 9. Revised second round (using only significant digital measure items) of mixed linear models for anxiety measure #2, distress, and resilience measure #2. ![Figure 11.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F11.medium.gif) [Figure 11.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F11) Figure 11. Coefficient plot for parameters of Model #7 – daily anxiety measure #2. Physical activity (app steps, locations visited, floors walked) and proximity to campus had slight positive associations with anxiety, while installed apps had a more positive association with anxiety. There was a negative association with weekly calendar events. ![Figure 12.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F12.medium.gif) [Figure 12.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F12) Figure 12. Coefficient plot for parameters of Model #8 – Psychological Distress Some measures of physical activity (app steps, locations visited) and daily calendar events had a negative association with distress. Proximity to campus, floors walked, and weekly calendar events had a positive association with distress. ![Figure 13.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F13.medium.gif) [Figure 13.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F13) Figure 13. Coefficient plot for parameters of Model #9 – daily resilience measure #2. App steps had a slight negative association with the second daily measure of resilience. System sleep time, free space also had negative associations with this measure of daily resilience. Locations visited, floors walked, proximity to campus, system uptime, and installed app count all had positive associations with this measure of resilience. #### Result of Model #10: Regressing daily total questionnaire sum on device digital measures View this table: [Table 10.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T10) Table 10. Revised second round (using only significant device digital measure items) of mixed linear model relating daily questionnaire sum to device digital measures. The first model is omitted for brevity. The initial model is included as an Appendix. ![Figure 14.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F14.medium.gif) [Figure 14.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F14) Figure 14. Coefficient plot for parameters of Model #10 – daily questionnaire sum. Floors walked, proximity to campus, and installed app count had positive associations with the daily questionnaire sum. System uptime and daily calendar events had negative associations with the daily questionnaire sum. ### Mixed linear models of SHC 2.0 weekly questionnaire variant In this section, the mixed linear model fits for the measures of the SHC 2.0 weekly questionnaire variant are presented. Note that for several weekly questions, mixed linear model fits did not yield any significant associations. This was the case for questions: 2 (psychological well-being), 5 (depression), and 6 (community connectedness). #### Regressing weekly questions 1, 3, and 4 on device digital measures Table 11 shows the results of three mixed linear models relating device digital measures to weekly life satisfaction, a first weekly measure of resilience, and a first weekly measure of anxiety. View this table: [Table 11.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T11) Table 11. First round of mixed linear models for weekly life satisfaction, weekly resilience, and weekly anxiety. #### Revised Models #11-13 using significant parameters only, with coefficient plots View this table: [Table 12.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T12) Table 12. Revised second round (using only significant digital measure items) of mixed linear models for weekly life satisfaction, weekly resilience measure #1, weekly anxiety measure #1. ![Figure 15.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F15.medium.gif) [Figure 15.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F15) Figure 15. Coefficient plot for parameters of Model #11 – weekly life satisfaction. App steps, calendar events on the day the weekly variant was completed, proximity to campus, and installed app count all had positive associations with weekly life satisfaction. There was a positive association with system sleep time which may be indicative of participants getting more sleep. There was a corresponding negative association with system uptime. ![Figure 16.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F16.medium.gif) [Figure 16.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F16) Figure 16. Coefficient plot for parameters of Model #12 – weekly resilience measure #1. The first measure of weekly resilience had a positive association with distance from campus and device free space. Weekly resilience had a negative association with installed app count, locations visited, and floors walked. ![Figure 17.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F17.medium.gif) [Figure 17.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F17) Figure 17. Coefficient plot for parameters of Model #13 – weekly anxiety measure #1. There was a slight positive association with locations visited and weekly anxiety. This weekly anxiety measure had negative associations with floors walked, installed app count, and proximity to campus. #### Regressing weekly questions 7, 8, and 9 on device digital measures Table 13 shows the results of three mixed linear models relating device digital measures to weekly physical activity, a second measure of anxiety, and a second measure of resilience. View this table: [Table 13.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T13) Table 13. First round of mixed linear models for weekly physical activity, weekly anxiety, and weekly resilience. #### Revised Models #14-16 using significant parameters only, with coefficient plots View this table: [Table 14.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T14) Table 14. Revised second round (using only significant digital measure items) of mixed linear models for weekly physical activity, a second weekly anxiety measure, and a second weekly measure of resilience. ![Figure 18.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F18.medium.gif) [Figure 18.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F18) Figure 18. Coefficient plot for parameters of Model #14 – weekly physical activity. As expected, app steps had a positive association with weekly physical activity. There was also a significant negative association in system up time, but a positive association with system sleep time, which may be indicative of physically active participants getting more sleep. Physically active participants had associations with more events on the day they completed the weekly survey, but a negative association with events throughout their week. ![Figure 19.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F19.medium.gif) [Figure 19.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F19) Figure 19. Coefficient plot for parameters of Model #15 – weekly anxiety measure #2. Like the first weekly anxiety measure, this measure of anxiety had a negative association with floors walked. Calendar events, on the day the weekly questionnaire was answered, also had a negative association with anxiety in this second measure. ![Figure 20.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F20.medium.gif) [Figure 20.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F20) Figure 20. Coefficient plot for parameters of Model #16 – weekly resilience measure #2. App steps had a negative association with this second measure of weekly resilience. There was a positive association with floors walked, and proximity to campus. #### Result of Model #17: Regressing weekly total questionnaire sum on device digital measures View this table: [Table 15.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/T15) Table 15. Revised second round (using only significant digital measure items) of mixed linear model for weekly questionnaire sum and device digital measures. The first model is omitted for brevity. The initial model is included as an Appendix. ![Figure 21.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/28/2023.03.22.23287598/F21.medium.gif) [Figure 21.](http://medrxiv.org/content/early/2023/03/28/2023.03.22.23287598/F21) Figure 21. Coefficient plot for parameters of Model #17 – sum of weekly questionnaire. Floors walked, proximity to campus, system sleep time, installed apps, and to a lesser extent, calendar events the day this was answered, all had positive associations with the weekly questionnaire sum. System uptime had a negative association with the weekly questionnaire sum. These results may suggest participants were getting more sleep, despite the weekly score seemingly being associated with an increase in installed apps. ## Discussion ### Principal Results The results from the SHC 2.0 study over this period did not yield as interesting results as an initial analysis of our SPE study data30, likely due to the smaller sample size (N=86 compared to N=315) and reduced number of observations (N=6518 compared to N=25985) in comparison. However, some significant associations were still identified. While there was a reasonable amount of data collected via the daily questionnaire, some results showed small effect sizes and were occasionally conflicting where there were multiple questions for the same measure, such as for resilience and anxiety. System uptime had a negative association with life satisfaction. Physical activity (app steps) appeared to have a negative association with daily psychological well-being, although this may be related to the small sample size of the study. Daily anxiety results were conflicting. The first measure suggested installed app count had a negative association with daily anxiety; the second measure showed the opposite of this. Other findings, such as the positive associations with app steps, locations visited, floors walked, proximity to campus, were minor. Daily resilience did appear to be associated with proximity to campus and negatively associated with use of the SHC 2.0 app. There also appeared to be a positive association with device uptime, although other results were conflicting. Daily community connectedness appeared to be associated with proximity to campus. System uptime had a positive association with daily depression. There were some significant associations with device digital measures and the daily questionnaire sum, although none were related to physical activity. For the weekly questionnaire, certain interesting results were found. For instance, device sleep time and install app count had a positive association with weekly life satisfaction. Device sleep time and daily calendar events had a positive association with physical activity. This may indicate participants were getting more sleep or just using their devices less when physically active. The significant associations found with both weekly resilience questions were inconsistent. Floors walked, installed app count, and calendar counts had a negative association with weekly anxiety. Like the daily questionnaire, there were some significant associations between device digital measures and the weekly questionnaire sum, although none were related to physical activity. For three weekly questions (2 - psychological well-being, 5 - depression, and 6 - community connectedness) the mixed linear model fits did not yield any significant associations. ### Limitations The models have some limitations which likely impacted the results. Missingness did vary on a weekly/daily basis and it likely impacted the results. For instance, there were about 2500 complete case observations from 46 participants daily and 1000-1500 complete case observations from 40-60 participants weekly. The sample size is relatively small compared to the total population of a mid-size university in Canada, although there are a reasonable number of observations for the participants. We were unable to analyze call/text message data from participants due to increasing privacy restrictions on Android/iOS. ### Conclusions Undergraduate studies (and pandemics) present unique conditions - and opportunities - for surveillance efforts with regards to mental health. This research found that there were some significant associations between phone digital measures and mental health-related self-reports from undergraduates in Ontario, Canada during pandemic conditions. Future opportunities for research include investigations into predictive modelling via machine learning or other means, the comparison of post-pandemic data, and the development of new interventions, such as interactive virtual assistants, or others of ranging complexity, which could potentially incorporate the evidence presented here. While an initial analysis of our related SPE study data30 appeared to produce more results as there were considerably more participants and digital measure samples (315 and approximately 25985, respectively) SPE was not designed as a long-term study and has concluded. There remains an opportunity to resume SHC 2.0 and compare data collected during the pandemic to post-pandemic data. Overall, the SHC 2.0 study identified some measures of lifestyle that appear to be associated with measures of mental health in undergraduate students. ## Data Availability The data for this project is not publicly available as it was an EMA-type study that repeatedly sampled the same participants for potentially sensitive data, such as location. ## Contributorship The authors provided have the entire authorship of this article. ## Declaration of conflicting interests None declared. ## Funding Study incentives were funded by FIMS, Western University, Canada. ## Ethical approval Ethics approval was provided by Western University’s Health Sciences Research Ethics Board (HSREB) in October 2020 (Project ID 116670). ## Informed consent Consent was obtained from all research participants prior to the commencement of the research. ## Supplemental material Supplemental material for this article is available online. ## Acknowledgements C.B. thanks Dr. Kevin Shoemaker for initial involvement in Smart Healthy Campus, and Ms. Arlene Fleischhauer for support of that work. C.B. thanks Dr. Lorie Donelle for suggestions to the EMAX software that were incorporated into the SHC 2.0 app. C.B. also thanks Dr. Eva Pila and MacLean Press for contributions to the SHC 2.0 study and app. ## Abbreviations CPU : Central Processing Unit EMA : Ecological Momentary Assessment EMAX1 : Ecological Momentary Assessment eXtensions 1st edition software EMAX2 : Ecological Momentary Assessment eXtensions 2nd edition software SHC : Smart Healthy Campus SHCv2 : Smart Healthy Campus Version 2 study SPE : Student Pandemic Experience * Received March 22, 2023. * Revision received March 22, 2023. * Accepted March 28, 2023. * © 2023, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. 1.Fried RR, Karmali S, Irwin JD, et al. Making the grade: Mentors’ perspectives of a course-based, smart, healthy campus pilot project for building mental health resiliency through mentorship and physical activity. Int J Evid Based Coach Mentor 2018; 16: 84–98. 2. 2.Council of Ontario Universities. In It Together: Taking Action on Student Mental Health, [https://cou.ca/wp-content/uploads/2017/11/In-It-Together-PSE-Mental-Health-Action-Plan.pdf](https://cou.ca/wp-content/uploads/2017/11/In-It-Together-PSE-Mental-Health-Action-Plan.pdf) (2017). 3. 3.Doreleyers A, Knighton T. COVID-19 Pandemic: Academic Impacts on Postsecondary Students in Canada. StatCan COVID-19: Data to Insights for a Better Canada. Stat Canada. 4. 4.Wall K. COVID-19 Pandemic: financial impacts on postsecondary students in Canada. Stat Canada. 5. 5.Hicks LJ, Caron EE, Smilek D. SARS-CoV-2 and Learning : The Impact of a Global Pandemic on Undergraduate Learning Experiences. Scholarsh Teach Learn Psychol. 6. 6.Tasso AF, Hisli Sahin N, San Roman GJ. COVID-19 disruption on college students: Academic and socioemotional implications. Psychol Trauma Theory, Res Pract Policy. Epub ahead of print 2021. DOI: 10.1037/tra0000996. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/tra0000996&link_type=DOI) 7. 7.Prati G, Mancini AD. The psychological impact of COVID-19 pandemic lockdowns: A review and meta-analysis of longitudinal studies and natural experiments. Psychological Medicine. Epub ahead of print 2021. DOI: 10.1017/S0033291721000015. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1017/S0033291721000015&link_type=DOI) 8. 8.Adlaf EM, Gliksman L, Demers A, et al. The prevalence of elevated psychological distress among canadian undergraduates: Findings from the 1998 Canadian campus survey. J Am Coll Health Assoc. Epub ahead of print 2001. DOI: 10.1080/07448480109596009. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/07448480109596009&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11590985&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000171507500003&link_type=ISI) 9. 9.Academic and Mental Health Needs of Students on a Canadian Campus. Can J Couns Psychother / Rev Can Couns psychothérapie. 10. 10.Giamos D, Lee AYS, Suleiman A, et al. Understanding Campus Culture and Student Coping Strategies for Mental Health Issues in Five Canadian Colleges and Universities. Can J High Educ. Epub ahead of print 2017. DOI: 10.7202/1043242ar. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7202/1043242ar&link_type=DOI) 11. 11.Wang R, Chen F, Chen Z, et al. Studentlife: Assessing mental health, academic performance and behavioral trends of college students using smartphones. In: UbiComp 2014 - Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2014. Epub ahead of print 2014. DOI: 10.1145/2632048.2632054. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1145/2632048.2632054&link_type=DOI) 12. 12.Brogly C, Lizotte DJ, Bauer MA. Ecological Momentary Assessment eXtensions 3 (EMAX3) Proposal: An App for EMA-Type Research. In: 2021 Symposium on Computers and Communications (ISCC): 26th IEEE Symposium on Computers and Communications - Workshop on ICT Solutions for eHealth (ICTS4eHealth) (ICTS4eHealth2021). Athens, Greece, 2021. 13. 13.Brogly C, Shoemaker JK, Lizotte DJ, et al. A Mobile App to Identify Lifestyle Indicators Related to Undergraduate Mental Health (Smart Healthy Campus): Observational App-Based Ecological Momentary Assessment. JMIR Form Res. Epub ahead of print 2021. DOI: 10.2196/29160. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/29160&link_type=DOI) 14. 14.Brogly C, Bauer MA, Lizotte DJ, et al. An app-based surveillance system for undergraduate students’ mental health during the covid-19 pandemic: Protocol for a prospective cohort study. JMIR Res Protoc. Epub ahead of print 2021. DOI: 10.2196/30504. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/30504&link_type=DOI) 15. 15.Mohr DC, Zhang M, Schueller SM. Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning. Annu Rev Clin Psychol. Epub ahead of print 2017. DOI: 10.1146/annurev-clinpsy-032816-044949. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1146/annurev-clinpsy-032816-044949&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28375728&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) 16. 16.Tugade MM, Fredrickson BL. Resilient Individuals Use Positive Emotions to Bounce Back From Negative Emotional Experiences. Journal of Personality and Social Psychology. Epub ahead of print 2004. DOI: 10.1037/0022-3514.86.2.320. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/0022-3514.86.2.320&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=14769087&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000188926800011&link_type=ISI) 17. 17.Inglehart, R., C. Haerpfer, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, P. Norris EP& BP, et al. (eds. World Values Survey: All Rounds - Country-Pooled Datafile Version: [http://www.worldvaluessurvey.org/WVSDocumentationWVL.jsp](http://www.worldvaluessurvey.org/WVSDocumentationWVL.jsp). Madrid: JD Systems Institute. World Values Survey: All Rounds - Country-Pooled Datafile. 18. 18.Berwick DM, Murphy JM, Goldman PA, et al. Performance of a five-item mental health screening test. Med Care. Epub ahead of print 1991. DOI: 10.1097/00005650-199102000-00008. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/00005650-199102000-00008&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=1994148&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1991EX22500008&link_type=ISI) 19. 19.Smith BW, Dalen J, Wiggins K, et al. The brief resilience scale: Assessing the ability to bounce back. Int J Behav Med. Epub ahead of print 2008. DOI: 10.1080/10705500802222972. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/10705500802222972&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18696313&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000258820800006&link_type=ISI) 20. 20.Spitzer RL, Kroenke K, Williams JBW, et al. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch Intern Med. Epub ahead of print 2006. DOI: 10.1001/archinte.166.10.1092. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/archinte.166.10.1092&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16717171&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000237720500008&link_type=ISI) 21. 21.Henry JD, Crawford JR. The short-form version of the Depression anxiety stress scales (DASS-21): Construct validity and normative data in a large non-clinical sample. Br J Clin Psychol. Epub ahead of print 2005. DOI: 10.1348/014466505X29657. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1348/014466505X29657&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16004657&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000229910200007&link_type=ISI) 22. 22.Mashek D, Cannaday LW, Tangney JP. Inclusion of community in self scale: A single-item pictorial measure of community connectedness. J Community Psychol. Epub ahead of print 2007. DOI: 10.1002/jcop.20146. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/jcop.20146&link_type=DOI) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000244427700008&link_type=ISI) 23. 23.Milton K, Bull FC, Bauman A. Reliability and validity testing of a single-item physical activity measure. Br J Sports Med. Epub ahead of print 2011. DOI: 10.1136/bjsm.2009.068395. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYmpzcG9ydHMiO3M6NToicmVzaWQiO3M6ODoiNDUvMy8yMDMiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8wMy8yOC8yMDIzLjAzLjIyLjIzMjg3NTk4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 24. 24.Ostroff JS, Woolverton KS, Berry C, et al. Use of the Mental Health Inventory with adolescents: A secondary analysis of the Rand Health Insurance Study. Psychol Assess. Epub ahead of print 1996. DOI: 10.1037/1040-3590.8.1.105. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/1040-3590.8.1.105&link_type=DOI) 25. 25.Heubeck BG, Neill JT. Confirmatory factor analysis and reliability of the mental health inventory for australian adolescents. Psychol Rep. Epub ahead of print 2000. DOI: 10.2466/pr0.2000.87.2.431. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2466/pr0.2000.87.2.431&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11086588&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) 26. 26.Sun S, Folarin AA, Ranjan Y, et al. Using smartphones and wearable devices to monitor behavioral changes during COVID-19. J Med Internet Res. Epub ahead of print 2020. DOI: 10.2196/19992. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/19992&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32877352&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F28%2F2023.03.22.23287598.atom) 27. 27.Kuznetsova A, Brockhoff PB, Christensen RHB. lmerTest Package: Tests in Linear Mixed Effects Models. J Stat Softw. Epub ahead of print 2017. DOI: 10.18637/jss.v082.i13. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.18637/jss.v082.i13&link_type=DOI) 28. 28.Huckins JF, da Silva AW, Wang W, et al. Mental health and behavior of college students during the early phases of the COVID-19 pandemic: Longitudinal smartphone and ecological momentary assessment study. J Med Internet Res. Epub ahead of print 2020. DOI: 10.2196/20185. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/20185&link_type=DOI) 29. 29.Mack DL, DaSilva AW, Rogers C, et al. Mental health and behavior of college students during the covid-19 pandemic: Longitudinal mobile smartphone and ecological momentary assessment study, part II. J Med Internet Res. Epub ahead of print 2021. DOI: 10.2196/28892. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/28892&link_type=DOI) 30. 30.Brogly C. Observations of positive mental health indicators in undergraduates using specialized mobile apps during the COVID-19 pandemic. Western University, [https://ir.lib.uwo.ca/etd/8532](https://ir.lib.uwo.ca/etd/8532) (2022).