AutoTriage - An Open Source Edge Computing Raspberry Pi-based Clinical Screening System

A. Febrile state detection

Infrared imaging has been used extensively for remote, contactless human body temperature estimation for the last two decades since SARS (18). The efficacy of using infrared imaging for mass fever screening was first validated in (19), which demonstrated an ability to detect hyperthermia and a good correlation between thermal scanner readings and ear temperature. The efficacy of using a handheld FLIR350 camera (20) and the FLIRONE camera (which uses the Lepton sensor) (21) for febrile state detection were then validated. Recently, a deep learning face detection method was introduced to detect the face in thermal images for febrile state detection (22). However, the deep learning approach is only used intermittently due to the high processing time, and the region of interest was arbitrarily defined, leading to potential limitations in the usage.

B. Cyanosis detection

Assessment of cyanosis is often conducted visually by doctors. The only known general method to semi-automatically detect cyanosis via a color correction and manual lips segmentation was proposed in a brief conference article (23).

C. Heart rate estimation

Heart rate is one of the most frequently measured human vital signs. It is usually measured using electrocardiography, pulse oximetry, or by counting by radial palpation (24). With advances in the field of computer vision, various camera-based heart rate estimation methods have emerged. These methods have the advantage of being contactless. Subtle head motion is caused by the Newtonian reaction to the influx of blood at each beat. In (25), the authors report a method which leverages this behavior to measure the heart rate. The motion of the head is extracted by feature tracking and principal component analysis is applied to decompose the trajectories of the features into its component motions. The component which has a frequency spectrum that corresponds to the cardiac frequency interval is selected. The motion in this component is analyzed to identify peaks, which correspond to heartbeats. However, various noise sources such as internal and external head motions, low facial frame quality from video or camera and abnormal posture affect the heart rate estimation. Some solutions to these issues are proposed in (26), which introduces a face quality assessment method to ensure that low-quality frames do not contribute to the estimation of heart rate. Feature points from the face are combined with facial landmarks in order to create stable trajectories that are used to estimate heart rate.

The first remote photoplethysmogram (rPPG) imaging method, which used ambient light to estimate heart rate, was introduced in (27). The red, green and blue (R, G, and B) channels are extracted from a region of interest (ROI) in the frame. A raw photoplethysmogram (PPG) signal is generated using the spatial average pixel values of the channels from each frame over time. This raw signal is bandpass filtered to remove noise.

In (28) authors extract PPG signals from facial videos using blind source separation. The mean R, G, B channel values are calculated for the ROI in each frame over time. Here the ROI is the entire face. These raw signals generated from the means are normalized and decomposed into three source signals using Independent Component Analysis. The source signal which corresponds closest with a PPG signal is used to measure the heart rate, which somewhat limits the utility, since the heart rate estimation must be seeded with a known heart rate in the first place. A detailed review of many other heart rate estimation methods is given in (29).

D. Respiratory effort estimation

From non-invasive methods to non-contact methods, various methods have been proposed to automatically estimate respiration rate and respiratory effort. Most early methods focused on replacing invasive methods like esophageal manometry via non-invasive methods, such as measuring the external breathing airflow via nasal cannula/pressure transducer system (30), measuring movement using diaphragm mechanomyography (31) or accelerometer (32), or measuring indirectly from surface electrocardiogram (33).

More recently, contactless approaches have gained popularity. A vast majority of them utilized either rPPG or a certain type of measurement for respiration induced motion. Karlen et al. acquired rPPG was via video in (34, 35) and proved to be feasible to estimate respiratory rates. However, respiratory effort estimation was not addressed in methods using the rPPG approach. The effectiveness of motion measurement based methods has also demonstrated with Doppler radar (36) and visible video (37). Additionally, it has been reported that respiratory pattern can be reconstructed with a motion based method (37), which enables estimation of respiratory effort.

Many other works have described attempts to derive physiology from video cameras, and an extensive comparison of the literature can be found in (38). In particular, the authors note that many approaches have been described in this field and that ‘the lack of standardization hinders comparability of these techniques and of their performance’. Notably they advise that sharing algorithms and/ or datasets would address this issue and potentially allow the application of newer techniques, such as deep learning’. Notably, none of the reviewed systems are available open source or have been tested in extremely large populations in noisy real-world settings. In this work we leverage deep learning algorithms developed on public data, and begin the work of applying these using a framework that we have publicly released to help reach critical mass with public data and repeatable techniques.

E. Hardware configuration

The system proposed is an off-body camera based system to detect symptoms of respiratory illnesses. A Raspberry Pi 4 (RasPi) with 4GB RAM is used as the main processor and is used to run most of the algorithms. A Google Coral USB Accelerator is used to perform person detection, which uses a deep learning based algorithm, and thus needs significantly more computational power. The accelerator is designed to run deep learning models optimally.

A Raspberry Pi camera v2, which is a visible light camera, is used to detect people and for cyanosis, heart rate, and respiration effort estimation. A FLIR Lepton 3.5 Radiometry Long-Wave Infrared Camera with its associated IO module is used for febrile state detection. The left panel in Figure 1 shows the proposed system. An optional temperature and humidity sensor is added to help with the thermal camera calibration. For this work, we tested the Gowoops 2 PCS DHT22 Temperature Humidity Sensor Module. An optional car battery power-source is also detailed in this work for use in remote and rugged locations.

• Download figure
• Open in new tab

Fig. 1.

Left: Hardware configuration consisting of a Raspberry Pi 4 (4 GB RAM), a Google Coral USB accelerator (top), a RasPi Camera v2 and a FLIR Lepton 3.5 Radiometry Long-Wave Infrared Camera. Right: Installation of system with all cabling including power and external monitor for visualization (not shown or needed for detection).

Our system works in real-time, with the visible light camera capturing frames at a rate of 25Hz and the thermal camera at a rate of 9Hz. The various estimation results can be displayed on an external monitor via an HDMI-microHDMI cable or on an LCD display screen directly connected to the RasPi. (We chose a $20 touch-screen 3.5 inch LCD for compactness.) The full setup can be seen in the right panel of Figure 1. The costs associated with various parts are included in the appendix. The total retail off-the-shelf cost of the system is around $480 (not including the car battery).

F. Electrical supply

The design of this system focuses on low-resource mass-triage scenarios, where power is limited or absent. We therefore chose to power our system from a car battery, which is a readily available power source across the planet. Since car batteries supply energy at 12V, a stepdown transformer is required. We chose a 12V to 5V DC step-down power converter that comes with a 15W output (3A at 5V). The transformer was equipped with a USB Type-C power supply output, which was connected to the RasPi.

Noting that the average car battery supplies 40 Ampere-hours or 144, 000 Coulombs at 12V for one charge cycle, and the RasPi together with the coral, transformer and LCD touchscreen draw 3A at 5V, the car battery can supply power for hours continuously before discharging. Assuming we do not want the car battery to go below 50% of its capacity, we expect the car battery to run the unit for 16 hours continuously before needing to be recharged.

G. Temperature and Humidity Detection

A temperature-humidity detection module was added to assist with the calibration of the FLIR lepton camera. Specifically, we used the DHT22 sensor module that is comprised of a capacitive humidity sensor and a thermistor that measures the surrounding air to provide calibrated temperature and humidity values.

This module comes with a digital board that houses three pins, namely VCC, GND, and OUT. The sensor has an operating voltage of 3.3/5V (DC), and the OUT can be read from a GPIO pin on the RasPi. The temperature range is −40 to 80°C, and the humidity range is 0 − 100%RH. The associated Python software that allowed us to integrate the sensor with the RasPi is open source and available in our Github repository (39).

H. Face and thorax detection

To estimate febrile state, cyanosis, heart rate, and respiration effort, it is necessary to detect people in a frame and segment out certain regions of the face. For febrile detection, the forehead-eye region is necessary. Heart rate estimation is performed on the area of the face below the eyes. For cyanosis estimation, the lips need to be segmented, and the thorax region is necessary for respiratory effort estimation.

We use PoseNet (40) to detect people in a frame. This is a convolutional neural network based algorithm which regresses keypoints of human beings in an image or video. Here, keypoints refer to image coordinates of certain key parts of the body, such as the elbows, knees, eyes, nose, etc.

We use the estimated keypoints of the left and right eyes to extract the various face segments mentioned above. If we define the distance between the eyes to be D pixels, then to determine the bounding box around the forehead-eye region, we use a rectangle that has width 2D and a height 1.2D. The base of the bounding box is 0.2D below the eyes. This ensures that the forehead-eyes area is captured. This is an important site since the inner canthus of the eye is consistently the warmest area on the head and the most suitable area for fever detection (41). To create a bounding box around the lips, we move a distance D below the eyes and create a bounding box with width D and height 0.5D. For heart rate estimation, the bounding box has a height of 0.8D, starting 0.2D below the eyes and extending downwards. It has width 1.2D, which extends from 0.1D left of the left eye to 0.1D right of the right eye. The bounding box for respiratory effort estimation is obtained by using the shoulder keypoints as reference. The top edge of the bounding box is formed by joining the shoulder keypoints. If we let the number of pixels between the shoulders be denoted, R, then the bottom edge is 1.5R pixels below the top edge. I.e., the bounding box has a width of R pixels and height of 1.5R pixels. Note that the coordinates obtained by applying this heuristic are rounded to the closest integer value

These values were set empirically (through trial and error using the authors as test points). With few subjects in a given lockdown space, large scale experiments in a short period of time were not possible. In future experiments, these values can be optimized on larger datasets. Figure 2 shows an example of the forehead and lip detection on one of the authors using this approach in the visible spectrum (upper plot) and the corresponding FLIR image (lower plot). Note that the images have slight FOV, image angle, and translational differences since the cameras cannot take images from the same location in space and operate at different (non-synchronous) sampling rates.

• Download figure
• Open in new tab

Fig. 2.

Bounding boxes detected for the forehead and lips from a 1m range in visible light video (upper image) and thermal video (lower image) using PoseNet.

I. Febrile state detection

Unlike the previous studies, our proposed system detects key points using visible light video and then transforms the coordinates of the bounding boxes of the ROI to coordinates in the thermal video. After finding the ROI in the thermal video, the ten pixels exhibiting the highest temperatures are averaged to produce a final temperature estimate. Lastly, a threshold is set to determine the febrile state.

I.1 Thermal output calibration

To achieve a more accurate measurement of the body temperature, we followed the guidelines from the FLIR Lepton 3.5 datasheet (42) and used a robust regression to map the temperature output to the ground truth within the desired range.

We used bottles (with open lids) of heated water with temperature ranging from 35− 40°C as a heat source and located them at one meter to the camera and approximately in the center of the field of view (FOV). (See figure 3.)

• Download figure
• Open in new tab

Fig. 3.

Illustration of the water calibration experiment. The bright spot in the lower left quadrant of the FLIR output represents the water heated to varying known temperatures.

The reference temperature of the water was measured three times using a Braun IRT6500 thermometer and averaged. The reference has an accuracy of ± 0.2°C within the 35 − 42°C measurement range. Also, the top ten pixels in the heat source were selected and averaged as the final output of the FLIR Lepton camera. The slope and the intercept are then fitted using the above-described experiment via Huber regression implemented in scipy 1.2.3. The Root Mean Square Error (RMSE) metric was used to evaluate the fitting error. The fitted line was then implemented to convert pixels values to temperature output.

J. Cyanosis detection

To detect cyanosis, we use the image of the lips, which is segmented as explained in section H. In this work, we have tested three classification algorithms: K nearest neighbor algorithm, logistic regression, and support vector classifier on a dataset of cyanotic and non-cyanotic lips. We picked the algorithm with the best performance to run in real-time on the RasPi. This algorithm classifies images of lips segmented from the frames coming from the visible light camera as cyanotic or non-cyanotic.

K. Heart rate estimation

Our implementation of heart rate estimation uses the visible light camera in our system. We have followed the algorithm detailed in (46) for our implementation. This method attempts to nullify the effects of the motion of the subject and illumination variation while measuring heart rate, which are common problems in a real-world setting. The steps involved are detailed below. It is worth noting that a similar pipeline can be applied to thermal video as well, and the fluctuation of the output in ROI can also be viewed as remote PPG.

L. Respiration effort estimation

Respiration rate is a difficult parameter to estimate, given that it can be non-periodic, spectral estimation techniques are likely to fail. In particular, we are looking to observe the pattern of respiration (i.e., struggling to breath), rather than any respiratory rate (although hyperventilation is important to identify). Rather than estimating a rate, we chose to derive the respiratory effort tracing that can then be used to derive further metrics (e.g., high sample entropy with a peak in energy between 0.05-2 Hz may identify disordered, but rapid, breathing). Of course, much more data is needed before reliable metrics and thresholds can be determined.

To determine the respiratory effort, we isolate the upper thorax using a bounding box that covers the regions from the shoulders to the diaphragm. Then we calculated the first difference between each pixel in subsequent frames. The final effort signal is then given as the average of each ‘difference’ image after a post-processing bandpass filter step (with a passband of 0.5Hz and 2Hz).

M. Febrile state detection

The preliminary test was conducted on a healthy male subject. A heated cloth was put on the subject’s forehead to raise the temperature of the subject’s forehead. The estimated temperature from the proposed system was recorded immediately after a measurement from the thermometer in the center of the subject’s forehead. The subject was sitting one meter away from the cameras. The RMSE between the estimated value and temperature from the thermometer was used to evaluate the accuracy of our proposed system.

N. Heart rate estimation

To test the algorithm in real time, one of the authors was used as the subject. The subject was seated in an indoor setting at distances of 50cm and 1m from the system. The heart rate was measured using the system, and the pulse reading was taken at the wrist simultaneously. The pulse readings are used as ground truth. Each reading was one minute long. This was repeated for different heart rate by increasing the heart rate using cardiovascular exercises. The subject tried to stay as still as possible to get accurate readings. 30 readings were recorded in total, with 15 readings recorded at a distance of 50cm from the system and 15 readings recorded at a distance of 1m from the system.

O. Respiratory effort estimation

To test this concept, one of the authors was used as a subject. The subject stood 0.5m away from the system, and the algorithm was run to estimate respiratory effort for 30 seconds in each trial. Respiratory rate was recorded simultaneously by counting the number of breaths per minute. There were a total of 14 trials where 7 were normal breathing, and the other 7 were faster and harder breaths to simulate shortness of breath. It was ensured that there was minimal illumination variation in the environment in order to avoid interference from varying ambient lighting.

P. Febrile state detection

The left subfigure in figure 4 shows the measurements used in the calibration process and the fitted results. The calibrated values of the slope and intercept were determined to be 0.0113 and 313.383, respectively, and the resultant RMSE was 0.57°C. The right subfigure in figure 4 shows the measured data points in the preliminary experiment, in which the proposed system achieved an RMSE of 0.41°C and a Pearson correlation coefficient of 0.96. When applying 37.4°C as the threshold for febrile state detection, an accuracy of 96.7% and an area under receiver operating characteristic curve (AUC) of 0.97 were achieved.

• Download figure
• Open in new tab

Fig. 4.

Upper: Calibration of FLIR images using a bottle of water: Best fitted line between the average temperature from the thermometer and the FLIR pixel values had a root mean square error (RMSE) of 0.57 °C. Lower: Estimated temperature vs. temperature measured from thermometer. The Pearson correlation between the parameters was found to be 0.96 and the RMSE difference between them was 0.41 °C.

Since the system described here was only created in an apartment with a narrow temperature and light range, which is not necessarily reflective of how this tool might be used in a real triage situation, we evaluated the system in Emory’s ED and discussed its utility with the ED team. When used in a bright cold environment, the temperature estimation appeared to run about 2 degrees °C lower than in the training environment, and the lip analysis triggered many false positives.

Q. Cyanosis detection

Tables 1 shows the confusion matrices for each classifier. Figure 5 shows the receiver operating characteristic curve and AUC of each classifier. Table 2 summarizes the accuracy, AUC, sensitivity and specificity of each of the three classification models.

• Download figure
• Open in new tab

Fig. 5.

Receiver operating characteristic curves for the three classifiers evaluated in this work for cyanosis detection. The filled circles represent the operating points resulting in the other performance statistics. (Small differences exist due to the leave-one-out cross-validation approach.)

To assess the relative importance of the features used for classification, we visualized the weights assigned to different features by the logistic regression classifier, since this is easier to interpret than the parameters of the SVR or KNN. Figure 6 shows the weights for the eighteen features used (six bins each for each of the three R, G and B channels). The first six features correspond to the red channel (R1 - R6). The next six features correspond to the green channel (G1 - G6) and the last six features correspond to the blue channel (B1 - B6).

• Download figure
• Open in new tab

Fig. 6.

Weighting of features from logistic regression. The features are the histogram values the R, G and B channels. Positive coefficients refer to cyanotic condition and negative refer to non-cyanotic.

R. Heart rate estimation

Figure 7 provides a Bland-Altman plot for the heart rate estimates. The mean of the absolute difference between the ground truth and estimate values is 16.31 bpm, and the standard deviation of this absolute difference is 14.42 bpm. It can be observed that the estimate struggles to provide an accurate hear rate above 70 bpm. This could be due to small movements within the ROI - the subject had to perform cardiovascular exercises to raise the heart rate. The breathing rate also increased, which caused movement in the face while inhaling and exhaling. Another issue that contributed to the errors was sudden changes in lighting.

• Download figure
• Open in new tab

Fig. 7.

Bland-Altman plot of estimated heart rates from visible camera from a single subject

S. Respiratory effort estimation

Figure 8 shows the time series signal obtained by the process described above (taking the mean value of all pixels of the difference between consecutive frames and passing them through a bandpass filter). On the left hand side is the signal obtained for rapid breathing (tachnypnea) at a respiratory rate of 20 breaths per minute. On the right side of the image is the signal obtained for hyperventilation at 60 breaths per minute.

• Download figure
• Open in new tab

Fig. 8.

Plot of respiratory effort estimate over time at tow different rates

Figure 9 shows the distribution of energy in the 0.05Hz to 2Hz frequency range of the signal along side the corresponding respiratory rate measured. It can be seen that the energy values approximately track the respiratory rate. The Pearson correlation coefficient between the respiration rate and energy is 0.63.

• Download figure
• Open in new tab

Fig. 9.

Plot of energy in signal against respiration rate.

T. Clinical feedback

Figure 10 illustrates the system being used (hand-held version, not on tripod) in a field test at the emergency department. A series of informal tests for varying lighting conditions were made, and informal discussions were conducted with the clinical staff. (Formal testing was not possible because no ethical review approval had been made at the time.)

• Download figure
• Open in new tab

Fig. 10.

Photo of one of the authors testing out the system at the emergency department during a preliminary field trial.

U. Forehead and lip segmentation

Detecting and segmenting out the forehead and lips is the first step in our pipeline. The accuracy of this stage can determine the accuracy of the remaining stages. This step is dependent on the performance of PoseNet, which sometimes provides false positive detection of individual’s faces. In this implementation, it is limited to detecting ten people in any given frame.

V. Febrile state detection

Previous meta-reviews suggest that peripheral temperature may not be sufficient to determine fever (52, 53). This suggests that our proposed system, along with all traditional methods that measure peripheral temperature, like an ear thermometer, are not suitable to be used to perform an accurate diagnosis of febrile state. However, the proposed system is useful to perform mass early screening of the febrile state as a triage tool. The body temperature varies throughout the day in accordance with the circadian rhythm ((14), chapter 218). Taking this into account and having a dynamic threshold can reduce the number of false positive and false negative fever detection. Besides, body temperature could vary based on the ambient temperature. Having a reference temperature can help solve this issue. Additionally, the thermal camera in selection does not innately have the level of accuracy required for this task. Thus, a calibration for the targeted temperature range needs to be performed.

However, the temperature calibration of the thermal camera is not a trivial task and can be inaccurate depending on various aspects, like environmental temperature and surface condition. Also, a previous study suggests that improper use and interpretation of the infrared camera can lead to inaccurate triage (54). Hence, it is important to understand that the proposed system is only reliable for the designated task under limited conditions. For example, the presence of common cosmetics or clothing such as a turban or hijab can affect the accuracy of the estimated temperature from thermal camera (55). With a higher budget, the use of a thermal camera with higher accuracy and, if possible, one which is pre-calibrated can lead to a more reliable system. But a higher cost will inevitably lower the accessibility of the system.

W. Cyanosis estimation

For the detection of cyanosis, out of the three classification approaches evaluated, the SVC exhibited the highest accuracy (77.1%), and KNN has the highest AUC (0.93), although this is not significantly higher than the AUC of 0.91 for the SVR. The SVR also produces the fewest false negatives (missed cyanosis), which at triage, is probably the most important feature of this system. For the open-source implementation, we, therefore, chose the SVR, although we note this is somewhat arbitrary at this point, given the size of the data set we used.

To visualize the effect of the features on the overall classification, we plot the LR coefficients for each feature (see figure 6).

It can be observed that the red channel exhibits a higher importance for cyanosis detection. When creating a histogram of the pixel values with six equally spaced bins for each of the three color channels, it can be seen that every channel (R, G, and B) contributes to the classification.

We note that the lip cyanosis dataset we used is relatively small and contains relatively good quality images. In the wild, the quality of images is not guaranteed to be high. This may be due to variations in lighting, occlusions, movement, angle of presentation, distances much greater than 1m, among other issues. Consistency of ambient lighting is an important factor in ensuring the correct classification of cyanosis (14). Camera parameters such as field of view and shutter speed can also influence the absolute color detected by the camera. Applying a color correction by using a color reference in the frame can solve this issue (56). Cosmetics applied to and around the lip can also interfere with the classification. In practice, we observed that if the mouth is open or teeth are visible, the cyanosis classifications tend to be inaccurate.

Perhaps the most important issue to consider is that of skin color and the variation of presentation of cyanosis across the human race. It has been reported that detection of peripheral cyanosis in individuals with distinct levels of cutaneous pigmentation can have different levels of difficulty, but can be considerably mitigated by selecting the appropriate skin site to perform the observations (57). Research into racial bias in facial recognition algorithms (44) also has highlighted just how dangerous it can be to use these algorithms out-of-the-box, without tuning to a population or thought about the bias it can create.

X. Heart rate estimation

From the results obtained from our experiments, it can be seen that the algorithm is capable of estimating heart rate at low heart rates, but it requires further tuning in order to increase accuracy. The deviation from ground truth increased when there was a movement in the ROI and when there were large and frequent illumination changes, such as from a bright screen. Note that in this experiment, the subject was very still, and illumination changes in the environment were relatively low. This indicates a need for an improved heart rate estimation approach.

The algorithm is still susceptible to noise from lighting variation and unwanted movement, which affects the HR estimates. It is also seen that the heart rate tracking is less accurate as the heart rate increases, which may be due to movement in the ROI due to rapid breathing after performing cardiovascular exercises to increase the heart rate.

Y. Respiratory effort estimation

Shortness of breath changes the breathing pattern when compared to normal breathing. This is evident from figure 8. Figure 9 shows that the energy in the respiratory effort signal is positively correlated with the respiratory rate measured. These results indicate that respiratory effort has the potential to be used in the classification of breathing as normal or abnormal.

Further studies will have to be conducted on a larger number of subjects with varied demographics in order to create an algorithm capable of performing this classification accurately. In the future, we aim to address issues that can cause interference with readings, such as lighting variation, movement of clothes, unwanted movement of the body, etc. This will make the algorithm more robust to noise.

Z. Clinical testing

During a rapid group session, conducted mostly over video connections, and partially in person, we determined that the system would best be deployed in a mass triage situation. For this reason we chose to add a car battery supply. We found varying temperatures required a temperature sensor to help with calibration. Finally, clinical feedback stressed the need for heart rate and oxygen saturation assessment. (Saturations below 92% and heart rate above 110 beats per minutes were identified as reasons for triage through to the emergency department.) While heart rate was implementable in a short period of time, oxygen saturation was left for future work. We also chose to add respiratory effort estimation as a key vital sign, since its implementation is similar to that of heart rate estimation. Never-the-less, the lack of test subjects, and rapidity of turn-around in development, limited the accuracy of these additional vital signs.