ABSTRACT
Background The demand for pulse oximeters is high in the current COVID-19 pandemic. Despite their popularity, clinical studies to evaluate the reliability in obtaining information on a patient’s oxygenation status, are lacking.
Aim To evaluate the performance of pulse oximeters under clinical conditions, with arterial blood gas measurement (SaO2) as reference standard.
Methods We studied the accuracy of ten top-selling pulse oximeters in Europe and the USA in an intensive care population at the Flevoziekenhuis in Almere, the Netherlands. The studied pulse oximeters were: AFAC FS10D, AGPTEK FS10C, ANAPULSE ANP 100, Cocobear, Contec CMS50D1, HYLOGY MD-H37, Mommed YM101, PRCMISEMED F4PRO, PULOX PO-200, and Zacurate Pro Series 500DL. Adult patients in whom an SaO2 measurement was obtained as part of routine care were asked for inclusion. Directly after obtaining the SaO2 blood sample we obtained the pulse oximeter readings (SpO2) of the investigational devices, in random order. Outcomes were bias (SpO2 – SaO2) mean, root mean square difference (ARMS), mean absolute error (MAE), and accuracy in identifying hypoxemia (SaO2≤90%). As a clinical index test, we included a hospital-grade SpO2-monitor.
Results In 35 consecutive patients we obtained 2,258 SpO2-readings and 234 SaO2-samples. Mean bias ranged from −0.6 to −4.8. None of the pulse oximeters had an ARMS≤3%, the requirement set by ISO-standards and required for FDA 501(k) clearance. The MAE ranged from 2.3 to 5.1, and 5 out of 10 pulse oximeters met the requirements of ≤3%. For hypoxemia, negative predictive values were 98-99%. Positive predictive values ranged from 11-30%. Highest accuracy (95%CI) were found for Contec CMS50D1; 91% (86-94) and Zacurate Pro Series 500DL; 90% (85-94). The hospital-grade SpO2-monitor had an ARMS of 3.0 and MAE of 1.9, and an accuracy of 95% (91-97%).
Conclusion Direct-to-consumer pulse oximeters do not meet ISO-standards required for FDA-clearance, but can accurately rule out hypoxemia.
BACKGROUND
Pulse oximetry has become an indispensable, low-cost, non-invasive, diagnostic tool to assess a patients’ oxygen saturation. Typically, these diagnostic instruments have a clip that can be put on a patient’s finger to obtain information on the peripheral arterial oxygen saturation (SpO2), which serves as a proxy for tissue oxygenation. [1] Over the past decade, pulse oximeters have evolved to play a pivotal role in routine medical care, and are an essential bedside tool in making treatment and/or referral decisions in community-based health care settings. In the past year, pulse oximeters have become even more important, as they are widely used to monitor for (silent) hypoxemia in patients with coronavirus disease 2019 (COVID-19). [2] To illustrate the uptake of pulse oximetry, the global pulse oximeter market was valued at 1.9 billion US dollars in 2019, and is predicted to grow 6.4% annually for the next decade. [3] Given its importance in guiding medical decision making, it is remarkable how little is known about the diagnostic accuracy of these devices when used under clinical conditions in actual patients. [4] As such, we aimed to evaluate whether popular direct-to-consumer fingertip pulse oximeters meet the standards for accuracy, as proposed by regulatory bodies under real-world conditions.
METHODS
We reported this diagnostic accuracy study in accordance with the Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 statement. [5]
Study design
We enrolled consecutive patients, who were at least 18 years of age, admitted to the Intensive Care Unit of a large community-based hospital (Flevoziekenhuis, Almere, the Netherlands. Eligible patients had a radial artery catheter for arterial blood oxygen saturation assessments (SaO2) as part of routine medical care. Exclusion criteria were patients without a clinical indication for arterial access, those with inherited forms of abnormal hemoglobin, and those who rapidly deteriorated due to acute hemodynamic compromise, in which the measurements required for this study could hinder medical interventions and thereby negatively affect patient safety.
Study procedures
Intensive Care personnel notified the site investigator (L.B.) of a potentially eligible patient for study enrolment. The site investigator enrolled each patient after verbal and written consent, either by the patient or his/her legal representative. During office hours, the site investigator was notified by the Intensive Care personnel when a blood gas sample was about to be performed. The site investigator positioned the pulse oximeters for readings directly after the arterial blood gas sample (SaO2) was obtained. In order to reduce detection bias the site investigator applied the pulse oximeters in a rotating fashion, maintaining a fixed device order but starting with a different device in each consecutive sample. The Intensive Care personnel (SaO2) and site investigator (SpO2 readings) reported their findings on separate digital forms, and were blinded for each other’s findings.
Index test and devices
We evaluated ten oximeters, which were selected from the top ten of most purchased pulse oximeters on Amazon in at least two of the following countries: United States, United Kingdom, Germany, Italy, or France. Amazon was chosen because of its dominance on the e-commerce market. In alphabetical order these oximeters were: AFAC FS10D, AGPTEK FS10C, ANAPULSE ANP 100, Cocobear, Contec CMS50D1, HYLOGY MD-H37, Mommed YM101, PRCMISEMED F4PRO, PULOX PO-200, and Zacurate Pro Series 500DL. These pulse oximeters cost between 20 and 50 euros each, and all claimed to meet ISO standards (see paragraph ‘outcomes of interest’ for specifics). As a clinical index test, we also included a hospital-grade pulse oximeter (Philips M1191BL sensor glove, Philips, The Netherlands), which was used as the clinical standard of care for continuous SpO2 monitoring at the study site, and has met ISO standards and received 510(k) clearance of the FDA (clearance number: K062455). We used the SpO2 value that was shown on the pulse oximeter’s display 30 seconds after placement on a patient’s fingertip. The same fingertip was used for each oximeter. When no result was displayed 30 seconds after placement, this was documented as an invalid reading.
Reference standard
The reference standard was a point of care testing analyser (ABL90 Flex Plus, Radiometer Medical ApS, Brønshøj, Denmark, calibrated as per regulatory standards), which was used to perform blood gas analysis on arterial blood gas samples to obtain the arterial oxygen saturation (SaO2) at the study site as part of routine Intensive Care.
Outcomes of interest
We formulated the following outcome measures: mean bias, root mean square difference (ARMS), the mean absolute error (MAE), and diagnostic accuracy for hypoxemia, defined as SaO2 ≤90%. Mean bias is calculated as SpO2 – SaO2. ARMS and MAE are derived from calculations involving mean bias and precision (standard deviation of bias). Because outliers have an excessive negative effect on results of the ARMS parameter, we also assessed the MAE, a measurement that is more robust in the presence of outliers. The formulas to calculate these outcomes can be found in the supplement. We evaluated the diagnostic accuracy of the pulse oximeters according to the standards defined by the International Organization for Standardization in ISO 80601-2-61:2017, which supersedes the ISO 80601-2-61:2011 standard advised by the American Food and Drug Administration (FDA) in their 510(k) Premarket Notification Submissions Guidance for pulse oximeters. This standard considers an ARMS of ≤3% in the SaO2 range of 70-100% acceptable. [6,7]
Data collection and sample size
We included data on SpO2, SaO2, heart rate, systolic- and diastolic blood pressure, sex, age, skin type (FitzGerald classification) as assessed by the site investigator, noradrenalin dose (mg/h), body temperature (°C) and hand temperature to touch as assessed by the site investigator. We followed the FDA advice for sample size determination, which states that at least 200 measurements per pulse oximeter should be obtained from at least 10 subjects, of which at least 2 or 15% have a dark skin type (FitzGerald class IV-VI). [7]
Statistical analysis
We assessed mean bias (SpO2 − SaO2) and standard deviation (SD), and subsequently calculated the ARMS and MAE using the formulas as described in the supplemental data. We created Bland-Altman plots for each pulse oximeter to graphically display its bias (SpO2-SaO2) to the reference standard (SaO2). We added a zero-line and upper and lower limits of agreement (±1.96 SD). To visualize the accuracy standards required by the regulatory bodies, ±3% - lines are also displayed in the figures. As multiple observations were performed per individual, we calculated the standard deviation by using the within-subject variance (σ2) and the between-subject variance (σμ2). Since ARMS can be easily affected by the presence of outliers, we added MAE, which is more robust in the presence of outliers, as well as a sensitivity analysis restricting the sample to measurements within 1.96 SD of each pulse oximeter’s mean and calculating ARMS and MAE on this sample. This sensitivity analysis is analogous to discarding extreme readings as done in routine clinical care when a reading does not coincide with the patient’s apparent clinical state. We also assessed the diagnostic performance (sensitivity, specificity, predictive values, accuracy) of pulse oximeters in detecting hypoxemia, which we defined as SaO2≤90%. Finally, we evaluated factors associated with poor performance of each pulse oximeter. We used bias as a continuous measurement for performance, and used logistic regression models in which we included relevant patient and pulse oximeter characteristics. We used SPSS (IBM SPSS, version 26.0, IBM, Armonk, USA), R software (R version 3.6.1, The R Foundation for statistical computing), and MedCalc Statistical Software version 18.5 (MedCalc Software bvba, Ostend, Belgium; http://www.medcalc.org; 2018)) to conduct the analyses. We assessed statistical significance at the 0.05 level in all analyses.
Ethics review
The study was approved by the local ethics committee and the board of directors of the Flevoziekenhuis. We obtained informed written consent from each participant. When a patient was unable to provide informed consent him-/herself, e.g. when sedated or unconscious, a legal representative provided informed consent on the patient’s behalf.
RESULTS
Baseline characteristics
We enrolled 35 consecutive patients, with a median age of 69 years, and 40% female. Table 1 displays the baseline characteristics. Patients were primarily admitted for respiratory failure due to COVID-19 or other pulmonary diseases (such as COPD). Other diagnoses were septic shock, pulmonary embolism, and arterial thrombosis in extremities, severe hyperglycemia, and suicide attempt.
Blood gas samples and Oximeter readings
We obtained a total of 234 arterial blood gas samples; SaO2 ranged between 85.6% and 99.8%, with the distribution illustrated in figure 1. Of those 234 samples, 12 samples (5.1%) were classified as hypoxemia. We obtained 2258 pulse oximeter readings, all within a 10 minute time window after acquiring the reference standard’s blood sample. Per patient we obtained on average 67 pulse oximeter measurements, ranging from 10 to 420. Of all readings, 82 (3.5%) were invalid as they did not display a result after 30 seconds. The percentage of invalid measurements varied considerably per pulse oximeter type, as shown in table 1.
Test results of pulse oximeters
The test characteristics of the tested pulse oximeters are shown in table 2 and are graphically displayed in figure 2 and figure S1. We found that most pulse oximeters overall had a negative mean bias, thus SpO2 was on average lower than SaO2. Moreover, none of the tested pulse oximeters met the requirement of diagnostic accuracy, using the ARMS ≤3% threshold. When using MAE, which is less affected by outliers, five oximeters performed within a ≤3% difference margin of the reference standard. When excluding extreme outliers (>1.96 SD), performance of Contec CMS50D1, Mommed YM101, Pulox-PO-200, and Zacurate Pro Series 500DL were within the ARMS ≤3% limit, and all but two oximeters (ANAPULSE ANP 100 and Cocobear) met the MAE ≤3% thresholds.
Performance of pulse oximeters compared with arterial blood gas (using the metrics: mean bias, root mean square difference and mean absolute error)
Bland-Altman plots of the bias compared with SaO2 of two pulse oximeters with the lowest mean bias (Contec CMS50D1 −0.6 and Zacurate Pro series 500DL −1.4).*
Legend figure 2: Bland-Altman plots for each pulse oximeter to graphically display its bias (SpO2-SaO2) to the reference standard (SaO2). We added a zero-line and upper and lower limits of agreement (±1.96 SD). To visualize the accuracy standards required by the regulatory bodies, ±3% - lines are also displayed in the figures.
*Bland-Altman plots of the other pulse oximeters can be found as a supplemental figure.
Detection of hypoxemia
Table 3 summarizes the accuracy of each pulse oximeter in detecting hypoxemia. Overall, the oximeters with the highest specificity were Contec CMS50D1 (93%) and Zacurate Pro Series 500DL (91%). Oximeters with the highest sensitivity were Hylogy MD-H37 (92%) and Anapulse ANP 100 (91%). In terms of predictive values, all pulse oximeters performed well in ruling out hypoxemia, with negative predictive values of 98-99% in a population with ≈5% hypoxemia measurements. Confirming the presence of hypoxemia was poor, with positive predictive values of 11 to 30%. For all reading, accuracy was highest for Contec CMS50D1 (91%) and Zacurate Pro Series 500DL (90%). As a sensitivity analysis, we also provided data on the performance of pulse oximeters at different abnormal oxygenation thresholds (SaO2≤92% and SaO2≤94), which can be found as supplemental table S1.
Accuracy of pulse oximeters in detecting abnormal oxygenation status, using either SaO2≤92% or SaO2≤94% as threshold.
Factors associated with poor pulse oximeter performance
Table 4 displays factors that were associated with bias (SpO2-SaO2) of each pulse oximeter. Overall, darker skin complexion and an inaccurate pulse rate measurement (difference between pulse oximeter pulse rate and “true” pulse rate as captured by ICU monitor), were associated with poorer SpO2 performance in five out of ten pulse oximeters. Other factors that negatively affected results were increased systolic blood pressure and cold hands to touch, in four and three pulse oximeters, respectively.
Test results and diagnostic accuracy of the SpO2 continuous monitor (clinical index test)
The hospital-grade Philips sensor glove SpO2 continuous monitor performed with an ARMS of 3.0 and MAE of 1.9 for all measurements, and ARMS of 2.1 and MAE of 1.6 when excluding outliers (1.96 SD). For detecting hypoxemia, the hospital grade Philips sensor glove SpO2 monitor had a sensitivity of 67% (35-90%), specificity of 96% (93-98%), positive and negative predictive values of 50% (31-69) and 98% (96-99%), respectively, and an accuracy of 95% (91-97%). The presence of cold hands/acra was associated with higher inaccuracy of the SpO2 continuous monitor compared with SaO2.
DISCUSSION
Our study found that the top ten of low-cost, popular direct-to-consumer pulse oximeters do not meet the requirements set by the regulatory bodies (ISO/FDA), when compared with the gold standard of arterial oxygen saturation, as obtained by blood gas samples, in a population of intensive care patients. However, when extreme outliers were disregarded, four pulse oximeters would meet the ARMS≤3% requirements, and eight pulse oximeters would meet MAE≤3% standards. The hospital grade Philips sensor glove SpO2 monitor performed slightly better. From a clinical perspective, we found that the pulse oximeters perform well in ruling out hypoxemia, but are not reliable in detecting manifest hypoxemia. As such, when a pulse oximeter indicates a low SpO2, confirmation is required. Caution is warranted when factors are present that negatively affect the reliability of pulse oximeters, such as an inaccurate pulse rate reading, darker skin pigmentation, high blood pressure, or cold extremities.
Strengths and limitations
Our study was performed under clinical conditions involving consecutive patients with direct access to arterial blood gas samples. Patients were diverse in terms of age, skin type, and underlying clinical conditions. We used popular pulse oximeter devices and the number of samples was sufficient to draw accurate conclusions. Moreover, we used a hospital grade SpO2 device as a clinical index test. However, there are also a number of limitations that should be mentioned. First, the performance of fingertip pulse oximeters may have been affected by the poor health status and poorer peripheral perfusion of intensive care patients, when compared with community-based patients. Moreover, the distribution of SaO2 is different in intensive care populations (i.e. oxygenation and ventilation settings are set to strive for SaO2 of 92%), when compared with community based populations, which would also affect performance. Finally, we did not capture detailed laboratory findings on our patients, such as hemoglobin or acid-base status. These variables are of relevance as SpO2 readings are based on photopletysmographic measurements using infrared wavelengths through (de)oxyhemoglobin. As such, a decrease in hemoglobin blood concentration may affect capturing a sufficient signal, whereas a pH shift could alter the oxyhemoglobin dissociation curve, both resulting in inaccurate SpO2 calculations.
Prior studies
Many commonly used direct-to-consumer pulse oximeters do not undergo rigorous in vivo testing, and thus, little is known about the accuracy of these devices. An important study which did perform in vivo testing of inexpensive pulse oximeters was published by Lipnick et al in 2016. [4] In this study six finger pulse oximeters (not cleared by the FDA) were evaluated in 22 healthy subjects, in which stable SaO2 plateaus between 70-100% were achieved under controlled conditions via a partial rebreathing circuit. The study found that two pulse oximeters tested (Contec CMS50DL and Beijing Choice C20) demonstrated an ARMS of ≤3%, hereby meeting the ISO criteria for accuracy. Of the tested oximeters, the Contec CMS50DL may be comparable with the Contec CMS50D1 model that we tested. In our study, which was performed under clinical conditions, the Contec device did not fare as well, although it was still one of the better pulse oximeters that we tested. Prior studies found that oximeters performed worse in hypoxic conditions, with mean bias increasing at lower oxygen saturations compared with arterial blood gas [4] or a conventional bedside pulse oximeter [8,9] In our study we found a similar observation in some, but not all oximeters. Still, from a clinical perspective, it is particularly important to have minimal bias in the range of 90 to 95%, as this is where a hypoxemic state should be differentiated from a non-hypoxemic state.
Implications for practice
Due to the current COVID-19 pandemic, pulse oximeters have become more popular than ever before. Despite their limitations, these devices present a welcome tool for remote monitoring of patients and for ruling out hypoxemia, particularly in a population where the prevalence of hypoxemia is low. In our population, in which approximately 5% was hypoxemic, these low-cost devices were able to safely rule out hypoxemia in virtually all cases (98-99%). This percentage would even further approach 100% in low prevalence settings. Still, our findings, as those of other studies [4,8,9] illustrate that in patients with other symptoms suggestive of hypoxemia, physicians should remain alert, particularly in high-risk patients with preexisting pulmonary disease. In these scenarios, devices that are FDA-cleared, should be preferred as they show a smaller degree of increasing bias during lower SaO2 conditions. [10] Finally, irrespective of the device used, it is important for clinicians to realize that there are a number of factors that negatively impact the reliability of pulse oximetry. The most important ones include anemia, nail polish, and dark skin pigmentation [11,12], as well as others such as poor peripheral circulation, and inaccurate pulse rate measurement, as shown in our study.
CONCLUSION
Low-cost pulse oximeters are widely available and in high demand. Most pulse oximeters have not been rigorously tested. In this study, we tested ten popular pulse oximeters in ICU patients with direct access to arterial blood gas. Overall, we found that the tested pulse oximeters would not meet strict ISO requirements used by the FDA in their 510(k) premarket notification clearance process. However, most devices can safely rule out hypoxemia in the vast majority of patients, particularly in community-based populations with a lower a priori hypoxemia risk. Future studies are warranted to further assess the accuracy of pulse oximeters in community-based patients, and to gain insight how to further improve this non-invasive, low-cost, and potentially life-saving technology.
Data Availability
All data relevant to the study are included in the article or uploaded as an online supplement. The datasets used during the current study are available from the corresponding author on reasonable request.
Contributors
RH and LB designed the study protocol. LB was involved in the recruitment of patients for participation in data collection, supervised by MS. LB performed the statistical analysis and interpreted all data with statistical guidance from JH, LdC, EK, WL, and RH. RH and LB drafted the manuscript and all authors contributed to its revision. All authors read and approved the final manuscript
Funding
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests
The authors report no competing interests with regards to the current study.
Data availability
All data relevant to the study are included in the article or uploaded as an online supplement. The datasets used during the current study are available from the corresponding author on reasonable request.
Supplemental data
Definition of Root mean square difference
ARMS: root mean square difference;
n: number of samples;
yj: Pulse oximeter measurement;
ŷ j: Reference standard measurement
We squared the difference (either positive, negative or 0) between a pulse oximeter measurement and the reference standard measurement. Then we calculated the mean of the sum of squares, and determined the root of that mean. As the difference between the pulse oximeter and the reference standard is squared, the ARMS is always positive.
Definition of Mean absolute error
MAE: mean absolute error;
n: number of samples;
y j: Pulse oximeter measurement;
ŷ j: Reference standard measurement
We calculated the absolute difference (either positive or 0) between a pulse oximeter measurement and the reference standard measurement. We then determined the mean of these absolute differences. Similarly to the ARMS, the MAE is always positive, as it concerns an absolute difference.
Acknowledgements
We would like to thank all participating patients, their families, as well as the personnel of the Intensive Care of FlevoZiekenhuis.
Footnotes
↵* shared first authorship
REFERENCES
References
- [1].
- [2].