ABSTRACT
Point-of-care testing (POCT) offers several advantages over traditional laboratory testing. Offering less invasive testing with a faster turnaround time is not enough if not associated with an acceptable level of accuracy. Here, we show the analytical validation behind the multi-analyte POCT immunochromatography analyser, Hilab Flow (HiF). Analyses from 4,518 clinical samples were compared to College of American Pathologists accredited laboratories for ten quantitative and thirteen qualitative exams. Compatibility between methods was evaluated in terms of association/correlation and clinical agreement. Strong correlation/ concordance was observed between quantitative (CHOL, HDL-c, TG, HbA1c, Glycemia, 25-Hydroxy Vitamin D, TSH, Uric Acid, Creatinine, Urea) and qualitative methods (COVID-19 IgG/ IgM, Beta-hCG, Syphilis, Anti-HBsAg, Zika IgG/ IgM, Influenza A/B, HIV, HCV, HBsAg, Dengue NS1, COVID-19 Ag, Dengue IgG/ IgM, PSA). Approval criteria was obtaining a kappa agreement > 0.8 or a Pearson correlation > 0.9 depending on the exam. Overall percentage agreement was greater than 95% for all exams, indicating a good clinical agreement to gold-standard laboratory-based tests. Results indicate all exams are suitable for POCT and present a reliable performance. Data support the analyser is a useful tool to aid decision-making at the clinical setting, with potential to contribute with healthcare solutions in diagnostic medicine worldwide.
1. INTRODUCTION
Point-of-care testing (POCT) employs remote clinical diagnostics with faster turnaround time, next to the patient [1-2]. It is a tool evolving as fast as innovation allows, with a potentially transformative impact on health care [3]. Devices for POCT diagnostics are now equipped with embedded technology and its application in medicine is steadily growing [4], since it may optimize clinical decisions, patient outcomes and even provide financial benefits [5-6].
POCT offers an alternative to traditional laboratory testing, especially in locations with limited infrastructure, but it is also used at wealthy settings in developed countries [7]. Advantages of the testing model includes the use of portable equipment, specimen collection at the test site, immediate identification of biological samples, small sample volume needed and fast results, features that can minimize pre-analytical errors and risks related to the transport and identification of biological samples [1-2,8].
One of the commercially available POCT devices is the Hilab Flow (HiF), a patented analyser [9-13] dedicated to read immunochromatography results from lateral flow assays and colorimetric results from vertical flow assays. The small handheld POCT device combines artificial intelligence, machine learning and deep learning techniques to enable a shorter turnaround time, but without compromisings the exam’s performance, precisely the main drawback in current POCT [14]. The HiF analysers are connected to a central laboratory for real-time monitoring and management of POCT performed remotely. Another important feature of this analyser is that it is able to perform a virtually unlimited number of exams, due to its system’s flexibility.
The main concern regarding POCT is to guarantee reliable results, equivalent to those acquired with traditional standard laboratory-based methods, and according to international clinical guidelines [15-17]. Rapid test suppliers usually offer limited information about test performance thus performing a robust performance analysis is crucial to offer a quality service which meets consumer expectations. Therefore, this study aims to provide data from analytical validation to assess the system performance for POCT use and its potential to aid medical decision-making.
2. MATERIAL AND METHODS
2.1 Sample and Data
Internal data from the routine analysis of clinical laboratory service were used retrospectively for this in-house validation study. Evaluation of respiratory viruses exams evaluation, such as COVID-19 Ag, were conducted with nasopharyngeal samples, while other exams were evaluated from whole blood, serum and/or plasma specimens. The study was approved by the Beneficência Portuguesa Research Ethics Committee: CAEE 33490420.9.0000.5483.
The clinical samples were evaluated by lateral or vertical flow assays, depending on the type of the exam. Lateral flow assays are membrane-based tests which combine colloidal gold labeled particles to detect the analyte of interest from nasopharyngeal secretion, whole blood, serum or plasma specimens. Analyte’s molecules react and form a complex with labeled antibodies or antigens and, as this conjugate moves through capillarity across the membrane, the anti-analyte antibody or antigen immobilized in the membrane binds to the complex, revealing a colored line of varying intensity, that can be measured by its optical density.
For qualitative exams, a colored line in the test region, accompanied by a line in the control region, indicates a positive result (reagent). For quantitative exams, the intensity of the coloured line in the test region relates to analytes’ concentration. Other analytes were evaluated by vertical flow assays. The presence of the analyte of interest promotes a measurable color change on the test membrane which is proportional to the analyte concentration and can be measured similarly to lateral flow assays.
A list of all validated exams and the respective employed tests are shown in Table 1.
2.3 HiF Analyser
The HiF reader (Hilab, Curitiba-PR) is a laboratory analyser platform for professional POCT used for detection and/or quantification of various analytes. The system processes immunochromatography results from lateral flow assays and colorimetric results from vertical flow assays by measuring optical density or color model values, respectively. The equipment is composed of two main parts, a portable handheld analyser (12 cm x 12 cm x 13 cm, 450 g), which incorporates a camera-equipped light detector, and sample integrated capsules/cartridges. Such a device communicates with the laboratory’s server, where artificial intelligence tools and biomedical/biochemical specialists analyze the results. The system applies artificial intelligence techniques, such as computer vision and image processing tools to find regions of interest, improve image quality and detect objects. Moreover, machine learning and deep learning techniques perform classification and regression tasks to assist the analysis of quantitative and qualitative exams, respectively. Finally, human specialists also analyze the exams. If there is divergence between the automatic and human results, a senior specialist also analyzes the exam to achieve the final result, improving the overall quality of exams. For this study, HiF analysers were used for the analytical validation of 27 different analytes.
2.4 Operation
Sample (5 µL to 80 µL) is introduced into the capsule by using a capillary tube, pipette or a medicine dropper device, with or without an adjuvant buffer (depending on the exam). The capsule, identified by a unique QR code, is then inserted into the analyser, which measures the compatible signal of the lateral/vertical flow test within the capsule. Internet Of Things (IoT) technology is used to recognize the unique QR code from each test sample and sends the reaction information via cloud to a main laboratory. There, an artificial intelligence (AI) software analyses the reaction, and a licensed health professional confirms and signs the clinical diagnosis report. The test result is released through the cloud system to the health care service location where the test was performed and via email and/or text message to the patient. Data management and protection is ensured by a robust system, enabling the tracking of processed samples based on the investigated analyte [12]. The full process occurs within 30 min.
2.5 Clinical Agreement Evaluation
To ensure a thorough analysis, each exam was evaluated in terms of clinical agreement and compatibility between methods.
Agreement between quantitative methods was assessed through regression analysis of the plotted curves. Interpretation of Pearson’s correlation was only secondary in our analysis especially because testing in a wider range might show strong correlation, but not necessarily indicate that both methods are compatible.
2.6 Performance Evaluation
For the exam’s performance evaluation, measurements from 10 quantitatives and 13 qualitatives exams were compared to those from College of American Pathologists’ (CAP) accredited laboratories. Some exams are multi-analyte (e.g. renal function and lipid profile) which explains a total of 27 evaluated analytes. Analytes, specimens, gold-standard assay and equipment used for comparison and validation of the POCT assays are described in the Supplementary Data.
Quantitative clinical correlation analyses were performed for the following analytes: blood glucose (glycemia), glycosylated hemoglobin (HbA1c), total cholesterol (CHOL), high-density lipoprotein cholesterol (HDL-c), triglycerides (TG), 25-Hydroxy (25-OH) Vitamin D, thyroid-stimulating hormone (TSH), uric acid (UA), creatinine (C), and urea (UR). Other analytes from the lipid panel are indirectly estimated through the difference between CHOL and HDL-c (NHDL-c), Martin’s equation (LDL-c), Friedewald equation (VLDL), but not reported here because any interpretation on correlation and analytical accuracy would be derived from the relationship between CHOl, HDL-c and TG.
Qualitative clinical correlation analyses were performed for: COVID-19 IgG / IgM, COVID-19 antigen (Ag), Influenza A / B virus Ag, Syphilis antibody (Ab), human immunodeficiency virus (HIV) Ab, hepatitis C virus (HCV), surface antigen (HBsAg), hepatitis B virus surface antibody (Anti-HBsAg), Dengue virus NS1 Ag, Dengue virus IgG / IgM, Zika virus IgG / IgM, Beta-human chorionic gonadotropin (Beta-hCG), and prostate-specific antigen (PSA).
2.7 Statistical Analysis
The adjusted Wald Interval method (Agresti–Coull interval) was applied to estimate the confidence intervals for the clinical agreement parameters of the tests: positive percentage agreement (PPA), negative percentage agreement (NPA), overall percentage agreement (OPA), and overall clinical agreement (OCA). Other approaches might render inaccurate results due to showing proportions (PPA, NPA) close to 0, thus the adjusted Wald interval estimate was chosen to prevent such behavior [18].
OCA and OPA differed in exams with more than two reference ranges, which is the case for most quantitative assays. While qualitative assays comprise binary outcomes (e.g. normal VS disease), several quantitative assays might possess multiple reference ranges depending on analyte concentration and reported clinical impact (e.g. diabetes). Thus, OCA was based on the hit rate for all reference ranges of the analyte. Conversely, OPA was calculated only for samples matching the reference ranges present in the analyte individual contingency tables. The reference ranges used for plotting the contingency tables and calculating each exam’s performance are shown in Table 2.
Compatibility between methods was evaluated in terms of association (Pearson’s correlation coefficient for quantitative or Cohen’s Kappa agreement for qualitative assays) and performance parameters (PPA and NPA). For quantitative assays, the systematic difference was also evaluated by applying a compatible paired test (t-test for parametric distribution or Mann-Whitney for non-parametric) considering the null hypothesis as 0 for the true mean difference. Finally, the regression analysis was employed for estimating the slope (and its respective confidence intervals) and correlation coefficient [19]. Samples with concentrations outside of the exam’s measuring range were excluded from the linear regression analysis to avoid misinterpretation of the data. The statistical significance for all analyzes was p < 0.05.
3. RESULTS
Due to lack of consensus in literature, we followed IUPAC and CLSI definition for “analytical sensitivity” which refers to the assay sensitivity to slight changes in analyte concentration [20]. That is, the closer to 1 is the slope of the regression curve, the more precise is the method. Several guidelines on method validation highlight the importance of calculating regression parameters (e.g. correlation coefficient, y-intercept) for a comprehensive method validation [21-22], but no clear criteria is established for the acceptable range of estimated slope, especially for POCT [23]. Thus, our acceptance criteria for assay precision was obtaining a slope anywhere between 0.85 - 1.15, which is a common rule-of-thumb for most rapid test strip suppliers (Figure 1).
All exams matched our criteria, since estimated slopes (with confidence intervals) ranged from 0.85 - 1.15. The closer to 1 is the slope the higher is the precision of an exam in relation to changes in analyte content. For the remaining exams, the variation in slope did not impact the clinical agreement of the exams even at decision limits, as indicated by the obtained PPA and NPA.
PPA, NPA and OCA were greater than 90% for all analytes, suggesting a good agreement for the proposed methods (Tables 3 and 4). Clinical differences between measurements (proposed method VS POCT) were considered acceptable for any result within a defined % of a category boundary. Ferreira et al. (2015) [24] employed a similar approach to avoid overestimation of clinical differences. However, we considered analyte-specific variation – reported by the CLIA [25] – as a standard other than an arbitrary value (e.g. 5%) for a more accurate estimate of exam’s performance.
4. DISCUSSION
Universal health care availability depends on decentralized diagnostic POCT systems in order to solve healthcare bottlenecks both in wealthy and developing countries [26-27]. Technology has incremented POCT for the last two decades, so modern devices are now smaller, smarter, easier to use, less prone to errors and go without additional testing for clinical purposes [4]. POCT is a patient-oriented technology, but health professional expertise is still required for sensitive diagnostic and clinical decision-making. Importantly, patients that used POCT reported greater levels of confidence in their health professional and motivation to manage their health condition [28]. For minor conditions and highly prevalent diseases, POCT results may be sufficient for health decisions in the clinical setting [8], without additional confirmation from a traditional laboratory. However, this scenario requires that POCT is reliable and robust regarding clinical agreement with an established laboratory method [29-30]. Combining a robust analytical validation and a reliable quality control in the clinical laboratory is the key for improving healthcare access, even in remote regions [14].
In the point of care industry, equipments are usually designed to analyse a small set of analytes which limits the spread of POCT and, consequently, offers a service at competitive prices to the consumer. The HiF analyser goes further by enabling a multi-analyte evaluation on a single device of quantitative and qualitative exams. Also, employing a dual verification of analyzed samples (by the AI and the biomedical specialist) helps minimize common analytical errors from decentralized laboratory settings. Previous works from our group show AI can improve the analytical precision of a method, thus enabling epidemiological conclusions on a populational level [13,31-32].
The integrated laboratory management model, especially developed for POCT with the HiF, is a distinguishing feature that enables a continuous enhancement of the test accuracy. These features are aligned with the understanding that POCT test performance should be compatible with any traditional tests, as required by international and regional standards [33]. POCT can be used to improve patient flow within the clinical setting and may even provide indirect cost effectiveness advantages compared to centralized laboratory testing, considering increased resource management and more efficient patient care [3,34]. Shortening the turnaround time is another asset of the HiF system, specially when delays compromises the patient’s health: on an intensive care unit setting, the time of analysis shows great influence over a physician’s conduct [35], thus the patient can greatly benefit from a faster, yet precise result.
Modern POCT health systems may also create a valuable database for precision public health management using Big Data. Precision public health uses data from traditional and emerging sources to deliver assertive interventions to populations, focusing on social and environmental health aspects [36-38]. Big Data analytics aims to implement evidence-based decision-making in clinical and population settings [38]. This scenario depends on available population-level data on disease surveillance [40], and requires a solid collaboration among health care systems, clinicians, governments and communities [41]. Precision public health can provide doctors and public health practitioners with new insights, with the potential of revolutionizing health care, through the use of new technologies.
5. CONCLUSION
This study presents data from 4,518 clinical samples used to evaluate the analytical accuracy and clinical agreement between a CAP certified laboratory and a POCT based laboratory (HiF analyser). Combining effective IoT and AI tools is essential for offering POCT alternatives with a faster turnaround time and a reliable performance to benefit both health professionals and patients. Our results show that the evaluated POCT system is a useful tool to aid medical decision-making at the clinical setting, with potential to contribute with healthcare solutions in diagnostic medicine.
Data Availability
All data produced in the present work are contained in the manuscript.
CRediT author statement
Lucca Centa Malucelli: Project Administration, Writing, Editing, and Review (Original and Final draft).
Gabriele Luise Neves Alves: Visualization, Writing, Editing, and Review (Original and Final draft).
Carolina Melchioretto dos Santos: Supervision.
Matheus Gonçalves Severo: Editing and Review (Original and Final draft).
Victor Henrique Alves Ribeiro: Writing, Editing, and Review (Final draft).
Bernardo Montesanti Machado de Almeida: Writing, Editing, and Review (Original and Final draft), Supervision.
Caio Corsi Klosovski: Editing and Review (Original and Final draft).
Tania Leme da Rocha Martinez: Editing and Review (Original and Final draft), Supervision.
Marileia Scartezini: Editing and Review (Original and Final draft), Supervision.
Tereza Luiza Bellincanta Fakhouri: Editing and Review (Original and Final draft).
Marcus Vini cius Mazega Figueredo: Funding Acquisition and Supervision.
ACKNOWLEDGMENT
We acknowledge funding for the analytical method validation of the exams reported here.
Footnotes
↵□ Nephrology Department, BP - A Beneficência Portuguesa de São Paulo, Brazil
REFERENCES
