PT  - JOURNAL ARTICLE
AU  - Owusu-Adjei, Michael
AU  - Hayfron-Acquah, James Ben
AU  - Frimpong, Twum
AU  - Abdul-Salaam, Gaddafi
TI  - A systematic review of prediction accuracy as an evaluation measure for determining machine learning model performance in healthcare systems
AID  - 10.1101/2023.06.01.23290837
DP  - 2023 Jan 01
TA  - medRxiv
PG  - 2023.06.01.23290837
4099  - http://medrxiv.org/content/early/2023/06/04/2023.06.01.23290837.short
4100  - http://medrxiv.org/content/early/2023/06/04/2023.06.01.23290837.full
AB  - Background Focus on predictive algorithm and its performance evaluation is extensively covered in most research studies. Best predictive models offer Optimum prediction solutions in the form of prediction accuracy scores, precision, recall etc. Prediction accuracy score from performance evaluation have been used as a determining factor for appropriate model recommendations use. It is one of the most widely used metric for identifying optimal prediction solutions irrespective of context or nature of dataset, size and output class distributions between the minority and majority variables. The key research question however is the impact of using prediction accuracy as compared to balanced accuracy in the determination of model performance in healthcare and other real-world application systems. Answering this question requires an appraisal of current state of knowledge in both prediction accuracy and balanced accuracy use in real-world applications including a search for related works that highlight appropriate machine learning methodologies and techniques.Materials and methods A systematic review of related research works through an adopted search strategy protocol for relevant literature with a focus on the following characteristics; current state of knowledge with respect to ML techniques, applications and evaluations, research works with prediction accuracy score as an evaluation metric, research works in real-world context with appropriate methodologies. Excluded from this review search is defining specific search timelines and the motivation for not specifying search period was to include as many important works as possible irrespective of its date of publication. Of particular interest was related works on healthcare systems and other real-world applications (spam detections, fraud predictions, risk predictions etc).Results Observations from the related literature used indicate extensive use of machine learning techniques in real-world applications. Predominantly used machine learning techniques were Random forest, Support vector machine, Logistic regression, K-Nearest Neighbor, Decision trees, Gradient boosting classifier and some few ensemble techniques. The use of evaluation performance metrics such as precision, recall, f1-score, prediction accuracy and in some few instances; predicted positive and predicted negative values as justification for best model recommendation is also noticed. Of interest is the use of prediction accuracy as a predominant metric for assessing model performance among all the related literature works indentified.Conclusions In the light of challenges identified with the use of prediction accuracy as a performance measure for best model predictions, we propose a novel evaluation approach for predictive modeling use within healthcare systems context called PMEA (Proposed Model Evaluation Approach) which can be generalized in similar contexts. PMEA, addresses challenges for the use of prediction accuracy with balanced accuracy score derived from two most important evaluation metrics (True positive rates and True negative rates: TPR, TNR) to estimate more accurately best model performance in context. Identifying an appropriate evaluation metric for performance assessment will ensure a true determination of best performing prediction model for recommendation.Competing Interest StatementThe authors have declared no competing interest.Funding StatementNo external or internal funding has been used.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:N/AI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesApplicable upon request