Abstract
The recent COVID-19 crisis highlighted the inadequacy of human forecasting. We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n=1822) and Next Generation Social Science (NGS2) platform (n=103) were utilised. A 43-feature model predicted top quintile relative Brier accuracy scores in two out-of-sample datasets (pboth<1×10−9). Trades graded as high machine accuracy quality vs. other trades had a greater AUC temporal gain from before to after trade. Hybrid human-machine forecasts had higher accuracy than human forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial AUC gains of 13.2%, p=1.35×10−14 and 13.8%, p=0.003 in the out-of-sample Almanis B and NGS2 datasets respectively. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for human-only models, p=0.007. This net classification benefit was replicated in the separate Almanis B dataset, p=2.4×10−7. Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. Implementation may allow improved anticipation of and response to emerging risks and improved human collective efforts generally.
Significance Statement Human-machine hybrid approaches have been identified as a new frontier for event prediction and decision making in the artificial intelligence and collective human intelligence fields. For the first time, we present the successful development and validation of a human-machine hybrid prediction market approach and demonstrate its superior accuracy when compared to prediction markets based on human forecasting alone. The advantages of this new hybrid system are demonstrated in the context of COVID-19-related event prediction.
Competing Interest Statement
Dysrupt Labs, a subsidiary of Slowvoice Pty Ltd, supplied the Almanis prediction market database for this research. K.M., F.B., and C.N. are employees of Dysrupt Labs. K.M., F.B., C.N., and A-L.P. have stocks in Dysrupt Labs. A.G. and E.M. have no conflicts of interest.
Funding Statement
This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources to SlowVoice Pty Ltd. (IR2101990) and an Investigator grant (N1197234) and Fellowship to A-L Ponsonby by the National Health and Medical Research Council of Australia (GNT1110200).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The Human Research Ethics Committee of the Royal Childrens Hospital, Melbourne, Australia, gave ethical approval for this work.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Data Availability
Data availability. The dataset from Almanis prediction markets was used under license for the current study and is not publicly available. Applications for access to the Almanis database can be made from Dysrupt Labs, Slowvoice Pty. Ltd. (info@slowvoice.com). NGS2data can be accessed through an R package found at: https://github.com/MichaelbGordon/pooledmaRket. Code availability. Data collection: Provision and use of the code for the Almanis platform is subject to a Commercial License Agreement with Dysrupt Labs, Slowvoice Pty. Ltd. (info@slowvoice.com). Data analysis: Provision and use of the code is generally subject to a Commercial License Agreement with Dysrupt Labs, Slowvoice Pty. Ltd. (info@slowvoice.com). For academic and government purposes, please contact the corresponding author. All experiments and implementation details are described in sufficient detail in the appendix to support replication with non-proprietary libraries.
