Abstract
Background Pneumonia remains a leading cause of hospitalization and death among young children worldwide, and the diagnostic challenge of differentiating bacterial from non-bacterial pneumonia is the main driver of antibiotic use for treating pneumonia in children. Causal Bayesian networks (BNs) serve as powerful tools for this problem as they provide clear maps of probabilistic relationships between variables and produce results in an explainable way by incoporating both domain expert knowledge and numerical data.
Methods We used domain expert knowledge and data in combination and iteratively, to construct, parameterise and validate a causal BN to predict causative pathogens for childhood pneumonia. Expert knowledge elicitation occurred through a series of group workshops, surveys and one-on-one meetings involving 6-8 experts from diverse domain areas. The model performance was evaluated based on both quantitative metrics (area under the receiver-operating characteristics curve (AUROC) and log loss) and qualitative expert validation. Sensitivity analyses were conducted to investigate how the target output is influenced by varying key assumptions of particular high degree of uncertainty around data or domain expert knowledge.
Results Designed to apply to a cohort of children with X-ray confirmed pneumonia who presented to a tertiary paediatric hospital in Australia, the resulting BN offers explainable and quantitative predictions on a range of variables of interest, including the diagnosis of bacterial pneumonia, detection of respiratory pathogens in the nasopharynx, and the clinical phenotype of a pneumonia episode. Satisfactory numeric performance has been achieved including an AUROC of 0.8 in predicting the clinical diagnosis of bacterial pneumonia. Three commonly encountered scenarios were presented to demonstrate the potential usefulness of the BN outputs in various clinical pictures.
Conclusions To our knowledge, this is the first causal model developed to help determine the causative pathogen for paediatric pneumonia. It can be utilized to derive recommendations to support more directed and judicious use of antimicrobials for relevant cohorts. The BN needs further validation before it can be clinically implemented. Our model framework and the methodological approach can be adapted beyond our context to broad respiratory infections and geographical and healthcare settings.
1 Introduction
Pneumonia, infection of the lower airways, remains a leading cause of hospitalisation and death among young children worldwide [1, 2]. In recent decades, marked reductions in the burden of disease have occurred thanks to public health strategies including immunisation targeting Haemophilus influenzae type b, Streptococcus pneumoniae, and influenza virus [3, 4], however substantial morbidity and mortality remains. These interventions has led to a decline in the contribution of bacterial pathogens in particular, challenging the appropriateness of routine empiric antibiotic use for treating community-acquired pneumonia (CAP) [5]. Viruses are increasingly considered a major cause of paediatric pneumonia especially in high income settings, and children with viral infection are unlikely to benefit from antibiotic therapy [6, 7]; rather, unnecessary antibiotic use may lead to otherwise avoidable side effects, and be a driver of antimicrobial resistance [8, 9]. Despite this, antibiotics are still widely prescribed for children with pneumonia [5] possibly due to the difficulty in excluding bacterial infection as well as clinician concern regarding the potential consequences of under-treatment of bacterial pneumonia whether caused by ‘typical’ bacterial pathogens such as Streptococcus pneumoniae (pneumococcus), atypical bacteria such as Mycoplasma pneumoniae (mycoplasma), or bacterial super-infection following viral infection [10].
The aetiology of CAP in children has been studied extensively [11]. A number of clinical and laboratory factors have been shown to be associated with pneumonia in sick children [12–14], and many of these features (including cough, increased respiratory effort and raised inflammatory markers) are common in both bacterial and non-bacterial pneumonia. However, no single predictor exists that is sufficiently sensitive or specific to reliably differentiate bacterial (including bacterial-viral co-infection) from non-bacterial pneumonia [9]. Also, the performance of possible predictors may vary by context, e.g., in early versus late stage illness, or where the underlying prevalence of a pathogen varies [15]. The contribution of bacteria and viruses to pneumonia has been reported to vary by age, season, vaccine coverage, socioeconomic status, and across countries [6, 16, 17], although a recent review found evidence that viruses associated for a similar proportion of pneumonias across settings [7]. Isolation of pathogens directly from the lower respiratory tract is highly specific and may be sensitive, but sampling is typically invasive, challenging in children, and rarely indicated clinically and consequently rarely done [18]. Although less specific, nasopharyngeal samples are frequently used for extrapolating pneumonia aetiology because of ease of collection, however any pathogens detected in the nasopharynx might not be the actual cause of the pneumonia [19]. Due to the dynamic nature of disease epidemiology, non-specific clinical presentation and diagnostic limitations of laboratory tests, the timely and accurate identification of pathogens causing pneumonia remains an ongoing challenge.
Mathematical prediction models have been developed to aid clinical diagnosis, and existing approaches are usually based on quantifying associations between the target (e.g. bacterial pneumonia) and certain input variables in specified patient groups. Typical statistical regression methods are agnostic to underlying causal mechanisms and ignore complex interactions, such as those that occur between epidemiological, clinical, microbiological and immunological factors in pneumonia. Nevertheless, reasonable diagnostic performance has been achieved [20, 21]. However, the implementation and uptake of these models has been limited for several reasons. First, validation and clinical implementation of models are often challenged by missing data [22], such as biomarkers which might be useful predictors but may not be available in all cases to aid timely clinical decision-making [21]. Second, the models may not be transportable from the training context to other clinical contexts, because the predictive values of input variables are often driven by baseline prevalence [15], infection severity and testing techniques employed in the validated study population. Third, association-driven predictions that lack causal explanation may not be accepted by their intended end-users (i.e., clinicians) [23]. The clinical implementation of any model for decision support is largely dependent on trust which, in turn, depends on end-users understanding how the model works.
Bayesian networks (BNs) may offer a solution to the challenges of predictive model design and imple-mentation into clinical practice [24]. BNs have been used to clarify complex medical problem domains including diagnosing CAP and ventilator-associated pneumonia [25–27], by facilitating probabilistic and causal reasoning using directed acyclic graphs (DAGs) [28, 29]. The graphical representation of BNs facilitates the translation of knowledge from domain experts and that acquired from clinical datasets into a causal inference framework, with quantitative relationships between variables. BNs provide probabilistic predictions that are transparent and explainable. The development of BNs relies on collaboration between domain experts and modellers to synthesize an understanding of the problem domain with the modelling technique, creating model outputs that are meaningful for clinical practice. BN model outputs can be applied to a variety of patient subgroups (such as specific age groups or geographic locations), and can robustly account for missing input variables like biomarker measurements from laboratory data. Our previous work has demonstrated how epidemiological, procedural and laboratory observations can be organised under a BN designed for predicting the causative pathogen in other childhood infections, and how such a model may be presented to aid clinical decision making [30, 31].
Here we present a causal BN model that depicts the pathophysiology of pneumonia in children, which could be used to help distinguish bacterial (including typical and atypical bacteria, and mixed viral-bacterial infections) from non-bacterial infections. The model was constructed, parameterised, and validated using both domain expert knowledge and data. We used data obtained from 230 children admitted to a tertiary hospital in Western Australia from 2015-2018 with pneumonia who underwent clinical, microbiological, immunological and radiological assessment [5, 32]. A diverse group of domain experts participated in the knowledge elicitation process. Based on numeric evaluation metrics, the outcome model showed stable and good performance in predicting clinical diagnosis of bacterial pneumonia as well as the detection of pathogens in the nasopharynx. The model also produced clinically meaningful outputs as assessed by domain experts via evaluation workshops and surveys. To demonstrate the potential utility of the model, we apply it to three representative clinical scenarios and discuss how our model may be implemented as a tool for decision support to improve patient management and encourage judicious use of antibiotics for paediatric pneumonia.
2 Methods
A BN model consists of two components, 1) a DAG that qualitatively describes how variables (nodes) interact with each other, and 2) a joint conditional probability distribution that quantitatively specifies how changes in each parent variable probabilistically drive changes in their child variables. The arrows (arcs) of the DAG indicate the presence of a direct influence of predecessor (or parent) variables on their child nodes (nodes extending from other nodes). A causal BN is one in which the arcs represent influences that are causal; sometimes with minor exceptions for associations that are required, but are not a core part of the causal process being modelled. Developing a causal BN involves constructing the model structure (the DAG) that describes the problem domain of interest, and quantifying the probabilistic effect that parents have on their children in the model as a series of conditional probability tables (CPTs), i.e., parameterisation. In this project we used domain expert knowledge and data in combination and iteratively, to build a causal BN to predict causative pathogens for childhood pneumonia. Fig. 1 shows a schematic of the BN model development process.
The top blue row presents the development of the BN from its initial form (left) to the fourth iteration (right), and how this iteratively interacts with the data analysis (middle grey row) and the knowledge elicitation (bottom green row). The four workshops were organised between December 2019 and August 2021.
2.1 Knowledge elicitation
Expert knowledge elicitation occurred through a series of group workshops, surveys and one-on-one meetings (Fig. 1) involving 6-8 domain experts (with backgrounds in infectious disease, clinical microbiology, emergency medicine, general paediatrics, immunology and epidemiology). Workshops and surveys covered the development of the model structure, elicitation of key model parameters, data interpretation, model structure validation, and validation of the application use cases. One-on-one meetings were organised when more detailed discussion was considered necessary by the modellers, such as to consolidate understanding of a particular topic discussed during a workshop. As shown in Fig. 1, the theme of the workshop and survey was often driven by the evolving modelling progress—questions arising from data analyses, development of model structure and parameterisation. In Additional File 1, we provide a summary of survey questions.
2.2 BN structure and parameterisation
The final model structure was achieved through an iterative process of synthesising expert knowledge, data and learnings from earlier BN revisions (Fig. 1). Parameterisaton of the model was conducted using the expectation maximisation (EM) algorithm [33], with one of the following three methods applied to each node, depending on the quality of the data available for that node:
A data-driven approach was used to derive CPTs directly from the observed distribution of relevant variables in the data, when data for those variables was sufficient.
An expert-driven approach was used for conceptual variables that were not or could not be recorded in the data. In this approach, the CPT was fully specified by domain expert knowledge alone and not further updated with data. For example, susceptibility to progression describes the extent to which an infected child is at risk of progressing from mild to severe manifestations of pneumonia. In the model, this susceptibility is higher in younger children, in those with chronic respiratory disease, and in those who are immunocompromised.
A hybrid expert-data approach was used in which expert-driven CPTs were used as a Bayesian prior (i.e., a starting point) and subsequently updated using available data. This method was preferred for variables that were strongly associated with other data, but could not be directly observed or mapped to any individual variable (i.e., latent), as well as for observable variables where data were sparse for certain patient subgroups.
2.3 BN performance and evaluation
We evaluated the performance of the BN in two ways. Quantitatively, numerical evaluation ensured the model predictions were consistent with the observed data. While metrics that predict accuracy such as area under the receiver-operating characteristics curve (AUROC) have been widely used to summarise the performance of classifiers regarding their true positive/negative and false positive/negative rates, they can be misleading. These measures are more suitable for definite classifications rather than the probabilistic predictions that some models are able to provide. For probabilistic models like BNs, we can create a binary classification by applying a threshold. For example, if we choose a cut-off threshold of 0.5, the model predicted probabilities of 0.51 and 0.99 would be collapsed together; if we instead shifted the threshold to 0.55, then 0.51 and 0.01 would be collapsed together. AUROC mitigates this problem by evaluating all possible thresholds, but still rests on the expectation that outcomes will be dichotomised one way or another — when in fact end-users of the model will see the probability, not a deterministic prediction. For this reason, log loss (and similar probability-sensitive metrics) is considered more informative for evaluating models that generate probabilistic predictions [34]. Log loss is calculated, for a given case, as the negative log of the model’s estimated probability of what was actually observed (i.e. −log {P (observed outcome)}) with a lower log loss indicating a better fit of the model to the observed data. The log loss metric rewards accurate probability estimates most highly. For example, if a positive blood culture is truly observed, a model predicted probability for positive blood culture of 0.51 yields a log loss of –log (0.51) = 0.29, while a model predicted probability of 0.99 yields a much lower (and hence better) log loss of 0.004, reflecting a reward for the greater confidence in a positive culture if correct, but punishing this greater confidence if incorrect. Accumulating log loss over a number of cases then rewards the most accurate probabilistic predictions.
Sensitivity analyses were conducted using variance-based sensitivity analysis (VBSA) [35,36], to investigate how the target variable of the model is influenced by varying the assumptions for a range of key variables, especially those affected by a high degree of uncertainty around data or domain expert knowledge.
The approaches above are unfortunately of limited value in evaluating the causal features of a model. For example, its common for two BNs to differ in their causal features but be identical in their statistical properties [29, 37]; such models would be assessed as equivalently good by any of the above metrics. More commonly, major differences in causal features may only lead to minor differences in predictive performance. Thus, while predictive performance can help identify significant issues (such as major missing pathways or dependencies), other approaches must be used to evaluate the individual causal features of a BN. This can be achieved in a number of ways (e.g., comparing to literature, conducting experiments, etc.); here we performed this validation with experts, by walking through the DAG structure afresh at each workshop, and by soliciting feedback on the DAG structure via followup emails.
In addition, we performed qualitative expert validation of the model outputs, allowing us to assess if the model behaves in a way that meets clinician expectations and offers clinically informative predictions. We simulated three typical scenarios encountered in clinical practice, and presented BN predictions in both survey format (Survey 4) and during workshops (Workshops 3 and 4) for expert review.
Ethics approval was granted by the Child and Adolescent Health Service Human Research Ethics Committee (RGS2477). We used the GeNIe BN (https://www.bayesfusion.com/) software to elicit the BNs presented here, and Netica (https://www.norsys.com/) to develop and parameterise subsequent models. Data, evaluation and sensitivity analyses were conducted using R [38] and Python [39]. Throughout the paper, we label the name of a BN variable in bold, and its state name in italics where referred for the first time.
3 Results
Fig. 2 presents the high-level structure of the BN (v4.5), a model of paediatric pneumonia at the point of hospital presentation. We explicitly model the existence of infection (which is latent) separately from its clinical diagnosis (white node) which may be operationally defined by a set of clinical or laboratory observations. A more detailed description of the model is provided in § 3.2.
Variables were organised into five highly interdependent groups (shown as rectangle boxes). In the centre of the diagram sits the core part of the model, infection of the respiratory tract (white box), which is predominantly influenced by the presence of a range of pneumonia-causing pathogens in a child’s nasopharynx (yellow), as well as a number of background factors (blue). Infection manifests as clinical signs, symptoms and abnormal laboratory markers (salmon). Interventions (purple) refer to investigations and treatments which are relevant to the episode of pneumonia, and which may either occur before presentation (e.g., prior antibiotic use) potentially affecting variables like the culture result, or after presentation (e.g., supplemental oxygen) which may be affected by variables like the infection phenotype. Additional File 2 includes a BN dictionary which defines each variable.
3.1 Summary of model variables
The model consists of 63 variables including 14 which are background factors, six pathogen-specific variables describing the presence of pathogens in the nasopharynx, nine variables representing infection or the diagnosis of infection, 24 variables representing specific signs and symptoms, six laboratory variables, and four intervention variables. Alongside the 53 observable variables, nine latent variables were created for modelling purposes; latent variables help to explicitly describe important underlying mechanisms that cannot be directly observed and thus captured by data. One variable (current clinical phenotype (52)) was introduced as a summary of the presenting clinical phenotypes of pneumonia. This variable was derived based on a separate clustering using the EM algorithm to determine a child’s most probable pneumonia phenotype based on their clinical observations. The analysis resulted in two phenotypes, where phenotype 1 represented a more severe type which increased the probability of all observable signs and symptoms compared to phenotype 2, and this information was added as an additional column in the cohort data and subsequently used as an observed variable. We discuss the role of this variable further below § 3.2.3.
In the BN dictionary, we provide information on each model variable, including identifiers (ID and variable name), what it means and how it’s mapped to data (description), how it’s discretised in the BN (states), its parents in the BN, and how it’s affected (mechanistically rather than statistically) by those parent nodes. The dictionary also notes the observational status (whether observable, latent or derived), and the parameterisation method used (data-driven, expert-driven or hybrid expert-data) for each variable. Fig. 3 shows a sample of the BN dictionary; the full dictionary is provided in Additional File 2.
The full dictionary is provided in Additional File 2.
3.2 Model description
3.2.1 The epidemiological context
The model introduced three latent concepts (light blue) to simplify and describe the complex nature of pneumonia epidemiology in children; namely the level of exposure (12), the susceptibility to colonisation (13), and the susceptibility to progression (14) (Fig. 4). The susceptibility to colonisation summarises the level of a host’s susceptibility to being colonised in the nasopharynx with ‘typical’ bacterial pathogens, which is higher in the younger age group (1), in children with impaired immunity (6), and when there is a smoker in the household (3). The level of exposure refers to the host’s exposure to transmissible pathogens. Greater exposure is associated with older age, more childcare days per week (5), and Indigenous Australian ethnicity (2), which may in turn be surrogates of frequent and/or close or prolonged social contact and therefore greater opportunity for transmission. The susceptibility to progression describes the propensity of the host to pneumonia progression if infected, which is higher if the child is younger, is reported to have impaired immunity or has chronic respiratory disease (7).
Due to the number of arcs emanating from Age group, a lighter shade of blue has been used to make the graph easier to read. The detailed model structure and definition for each variable are provided in Additional File 2. Source model files can be accessed via Open Science Framework (https://osf.io/m97vb/).
This BN explicitly describes the presence of six pathogens in the nasopharynx: influenza, respiratory syncytial virus (RSV), human metapneumovirus (HMPV), parainfluenza, mycoplasma and typical bacteria (e.g., pneumococcus) (Fig. 4, yellow nodes (15) - (20)). Various epidemiological risk factors interact resulting in varied prevalence of each pathogen in the nasopharynx, which predispose children to respiratory tract infection including pneumonia, leading to the core part of the model presented in Fig. 5. For example, while seasonality can influence the prevalence of many pathogens, in this model we use influenza season (9) (defined as June to September based on the expected pattern of influenza in the southern hemisphere in the pre-COVID-19 era) as a surrogate of the temporal increase in the prevalence of influenza, RSV and HMPV, with the model estimating increases of 220%, 173% and 177%, respectively. Influenza vaccine (11) reduces the probability that influenza is present in the nasopharynx by a modelled estimate of 67%. Recent antibiotic exposure (60) reduces the probability of typical bacteria being present in the nasopharynx from 75% to 49%.
The detailed model structure and definition for each variable are provided in Additional File 2. Source model files can be accessed via Open Science Framework (https://osf.io/m97vb/).
3.2.2 The infection
Fig. 5 shows how the BN depicts infection, based on whether viruses or typical or atypical bacteria are involved in a particular pneumonia episode, and whether upper respiratory sites (nasopharynx and/or throat) are involved when pneumonia is present. Mycoplasma is an intracellular bacterial pathogen with similarities to viruses with respect to mechanism of acquisition and some clinical manifestations. For modelling purposes, we grouped it with the viruses (described as the viral-like group) while keeping all the other bacteria in the typical bacterial group. For viral-like pathogens, the BN assumes infection of the lung (viral-like pneumonia (23)) can either occur directly without observed upper respiratory tract infection (URTI), or it can occur following observed URTI including viral-like nasopharyngeal infection (21) and viral-like throat infection (22). The exact probability of each virus causing a respiratory infection is influenced by its pathogenicity (parameters estimated via the hybrid expert-data approach). Viral infection of the nasopharynx or throat increases the probability of viral pneumonia. Unlike viral-like pathogens, typical bacteria frequently colonise in the nasopharynx (62% in the model), described by Typical bacteria in nasopharynx (20), and spontaneously cause pneumonia (typical bacterial pneumonia (24)) at a much lower frequency than viruses. In addition, pneumococcal vaccination, the presence of viral-like infection, and the age of the child, can all directly influence the probability of typical bacterial pneumonia.
We introduced the causative pathogen for pneumonia (25) and upper airway involvement (26) variables to summarise the above. We defined the causative pathogen for pneumonia to be typical bacterial if typical bacterial pneumonia (24) is present, viral-like if viral-like pneumonia (23) is present and typical bacterial pneumonia is absent, or no pneumonia if both the typical bacterial and viral-like pneumonia are absent. Note that the probability of no pneumonia should always be 0% in our model as it was parameterised using data from a cohort of children who all had X-ray confirmed pneumonia (27). The operational definition of clinical diagnosis of bacterial pneumonia (28) in the cohort study was either having a pleural effusion (46) or a positive blood culture result (53), comprising 11% of children in the cohort. As illustrated in Fig. 5, this definition is subject to potential measurement and selection biases introduced by prior antibiotic exposure, the decision to perform blood culture (61) and hospital transfer (59).
The BN predicted that of all pneumonia presentations, 28.4% involved infection with typical bacterial pathogens with a mean of 25.2% based on the 10-fold cross validation (10-fold mean), while 92.8% (10-fold mean 94.6%) involve infection with viruses or viral-like pathogens. Among all pneumonia presentations involving infection with viruses or viral-like pathogens, 22.8% (10-fold mean 20.1%) were predicted to also involve infection with typical bacterial pathogens. From the decision-support perspective, it is important to differentiate infection with mycoplasma from viruses, and this can be addressed by combining information provided by two nodes, the causative pathogen for pneumonia (25) and mycoplasma in nasopharynx (19). In other words, if the model predicts a high probability of infection with a viral-like pathogen and also a high probability of mycoplasma in nasopharynx, there is a high probability that mycoplasma is the causative pathogen (with possible implications for management).
3.2.3 The evidence
From a clinical perspective, a child’s causative pathogen for pneumonia (25), current clinical phenotype (52) and upper airway involvement (26) are not directly observed (i.e., they are latent), and can only be inferred from clinical observations and the results of laboratory investigations. From a causal perspective, these latent variables interact to give rise to the various types of evidence including clinical signs, symptoms, and laboratory results (Fig. 6). The potential causes of signs, symptoms, and laboratory results are rarely deterministic, singular or mutually exclusive. For example, cough (29) can be driven by upper airway irritation, inflammation, or mucus production (e.g. postnasal drip secondary to rhinorrhoea/rhinitis) or can be from pneumonia affecting the lower airways. Similarly, increased respiratory rate (53) can be a consequence of airway inflammation or low oxygen saturation (58) and can vary by the child’s age.
Causative pathogen for pneumonia, current clinical phenotype and upper airway involvement interact to give rise to the various types of evidence including clinical signs, symptoms, and laboratory results. The BN dictionary (Additional File 2) details how these latent variables may give rise to each type of evidence. Source model files can be accessed via Open Science Framework (https://osf.io/m97vb/).
Background (dark blue) and intervention (purple) factors may also influence certain types of evidence. For example, age is an important determinant of the report of pain (e.g., headache (30), abdominal pain (47) and sore throat (32)) because children need to be old enough to communicate these symptoms. Recent antibiotic exposure (60) may reduce the probability of a successful blood culture (blood culture result (53)) even if a child is bacteremic. Knowing the type of oxygen (62) (supplemental or room air) is essential for interpreting oxygen saturation in relation to lung function. Symptoms and signs may also interact, e.g., vomiting (39) and diarrhoea (40) may lead to reduced oral intake (41) which could subsequently lead to lethargy (42).
The classification of the current clinical phenotype of pneumonia is analogous to a count-based metric for disease severity, where those with phenotype 1 had a median of 12 pneumonia-relevant signs and symptoms, compared to a median of 8 for phenotype 2. The inclusion of this variable allowed us to separate the differential influence of causative pathogen (typical bacterial or viral-like) from the severity of clinical manifestations conditional on the causative pathogen. For example, the overall probability of a child being phenotype 1 (severe) was almost identical for pneumonia involving viruses and viral-like pathogens (14.0%) and those involving typical bacterial pathogens (13.6%). For a child who had symptom onset within two days and been transferred from another hospital, the probability of phenotype 1 was much higher for viral-like pneumonia (79.4%) than typical bacterial case (42.6%). In contrast, if the child presented to ED directly (not transferred), the overall probability of phenotype 1 is lower (15.5%), however, typical bacterial pneumonia has a more severe phenotype (23.8%) than their viral-like counterparts (12.1%).
3.3 Model evaluation and sensitivity analyses
Fig. 7 presents the BN predicted probabilities of clinical diagnosis of bacterial pneumonia for every pneumonia episode for children in the cohort, and compares these against their final diagnosis (i.e., Bacterial Diagnosis: Yes/No). Note that we use the recorded clinical diagnosis as the best estimate of the causative pathogen type, acknowledging that some of these diagnoses may be incorrect. So long as the number of incorrect diagnoses is not great, this will not affect our conclusions. The graphs represent four input scenarios (a-d), each providing more information to the model than its preceding scenario:
Predicted probabilities of clinical diagnosis of bacterial pneumonia are compared with the final bacterial diagnosis for four input scenarios, with mean log loss and AUROC based on 10-fold cross-validation. For each scenario, available information from each participant was entered as input for the BN to predict the clinical diagnosis of bacterial pneumonia.
provides the model with available basic background factors of the partipants; these can include age group, childcare days per week, chronic respiratory disease, ethnicity, influenza season, prematurity, smoker in the household, impaired immunity, influenza vaccine, pneumococcal vaccine, and previous significant infection.
uses the information in (a) plus signs, symptoms and interventions if reported; these can include duration of symptom/s onset, rhinorrhoea, cough, respiratory rate, oxygen saturation, breathing difficulty, crackles, wheezing, (parent-reported) fever, highest temperature, chill, sweats, irritability, vomiting, diarrhoea, reduced oral intake, lethargy, chest pain, abdominal pain, other pain, earache, sore throat, headache, rash, (prior) antibiotic exposure, type of oxygen, hospital transfer, and (decision to) perform blood culture.
uses information in (b) plus any detection (or no detection) of respiratory pathogens in the nasopharynx; these can include RSV, influenza, HMPV, parainfluenza, mycoplasma, and any typical bacteria.
includes (c) plus all other available results; these can include C-reactive proteins (CRP), white cell count (WCC), and neutrophil proportion (of total WCC).
Graphs that show a strong overlap (such as in Fig. 7a) indicate the model is only able to achieve weak discrimination between bacterial diagnoses and others. However, as further information is provided (through scenarios (b), (c) and (d)), the discrimination improves, as evidenced by both the graphs and the improving AUROC scores. In contrast, the log score does not improve over the first 3 scenarios; coupled with the improving AUROC, this suggests the initial probabilistic prediction is already quite close to the true class. Log loss does improve in the final scenario, as does AUROC again and the greatest discrimination is visible in this graph. Table 1 additionally presents the evaluation metrics for other targets including pleural effusion, positive blood culture result, and the detection of viruses or mycoplasma in the nasopharynx.
The performance of BN v4.5 in predicting a range of evaluation targets under different input scenarios, in the form of mean log loss and AUROC based on 10-fold cross-validation. Note for each target and scenario, the variables used to operationally define the evaluation target were removed from the input variables.
We varied relevant parameters by 20% to investigate how various factors affect the predicted probability of typical bacterial pneumonia (Fig. 8). This target prediction is most sensitive to parameters describing the pathogenicity of the potential pathogens in causing pneumonia, ranging from 12% to 50%. Varying the prevalence of various pathogens in the nasopharynx caused the predicted probability of the involvement of typical bacteria to vary from 22% to 30%. Varying the assumptions regarding the latent epidemiological concepts (e.g., level of exposure) had a small influence on the probability of involvement of typical bacteria (25% to 27%).
Relevant parameters were varied by 20% to investigate how they affected the predicted probability of involvement of typical bacteria. This target prediction was most sensitive to the pathogenicity of potential pathogens in causing pneumonia (white), followed by the prevalence of various pathogens in nasopharynx (yellow), and was least sensitive to the latent epidemiological concepts (blue).
3.4 Demonstrative clinical scenarios
When serving as a clinical decision support tool, the available background factors, clinical observations and investigation results would be entered as model inputs for making inferences about the pathogen causing pneumonia in a particular episode. We presented three scenarios to demonstrate the potential use of this model in a clinical context (Fig. 9-11). The BN predicted probability of typical bacterial pneumonia is 28% for the cohort of children with X-ray confirmed pneumonia (labelled in the blue box at the top right corner). For each scenario, this prediction is updated, branching in multiple possible ways that are conditional on information that may become available over time along the child’s clinical course. In parentheses we provide the change in the target prediction in relation to the cohort average; we use red text to indicate if the predicted probability of typical bacterial pneumonia is higher than the cohort average, and use green text to indicate predicted target probabilities which are more than 50% lower than the cohort average.
This use case presents an infant with X-ray confirmed pneumonia, moderately high temperature, cough and runny nose. The BN predicts a low probability of typical bacterial pneumonia for this child, and a reported history of diarrhoea alone does not largely affect the prediction. The target prediction remains low unless both the CRP and WCC are moderately elevated.
Fig. 9 presents a scenario in which the BN predicts a low probability (13%) of a typical bacterial cause of pneumonia for a 4 month old infant with X-ray confirmed pneumonia with moderate signs and symptoms (i.e., highest temperature of 38.5 degrees Celcius, runny nose and cough). The predicted probability remains low unless both the CRP and WCC are high (at least 100 mg/L and 12 × 109/L, respectively). In contrast, if instead a 9 year old (school-aged) child with X-ray confirmed pneumonia presents various signs and symptoms (Fig. 10), the probability of typical bacterial pneumonia is predicted to be higher and similar to the cohort average. The predicted probability of typical bacterial pneumonia is almost doubled if the child has no breathing difficulty, and a moderately high CRP (40 mg/L) can be indicative for a bacterial cause. In other words, the significance of a raised CRP varies under various scenarios. The third scenario (Fig. 11) presents a 3.5 year old (pre-school aged) child with X-ray confirmed pneumonia and very high temperature during the influenza season. The probability of typical bacterial pneumonia increases if the child is under-vaccinated for pneumococcus, regardless of whether RSV is detected in their nasopharynx. If fully-vaccinated, the presence of wheezing will lead to a decrease in the predicted probability of typical bacterial pneumonia, and this prediction will drop to only 4% if RSV is detected in the child’s nasopharynx.
This scenario presents a school-aged child with X-ray confirmed pneumonia and with moderately high temperature, runny nose and high respiratory rate. The BN predicts a 22% probability of typical bacterial pneumonia for this child, and this prediction increases to 42% if the child has no breathing difficulty. Moderate and very high CRP and WCC further increase the probability of a typical bacterial cause for this pneumonia.
This scenario shows a pre-school aged child with X-ray confirmed pneumonia and very high temperature during the influenza season. The probability of typical bacterial pneumonia increases if the child is under-vaccinated for pneumococcus, regardless of whether RSV is detected from their nasopharynx or not. If fully-vaccinated, the presence of wheezing will lead to a decrease in the predicted probability of typical bacterial pneumonia, and this prediction will drop to only 4% if RSV is detected in the child’s nasopharynx.
4 Discussion
The diagnostic challenge of differentiating bacterial from non-bacterial pneumonia is the main driver of antibiotic use for treating pneumonia in children. We have built a causal BN that captures the complex epidemiological context of paediatric pneumonia and how this gives rise to the variable clinical manifestations observed in the hospital setting. Based on the combined use of domain expert knowledge and data, the model was designed to apply to a cohort of children with X-ray confirmed pneumonia who presented to a tertiary paediatric hospital in Australia. The resulting BN offers explainable and quantitative predictions on a range of variables of interest, including the diagnosis of bacterial pneumonia, detection of respiratory pathogens in the nasopharynx, and the clinical phenotype of a pneumonia episode. These outcomes can be utilized to derive recommendations to support more directed and judicious use of antimicrobials for this cohort.
Quantitatively, the model achieved stable and good performance (Table 1) in predicting the directly observable variables like the diagnosis of pleural effusion, positive blood culture and detection of RSV and mycoplasma in the nasopharynx, based on 10-fold cross-validation. Qualitatively, we used commonly encountered scenarios to assess the potential usefulness of the model outputs in various clinical pictures. To our knowledge, this is the first causal model developed to help determine the causative pathogen for paediatric pneumonia. Our model framework and the methodological approach can be adapted beyond our context to broad respiratory infections and geographical and healthcare settings.
4.1 Using latent concepts to reveal the epidemiological context
When a child has X-ray confirmed pneumonia, the causative pathogen for pneumonia usually remains unobserved, i.e., latent. Patient background factors such as basic demographics, seasonality, vaccination and comorbidities can help predict the latent causative pathogen because they often causally influence the presence of a respiratory pathogen in the child’s nasopharynx or respiratory tract which in turn predisposes to pneumonia. For example, RSV is more prevalent in influenza season owing to its elevated circulation in the community, and bacterial infection is more probable in a younger child due to the interplay between their developing immunity and respiratory microbiota [40, 41]. The difficulties faced by clinicians arises from the dynamic and complex interactions amongst these factors, as we saw in Fig. 11 for a young child who presents to hospital in the influenza season.
We addressed this challenge by firstly organising relevant variables under a causal framework (Fig. 4), and introducing several latent concepts such as the level of exposure and the susceptibility of a child to colonisation. This approach helped explain the potentially complex role of each variable in the problem domain. For example, older children may have lower susceptibility to bacterial colonisation as well as to pneumonia progression, with both leading to a reduced probability of typical bacterial pneumonia [41]. However, they are also more likely to have a high exposure through attendance at childcare or school [42], leading to an increase in the nasopharyngeal presence of all pathogens, thereby increasing the potential for subsequent infection. Being explicit about these underlying mechanisms allowed us to capture and understand the role of each factor in the context of each pneumonia episode, which in turn gave us further insight into the epidemiological context and more precise predictions of the causative pathogen for pneumonia.
4.2 Mapping clinical and laboratory evidence to explicit causal paths
Model predictions can be further improved by utilising additional information which is likely to be available shortly after presentation, namely, clinical observations and laboratory investigation results. By modelling the overlap and divergence in the pathophysiological pathways indicated by each piece of clinical and laboratory evidence (Fig. 6), the causal BN approach can estimate the episode-specific probability of each causative pathogen based on any input variables available. As illustrated in clinical scenario 1 (Fig. 9), while diarrhoea can be a feature of either viral or bacterial pneumonia, knowing about diarrhoea alone does not greatly affect the predicted probability of typical bacterial pneumonia. However, when additional information such as CRP and WCC becomes available, the prediction can vary from close to 0% up to 82%.
A challenge faced by clinicians is the variable predictive value of specific clinical and laboratory observations when the underlying epidemiology changes [15]. More specifically, typical bacterial pneumonia is more likely to drive elevated WCC and CRP [32], however, elevated biomarkers can also be caused by viral pathogens, especially when disease manifestation are severe, leading to reduced specificity for bacterial infection. Our BN addressed this issue by introducing a variable to summarise the clinical phenotype for each episode, which functioned to separate the influence of the causative pathogen from the clinical progression of the presenting episode based on the patient’s susceptibility to progression and the duration and severity of the presenting illness. This was illustrated in Scenario 1 (Fig. 9) and Scenario 2 (Fig. 10), in which the same blood results could lead to significantly different predictions of typical bacterial pneumonia for the two demographically and clinically distinct presenting episodes.
4.3 The challenge and need to better understand pathogenicity
It became apparent since early in the model development (after Workshop 1) that the pathogenicity of different organisms plays a central role in driving the probability that a pathogen is causative. There are a large number of organisms (viral and bacterial) that are capable of causing pneumonia, while there are also frequent detections of multiple potentially pathogenic organisms in the nasopharynx. Hence, it is important to understand the prevalence of relevant organisms in the respiratory tract (including the nasopharynx) as well as the probability of being causative of pneumonia given they are present there. These issues affect how the model should be structured (e.g., to what extent we should separate or group pathogens) and how we should parameterise the model (e.g., given the co-presence of more than one pathogen, which one is the more likely to be the causative agent). Many of these issues remain the subject of debate and ultimately unresolved by the scientific community [43].
We designed a series of survey questions to elicit the current understanding and knowledge of domain experts (Survey 2, 3 and 5). Finding the best way to frame questions proved challenging given the complexity of the problem domain. Consequently, we explored a variety of approaches based on modelling needs, facilitated by graphical illustration, data visualisation, and various question formats. This process also helped clarify the modelling requirements themselves, helping us to better manage our current imperfect understanding of the problem domain. In the areas of greatest need, sensitivity analyses were subsequently conducted to understand how much the uncertainty in our understanding could impact on the BN predicted outcomes (Fig. 8).
4.4 Implications for providing decision support for antibiotic prescription
Satisfactory numeric predictive performance has been achieved by the BN in predicting directly observable variables, especially those used to operationally define bacterial pneumonia. The model was qualitatively reviewed in various forms, including in Workshops 3 and 4 and Survey 4. The review outcomes from our experts from diverse domain areas provided confidence in the potential clinical value that this BN offers as a decision support tool.
Among the 230 children enrolled in our cohort of X-ray confirmed pneumonia, 99% received antibiotics during their hospital stay, although only 11% were confirmed with a clinical diagnosis of bacterial pneumonia [32]. This indicates that the current approach to clinically diagnose bacterial pneumonia fails to provide clinicians the information they need to confidently defer antibiotics. Applied to this cohort, the outcome BN predicted that 28% had typical bacterial pneumonia (either bacterial or viral-bacterial co-infection), indicating the potential to greatly rationalise use of antibiotics.
The three scenarios in § 3.4 provide an insight into how we envisage the model outputs could support decisions under various clinical contexts, leading to numerous questions about how model outputs should be presented. Even if causal models have good performance and produce explainable outputs, they will not be useful tools unless their outputs can be presented in a form that end-users can understand, interpret and use. The barriers to implementation and uptake that decision support tools including BNs face are widely reported [44, 45]
4.5 Study limitations and future direction
The BN needs further validation before it can be clinically implemented, particularly examining the performance of the model in additional cohorts that differ in both key contextual factors and in the availability and operational definitions of superficially equivalent variables. Examples of such differences include: whether the definition of pneumonia is distinct across studies; whether a different spectrum of respiratory infections are involved; and how the clinical epidemiology differs based on the geographically distinct transmission dynamics or variations across healthcare systems. The results of these additional cohort validations should in turn feed back and inform how the BN should be further developed and adapted to other settings.
It is important to note there is a distinction between the actual presence of an organism at a body site and its successful detection by sampling at that body site; we only ever know about the former via evidence of the latter. In the model, we used the detection of organisms as a surrogate for their actual presence in the nasopharynx, in order to reduce the number of latent variables. For our model development, these data were collected prospectively and PCR testing was performed systematically to identify relevant pathogens over and above the routine clinical practice, generating high quality information with little missing data. However, the quality of model predictions will not be as robust if data from routine clinical review and investigation are used alone. Future work should consider the sensitivity and specificity of the laboratory techniques used to capture this information.
The BN can be further improved using richer data with a higher level of granularity. For example, information on the severity of reported symptoms was limited. Furthermore, other comorbidities were identified but are yet to be included in the model, such as the history of neurological conditions that may affect susceptibility to colonisation and disease progression. There is also the potential to include further information on the use of intravenous fluid, intensive care management and patient progression over time, which would enable the model to predict the likely effects of specific interventions. Integrating such models into electronic medical record systems to enable efficient data capture is an important future direction, as it would significantly improve the ability to evaluate, maintain and use these models and, ultimately, to derive the most value from them.
Data Availability
All data produced in the present study are available upon reasonable request to the authors, the source BN model with associated dictionary can be accessed via Open Science Framework at https://osf.io/m97vb/.
Acknowledgments
This project was supported by Perth Children Hospital Foundation (Grant ID 9900). YW is supported by the Western Australian Health Translation Network Early Career Fellowship and the Australian Government’s Medical Research Future Fund (MRFF) as part of the Rapid Applied Research Translation program. CCB is supported by an Investigator grant from the National Health and Medical Research Council (NHMRC, APP1173163). TS is supported by a NHMRC Career Development Fellowship (GNT1111657). LK is supported by a Perron Foundation fellowship and NHMRC Ideas Grant (APP2003876). AOM is supported by a NHMRC Postgraduate Scholarship (APP1191465) and an Australian Government Research Training Program Fees Offset. CM is supported by a RAINE Clinician Research Scholarship. PCMW is supported by a NHMRC Investigator grant (APP1197335).
Footnotes
Typos fixed, including 'receiver-operating characteristics curve', 'practice', etc. Units of temperature, CRP, WCC, respiratory rate clarified in both main manuscript and Additional File 2. Additional File 1, Survey 4 updated, with model version number clarified.
References
