A Novel Application of Neural Networks to Identify Potentially Effective Combinations of Biologic Factors for Enhancement of Bone Fusion/Repair

INTRODUCTION

Bone repair is a highly multifactorial process involving a wide array of molecular and cellular factors.[1] Orthopaedic surgeons have manipulated these factors by administering various biologic agents in order to augment bone repair.[2] In most cases, surgeons have administered only one biologic agent. Considering the physiological complexity of the process, it is reasonable to suggest that superior bone repair could be achieved using biologic factors in combination.

Combinatorial explosion prohibits exhaustive experimental evaluation of the full set of possible combinations. An alternative is to use computational methods to extrapolate, or generalize, from existing data and predict which combinations would be the most effective, and then expend experimental resources to evaluate only those. The most powerful AIs in use today for making accurate predictions are artificial neural networks (ANNs).[3, 4] ANNs are composed of many, highly interconnected neuron-like elements known as units, which can be arranged in layers or circuits.

ANNs are computational devices that process information from their input units to produce a pattern of activation at their output units. Feedforward networks have their units arranged in layers. The simplest feedforward networks have only two layers of units: input and output. More complex feedforward networks have one or more layers of hidden units, so called because they are interposed between the input and output layers. Feedforward ANNs are considered deep if they have more than two hidden layers.[5] Recurrent networks have their units arranged in circuits. Multiple processing layers, or circuits, are needed when the production of useful output patterns requires the processing of complex interactions among the inputs.

ANNS are trained via machine learning on a set of input/desired-output examples. They have been applied in many domains of biomedicine.[6-8] The most extensive medical applications of ANNs have been in radiology.[9-12] Generally in these applications the inputs are the pixels of (usually MRI) images, and the desired outputs are the components of known radiological diagnoses. Once trained, the ANN could generalize from its training data and could make a diagnosis from an image on which it has not been trained. It is likely that clinicians will soon use ANNs adjunctively in radiological diagnosis.

Other applications under development of ANNs in medicine include cancer diagnosis from gene expression data[13, 14], heart-disease diagnosis from electrocardiogram data[15], osteoporosis diagnosis from bone-density data[16], and diabetes diagnosis from blood chemistry data[17, 18]. More recent applications involve patient records as inputs.[19-21] ANNs have been applied to orthopaedic patient records, to predict outcomes such as bone fracture healing or mortality following hip fracture.[22, 23]

The application of various biologic agents to bone repair is a rapidly growing subfield of orthopedics. Over the past few decades, many reports have demonstrated the benefits of specific agents on post-surgical bone fusion rates. Still, to our knowledge, AI has not been applied in this domain.

The usefulness of an ANN derives from its ability to generalize beyond its training data, so that it can predict the correct output for inputs on which it has not been trained. We collected a large amount of the available data, organized it into a form suitable for machine learning, and used it to train an ANN with an architecture that we had determined beforehand would generalize well over the dataset. We used this ANN to explore potential combinatorial therapies within the realm of orthobiologic adjuncts.

METHODS

Our study design consisted of six steps: (1) assemble a dataset on the outcomes associated with the use of various agents and organize them into a form suitable for machine learning; (2) build a series of ANNs with increasingly complex architectures and processing potentials; (3) determine for each network type its optimal machine learning parameters; (4) assess the ability of each network type to generalize over the dataset; (5) train the best generalizing network on the dataset and use it to predict the efficacy of a large set of combinations of the factors on which it had been trained; and (6) analyze the predictions and determine which combinations of factors are potentially the most effective, and should be experimentally verified. The first five steps are methodological and are summarized here in the Methods section. Further methodological details are available in Supplementary Texts S1 – S4. Step (6) is elaborated in the Results section.

Steps 3 and 4

We optimized the parameters of the machine-learning algorithm used to train each of the network types, and then tested the ability of each type to generalize over the dataset following training. We found that the feedforward ANN composed of an input layer, an autoencoder layer, 2 hidden layers, and an output layer exhibited the best generalizability. A diagram of this ANN is shown in Figure 1.

• Download figure
• Open in new tab

Figure 1.

Diagram of the artificial neural network used to predict the efficacy of 2400 combinations of orthobiologic factors. The feedforward network has two layers of hidden units. The first hidden layer receives the input not directly but only after it has been processed by a layer of autoencoder units. The 17 input units project to a layer of 50 autoencoder units, which project to the first layer of 100 hidden units, which in turn project to the second layer of 100 hidden units, which finally project to the set of 26 output units. The connectivity between layers is complete in that each unit in a previous layer projects to every unit in a subsequent layer. The weights from the input layer to the autoencoder layer are trained separately, and then held fixed while the other weights in the network are trained.

RESULTS

We trained the ANN with the best generalizability (Figure 1) to achieve a good but not perfect match between its actual and desired outputs, because the overtraining required to achieve a perfect match would impair its ability to generalize. Comparison of the desired and actual output images for an example ANN (Figure 2) shows that the agreement is good but not perfect. Precisely this sort of relationship would be expected for an ANN that could generalize beyond its training data.

• Download figure
• Open in new tab

Figure 2.

Desired and actual outputs of the neural network after training. The values of the 26 output units for each of the 225 input/desired-output patterns in the training dataset are shown as images, separately for the desired (left) and actual (right) outputs. The unscaled values of the outputs range from 2 (deep blue) to 277 (bright yellow). Close inspection reveals that the match between the desired and actual outputs is good but not perfect. This is expected due to ambiguity in the training dataset, which is derived directly from the experimental data of multiple labs that often reported different outputs for the same inputs. The pattern of agreement in general, with disagreement in detail, indicates that the neural network has learned to generalize from the data in the training dataset. The ability to generalize is central to the ability of the neural network predict the outputs for combinations of inputs on which it has not been trained.

We rank-ordered the predicted efficacy scores for the 2400 combinations (Figure 3). They ranged from about 0.30 to almost 0.75 and so covered almost half of the possible [0, 1] range. The efficacy scores seemed to plateau for the most effective several hundred combinations.

• Download figure
• Open in new tab

Figure 3.

The efficacies of 2400 combinations of orthobiologic factors as predicted by the neural network. The scores are sorted from least to most effective. Effective combinations were defined as those that were high in 12 / 26 outputs (DR, BF3, BF6, MV/TV, PLF-FR, ODI, FR, FH, OW, RO, HO, and IS), and low in 5 / 26 outputs (TWC/CH, TRU, RBG, NH, and DY). The remaining 9 / 26 outputs were not included in the efficacy measure (see text). For the purposes of ranking, the outputs that should be low were flipped, all outputs were scaled in the range [0, 1], and the output values were averaged. By this measure, which is relative to the maximal and minimal output values, the highest possible efficacy of 1 would be obtained if all of the outputs that should be high / low were at their maximal / minimal levels for that output, and vice-versa for the lowest possible efficacy of 0. The predicted efficacies for the 2400 combinations in the screen varied widely over the [0, 1] range and nearly plateaued for the most effective several hundred.

The 2400 rank-ordered combinations are shown in two separate images in Figure 4: one for all 2400 combinations and another for the top 200. Analysis of the top 200 reveals some statistically significant, pairwise correlations among the 9 factors (Table 1). BMP2 and BMP7 are positively correlated, while OG and PDGF are negatively correlated. PDGF is negatively or positively correlated with BMAC or PRP, respectively. PU and ES, and likewise BMAC and PRP, are also negatively correlated, but this is due largely to constraints in the design of the combination screen (see Methods).

• Download figure
• Open in new tab

Figure 4.

The combinations of orthobiologic input factors ordered by predicted efficacy. The combinations are presented in descending order, so the combinations are most / least effective at the top / bottom of either plot. The plot on the left shows all 2400 combinations while the plot on the right shows the top 200 combinations. The factors ES, BMAC, and PRP are either present or absent; EBG is always present. The factor BMP7 takes 2 levels; BMP2, OG, and PU each take 4 levels; and PDGF takes 5 levels. Input factor levels varied over a broad range but were normalized into [0, 1] for purposes of illustration. In the images, yellow and deep blue correspond to 1 and 0, respectively.

The 10 best factor combinations show some consistent similarities, and some consistent differences with the 10 worst combinations. BMP2 and BMP7 are at their highest levels in the 10 best combinations, while they are 0 in the 10 worst combinations, and this is consistent with the positive correlation observed between BMP2 and BMP7 in the top 200 combinations. OG tends to be at its highest levels in the 10 best combinations but is 0 in the 10 worst combinations. In contrast, PDGF tends to be at its lowest or highest levels in the 10 best or worst combinations, respectively, and this is consistent with the negative correlation observed between OG and PDGF in the top 200 combinations.

The analysis suggests that the most effective combinations are high in BMP2, BMP7, and OG, but low in PDGF (see Table 2 and its caption for quantification of amounts). These 4 factors seem to be the most determinative of the best factor combinations. BMAC appears in some of the 10 best combinations but in none of the 10 worst, and this is consistent with the negative correlation between PDGF and BMAC. PRP is absent from all 10 best and worst combinations. The 10 best and 10 worst combinations seem indifferent to the levels of PU and ES, with the potentially important exception that ES appears in some of the 10 best combinations that lack OG.

The 2400 combinations in the screen include the null combination (ie none of the 9 factors are present except for EGB, the common vehicle), and all combinations in which EGB and 1 other factor only are present. The analysis clearly indicates that combinations of several orthobiologic factors would be more effective than any single factor alone. The analysis indicates that combinations of BMP2, BMP7, and OG, perhaps including ES or BMAC, each at the high end of their ranges as reported in relevant studies, should outperform combinations that lack those components. Experimental verification of these predictions could lead to the development of orthobiologic factor combinations that outperform single factors for the enhancement of bone repair.

DISCUSSION

To properly situate our model within the orthopaedic literature, it is necessary to distinguish between process-driven and data-driven models. Process-driven models represent processes explicitly. There is a long tradition of process-driven modeling in bone fracture healing (see [1] for review). Process-driven models are valuable in that they explicitly describe the processes involved, but they are limited to what is known about the processes themselves. This limits their predictive power.

Data-driven models are built almost entirely on observed input-output relationships, without regard for the specifics of the underlying processes. Data-driven models offer little mechanistic insight, but they provide a powerful means to leverage all available data for predicting the outputs to novel inputs. Deep neural networks are the premier form of data-driven modeling in AI today. The multilayered ANN we chose to make our predictions (Figure 1) is a deep neural network.[5] To our knowledge, our model is the first data-driven, deep neural network model of the relationship between biologic factors and bone repair.

Even though it is data-driven, our model may indicate avenues for further research into the molecular physiology of bone healing. For example, OG (osteogenin, or bone morphogenic protein-3 (BMP3)) and ES seem to act interchangeably in our model. Interestingly, ES has been shown to upregulate BMPs 2 through 8, and is effective in upregulating BMP3 (also called OG) in cultured bone cells.[27] The fact that our model is able to postdict previously known molecular pathways advocates for its use in predicting previously unknown molecular interactions.

The main limitation in our study was in the size and composition of the dataset on which we trained our ANN. At 225 input/desired-output training patterns, our dataset is large in comparison with other datasets that are curated from the literature, but still very small in comparison with datasets used to train ANNs in many applications. Also, most of the input/desired-output patterns in our dataset included only one active orthobiologic factor. The risk in training mainly on single factors is that the network would fail to learn interactions among them but in our case, it seems that this did not occur.

If machine learning failed to pick up interactions, then the simplest ANN, that composed only of input and output layers, would have generalized as well as, if not better than, ANNs with hidden layers (or circuits) intervening between input and output (see Supplemental Text S4). The fact that the ANN that generalized best over our dataset was a multilayered network strongly suggests that it did learn some of the interactions between the factors.

The best way to remedy the main limitation in this analysis is to train deep neural networks on larger datasets containing more combinations of factors. The analysis already suggests both good and bad combinations that could be explored experimentally. Any and all new data on the outcomes for bone healing associated with orthobiologic factors administered alone, or better, in combination could be added to the training dataset and would improve the ability of an ANN to identify combinations of factors with the potential to outperform single agents.

CONFLICTS OF INTEREST

The authors have no conflicts of interest to declare.