Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

Dataset

The National Electronic Injury Surveillance System (NEISS) is a national sample of hospitals in the United States and its territories.[10] Patient information was collected from each NEISS hospital for every emergency visit involving an injury associated with consumer products. NEISS injury data are gathered from the emergency departments (ED) of approximately 100 hospitals selected as a probability sample of all 5,000+ U.S. hospitals. The system’s foundation rests on ED surveillance data, but the system also has the flexibility to gather additional data at either the surveillance or the investigation level. We choose this dataset for three reasons: 1) it contained both structured and unstructured data; 2) the length of the narratives was suitable to apply BERT-based models given the limitations of BERT models to handle long sequences; 3) it was a nationally representative dataset with low missing rates.

NEISS data contains the individual’s age, sex, race, ethnicity, injury diagnosis, affected body parts, incident locale, as well as a brief narrative description of the incident. For each case in NEISS, both structured and unstructured data were recorded. The structured data includes personal information such as age, sex, and race, and injury information such as location of injury, injured body part, fire involvement, and injury diagnosis. Figure 1 shows several examples of the unstructured data in the narratives. A complete narrative included three parts: basic information, injury description, and diagnosis, which could be identified by terms such as ‘YOM’ (i.e., Year of Male) and ‘DX’ (i.e., diagnosis). We removed the age, and sex, and the diagnosis, and only kept the information of products and events involved in the injury.

• Download figure
• Open in new tab

Figure 1. Methods pipeline and flowchart.

Figure 1 demonstrates the examples of the unstructured data in the narratives. The narratives should comply with the following format:

• Age and sex should be at the start of the narratives (i.e., 10YOM, 11MOF)

• Being descriptive and including details about the incident (i.e., who, what, why, when, where, how).

• Describing the sequence of events, alcohol/drug involvement, and the affected body part(s) in the middle of the narrative.

• Including the patient’s blood alcohol concentration/level (BAC/BAL) or breath alcohol results whenever alcohol use was associated with the incident. If a BAC/BAL or breath alcohol was not taken or recorded; the narrative should state this.

• The relevant clinician’s diagnoses should be at the end of the comment exactly as written in the ED record, and denoted with “DX:”. This abbreviation helped to distinguish the clinician’s diagnoses from other details about symptoms and complaints. If there were no clinical diagnoses in the ED record, “NO DX” were put at the end of comment.

• Quoting verbatim the words actually used within the ED record.

Examples of comments following these guidelines:

• 60YOF woke up on floor, unsure if fell from bed or if woke up and fell, heavy alcohol use night before. +5cm laceration thru all subcutaneous layers. BAC 0.19%. DX: laceration of forehead.

• 25YOM cut his hand when picking up broken glass from a bowl that dropped on the floor. DX: laceration of superficial palmar arch of right hand; laceration of left index finger without foreign body.

• 80YOF was vacuuming when walking from bedroom to hallway and caught her foot on an extension cord, falling to floor and heard pop in hip. Admitted. DX: fall; left femoral neck fracture.

In this study, the prediction outcome was the injury diagnosis. The structured personal information, structured injury information, and unstructured narrative information were used as model inputs. Records with missing values of diagnosis and/or without ‘DX’ in the narratives were dropped from the raw dataset.

Language model

The complex nature of unstructured narrative data, primarily characterized by long sentences with intricate word relationships, calls for a potent language model capable of decoding semantic information to extract diagnosis-related data. Recent advancements have witnessed the application of pre-trained transformer-based language models, such as BERT [3] and RoBERTa (A Robustly Optimized BERT Pretraining Approach),[11] in text analysis. RoBERTa took the BERT architecture a step further by experimenting with different training approaches and hyperparameters, leading to improved performance. These models, noted for their commendable ability to capture contextual information in long sequence data through attention mechanisms, are becoming more prevalent. In the realm of health care and clinical fields, language models enriched with clinical knowledge like Clinical-BERT [12] and Clinical-Longformer[13] have emerged. ClinicalBERT is a modified BERT model to clinical corpora to address the challenges of clinical text and the representations are learned using medical notes and further processed for downstream clinical tasks.[12] Clinical-Longformer is a clinical knowledge enriched version of Longformer that was further pre-trained using clinical notes; it can allow up to 4,096 tokens as the model input. [13] These models, pre-trained on clinical texts as well as resources like BookCorpus, are engineered to outperform their counterparts in handling clinical downstream tasks.

Multimodal modality fusion

Unstructured data and structured data can be considered as two data modalities. Unstructured data, such as the descriptions of injury causes and symptoms in the clinical narratives, typically contains abundant information. However, the integration of varying content types within a single sentence can present certain challenges for semantic understanding and information extraction. Contrarily, structured data (e.g., affected body parts) offers a more organized format, where diverse information types such as incident locations or injured body parts are clearly categorized under distinct classifications. Unimodal models leverage either structured or unstructured data for inference and prediction, whereas multimodal models strive to integrate both types to enhance predictive performance.

In multimodal analysis, there are two common modality fusion strategies: early fusion and late fusion.[14, 15] Early fusion, a data-level fusion method, amalgamates data from diverse modalities prior to model input. While it is noted for its superior ability to capture the interaction of different modalities, it necessitates a multimodal encoder capable of processing multimodal inputs, making the application of pre-trained unimodal models challenging in this context.[16] Different from early fusion, late fusion is an embedding level fusion approach, which first uses independent encoders to process different modalities, and then fuses embeddings from multiple encoders in a post-processing model.[16, 17] Pre-trained unimodal models can be easily applied as independent encoders in a late fusion. With its flexibility of leveraging pre-trained unimodal models, late fusion is more commonly used in multimodal analysis field.[18]

To use both unstructured narrative text and structured patient information to predict injury diagnosis, we converted structured data into unstructured text for effective early fusion. While leveraging powerful pre-trained language models for processing text data, a late fusion model was first proposed as the baseline model. As shown in Figure 2 (i), this model utilized a pre-trained language model as the text encoder and a randomly initialized fully-connected block as the structured data encoder. The embeddings from these two encoders are then concatenated and subsequently fed into the post-prediction layers to predict the final diagnosis. However, this late fusion model has several disadvantages: 1) the structured data encoder, not having been pre-trained, might exhibit instability and lack of generalization during the training phase; and 2) the interaction between structured data and unstructured text information could be constrained by the use of independent encoders, thus potentially undermining the overall model performance.

• Download figure
• Open in new tab

Figure 2. The frameworks of late fusion baseline(i) and proposed hybrid fusion method(ii).

In an attempt to leverage the benefits of both early and late fusion strategies while surmounting their limitations, we proposed a hybrid fusion approach for multimodal injury diagnosis. As illustrated in Figure 2 (ii), this system amalgamated the techniques of early and late fusion. Given that structured data bear significant semantic information and possess the potential to be interpreted by the language model, we integrated an early fusion module to incorporate structured data into raw narrative contents. Structured data were initially transformed into unstructured words based on their semantic meanings and processed jointly with unstructured text by the pre-trained language model. To elaborate, categorical variables were converted into semantic annotations; for instance, a “sex” value of 1 was translated to its original definition “male”. Numerical variables were transformed into cross-grained categories in accordance with commonly applied designation standards. For instance, ages ranged from 0-1 was defined as infant, 1-12 were children, 12-19 were teenager, 19-45 were young adults, 45-65 was middle age adults, and over 65 were old adults. However, this method might fall short in capturing fine-grained regression information. To exploit the fine-grained numerical information in the structured data, numerical variables were directed into a fully-connected encoder and subsequently fused with the output of the text encoder via a late fusion strategy. Unstructured words from narratives were concatenated with unstructured data and then fed into a pre-trained language model as text encoder. The resultant concatenated embedding was then fed into a post-processing block for the final diagnosis prediction. This strategy utilized pre-trained language models to understand semantic information in the structured data and added more interaction within structured data, and between structured data and unstructured data by self-attention mechanism in the transformer-based language model. In addition, inspired by a location-aware sound event detection model,[19] we used the language model to encode categorical location variable to handle unseen sound scenes in test data, which could also potentially handle unseen categories in the structured data.

Experimental Setup

To mitigate unevenly distributed injury categories in this dataset, a random sampling approach was first applied in each diagnosis category for each year. For categories with more than 1,000 records, only 1,000 records were randomly sampled; for categories with less than 1,000 records, all records were sampled. To evaluate and compare model performance, a stratified random sampling based on categories distribution was applied to split training, validation, and testing dataset with an 80:10:10 ratio.

To compare the effects of unstructured and structured data for injury prediction and validate the effectiveness of multimodal fusion, we compared the unimodal systems using either unstructured or structured data with the multimodal systems using both modalities. For multimodal systems, the late fusion strategy was set as the baseline and compared with the proposed hybrid fusion method.

For the language module used to extract information from the unstructured data, the pre-trained BERT model was employed in both the unimodal system with unstructured data and the multimodal systems. For the proposed hybrid fusion approach, we expanded our comparison set by integrating three additional language models alongside BERT: RoBERTa, Clinical-BERT, and Clinical-LongFormer.

To further improve the prediction performance on unbalanced categories with less data, we minimized focal loss function (as shown in function below) [20], which was designed to address class imbalance issues. Where n denotes the number of categories, p_i denotes the probability distribution of the prediction, the t_i denotes the real probability distribution, and the 𝛾 denotes the focusing parameter.

For evaluation, we used the Top 1 injury prediction accuracy on the validation dataset to identify the best-performed model checkpoint. We assessed and compared the best testing performance using multiclass classification metrics, including Top 1 injury’s accuracy, macro F1-score, and weighted macro F1-score. Because there were 24 injury diagnosis categories in this multiple classification task, which made it challenging to achieve an extremely high accuracy for the Top 1 injury. In addition, the causes and characteristics of some injuries might be similar or intercorrelated, which made it also difficult to distinguish between these categories. Therefore, we introduced Top 2 and Top 3 injuries’ accuracy metrics into our evaluation framework, reflecting real-world application scenarios.

Limitation

This study has a few limitations. First, the success of the proposed hybrid fusion approach heavily relied on the availability and quality of multimodal clinical data. The number of features was small in the NEISS datasets. Second, even though NEISS datasets are nationally cross-sectional, the performance of the proposed approach may vary across different clinical settings or health care institutions. In the future, we will validate this approach in other datasets with diverse data modalities. Third, integrating multiple modalities and applying natural language processing techniques may increase the computational complexity and resource requirements of the proposed approach; this may limit its scalability, particularly when dealing with large-scale datasets or real-time prediction scenarios. The proposed approach may need high computational resources and the time-consuming processing may restrict the practical implementation of the hybrid fusion approach in resource-constrained healthcare environments.