FHIR-GPT Enhances Health Interoperability with Large Language Models

Data Annotation

The HAPI FHIR public test server²⁵ hosts millions of examples of converted FHIR resources. However, we are unable to retrieve their source data before the conversion. To the best of our knowledge, there is no largely publicly available dataset in the FHIR standard that has been generated from the clinical notes. Therefore, we have decided to annotate a dataset that contains both free-text input and structured output in FHIR resources. The latter will serve as the ground truth against which we can evaluate the performance of our LLMs in FHIR transformation.

We manually annotated the medication-related clinical narratives to adhere to the MedicationStatement resource as per FHIR v6.0.0: R6 implementation guide²⁶. According to the official FHIR definition, a MedicationStatement indicates that a patient may currently be taking a medication, has taken it in the past, or will take it in the future. This transformation holds particular significance because many medication-related details, such as the reasons for administration and dosage instructions, often remain absent in structured data. Clinical notes within the Electronic Health Record (EHR) system frequently represent the sole available source for retrieval and conversion into a standardized format. Clinical notes within the EHR system might be the sole source available for the retrieval and transformation of this information into a standardized format. The MedicationStatement encompasses various contents of medication, including dosage, schedule, reason, form, route, strength, and more. For detailed examples of the elements in the MedicationStatement resource, please refer to Table 1.

The clinical text input is obtained from the discharge summaries in the MIMIC-III dataset²⁷. The 2018 n2c2 medication extraction challenge²⁸, essentially a named entity recognition task, provided mentions of medications and the word spans of the medications’ associated entities (including drug routes, frequencies, durations, adverse effects, forms, strengths, dosages, and reasons) within the discharge summaries in the MIMIC-III dataset. All entities were manually annotated by clinical experts. We extracted text snippets, each containing mentions of one medication and all its associated entities, from the discharge summaries. We also included some buffer words from the original discharges summaries before and after the extracted word spans to ensure that these snippets are complete sentences. These extracted snippets, each related to a specific medication, serve as input for both annotations and transformations.

The human annotation for transformation to the FHIR standard consists of three key steps. The first step involved identifying the elements associated with each medication, and this task was effectively addressed by re-using expert annotations from the n2c2 dataset, which accurately pinpointed the word spans of each element. The second step required standardizing the elements from free-text into clinical terminology coding systems. The elements were linked to different coding systems, and we have provided a detailed description of which code systems were used in Table 1. Notably, the medication name was encoded in three distinct coding systems. Initially, the medication name was mapped to the patient’s prescription table in MIMIC-III, where NDC codes were provided. Input data, for which the medication name couldn’t be mapped to the patient’s prescription table, were excluded from the dataset. Subsequently, NDC codes were mapped to RxNorm codes and SNOMED CT Medication Codes using the APIs provided by the RxNav toolkit²⁹. For all other elements, such as reasons, routes, and forms, the SNOMED CT coding system was primarily used, unless HL7.org provided its own code set. The transformation of these codes relied primarily on manual lookup. We looked up the display names, codes, and other SNOMED CT terminology details form the SNOMED CT Browser, International Edition ³⁰. . The third step involved assembling the identifiers, codes, texts, extensions, and structures into a complete MedicationStatement resource. Throughout the study, we utilized the .json structure format. The converted FHIR medication statements undergo validation by the official FHIR validator³¹ to ensure compliance with FHIR standards, including structure, datatypes, cardinalities code sets, display names, etc.

The annotation tasks were conducted by Y. Li, and H.W., who worked collectively to resolve ambiguities or uncertainties. We will make the annotated dataset available to the public for authorized use upon paper acceptance.

Large Language Models

The LLMs we experimented with include OpenAI GPT-4³², Llama-2-70B³³, and Falcon-180B³⁴. We accessed the GPT-4 APIs through the Azure OpenAI service, as recommended by the responsible use guideline of MIMIC data. The specific model we used is gpt-4-32k in its 2023-05-15 version. To enhance efficiency, we made multiple asynchronous API calls. For Llama-2-70B and Falcon-180B, we deployed them on our HIPAA-compliant firewalled local servers with multiple GPU backends. GPTQ ³⁵ was used to accelerate the inference time for Llama-2-70B and Falcon-180B.

We required these Language Models (LLMs) to transform the free-text entries into MedicationStatements conforming to the FHIR standard, employing the few-shot prompt settings. Each clinical snippet was individually input into the LLMs to generate the MedicationStatement resource. We used five separate prompts to instruct the LLM to transform the free-text input into the elements of a MedicationStatement resource, including medication details (such as drug name, strength, and form), route, timing, dosage, and reason, respectively. All few-shot prompts adhered to a template with the following order: task instructions, expected output FHIR templates in .JSON format, 4-5 examples of transformations, a comprehensive list of codes from which the model could make selections, and the input text to be transformed. As there was no fine-tuning or domain-specific adaptation in our experiments, we initially had the LLM generate the FHIR format for a small subset of the dataset (N=∼100). Then, we manually reviewed the discrepancies between the LLM-generated FHIR output and our human annotations. Common mistakes were identified and used to refine the prompts. There were slight differences in the prompts for each LLM, as different LLMs may be sensitive to different prompts. It’s important to note that we did not have access to comprehensive lists of NDC, RxNorm, and SNOMED Medication codes for all medication names, as well as SNOMED Finding codes for reasons. We did not instruct the LLMs to look up the SNOMED codes for the ’medication’ and ’reason’ elements, as the complete list of SNOMED CT Medication and Finding codes, numbering in the thousands or more, exceeds the token limits of LLMs. Instead, our instructions were for them to identify the contexts mentioned in the input text and convert them into the appropriate JSON format. For instance, the expected output is {“reason”: [{“concept”: {“text”: “Headach”}}]} rather than the more detailed {“reason”:[{’concept’: {’text’: ’headache’, ’coding’: [{’system’: ’SNOMED’, ’code’: ’25064002’,’display’: ’Headache’}]}}]}.. For other code sets, such as SNOMED CT Form codes, numbering in the hundreds, we allowed LLMs to directly code them. Please see the appendix for prompts.

Evaluation

We compared the transformed resources with the outputs from two existing approaches: NLP2FHIR ¹⁸ and Google Healthcare Natural Language (NL) API ²². The transformation results from both approaches lacked some elements covered by our human annotation and LLMs generation. NLP2FHIR was built based on a previous version of the FHIR implement guide, and the Google Healthcare NL API primarily standardized concepts to UMLS CUIs, rather than SNOMED CT codes, which are used in our annotations and LLMs’ transformations. We made adaptations and conversion to ensure a fair comparison. We deployed the NLP2FHIR pipeline on our HIPAA-compliant firewalled local servers. We accessed the Google Healthcare NL API through the Google Cloud Healthcare API, which is also compliant with HIPAA regulations.

When evaluating the FHIR resources generated by the LLMs, our initial step was to verify that the output was in valid JSON format. Once the JSON format check was successfully passed, our primary criterion for evaluation was the exact match rate. This criterion required that the resources generated by the LLMs exactly matched the human annotations in all aspects, including structures, codes, and cardinality. Unlike previous studies that reported word scan F1, precision, and recall scores, which considered the transformation as a NER (Named Entity Recognition) task, we did not use these metrics. This decision was made because those metrics may overlook the essential aspects of inferring and standardizing the content based on contexts. Exact identification of the word span does not guarantee the correct corresponding codes can be identified and that the accurate FHIR schema can be derived.