Genetic Transformer: An Innovative Large Language Model Driven Approach for Rapid and Accurate Identification of Causative Variants in Rare Genetic Diseases

Fine-tuned LLMs enable accurate identification of disease-causing variations

We employed a training dataset TIO-1 and a testing dataset TIO-2, comprising synthetic cases from HGMD, OMIM, GPCards and 1000 Genomes Project, for fine-tuning and evaluating the LLMs (detail see methods). This fine-tuning process aimed to enhance the model’s proficiency in identifying candidate causative variants. Fine-tuning was performed on six pre-trained open source LLMs, including Qwen1.5-0.5B, Qwen1.5-1.8B, Qwen1.5-4B, Mistral-7B, Meta-Llama-3-8B, and Meta-Llama-3-70B. The training dataset, consisting of 12 gradients, ranges from 40 to 20,000 cases for the analysis of the scaling law of those models (Table 1). Figure 2 depicts the F1 scores of each model across varying training dataset sizes. When the training dataset size is 0, i.e., when the base model is used without fine-tuning, the F1 scores of all models are 0. These results are unable to effectively discern the pathogenicity of the variations, and within the scope of our study, they are consistently characterized as hallucinations. As the dataset size exceeds 800, the performance of those models stabilizes, with subsequent enhancements becoming insignificant, yielding F1 scores and AUC exceeding 0.9 (Figure 2 A; Supplementary Table S1). It is noteworthy that despite their smaller parameter size, both Qwen1.5-0.5B and Qwen1.5-1.8B exhibit a F1-score comparable to, or even superior to, that of other large-parameter models.

• Download figure
• Open in new tab

Figure 2: Fine-tuned LLMs enable accurate identification of disease-causing variations.

A) Performance of fine-tuned LLMs across varying training dataset sizes. B) The F1 scores of the fine-tuned LLMs across different features.

Our model’s input prompts encompass a diverse array of variant features, including basic features (such as gene symbol, transcript, zygosity and HGVS name), functional data, ClinVar clinical significance, gene-associated disease information, population data, and ACMG interpretation information (Supplementary Table S2). To delve deeper into the learned feature importance of the model, we sequentially incorporated additional features into the inference of those fine-tuned models under the condition of variant basic information, to observe the changes in the F1 score. When only basic information is provided, we observed that models with a larger size of parameters perform better. For example, the model Qwen1.5-0.5B with a smaller size of parameters achieved an F1-score of only 0.0119, while Meta-Llama-3-70B reached 0.6969 (Figure 2B; Supplementary Table S2). This may be due to the fact that larger models possess a certain reservoir of pathogenic variant knowledge, which plays a positive role in determining pathogenic variants. Subsequently, the inclusion of various variant features led to an improvement in the F1 score, with the addition of gene-associated disease information producing the most significant enhancement. For instance, for Qwen1.5-0.5B, its F1 score increased from 0.0119 to 0.9188 (Figure 2B). Similar trends were observed in other models. This underscores the pivotal role of gene-associated disease information in the model’s assessment of pathogenicity.

GeneT outperforms state-of-the-art methods on synthetic datasets

We then fine-tuned the six pre-trained LLMs, using the diversity training dataset TIO-3 (see methods). In GeneT, patient phenotypes are associated with variant information through prompt, and phenotypes are provided in various forms, such as disease names, free-text, and HPO/phenotype terms. We generated 200 synthetic whole-genome sequencing (WGS) samples for each of the three categories by leveraging data sourced from the HGMD, OMIM, and GPCards databases, alongside samples obtained from the 1000 Genomes Project (see methods). This methodology was employed to assess the model’s performance across diverse phenotypic data sets.

After the implementation of GeneT, the number of variants in the three test set samples decreased significantly, most notably for Qwen1.5-1.8B, from an average of 421, 414, and 419 to 9, 5, and 10, respectively. The Mistral-7B performed the least well, with more false-positive variants detected (Figure 3 A, B, C). Meanwhile, the fine-tuned Qwen1.5-1.8B achieved positive variant recalls of 0.98, 0.99 and 0.98 in the three test sets, respectively. Other fine-tuned LLMs exhibit comparable or marginally inferior performance (Supplementary Tables S3-8). These results indicate that GeneT has the capacity to substantially decrease the number of candidate causative variants while upholding a high recall rate for causative variants.

• Download figure
• Open in new tab

Figure 3: Performance evaluation on synthetic datasets.

The number of candidate variants before and after the utilization of GeneT and TopK CDF plots on AD) the HGMD-1000Genomes dataset (HKG-I), BE) the OMIM-1000Genomes dataset (OKG-I), CF) the GPCards-1000Genomes dataset (GKG-I). CDF: Cumulative Distribution Function. The numerical values adjacent to each boxplot represent the mean number of candidate variations.

GeneT was then compared with existing state-of-the-art phenotype-driven variant prioritization methods on the synthetic samples for performance evaluation. We ranked the positive variants identified by GeneT using the predicted token probabilities to calculate the cumulative distribution of TopK recall performance. The results suggest that GeneT has demonstrated significantly superior performance compared to other phenotype-driven methods across the three test sets (Figure 3D, E, F, Supplementary Tables S3-8). Except for Mistral-7B, the performance of other fine-tuned LLMs is comparable, achieving high cumulative recall (>0.9) at lower TopK thresholds. Additionally, the token probabilities predicted by GeneT can be employed to prioritize all candidate variants within each sample, extending beyond the variants predicted as positive by GeneT. Upon prioritizing all variants based on these probability values, we noted that variants predicted as negative but actually positive were positioned at relatively higher ranks (Supplementary Figure S1, Tables S3-8). This indicates that GeneT can also serve as an alternative for prioritizing variant ranking tools.

An application in real-world clinical setting confirms GeneT markedly boosts expert efficiency and precision in identifying candidate variants

We provided GeneT as a plugin in our interpretation system for interpretation experts in real-world clinical setting to assist them in identification candidate causative variants. The efficacy of the GeneT was then evaluated over a one-month period within the real-world clinical setting. In the realm of clinical genetic testing, the assurance of interpretation result reliability often involves the allocation of expert 1 and expert 2 for the initial screening of candidate variations and subsequent review of the preliminary screening outcomes. GeneT is principally employed to aid expert 1 in the process of screening for genetic variations. The results indicated a significant reduction in analysis time when utilizing the GeneT results while adhering to the standard operating procedure (SOP). The average analysis time decreased from 63.22 minutes to 44.11 minutes per sample (p=2.57e-13) (Figure 4A). Additionally, if GeneT results are used directly, there is potential for a 20-fold increase in speed, reducing the time required from approximately 60 minutes to around 3 minutes per sample. Furthermore, a marginal enhancement in the recall rate was observed (p=0.0846), rising from 94.36% to 97.40%. Concurrently, the variance in recall rates among interpretation experts decreased (Figure 4B). The outcomes of this prospective investigation validate the capacity of GeneT to augment interpretive efficiency and precision. These findings emphasize the significance of GeneT as a valuable component within the genetic analysis workflow.

• Download figure
• Open in new tab

Figure 4. The effectiveness of GeneT in real-world clinical setting.

A) The Time consumption and B) Recall rates for interpretation experts with and without the use of GeneT, alongside with GeneT_only recall performance. Each data point in the left graph corresponds to a sample, whereas the points in the right graph denote interpretation experts. Without_GeneT: Experts in interpretation refrain from utilizing GeneT and instead interpret based on their Standard Operating Procedures (SOP). Without_GeneT: Interpretation experts utilize GeneT as a point of reference while adhering to their original SOP. GeneT_only: Solely employs GeneT analysis, with the time denoting the duration for analyzing a single sample using GeneT.

Hallucinations and reproducibility

Hallucination is a prevalent issue in LLMs, while reproducibility is a metric of significant importance in clinical applications. The assessment of hallucination and reproducibility is a crucial aspect of LLM utilization in clinical diagnostics. In this paper, we define the hallucination as outcomes incongruent with the anticipated outputs of “positive” or “negative”. We observed that the pre-training LLM exhibited significant hallucination issues before to fine-tuning, despite our constraints on the model’s output, which were notably improved after fine-tuning, particularly as the size of the fine-tuning dataset gradually increased (Figure 5, Supplementary Table S1). One of the primary reasons for this improvement lies in our transformation of the challenge of screening or ranking the entire pathogenic variation into an individual variation pathogenicity classification problem. Additionally, we enhanced the prompt by incorporating the “COT” directive, enabling the model to engage in step-by-step self-verification and improving its performance in multi-step reasoning tasks. The prompt also served to restrict the model’s output, allowing it to produce only “negative” or “positive” results. Furthermore, the integration of the RAG strategy, which links to external knowledge bases, bolstered the model’s ability to validate answers and to some extent, mitigated the generation of hallucinations.

• Download figure
• Open in new tab

Figure 5. Hallucinations of six LLMs.

The abscissa denotes the magnitude of the training dataset, while the ordinate signifies the occurrence of hallucinations. As the size of training data increases, the hallucinations of the model gradually dissipate.

During text generation, the LLM utilizes probability distributions to select the next word, thereby introducing a certain degree of randomness. To maximize the stability of LLM output, we set the temperature parameter of the softmax function in the LLM to 0. We utilized GeneT to perform inference analysis on a randomly chosen set of six genetic variants, executing a total of 10,000 replicate experiments for each to evaluate the variability in the predictive results. The results of these experiments exactly the same, indicate a high level of consistency and reproducibility in the identification of candidate variants by GeneT.

iGeneT: an interactive analysis platform incorporating GeneT

We developed an interactive analysis platform iGeneT (http://igenet.genomics.cn), which integrates GeneT, enabling users to perform variants analysis using an online web interface. Users can perform analysis by submitting the sample’s clinical phenotype and VCF file.

Datasets

We have established multiple training and testing datasets for model fine-tuning and testing, constructed from data sources, including HGMD[34], OMIM[35], GPCards[36], and the 1000 Genomes Project[37]. All variants have been annotated by an in-house pipeline, integrating tools such as BCFanno[38] and SIGVAR[39]. To mitigate the influence of the training set on the testing set, it is ensured that the testing set comprises variants not present in the training set.

Phenotype-driven methods

Four phenotype-driven tools were selected for comparison: PHIVE[40], PhenIX[11], Exomiser[9], and Xrare[14]. These have been chosen as benchmarks due to their demonstrated utility and the availability of comparative data, enabling a rigorous evaluation against other emerging tools in the field. This selection aims to provide a representative cross-section of the current state-of-the-art in phenotype-guided variant prioritization, facilitating a comprehensive analysis of their respective capabilities and shortcomings. The HPO terms input into the tools are uniformly extracted from the patient’s phenotype descriptions using PhenoTagger[41]. Xrare is executed using the xrare37-pub:2021 docker image. exomiser(v13.3.0) is executed using its default algorithm hiPhive, while phenIX and PHIVE are executed using the phenix and phive algorithms from the exomiser suite, respectively.

Pre-trained LLM

We have chosen six diverse pre-trained Large Language Models (LLMs) for fine-tuning purposes, encompassing a spectrum of parameter sizes ranging from 0.5 billion to 70 billion. Specifically, the models selected are:

1. “Qwen1.5-0.5B”, “Qwen1.5-1.8B” and “Qwen1.5-4B”, which are pre-trained LLMs open-sourced by Alibaba[42]. These models are characterized by a relatively modest parameter size, offering a compact yet robust foundation for fine-tuning endeavors.

2. “Mistral-7B”, a large-scale model developed by the French startup Mistral AI. The model adopts advanced training techniques and a large amount of corpus data, providing powerful language processing capabilities and a wide range of applications. Whether it’s text generation, sentiment analysis, or question-answering systems, Mistral-7B can deliver outstanding performance.

3. “Meta-Llama-3-8B” and “Meta-Llama-3-70B,” representing the state-of-the-art in LLM technology and recently open-sourced by Meta[43]. These models are distinguished by their extensive parameterization and have demonstrated superior performance in various language tasks, positioning them as exemplary candidates for advanced fine-tuning applications.

The selection criteria were designed to ensure a diverse representation of LLMs across different parameter sizes and linguistic focuses, allowing for a nuanced investigation into the fine-tuning outcomes and potential optimizations across a wide array of applications.

Prompt Engineering

We designed a structured prompts for LLM fine-tunning and inferencing, followed the GPT guidelines[44, 45] to design our prompts. We wrote clear instructions by following the suggested tactics (e.g., ask the model to adopt a role, use delimiters to clearly indicate distinct parts of the input, specify the output format).

Supplementary Table S9 shows the prompts with which we experimented. Prompts P1 is zero-shot, P2 is few-shot, P3 constitutes a few-shot, chain-of-thought prompt instructing the LLM specifically on how to perform the variant identification. While P4 constitutes a one-shot, chain-of-thought prompt and RAG. In P1, we instruct the LLM to identify whether the variant is negative or positive. In P2, additional a few outputs example is provided. In P3, the LLM is prompted to identify whether the variant is negative or positive based on the chain of thinking. Lastly, P4, a few-shot, chain-of-thought prompt augmented by variant rich-annotated information.

To ascertain the optimal prompt for subsequent research and application endeavors, we conducted an evaluation of the performance of four distinct prompts using GPT-4, and identified P4 as the selected prompt.

Fine-tuning LLM

In the process of fine-tuning Large Language Models (LLMs), we have systematically combined datasets, pre-trained LLMs to train a cohort of models for subsequent evaluation and application. This ensemble includes 78 models designated for gradient testing, encompassing 6 distinct LLMs, 13 variations across data gradients. Additionally, we developed another model that combine different training sets to enhance dataset diversity.

The training of these models was executed on a cluster comprising 2 high-performance GPU servers (8 CUDA-capable GPUs, namely NVIDIA A100 Tensor Core GPU with 80 GB graphic memory), leveraging the Swift training framework to facilitate the computationally intensive tasks associated with LLM fine-tuning. The full-parameter fine-tuning mode is used, with “sft_type” set to “full”, and training is set for 5 epochs.

Evaluation

We implemented a training data size gradient analysis. This approach involves incrementally varying the amount of data used in the fine-tuning process to observe its impact on the performance. Specifically, the gradient analysis examines how the performance changes with different data sizes, offering insights into the robustness and scalability of the fine-tuned LLMs across a spectrum of data availability.

We conducted an assessment of the variant ranking performance utilizing the TopK cumulative distribution function (TopK CDF) as a primary metric. This statistical measure is widely recognized for its ability to consolidate the true positive rate and false positive rate across various threshold settings into a single value, thereby providing a comprehensive evaluation of the ranking efficacy.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

The studies involving human participants were reviewed and approved by the Ethical Clearance the Institutional Review Board of BGI. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.