Novel “SMARTer” Clinical Trial Design Improves Odds of Approval and Can Reduce Study Size By Over 80%: Modeling Use of a ctDNA “Optimizing Diagnostic” for Early Therapy Switching in Immuno-Oncology Trials

Introduction

The classic randomized clinical trial (RCT) has evolved since Johannes Fibiger, in1898¹, designed a trial to determine the best treatment for future patients with acute diphtheria by comparing standard of care (SOC) to SOC plus ani-serum. Today it would be critiqued as both quasi-randomized and unblinded². However, to this day the principal stands: a new therapy is compared to the current SOC. The results are used to seek both regulatory approval of a new drug and provide convincing data supporting its use in future patients. This paper will focus on a novel designs to seek approval and choose therapies appropriate for many chronic diseases and especially for cancer.

The gold standard RCT has evolved into the double-blind, randomized controlled trial (DBRCT). A series of DBRCTs superbly determine which therapy is better in the population in the trial. In the situation of acute disease, it identifies the therapy to be chosen. However, in first-tier healthcare nations, death is overwhelmingly due to chronic disease. In clinical practice patients typically are given multiple drugs in sequence, or in combination. Neither the classic RCT or DBPCT typically used for approval, addresses this real-world medical practice of multiple sequential therapies. Murphy, in 2005 proposed Sequential Multiple Assignment Randomized (SMAR) trials for behavior modification, but noted they were best used for hypothesis generation about sequential treatment and had “insurmountable” high dimensionality issues. In 2007 Collins³ proposed an eHealth “Sequential Multiple Assignment Randomized Trial” (SMART) to experimentally test what sequence of eHealth therapies are best. Cardio-vascular disease (CVD) is the most frequent cause of death in many countries and a series of sequential therapies to reduce risk factors and fatal events is SOC. In 2022 Lambert⁴ applied formal SMART design to evaluated web-based stress management in CVD. Current prevention practice typically mandates starting with one drug (often a statin), prompt biochemically testing of effectiveness to determine if it reduces cholesterol-based risk factors adequately, and adjusting therapy, which often includes higher doses and/or additional drugs to achieve the desired (biochemical) result.

Section A: Introduction of an Optimizing Diagnostic without Sequential Randomization

Section B: Sequential Randomization in Addition to an Optimizing Diagnostic

With a IO success rate of 30%, many patients will be transferred to SOC. This presents an opportunity for a subsequent randomization to answer the critical question of whether SOC alone, or SOC with simultaneous IO is better in this IO alone resistant subgroup. As there can be classic randomization of this subgroup, and to the degree side effects don’t prevent it, blinding, the question of which is better can be answered. (Because it is a select sub-group, who may have more resistant disease, the outcome in these subgroups cannot be compared to either SOC alone or IO alone.) However, it is trivial to extrapolate what would have happened if everyone the optimizing diagnostic suggested would be unlikely to do well on IO alone had transferred to the more optimum therapy (SOC alone or SOC plus IO); using this extrapolated result, this optimum path for those initially started on IO can be compared with the SOC.

The trial flow is shown in Figure 2, it adds to that shown in Figure 1 a second randomization (in red) for those patients likely to fail IO alone based on the optimizing diagnostic.

• Download figure
• Open in new tab

Figure 1

illustrates a clinical trial path from left to right as time progresses. It starts with a companion diagnostic “Tailoring test” followed by classic randomization to SOC vs. a new IO treatment first. Patients always remain in these originally assigned groups, the statistical plan compares these groups. On the day of therapy initiation, before therapy, a quantitative ctDNA determination is performed. (Baseline imaging tests historically precede treatment by significantly longer and thus may provide a less accurate baseline.) During the initial therapy the quantitative ctDNA is repeated, it is designated the Optimizing Diagnostic. For convenience, this may be just prior to each IO infusion, which are typically 3 weeks apart. The likelihood of a patient responding is reassessed. If optimizing diagnostic testing predicts likely IO failure, that person is transferred to SOC but remains in the “IO First” group. While the companion diagnostic test may have selected a group anticipated to have, for example a 50% chance of response, (and SOC was anticipated to have a 30% chance of response) the on therapy optimizing diagnostic test might indicate the person now has only a 0-10% chance of response. Thus, this person will likely do better (and deserves) SOC, as some benefit, maybe still 30%, from SOC is anticipated. Exact models and numbers are presented below. Each New IO therapy first patient, and this trial group as a whole, benefits from this increasingly personalized care. They may experience significant benefit from the new IO and from SOC, the sum is greater than either alone. The target is that those patients who will be cured or improved with IO will achieve that benefit, while the IO hyper-progressors and progressors will get the benefit of SOC. The below analysis models if, and how much, the sum of IO plus sequential SOC, is better than SOC.

• Download figure
• Open in new tab

Figure 2

shows these results graphically. Dependence on both sensitivity and specificity are illustrated. Not surprisingly higher levels of both increase the benefit of using an optimizing diagnostic test.

• Download figure
• Open in new tab

When the optimizing diagnostic test indicates likely failure of IO alone (in many trials the most likely outcome) that patient is randomized to either switching to SOC, which has benefits as illustrated in section 1 above, or adding SOC while continuing IO. This sequential randomization (the above examples show sequential treatment based on the optimizing test but not a second randomization) allows direct comparison of benefit and side effects of 1) SOC alone and 2) SOC plus IO, in a group resistant to IO alone. From the outcome the best of these 2 can be chosen and calculations comparing the benefit of the optimal path if the total group that started on IO were to have had the optimal path for them. The numbers in the above table are also applicable to this analysis. The exact nature of the second therapy does not change the mathematical analysis.

Discussion

In situations of acute fatal disease, the scientific question of which drug is “better” and the medical question of which drug is best to administer, are the same. In contrast, for nonfatal chronic illness, physicians often start with the least toxic (medically and/or financially) and only proceed to the “more expensive/bigger gun” alternatives as needed. In many cases a sequence of drugs or “path” is used to achieve the best balance of toxicity and benefit. Historically regulatory bodies have approved drugs based on comparison with the previous SOC and clinical practice has established the path.

Cancer maybe an acute disease if cured early, but sadly it is often a chronic disease leading to death. The medical goal is typically to begin with the therapy most likely to lead to cure. IO has unique ability to cure, but does so only in the minority of cases. Historically trials that compared the average benefit of IO versus SOC that determine which is “better” have been the basis of approval. However, they do not address the most critical medical question of whether a patient’s path should begin with IO, to deliver the best long-term outcome. This paper models the ability of “SMARTer” design to answer this question.

In some situations, IO may have a greater chance of curing, but SOC may on average create as many or more responses, creating a dilemma for patients, physicians and third-party payers. The modeling of SMARTer design presented herein, shows that even when IO has no greater benefit than SOC, if it is used first and patients transferred to SOC based on a rapid biochemical analysis of the effectiveness of the IO (an “optimizing test”), the total benefit may be better. We propose that this trail design would generate data suitable for the approval of that treatment path.

Approval of a treatment path is not novel. Use of the companion diagnostic has established the role of a test in defining a treatment path. Those positive by the test get one therapy, those negative by the test another. The “optimizing diagnostic” used herein is completely analogous to a companion diagnostic, however it is more accurate since it truly measures a patient’s initial response, while companion diagnostic attempts to predict if a patient will respond. Like a companion diagnostic, an optimizing diagnostic need not be 100% accurate to be useful. We modeled various sensitivities and specificities ranging from a “good” test with 95% sensitivity and specificity to “poor” 75% sensitivity and specificity. Tests measuring effectiveness are typically more accurate than tests attempting to predict response.

A “companion diagnostic” informs the choice or IO vs. SOC. An “optimizing diagnostic” informs the choice to remain on IO or transfer to SOC. In some trials intensification of IO based on a cfDNA optimizing test, or switching of one thymidine kinase inhibitor (TKI) to another TKI (based on detection of a mutation that causes resistance to the original)¹⁴ has been done. However, we are not aware of use of an optimizing diagnostic to change to SOC, within the experimental arm, or define a group for a second/sequential randomization to SOC alone or SOC with continuing IO.

A simultaneous combination of SOC & IO is often used for naïve patients as it is often better than either SOC or IO alone. However, direct comparison to sequential therapy is rarely available. Further, in the above example, the transfer of a significant sized group from IO (often for about 66% of the patients a test will determine continuing IO is not their best option) presents the opportunity or a sequential randomization to [SOC] vs. [SOC plus IO].

We first modeled a situation in which IO was better than standard of care showing a 40% response compared to a 30% response for SOC. Power analysis, for a standard IO vs SOC trial, using typical parameters, indicated that a trial of 710 people (355 each arm) would be needed to show that benefit. In contrast, assuming that the SOC was neither more nor less effective due to being delayed by the 3-6 week initial IO evaluation, a good optimizing test reduced the number of patients required to 114 (57 each arm).

In addition to requiring fewer patients and less time, more of these patients would receive the optimal therapy for them, either IO alone if that was sufficient, or IO and transfer to SOC, if that was best for the person. Thus, in addition to answering the key medical question of should a patient start with IO, the new design does so much more cost-effectively and meets a higher ethical standard. It facilitates trial completion in a timely manner both by cutting the number of patients required and enhancing accrual as patients are informed that if the new therapy is not working for them, they will be transferred to best SOC (or SOC will be added).

The presented modeling also shows that in the situation where IO and SOC are equal in a head-to-head trial. If initial IO is given with a near term optimization test with good performance, the path of IO followed by SOC would lead to a 63% improvement compared to SOC, assuming that IO neither potentiated nor inhibited subsequent SOC. Even when subsequent SOC was assumed to be 33% less effective, a good test and rapid switching, resulted in the IO first path having a 40% benefit over SOC, despite them being equal in a head-to-head trial.

The literature is unclear concerning how delaying SOC would alter its benefit, thus we modeled multiple scenarios without favoring one. In clinical practice therapy often does not begin until 3 to 6 weeks after diagnosis; while delay may be assumed to disadvantageous, the magnitude of the change is unknown. To be conservative we modeled a 30% decrease in effectiveness as well as no change in effectiveness of SOC. Use of an optimizing diagnostic and SMARTer design is beneficial in both cases. We also modeled an enhancement of SOC as there is a growing body of evidence that initial IO sets the stage for better response to SOC; even if the IO itself fails. However the literature on this is complex and contradictory concerning if IO potentiates SOC, Maeda, et al., in 2019¹⁵ reviewed the general ineffectiveness of chemo for metastatic melanoma, with under 10% response rate and 1-2% durable responses, as well as multiple studies showing IO therapy does not decrease, and appears to increase, response rate to subsequent chemo; they reported unexpected good results with chemo after IO even in especially difficult to treat melanoma types. Also, in metastatic melanoma not expected to respond to chemo, Goldinger¹⁶ reported temozolomide after IO had a 3% response rate, consistent with no potentiation, but taxanes strikingly had a 27% response rate. Sato et al.¹⁷, reported IO potentiated taxanes in head and neck cancer. Schvartsman et al¹⁸. in 2017 reported ORR was as higher after failed IO than before. Park et al., in 2019¹⁹ found that last line salvage chemo after IO was more effective than the previous chemo before IO in NSCLC, while the expectation was that being 2 lines later it would be very much less effective. Mangin, 2021²⁰ noted IO appeared to enhance response to chemotherapy and reviews some studies that did not show this, concluding that it may depend on the subsequent chemotherapy type consistent with Goldinger’s findings. Karachaliou et al., in 2021²¹, found chemo unexpected caused durable responses after failed immunotherapy. Puza²² found unexpected good results of surgery after IO. Saint-Jean²³ reported potentiation of chemo following IO. Hadas-Bengad et al. in 2020²⁴ reported IO potentiates chemo as did Markovic, et al.²⁵ Asasa, et al.²⁶, reported IO re-established sensitivity to cetuximab. While the literature is mixed and limited due to the retrospective nature of the studies, a body of studies suggests that in at least some cases IO potentiates subsequent chemo. Herein we propose prospective studies to evaluate the optimal path. While simultaneous combination has become standard for NSCLC, at least certain diseases with certain therapies, it may be less beneficial than the optimal sequential path. Kwong²⁷ in preclinical work documented that chemo given at the same time as IO inhibited it, thus, the exact timing of IO is likely dependent on the specific chemo, specific IO and specific disease. As with all oncology investigations, prospective multiple randomized trials, in this case of sequences, are needed and we propose should be included in trials designed for registration.

Theories as to why potentiation may occur include that while activating the immune system alone is insufficient for a response, that, in combination with subsequent killing (potentially with antigen release resulting in further immuno-stimulation) results in unexpectedly good response to subsequent SOC, in the setting of the IO altered immune system. Another possibility is IO may increase growth (PD or HPD) in a manner that makes cells more susceptible to SOC. The proposed sequential randomization design may provide additional data relevant to the question of whether IO increases the benefit of SOC more if given first or if continued with SOC, albeit only in the subgroup of patients that do not have a good biochemical response to IO alone.

The same methodology proposed herein can also be used to determine if treatment path(s) initiated with SOC can be optimized, as we presume they can. Jones, et. al. presented an observational study (historical, not randomized) in an ASCO poster 2021, that observed no decrease in effectiveness of SOC if it was given after IO, but observed a potentiation of IO by SOC. However, no change in overall survival was seen. Zengin et al., 2021²⁸ discusses, based on a failed study of simultaneous combination therapy, that sequential is potentially better, in this case chemo before immuno. Their discussion illustrates that even a statistically significant increase in PFS with combined therapy vs chemo alone was not enough to change clinical practice, emphasizing the importance of seeing larger group differences with methods such as those we propose. Interpretation of such analysis of historical, non-randomized data, is limited. The proposed sequential multiple randomized trial (SMART) procedure proposed herein can be modified to provide hard data as to the best sequence of therapies. “Reinforced Learning”²⁹, “Deep Reinforcement learning”³⁰, “Q-learning”³¹ and a “jackknife estimator”³² have been described as methods to optimize therapy and sequence.

While we used the example of IO, the proposed methodology applies to any new test.

Not surprisingly, models assuming potentiation of SOC by IO (from 30% to 40%) plus an optimizing diagnostic showed benefit over SOC, even when in a head-to-head trial of SOC vs IO, no difference would be seen. Surprisingly, the improvement over SOC was a dramatic - 86% with a good test. Even a poor test showed 47% improvement.

In addition to a design incorporating just use of an optimizing test, the use of sequential randomization to determine if SOC, or SOC plus IO, was more beneficial after switching, (in the subgroup with less likely to respond to IO) is also described herein. The result of the calculation of the benefit of the IO first group, assuming all patients received what was determined as the optimum second therapy, can be compared with SOC. Approval of the optimal path would be sought.

This study’s limitations include being a modeling study, that proposes a new type of trial, but does not present patient data.

Another limitation is that the performance of SOC in a subgroup that fails IO is not known, and may be outside of the situations modeled. (Our code or our collaboration is available for those wishing to model other scenarios.)

Another limitation is the performance of cfDNA testing, or other optimizing diagnostics, for progression is not established. Numerous reports correlate ctDNA levels with prognosis, more importantly increasing ctDNA almost invariably predicts progression following immunotherapy, for instance Davis, et al.³³, Bratman et al.³⁴, and Herbreteau et al.³⁵, each noted every patient with increasing cfDNA progressed. In fact, the latter and Wang et al.³⁶, noted cfDNA testing had an AUC of 1, perfectly predicting response and detecting it, on average, 79 days earlier. However, as cancer therapies typically succeed for a period of time and then stopped working, it is difficult to measure the accuracy of the cfDNA test for success of a therapy. It may be hundred percent accurate at showing that at a certain moment the therapy is working, but since the “gold standard” imaging is insensitive to changes at that time, and is typically only measured later, after the therapy may have stopped working, classic measures to evaluate test performance (a gold standard done at the same point in time) are not available. Thus, while we used a theoretical specificity, calculation of the actual specificity for progression is problematic. We modeled sensitivities of 95% and down to 70% and this may be highly conservative given the above references, our modeling of specificities may also be conservative and difficult to validate. However, different types of cfDNA testing may have different performance characteristics, the models apply to any optimizing diagnostic test.

In implementing a SMARTer design, a decision concerning the cut off for switching to the SOC would need to be made. In order to give the IO the greatest chance of working, likely the predefined formulae for transfer would be if there is evidence of increase in cancer; biochemically stable disease would remain on the IO. However, other cutoff points may be chosen; for example all patients not showing biochemical improvement might be transferred.

At the time of writing SOC does not include an optimizing test; SOC change based on biochemistry, rather than on imaging may evolve. If so, this should be included in the SOC arm. This trial design is only appropriate for drugs that have a different MOA. The above discussion focuses on use of cfDNA as a sample optimizing test. It is equally applicable to any method that can sensitively monitor cancer amount, examples include cancer specific exosomes, or advanced imaging with tracers. The modeling and discussion herein is based on the endpoint of “events”; survival to a certain timepoint may be considered an event, however survival analysis modeling was not performed herein.

In conclusion, clinical trials that include rapid evaluation of the new therapy by biochemical methods, and transferring to SOC if biochemical evidence suggests the new therapy is failing (an optimizing test), allow a new path to potential approval of new therapies. This path is analogous to a companion diagnostic, both specify whether the proposed new therapy should be used. Companion diagnostics specify if it should be initiated, optimizing test/diagnostics specify if it should be continued. New therapies that would fail in a head-to-head comparison with the previous SOC may be documented to enhance the patient path and provide a better medical outcome, potentially leading to approval with this new trial design. In one likely scenario where the new therapy is 30% better than the SOC, a head-to-head study would require over five times as many patients as one including rapid evaluation and therapy switching based on an optimizing diagnostic. A second randomization is proposed to determine if the path should be a transfer to SOC or to SOC plus continuing the new therapy. Use of such designs decreases the cost of clinical trials, is more ethical, likely improves accrual, and better models actual clinical use.