Novel “SMARTer” Clinical Trial Design Improves Odds of Approval and Can Reduce Study Size By 80%: Modeling Use of a ctDNA “Optimizing Diagnostic” for Early Therapy Switching in Immuno-Oncology Trials
===============================================================================================================================================================================================================

* Floyd Elliot Taub
* Dexiang Gao

## Abstract

Novel pivotal trial designs, that more clearly demonstrate increased benefit over Standard of Care (SOC), especially in oncology and immuno-oncology (IO), are presented. The benefit of therapy is maximized and sample size is dramatically reduced. The novel methodology includes the introduction of a biochemical “Optimizing Diagnostic”, for example a cfDNA test that can detect poor response, when performed early after therapy is begun; this is used to change the therapy of the tested person early in the trial (typically to SOC), before clinical progression. Patients remain in “the new drug first” group, which is compared to SOC. An “Optimizing Diagnostic” is analogous to a “Companion Diagnostic”; both potentially allow approval of drugs that would otherwise fail. A companion diagnostic predicts benefit prior to therapy, the optimizing diagnostic (more accurately) predicts likelihood of benefit after initial therapy. Those patients deemed less likely to respond remain in the novel drug first arm, but are switched to SOC. A Sequential Multiple Assignment Randomized Trial” (SMART) design is proposed to evaluate if switching to SOC or SOC plus continuing the novel therapy is most beneficial. These designs will allow approval of therapy paths including novel agents when the novel agent could not be approved without this design. A good optimizing test may reduce the number of patients needed by 80%, dramatically reducing cost and time; more patients benefit and accrual is easier.

**Key Points**

*   A new clinical trial design focused on testing a path that begins with a novel regimen (IO is featured) is presented.

*   The path includes an “optimizing diagnostic” that determines, early during treatment, if a patient should remain on the new regimen.

*   Companion diagnostics define the path at a pre-treatment stage, optimizing diagnostics define the path early during treatment. Changes in therapy, typically to the SOC, is made based on the post-test probability of success.

*   The novel path is significantly more likely to lead to approval than the novel regimen alone.

*   Use of the novel method can reduce the size of a trial by 80%, and allow approval of the path, when approval of the novel regimen, based on a head-to-head trial vs SOC, would not be possible.

## Introduction

The classic randomized clinical trial (RCT) has evolved since Johannes Fibiger, in 18981, designed a trial to determine the best treatment for future patients with acute diphtheria by comparing standard of care (SOC) to SOC plus anti-serum. Today it would be critiqued as both quasi-randomized and unblinded2. However, to this day the principal stands: a new therapy is compared to the current SOC. The results are used to both seek regulatory approval of a new drug and provide convincing data supporting its use in future patients. This paper will focus on a novel designs to seek approval and chose therapies appropriate for many chronic diseases and especially for cancer.

The gold standard RCT has evolved into the double-blind, randomized controlled trial (DBRCT). A series of DBRCTs superbly determine which therapy is better in the population in the trial. In the situation of acute disease, it identifies the therapy to be chosen. However, in first-tier healthcare nations, death is overwhelmingly due to chronic disease. In clinical practice patients typically are given multiple drugs in sequence, or in combination. Neither the classic RCT or DBRCT typically used for approval, addresses this real-world medical practice of multiple sequential therapies. Murphy, in 2005 proposed Sequential Multiple Assignment

Randomized (SMAR) trials for behavior modification, but noted they were best used for hypothesis generation about sequential treatment and had “insurmountable” high dimensionality issues. In 2007 Collins3 proposed an eHealth “Sequential Multiple Assignment Randomized Trial” (SMART) to experimentally test what sequence of eHealth therapies are best. Cardio-vascular disease (CVD) is the most frequent cause of death in many countries and a series of sequential therapies to reduce risk factors and fatal events is SOC. In 2022 Lambert4 applied formal SMART design to evaluate web-based stress management in CVD. Current prevention practice typically mandates starting with one drug (often a statin), prompt *biochemically testing* of effectiveness to determine if it reduces cholesterol-based risk factors adequately, and adjusting therapy, which often includes higher doses and/or additional drugs to achieve the desired (biochemical) result.

**Cancer is the second leading cause of death; again, this is often a chronic disease. While some patients are cured initially, almost all of the 600**,**000 US patients who die of cancer have had sequential therapy**. As in CVD, clinical decision trees have evolved for cancer, but strikingly, except for testing no continuing therapy vs maintenance therapy5,6, the first discussion of a formal SMART design for oncology was in 20187 when Kidwell, et al., proposed an immuno-oncology (IO) trial that, consistent with Murphy’s original concern of high dimensionality, resulted in 7 to 10 comparisons. This had limited practicality and, as the authors noted, led to statistical complexity; it has not been executed.

Herein we propose the first practical ***SMART trial design to enhance registration* (SMARTer)** of a novel therapy path. It is the first design to use a biochemical “Optimizing Diagnostic”, measured before progressive disease, to revise therapy to include SOC. In Section A below, the design specifies changing therapy to SOC; in Section B, it specifies a second randomization to either SOC or SOC plus new drug(s). In all cases patients first treated with the new drug remain within the experimental arm for statistical analysis. Both these designs result in capturing within the experimental group some benefit of the new therapy and the SOC.

**The combined, for some patients sequential, benefit is compared to SOC**. Modeling shows this dramatically reduces the patients needed to prove benefit of a new therapy. Further it will identify benefit of a therapy path including the new therapy, when benefit of the new drug without this novel therapy path, is not present. Thus, it decreases the investment (patients, time and dollars) to approval for some therapies and can detect approvable response when none would be detected with standard trial designs.

## Section A: Introduction of an Optimizing Diagnostic without Sequential Randomization

### Trial Design Description

IO drugs have revolutionized oncology as they may cure even late-stage patients and are used as an example herein. While indications and situations vary greatly, they often give durable responses in only 10% of the population, even when appropriate patients are preselected with a predictive companion diagnostic. Long-term benefit varies from 0 to 100% depending on the type of cancer and clinical situation; in many situations about ∼20-50% show some benefit. Unfortunately, again depending on the situation, about, 15-20% may have more than doubling in the growth rate (Hyperprogressive Disease or HPD) of their cancer8, 9, 10, both Russo et al11, and Yildirim et al.12 report NSCLC patients have ∼30% incidence of HPD.

DNA methods, including cfDNA, have become SOC for detecting actionable mutations. They can also be used soon after therapy is started to monitor drug effectiveness. Herein we propose using such information to improve clinical trial as well as patient outcomes. cfDNA testing soon after therapy begins can allow much greater accuracy in determining if a patient will respond. While currently expensive and time consuming, there are many reports that increasing circulating tumor DNA, ctDNA, accurately predicts if therapy is doomed to fail. While cfDNA is not itself an approvable endpoint, the FDA allows, even encourages13, ctDNA levels being used in a clinical trial.

Figure 1 illustrates a clinical trial path from left to right as time progresses. It starts with a companion diagnostic “Tailoring test” followed by classic randomization to SOC vs. a new IO treatment first. Patients always remain in these originally assigned groups; the statistical plan compares these groups. On the day of therapy initiation, before therapy, a quantitative ctDNA determination is performed. (Baseline imaging tests historically precede treatment by significantly longer and thus may provide a less accurate baseline.) During the initial therapy the quantitative ctDNA is repeated; it is designated the Optimizing Diagnostic. For convenience, this may be just prior to each IO infusion, which are typically 3 weeks apart. The likelihood of a patient responding is reassessed. If optimizing diagnostic testing predicts likely IO failure, that person is transferred to SOC but remains in the “IO First” group. While the companion diagnostic test may have selected a group anticipated to have, for example a 50% chance of response, (and SOC was anticipated to have a 30% chance of response) the on therapy optimizing diagnostic test might indicate the person now has only a 0-10% chance of response. Thus, this person will likely do better (and deserves) SOC, as some benefit, maybe still 30%, from SOC is anticipated. Exact models and numbers are presented below. Each New IO therapy first patient, and this trial group as a whole, benefits from this increasingly personalized care. They may experience significant benefit from the new IO and from SOC; the sum is greater than either alone. The target is that those patients who will be cured or improved with IO will achieve that benefit, while the IO hyper-progressors and progressors will get the benefit of SOC. The below analysis models if, and how much, the sum of IO plus sequential SOC, is better than SOC.

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/06/29/2023.02.20.23286152/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/F1)

Figure 1. 
Introduction of an Optimizing Diagnostic

The benefit depends on the accuracy of the optimizing diagnostic test. Various levels of accuracy of the optimizing test are modeled in the examples below. **The sensitivity is defined as the proportion of patients who test positive for likely to progress, among the true disease progressive subjects; the specificity is defined as the proportion of test negative subjects among the true non-progressive subjects**. Various levels of sensitivity and specificity are presented in the below examples.

**Modeling Scenario 1: Benefit resulting from early treatment switching from IO to SOC for subjects found to be at high risk of progressing on IO based on an early optimizing diagnostic**

Within this scenario #1, we consider a situation in which IO is somewhat better than SOC as a single agents, two assumptions regarding potential IO effect on later SOC: in 1.1) the effectiveness of subsequent SOC is presumed to be the same as if the SOC was given initially; in 1.2) SOC effectiveness was presumed to decrease if given after IO (potentially due to delayed treatment). In scenario 2 we consider both the above scenarios but assume that as single agents the IO and SOC have the same performance. Based on literature reviewed in the discussion, there is a possibility that IO sensitizes to subsequent SOC, this is modeled in two scenarios: 3.1 models a situation when the IO is better than SOC as single agents, 3.2 models a situation in which they are equal as single agents.

In these examples it is assumed that the mechanism of action of the two drugs is distinct and cross resistance is not/or minimally present. This criterion is met for comparing IO to SOC chemotherapy or growth pathway targeted therapy.

One likely scenario is that the IO is more effective than SOC, however the difference is small; this is assumed in Scenario 1. Table 1.1 examines a scenario in which SOC is assumed to have an effectiveness of 30% (regardless of when administered), while IO is assumed to be 40% effective.

View this table:
[Table 1.1.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T1)

Table 1.1. 
Benefit of switching early after using “Optimizing Diagnostic” test in the IO First arm with the scenario IO response rate of 40% and SOC response rate of 30%; SOC effectiveness remains the same after IO.

The top row indicates that using sequential treatment based on an optimizing diagnostic test with a 95% sensitivity and specificity (95/95), the improvement in patient outcome an [IO first, test, continue treatment with IO or SOC as indicated] scenario compared to an [IO only] scenario would be 55.7%. Simple testing and using the result to decide if IO should be continued or if SOC is better for that patient improves the outcome in the experimental arm 55.7%. This would be a striking 85.67% increase over SOC, this contrasts with the classic head-to-head comparison (no optimizing test or switching) in which the benefit would be only 33%, a 2.6x improvement. Even a poor optimizing diagnostic test with sensitivity and specificity of only 70% (70/70) results in an “IO First” arm response rate of 44.2% and results in a 47.3% improvement over SOC.

Figure 2 shows these results graphically. Dependence on both sensitivity and specificity are illustrated. Not surprisingly, higher levels of both increase the benefit of using an optimizing diagnostic test.

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/06/29/2023.02.20.23286152/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/F2)

Figure 2. 

One implication of the improvement in outcome with a good optimizing diagnostic and sequential therapy is that fewer patients are required to see a statistically significant benefit. Table 1.2 is a power calculation for the above scenario.

View this table:
[Table 1.2.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T2)

Table 1.2. 
Benefit of switching early after using “Optimizing Diagnostic” test in the IO First arm with the scenario IO response rate of 40%, SOC response rate of 30%, and SOC response rate 20% if given later.

**Without an optimizing diagnostic and rapid switching, 355 patients** in each arm (710 total) are needed to test the 10% points difference between SOC and the original IO (no switching) with a 80% power using a two-sided test.

View this table:
[Table 1.1a](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T3)

Table 1.1a 
Sample Size calculation for case 1.1, with 80% power, 2-sided test with alpha of 0.05, assuming a 10% difference in benefit.

**Strikingly less than 1/5 the number of patients is required; the sample size required falls from 355 to 57 in each arm, (710 vs. 114 total, 84% reduction) using a good optimizing test (95/95)**. Even with a poor test (70/70) the number of patients needed is reduced from 355 in each arm to 180.

We next examined a scenario (designated **Scenario 1.2)** in which it was assumed that **SOC had a 33% decrease in effectiveness if delayed** (30% if given first, 20% if given later), to see if this negated the benefit of testing and sequential therapy in the “IO First” treatment arm.

While, as expected, the benefit of an optimizing diagnostic and rapid switching was lower than in scenario 1.1, with a test having a sensitivity and specificity of 95% there was still a 49.8% response rate in the sequential therapy group which is a surprisingly large 66% benefit over SOC. If the test was poor, (70/80 or less) and SOC lost effectiveness if delayed, there was still a benefit over SOC, but it was not better than IO alone in this scenario where IO alone was 30% better than SOC.

The sample size (SS) calculation for this example 1.2 is given below. A sample size of 355 in each arm, for the standard no optimizing test design, is needed; in contrast, only 95 patients per arm would be needed with a good optimizing test. This is a 73% savings.

View this table:
[Table 1.2a.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T4)

Table 1.2a. 
Sample size calculation for case 1.2, with 80% power, 2-sided test with alpha of 0.05, assuming a 10% difference in benefit.

**We next asked if this method could result in approval of a sequence starting with the new IO, even if the IO alone was no more effective than SOC alone**. Example 2.1 models a situation in which both the IO and SOC have a 30% effectiveness. Under this situation, without this trial design, approval would be unlikely without a major difference in side effects, convenience, etc. The first assumption in this section (2.1) is that SOC effectiveness does not change due to the delay in administering it.

View this table:
[Table 2.1.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T5)

Table 2.1. 
Benefit of switching early after using “Optimizing Diagnostic” test in the IO First arm with the scenario IO and SOC equal as single agents (30%) and the SOC effectiveness remains the same (30%) if given later, after IO.

**The top row indicates that an optimizing diagnostic with a 95% sensitivity and specificity would improve patient outcome to 48.9% in the experimental group (IO first) and thus this sequential treatment would have a 63% improvement of SOC**. This level of improvement is likely significant for patient care and approval. Even a poor test with sensitivity and specificity of only 70% improves new arm outcome by 43.7% and results in a 45.5% improvement over SOC, suggesting approval of this sequential path.

We next asked if this remarkable approval of a sequential path, when the first drug alone could not be approved, was still true if the effectiveness of SOC decreased if it was delayed. Table 2.2 models this.

View this table:
[Table 2.2.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T6)

Table 2.2. 
Benefit of switching early after using “Optimizing Diagnostic” test in the IO First arm with the scenario IO and SOC equal as single agents and the SOC effectiveness decreases after IO. SOC=IO=30%, SOC=20% if given later.

While the benefit of switching was less, with a test having a sensitivity and specificity of 95% there was still a 42.1% response rate in the sequential therapy group and a 40% benefit over SOC. While these numbers would significantly improve patient care and chance of approval, if the test was poor, 70% sensitivity and specificity, the benefit over SOC fell to only 8.67%. Thus, even if the new drug alone is un-approvable alone, as it has not benefit over SOC, with a good, or even moderately good optimizing diagnostic test, sequential therapy based on an optimizing diagnostic shows apparently approvable benefit over SOC.

While intuition suggests a delay in SOC would result in some decrease in benefit, as reviewed in the discussion section, a growing body of evidence suggests IO potentiates SOC in certain situations, even when the IO alone is not effective, thus we modeled this possibility. Below (Table 3) we model the benefit of sequential therapy if IO sensitizes to subsequent SOC.

View this table:
[Table 3.1.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T7)

Table 3.1. 
Benefit of switching early after using “Optimizing Diagnostic” test in the IO First arm with the scenario IO is better than SOC and sensitizes to SOC. SOC=30%, IO=40%, SOC=40% if given later.

Strikingly potentiation of SOC from 30% to 40% effective, in the setting of IO being 40% effective while SOC is 30% effective, results in 105% benefit over SOC with a good optimizing diagnostic test (95/95).

Lastly, we asked if an IO was no better than SOC but potentiated subsequent SOC from 30% to 40% would sequential therapy yield improvement significant for clinical use and potentially pproval.

View this table:
[Table 3.2.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/T8)

Table 3.2. 
Benefit of switching early after using “Optimizing Diagnostic” test in the IO First arm with the scenario IO is equal to SOC and sensitizes to SOC. SOC=IO=30%, SOC=40% if given later.

In this situation of IO being no better than SOC alone, but showing a potentiation of SOC from 30% to 40% effective, a 56% response and 86% improvement over SOC is seen with sequential therapy. Thus, we find another striking example of an un-approvable drug being potentially approvable if used to initiate a sequence and followed with an optimizing diagnostic.

## Section B: Sequential Randomization in Addition to an Optimizing Diagnostic

With a IO success rate of 30%, many patients will be transferred to SOC. This presents an opportunity for a subsequent randomization to answer the critical question of whether SOC alone, or SOC with simultaneous IO is better in this IO alone resistant subgroup. As there can be classic randomization of this subgroup, and to the degree side effects don’t prevent it, blinding, the question of which is better can be answered. (Because it is a select sub-group, who may have more resistant disease, the outcome in these subgroups cannot be compared to either SOC alone or IO alone.) However, it is trivial to extrapolate what would have happened if everyone the optimizing diagnostic suggested would be unlikely to do well on IO alone had transferred to the more optimum therapy (SOC alone or SOC plus IO); using this extrapolated result, this optimum path for those initially started on IO can be compared with the SOC.

The trial flow is shown in Figure 3; it adds to that shown in Figure 1 a second randomization (in red) for those patients likely to fail IO alone based on the optimizing diagnostic.

![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/06/29/2023.02.20.23286152/F3.medium.gif)

[Figure 3.](http://medrxiv.org/content/early/2023/06/29/2023.02.20.23286152/F3)

Figure 3. 
Optimizing diagnostic with sequential randomization

When the optimizing diagnostic test indicates likely failure of IO alone (in many trials the most likely outcome) that patient is randomized to either switching to SOC, which has benefits as illustrated in section 1 above, or adding SOC while continuing IO. This sequential randomization (the above examples show sequential treatment based on the optimizing test but not a second randomization) allows direct comparison of benefit and side effects of 1) SOC alone and 2) SOC plus IO, in a group resistant to IO alone. From the outcome the best of these two can be chosen and calculations comparing the benefit of the optimal path if the total group that started on IO were to have had the optimal path for them. The numbers in the above table are also applicable to this analysis. The exact nature of the second therapy does not change the mathematical analysis.

## Discussion

In situations of acute fatal disease, the scientific question of which drug is “better” and the medical question of which drug is best to administer, are the same. In contrast, for nonfatal chronic illness, physicians often start with the least toxic (medically and/or financially) and only proceed to the “more expensive/bigger gun” alternatives as needed. In many cases a sequence of drugs or “path” is used to achieve the best balance of toxicity and benefit. Historically, regulatory bodies have approved drugs based on comparison with the previous SOC and clinical practice has established the path.

Cancer may be an acute disease if cured early, but sadly it is often a chronic disease leading to death. The medical goal is typically to begin with the therapy most likely to lead to cure. IO has unique ability to cure, but does so only in the minority of cases. Historically, trials that compared the average benefit of IO versus SOC that determine which is “better” have been the basis of approval. However, they do not address the most critical medical question of whether a patient’s path should begin with IO to deliver the best long-term outcome. This paper models the ability of “SMARTer” design to answer this question.

In some situations, IO may have a greater chance of curing, but SOC may on average create as many or more responses, creating a dilemma for patients, physicians and third-party payers. The modeling of SMARTer design presented herein, shows that even when IO has no greater benefit than SOC, if it is used first and patients transferred to SOC based on a rapid biochemical analysis of the effectiveness of the IO (an “optimizing test”), the total benefit may be better. We propose that this trial design would generate data suitable for the approval of that treatment path.

Approval of a treatment path is not novel. Use of the companion diagnostic has established the role of a test in defining a treatment path. Those positive by the test get one therapy, those negative by the test another. The “optimizing diagnostic” used herein is completely analogous to a companion diagnostic, however it is more accurate since it truly measures a patient’s initial response, while companion diagnostic attempts to predict if a patient will respond. Like a companion diagnostic, an optimizing diagnostic need not be 100% accurate to be useful. We modeled various sensitivities and specificities ranging from a “good” test with 95% sensitivity and specificity to “poor” 75% sensitivity and specificity. Tests measuring effectiveness are typically more accurate than tests attempting to predict response.

A “companion diagnostic” informs the choice of IO vs. SOC. An “optimizing diagnostic” informs the choice to remain on IO or transfer to SOC. In some trials intensification of IO based on a cfDNA optimizing test, or switching of one thymidine kinase inhibitor (TKI) to another TKI (based on detection of a mutation that causes resistance to the original)14 has been done. However, we are not aware of use of an optimizing diagnostic to change to SOC, within the experimental arm, or define a group for a second/sequential randomization to SOC alone or SOC with continuing IO.

A simultaneous combination of SOC & IO is often used for naïve patients as it is often better than either SOC or IO alone. However, direct comparison to sequential therapy is rarely available. Further, in the above example, the transfer of a significant sized group from IO (often for about 66% of the patients a test will determine continuing IO is not their best option) presents the opportunity or a sequential randomization to [SOC] vs. [SOC plus IO].

We first modeled a situation in which IO was better than standard of care showing a 40% response compared to a 30% response for SOC. Power analysis, for a standard IO vs SOC trial, using typical parameters, indicated that a trial of 710 people (355 each arm) would be needed to show that benefit. In contrast, assuming that the SOC was neither more nor less effective due to being delayed by the 3-6 week initial IO evaluation, a good optimizing test reduced the number of patients required to 114 (57 each arm).

In addition to requiring fewer patients and less time, more of these patients would receive the optimal therapy for them, either IO alone if that was sufficient, or IO and transfer to SOC, if that was best for the person. Thus, in addition to answering the key medical question of should a patient start with IO, the new design does so much more cost-effectively and meets a higher ethical standard. It facilitates trial completion in a timely manner both by cutting the number of patients required and enhancing accrual as patients are informed that if the new therapy is not working for them, they will be transferred to best SOC (or SOC will be added).

The presented modeling also shows that in the situation where IO and SOC are equal in a head-to-head trial. If initial IO is given with a near term optimization test with good performance, the path of IO followed by SOC would lead to a 63% improvement compared to SOC, assuming that IO neither potentiated nor inhibited subsequent SOC. Even when subsequent SOC was assumed to be 33% less effective, a good test and rapid switching resulted in the IO first path having a 40% benefit over SOC, despite them being equal in a head-to-head trial.

The literature is unclear concerning how delaying SOC would alter its benefit, thus we modeled multiple scenarios without favoring one. In clinical practice therapy often does not begin until 3 to 6 weeks after diagnosis; while delay may be assumed to disadvantageous, the magnitude of the change is unknown. To be conservative we modeled a 30% decrease in effectiveness as well as no change in effectiveness of SOC. Use of an optimizing diagnostic and SMARTer design is beneficial in both cases. We also modeled an enhancement of SOC as there is a growing body of evidence that initial IO sets the stage for better response to SOC, even if the IO itself fails. However the literature on this is complex and contradictory concerning if IO potentiates SOC. Maeda, et al., in 201915 reviewed the general ineffectiveness of chemo for metastatic melanoma, with under 10% response rate and 1-2% durable responses, as well as multiple studies showing IO therapy does not decrease, and appears to increase, response rate to subsequent chemo; they reported unexpected good results with chemo after IO even in especially difficult to treat melanoma types. Also, in metastatic melanoma not expected to respond to chemo, Goldinger16 reported temozolomide after IO had a 3% response rate, consistent with no potentiation, but taxanes strikingly had a 27% response rate. Sato et al.17 reported IO potentiated taxanes in head and neck cancer. Schvartsman et al.18 in 2017 reported ORR was higher after failed IO than before. Park et al., in 201919 found that last line salvage chemo after IO was more effective than the previous chemo before IO in NSCLC, while the expectation was that being 2 lines later it would be very much less effective. Mangin, in 202120 noted IO appeared to enhance response to chemotherapy and reviews some studies that did not show this, concluding that it may depend on the subsequent chemotherapy type, consistent with Goldinger’s findings. Karachaliou et al., in 202121, found chemo unexpectedly caused durable responses after failed immunotherapy. Puza22 found unexpected good results of surgery after IO. Saint-Jean23 reported potentiation of chemo following IO. Hadas-Bengad et al. in 202024 reported IO potentiates chemo as did Markovic, et al.25 Asasa, et al.26, reported IO re-established sensitivity to cetuximab. While the literature is mixed and limited due to the retrospective nature of the studies, a body of studies suggests that in at least some cases IO potentiates subsequent chemo. Herein we propose prospective studies to evaluate the optimal path. While simultaneous combination has become standard for NSCLC, at least certain diseases with certain therapies, it may be less beneficial than the optimal sequential path. Kwong27 in preclinical work documented that chemo given at the same time as IO inhibited it, thus, the exact timing of IO is likely dependent on the specific chemo, specific IO, and specific disease. As with all oncology investigations, prospective multiple randomized trials, in this case of sequences, are needed and we propose should be included in trials designed for registration.

Theories as to why potentiation may occur include that while activating the immune system alone is insufficient for a response, that, in combination with subsequent killing (potentially with antigen release resulting in further immuno-stimulation) results in unexpectedly good response to subsequent SOC, in the setting of the IO altered immune system. Another possibility is IO may increase growth (PD or HPD) in a manner that makes cells more susceptible to SOC. The proposed sequential randomization design may provide additional data relevant to the question of whether IO increases the benefit of SOC more if given first or if continued with SOC, albeit only in the subgroup of patients that do not have a good biochemical response to IO alone.

The same methodology proposed herein can also be used to determine if treatment path(s) initiated with SOC can be optimized, as we presume they can. Jones, et al. presented an observational study (historical, not randomized) in an ASCO poster 2021, that observed no decrease in effectiveness of SOC if it was given after IO, but observed a potentiation of IO by SOC. However, no change in overall survival was seen. Zengin et al., 202128 discusses, based on a failed study of simultaneous combination therapy, that sequential is potentially better, in this case chemo before immuno. Their discussion illustrates that even a statistically significant increase in PFS with combined therapy vs chemo alone was not enough to change clinical practice, emphasizing the importance of seeing larger group differences with methods such as those we propose. Interpretation of such analysis of historical, non-randomized data, is limited. The proposed sequential multiple randomized trial (SMART) procedure proposed herein can be modified to provide hard data as to the best sequence of therapies. “Reinforced Learning”29, “Deep Reinforcement learning”30, “Q-learning”31 and a “jackknife estimator”32 have been described as methods to optimize therapy and sequence.

While we used the example of IO, the proposed methodology applies to any new test.

Not surprisingly, models assuming potentiation of SOC by IO (from 30% to 40%) plus an optimizing diagnostic showed benefit over SOC, even when in a head-to-head trial of SOC vs IO, no difference would be seen. Surprisingly, the improvement over SOC was a dramatic - 86% with a good test. Even a poor test showed 47% improvement.

In addition to a design incorporating just use of an optimizing test, the use of sequential randomization to determine if SOC, or SOC plus IO, was more beneficial after switching, (in the subgroup with less likely to respond to IO) is also described herein. The result of the calculation of the benefit of the IO first group, assuming all patients received what was determined as the optimum second therapy, can be compared with SOC. Approval of the optimal path would be sought.

This study’s limitations include being a modeling study, that proposes a new type of trial, but does not present patient data.

Another limitation is that the performance of SOC in a subgroup that fails IO is not known, and may be outside of the situations modeled. (Our code or our collaboration is available for those wishing to model other scenarios.)

Another limitation is the performance of cfDNA testing, or other optimizing diagnostics, for progression is not established. Numerous reports correlate ctDNA levels with prognosis; more importantly, increasing ctDNA almost invariably predicts progression following immunotherapy, for instance Davis, et al.33, Bratman et al.34, and Herbreteau et al.35, each noted every patient with increasing cfDNA progressed. In fact, the latter and Wang et al.36, noted cfDNA testing had an AUC of 1, perfectly predicting response and detecting it, on average, 79 days earlier. However, as cancer therapies typically succeed for a period of time and then stop working, it is difficult to measure the accuracy of the cfDNA test for success of a therapy. It may be hundred percent accurate at showing that at a certain moment the therapy is working, but since the “gold standard” imaging is insensitive to changes at that time, and is typically only measured later, after the therapy may have stopped working, classic measures to evaluate test performance (a gold standard done at the same point in time) are not available. Thus, while we used a theoretical specificity, calculation of the actual specificity for progression is problematic. We modeled sensitivities of 95% and down to 70% and this may be highly conservative given the above references; our modeling of specificities may also be conservative and difficult to validate. However, different types of cfDNA testing may have different performance characteristics, the models apply to any optimizing diagnostic test.

In implementing a SMARTer design, a decision concerning the cut off for switching to the SOC would need to be made. In order to give the IO the greatest chance of working, likely the predefined formulae for transfer would be if there is evidence of increase in cancer; biochemically stable disease would remain on the IO. However, other cutoff points may be chosen; for example all patients not showing biochemical improvement might be transferred.

At the time of writing SOC does not include an optimizing test; SOC change based on biochemistry, rather than on imaging may evolve. If so, this should be included in the SOC arm. This trial design is only appropriate for drugs that have a different MOA. The above discussion focuses on use of cfDNA as a sample optimizing test. It is equally applicable to any method that can sensitively monitor cancer amount; examples include cancer specific exosomes, or advanced imaging with tracers. The modeling and discussion herein is based on the endpoint of “events”; survival to a certain timepoint may be considered an event, however survival analysis modeling was not performed herein.

In conclusion, clinical trials that include rapid evaluation of the new therapy by biochemical methods, and transferring to SOC if biochemical evidence suggests the new therapy is failing (an optimizing test), allow a new path to potential approval of new therapies. This path is analogous to a companion diagnostic, both specify whether the proposed new therapy should be used. Companion diagnostics specify if it should be initiated, optimizing test/diagnostics specify if it should be continued. New therapies that would fail in a head-to-head comparison with the previous SOC may be documented to enhance the patient path and provide a better medical outcome, potentially leading to approval with this new trial design. In one likely scenario where the new therapy is 30% better than the SOC, a head-to-head study would require over five times as many patients as one including rapid evaluation and therapy switching based on an optimizing diagnostic. A second randomization is proposed to determine if the path should be a transfer to SOC or to SOC plus continuing the new therapy. Use of such designs decreases the cost of clinical trials, is more ethical, likely improves accrual, and better models actual clinical use.

## Data Availability

All data produced in the present study are available upon reasonable request to the authors

## Footnotes

*   The clarity is improved; the concepts are unchanged.

*   Received February 20, 2023.
*   Revision received June 28, 2023.
*   Accepted June 29, 2023.


*   © 2023, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## References

1.  Fibiger J. Om Serumbehandling af Difteri. Hospitalstidende. 1898;6:309–325., 337-50.
    
    

2.  Hróbjartsson A, Gøtzsche PC, Gluud C. The controlled clinical trial turns 100 years: Fibiger’s trial of serum treatment of diphtheria. BMJ. 1998;317(7167):1243–1245. doi:10.1136/bmj.317.7167.1243
    
    [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMTcvNzE2Ny8xMjQzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjMvMDYvMjkvMjAyMy4wMi4yMC4yMzI4NjE1Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.  Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med. 2007 May;32(5 Suppl):S112–8. doi: 10.1016/j.amepre.2007.01.022. PMID: 17466815; PMCID: PMC2062525
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.amepre.2007.01.022&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17466815&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000246527200007&link_type=ISI) 

4.  Lambert SD, Grover S, Laizner AM, McCusker J, Belzile E, Moodie EEM, Kayser JW, Lowensteyn I, Vallis M, Walker M, Da Costa D, Pilote L, Ibberson C, Sabetti J, de Raad M. Adaptive web-based stress management programs among adults with a cardiovascular disease: A pilot Sequential Multiple Assignment Randomized Trial (SMART). Patient Educ Couns. 2022 Jun;105(6):1587–1597. doi: 10.1016/j.pec.2021.10.020. Epub 2021 Oct 23. PMID: 34753612.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.pec.2021.10.020&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34753612&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

5.  Tummarello D, Mari D, Graziano F, Isidori P, Cetto G, Pasini F, Santo A, Cellerino R. A randomized, controlled phase III study of cyclophosphamide, doxorubicin, and vincristine with etoposide (CAV-E) or teniposide (CAV-T), followed by recombinant interferonalpha maintenance therapy or observation, in small cell lung carcinoma patients with complete responses. Cancer. 1997 Dec 15;80(12):2222–9. PMID: 9404698.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/(SICI)1097-0142(19971215)80:12<2222::AID-CNCR2>3.0.CO;2-W&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=9404698&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1997YK93400002&link_type=ISI) 

6.  Ruppert AS, Yin J, Davidian M, et al. Application of a sequential multiple assignment randomized trial (SMART) design in older patients with chronic lymphocytic leukemia. Ann Oncol. 2019;30(4):542–550. doi:10.1093/annonc/mdz053
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/annonc/mdz053&link_type=DOI) 

7.  Kidwell KM, Postow MA, Panageas KS. Sequential, Multiple Assignment, Randomized Trial Designs in Immuno-oncology Research. Clin Cancer Res. 2018 Feb 15;24(4):730–736. doi: 10.1158/1078-0432.CCR-17-1355. Epub 2017 Aug 23. PMID: 28835379; PMCID: PMC5963278.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImNsaW5jYW5yZXMiO3M6NToicmVzaWQiO3M6ODoiMjQvNC83MzAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8wNi8yOS8yMDIzLjAyLjIwLjIzMjg2MTUyLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

8.  Saâda-Bouzid E., Defaucheux C., Karabajakian A., Coloma V.P., Servois V., Paoletti X., Even C., Fayette J., Guigay J., Loirat D. Hyperprogression during anti-PD-1/PD-L1 therapy in patients with recurrent and/or metastatic head and neck squamous cell carcinoma. Ann. Oncol. 2017;28:1605–1611. doi: 10.1093/annonc/mdx178.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/annonc/mdx178&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

9.  Wakiyama H, Kato T, Furusawa A, Okada R, Inagaki F, Furumoto H, Fukushima H, Okuyama S, Choyke PL, Kobayashi H. Tregdominant tumor microenvironment is responsible for hyperprogressive disease after PD-1 blockade therapy. Cancer Immunol Res. 2022 Sep 28:CIR-22-0041. doi: 10.1158/2326-6066.CIR-22-0041. Epub ahead of print. PMID: 36169564.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1158/2326-6066.CIR-22-0041&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36169564&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

10. Yildirim HC, Guven DC, Aktepe OH, Taban H, Yilmaz F, Yasar S, Aksoy S, Erman M, Kilickap S, Yalcin S. Blood Based Biomarkers as Predictive Factors for Hyperprogressive Disease. J Clin Med. 2022 Sep 1;11(17):5171. doi: 10.3390/jcm11175171. PMID: 36079101; PMCID: PMC9457309.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/jcm11175171&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36079101&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

11. . Russo G.L., Moro M., Sommariva M., Cancila V., Boeri M., Centonze G., Ferro S., Ganzinelli M., Gasparini P., Huber V. Antibody– Fc/FcR interaction on macrophages as a mechanism for hyperprogressive disease in non–small cell lung cancer subsequent to PD-1/PD-L1 blockade. Clin. Cancer Res. 2019;25:989–999.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImNsaW5jYW5yZXMiO3M6NToicmVzaWQiO3M6ODoiMjUvMy85ODkiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8wNi8yOS8yMDIzLjAyLjIwLjIzMjg2MTUyLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. Yildirim HC, Guven DC, Aktepe OH, Taban H, Yilmaz F, Yasar S, Aksoy S, Erman M, Kilickap S, Yalcin S. Blood Based Biomarkers as Predictive Factors for Hyperprogressive Disease. J Clin Med. 2022 Sep 1;11(17):5171. doi: 10.3390/jcm11175171. PMID: 36079101; PMCID: PMC9457309.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/jcm11175171&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36079101&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

13. [https://www.fda.gov›media›download](https://www.fda.gov/media/93053/download)
    
    

14. Remon J, Menis J, Hasan B, Peric A, De Maio E, Novello S, Reck M, Berghmans T, Wasag B, Besse B, Dziadziuszko R. The APPLE Trial: Feasibility and Activity of AZD9291 (Osimertinib) Treatment on Positive PLasma T790M in EGFR-mutant NSCLC Patients. EORTC 1613. Clin Lung Cancer. 2017 Sep;18(5):583–588. doi: 10.1016/j.cllc.2017.02.005. Epub 2017 Mar 1. PMID: 28341106.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.cllc.2017.02.005&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28341106&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 
    
    See also Annals of Oncology (2022) 33 (suppl_7): S808–S869. 10.1016/annonc/annonc1089 for initial results.
    
    

15. Maeda T, Yoshino K, Nagai K, Oaku S, Kato M, Hiura A, Hata H. The efficacy of platinum-based chemotherapy for immune checkpoint inhibitor-resistant advanced melanoma. Acta Oncol. 2019 Mar;58(3):379–381. doi: 10.1080/0284186X.2018.1541252. Epub 2019 Jan 7. PMID: 30614355.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/0284186X.2018.1541252&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30614355&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

16. Goldinger SM, Buder-Bakhaya K, Lo SN, Forschner A, McKean M, Zimmer L, Khoo C, Dummer R, Eroglu Z, Buchbinder EI, Ascierto PA, Gutzmer R, Rozeman EA, Hoeller C, Johnson DB, Gesierich A, Kölblinger P, Bennannoune N, Cohen JV, Kähler KC, Wilson MA, Cebon J, Atkinson V, Smith JL, Michielin O, Long GV, Hassel JC, Weide B, Haydu LE, Schadendorf D, McArthur G, Ott PA, Blank C, Robert C, Sullivan R, Hauschild A, Carlino MS, Garbe C, Davies MA, Menzies AM. Chemotherapy after immune checkpoint inhibitor failure in metastatic melanoma: a retrospective multicentre analysis. ASCO Abstract 2018 DOI: 10.1200/JCO.2018.36.15_suppl.e21588
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1200/JCO.2018.36.15_suppl.e21588&link_type=DOI) 
    
    Journal of Clinical Oncology - published online before print June 1, 2018and Eur J Cancer. 2022 Feb;162:22–33. doi: 10.1016/j.ejca.2021.11.022. Epub 2021 Dec 21. PMID: 34952480.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ejca.2021.11.022&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34952480&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

17. Sato Y, Fukuda N, Fujiwara YU, et al. Efficacy of Paclitaxel-based Chemotherapy After Progression on Nivolumab for Head and Neck Cancer. In Vivo. 2021;35(2):1211–1215. doi:121873/invivo.12371
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiaW52aXZvIjtzOjU6InJlc2lkIjtzOjk6IjM1LzIvMTIxMSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIzLzA2LzI5LzIwMjMuMDIuMjAuMjMyODYxNTIuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

18. Schvartsman G, Peng SA, Bis G, Lee JJ, Benveniste MFK, Zhang J, Roarty EB, Lacerda L, Swisher S, Heymach JV, Fossella FV, William WN. Response rates to single-agent chemotherapy after exposure to immune checkpoint inhibitors in advanced non-small cell lung cancer. Lung Cancer. 2017 Oct;112:90–95. doi: 11016/j.lungcan.2017.07.034. Epub 2017 Aug 3. PMID: 29191606
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.lungcan.2017.07.034&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

19. Park SE, Lee SH, Ahn JS, Ahn MJ, Park K, Sun JM. Increased Response Rates to Salvage Chemotherapy Administered after PD-1/PD-L1 Inhibitors in Patients with Non-Small Cell Lung Cancer. J Thorac Oncol. 2018 Jan;13(1):106–111. doi: 10.1016/j.jtho.2017.10.011. Epub 2017 Oct 31. PMID: 29101058.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jtho.2017.10.011&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

20. Mangin MA, Boespflug A, Maucort Boulch D, Vacheron CH, Carpentier I, Thomas L, Dalle S. Decreased survival in patients treated by chemotherapy after targeted therapy compared to immunotherapy in metastatic melanoma. Cancer Med. 2021 May;10(10):3155–3164. doi: 10.1002/cam4.3760. Epub 2021 May 1. PMID: 33932099; PMCID: PMC8124115.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/cam4.3760&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33932099&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

21. Karachaliou GS, Ayvali F, Collichio FA, Lee CB, Ivanova A, Ollila DW, Moschos SJ. Chemotherapy Following PD-1 Inhibitor Blockade in Patients with Unresectable Stage III/Stage IV Metastatic Melanoma: A Single Academic Institution Experience. Oncology. 2020;98(3):174–178. doi: 10.1159/000504578. Epub 2019 Dec 3. PMID: 31794964.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1159/000504578&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31794964&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

22. Puza CJ, Bressler ES, Terando AM, Howard JH, Brown MC, Hanks B, Salama AKS, Beasley GM. The Emerging Role of Surgery for Patients With Advanced Melanoma Treated With Immunotherapy. J Surg Res. 2019 Apr;236:209–215. doi: 10.1016/j.jss.2018.11.045. Epub 2018 Dec 20. PMID: 30694757.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jss.2018.11.045&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30694757&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

23. Saint-Jean M, Fronteau C, Peuvrel L, Khammari A, Varey E, Quéreux G, Dréno B. Chemotherapy efficacy after first-line immunotherapy in 18 advanced melanoma patients. Medicine (Baltimore). 2020 Jul 17;99(29):e21329. doi: 10.1097/MD.0000000000021329. PMID: 32702928; PMCID: PMC7373626.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/MD.0000000000021329&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32702928&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

24. Hadash‐Bengad R, Hajaj E, Klein S, et al. Immunotherapy potentiates the effect of chemotherapy in metastatic melanoma – a retrospective study. Front Oncol. 2020;10:70
    
    

25. Markovic SN, Suman VJ, Javed A, et al. Sequencing ipilimumab immunotherapy before or after chemotherapy (Nab‐Paclitaxel and Bevacizumab) for the treatment of BRAFwt (BRAF Wild‐Type) metastatic malignant melanoma: results of a study of academic and community cancer research united (ACCRU) RU261206I. Am J Clin Oncol. 2020;43(2):115–121.
    
    

26. Asada Y, Teramura C, Wada T, Machida Y, Koshinuma S, Yamamoto G. Successful rechallenge with cetuximab after progression with nivolumab for recurrent cervical lymph node metastasis from carcinoma of the tongue. Clin Case Rep. 2021;9(7): e04464. Published 2021 Jul 16. doi:11002/ccr3.4464
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=11002/ccr3.4464&link_type=DOI) 

27. Kwong TT, Wong CH, Zhou JY, Cheng ASL, Sung JJY, Chan AWH, Chan SL. Chemotherapy-induced recruitment of myeloid-derived suppressor cells abrogates efficacy of immune checkpoint blockade. JHEP Rep. 2020 Dec 20;3(2):100224. doi: 10.1016/j.jhepr.2020.100224. PMID: 33604533; PMCID: PMC7876565.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jhepr.2020.100224&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33604533&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

28. Zengin ZB, Meza L, Pal SK, Grivas P. Chemoimmunotherapy in urothelial cancer: concurrent or sequential? Lancet Oncol. 2021 Jul;22(7):894–896. doi: 10.1016/S1470-2045(21)00284-9. Epub 2021 May 26. PMID: 34051179.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1470-2045(21)00284-9&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34051179&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

29. Liu S, See KC, Ngiam KY, Celi LA, Sun X, Feng M Reinforcement Learning for Clinical Decision Support in Critical Care: Comprehensive Review J Med Internet Res 2020;22(7):e18477 doi: 10.2196/18477 PMID: 32706670 PMCID: 7400046
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/18477&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32706670&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

30. Liu M, Shen X, Pan W. Deep reinforcement learning for personalized treatment recommendation. Stat Med. 2022 Sep 10;41(20):4034–4056. doi: 10.1002/sim.9491. Epub 2022 Jun 18. PMID: 35716038; PMCID: PMC9427729.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/sim.9491&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=35716038&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

31. Zhao Y, Zeng D, Socinski MA, Kosorok MR. Reinforcement learning strategies for clinical trials in nonsmall cell lung cancer. Biometrics. 2011 Dec;67(4):1422–33. doi: 10.1111/j.1541-0420.2011.01572.x. Epub 2011 Mar 8. PMID: 21385164; PMCID: PMC3138840.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1541-0420.2011.01572.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21385164&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

32. Honvoh GD, Cho H, Kosorok MR. Model selection for survival individualized treatment rules using the jackknife estimator. BMC Med Res Methodol. 2022 Dec 22;22(1):328. doi: 10.1186/s12874-022-01811-6. PMID: 36550398; PMCID: PMC9773469.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12874-022-01811-6&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36550398&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

33. Davis AA, Iams WT, Chan D, Oh MS, Lentz RW, Peterman N, Robertson A, Shah A, Srivas R, Wilson TJ, Lambert NJ, George PS, Wong B, Wood HW, Close JC, Tezcan A, Nesmith K, Tezcan H, Chae YK. Early Assessment of Molecular Progression and Response by Whole-genome Circulating Tumor DNA in Advanced Solid Tumors. Mol Cancer Ther. 2020 Jul;19(7):1486–1496. doi: 10.1158/1535-7163.MCT-19-1060. Epub 2020 May 5. PMID: 32371589; PMCID: PMC8142024.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6Im1vbGNhbnRoZXIiO3M6NToicmVzaWQiO3M6OToiMTkvNy8xNDg2IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjMvMDYvMjkvMjAyMy4wMi4yMC4yMzI4NjE1Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

34. Bratman, S.V., Yang, S.Y.C., Iafolla, M.A.J. et al. Personalized circulating tumor DNA analysis as a predictive biomarker in solid tumor patients treated with pembrolizumab. Nat Cancer (2020).
    
    

35. Herbreteau G, Vallée A, Knol AC, et al. Quantitative monitoring of circulating tumor DNA predicts response of cutaneous metastatic melanoma to anti-PD1 immunotherapy. Oncotarget. 2018;9(38):25265–25276. Published 2018 May 18. doi:10.18632/oncotarget.25404
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.18632/oncotarget.25404&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom) 

36. Wang C, Chevalier D, Saluja J, Sandhu J, Lau C, Fakih M. Regorafenib and Nivolumab or Pembrolizumab Combination and Circulating Tumor DNA Response Assessment in Refractory Microsatellite Stable Colorectal Cancer. Oncologist. 2020 Aug;25(8):e1188–e1194. doi: 10.1634/theoncologist.2020-0161. Epub 2020 May 30. PMID: 32406541; PMCID: PMC7418365.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1634/theoncologist.2020-0161&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32406541&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F06%2F29%2F2023.02.20.23286152.atom)