A Discussion with Michael Proschan on Response-Adaptive Randomization

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Scott Berry: All right.

Welcome everybody.

Back to in the interim today
I have a really cool topic.

Uh, topic is response-adaptive
randomization, and I'll talk a

little bit about the structure.

But I have, uh, a guest with me
today, an expert in clinical trials.

An expert statistician, um, Michael
Proschan has been at the NIH for 19 years.

He's a mathematical statistician
in the office of biostatistics

research of NIH and specifically
N-I-A-I-D, the, uh, Institute for

Allergy and Infectious Diseases.

And Mike has three books to his credit.

Uh, essentials of Probability Theory
for Statisticians, statistical

Monitoring of Clinical Trials, and
Statistical Thinking in Clinical Trials.

So welcome to in the interim, Mike.

Michael Proschan: Thank you very much.

Good to be here.

Scott Berry: All right, so Mike and I
are probably known to be on opposite

sides of response-adaptive randomization.

Mike uh, and Scott Evans wrote
an article, A Resist, resist.

The temptation of
Response-Adaptive randomization.

I've used response-adaptive randomization.

So I thought it would be great
if we talked about what do we

agree on, what do we disagree on?

What are the issues around
response-adaptive randomizations,

uh, are there good?

Are there bad?

Are there ugly?

And I hear what we agree and disagree on.

You sound good, Mike?

Michael Proschan: Yeah.

No, that sounds like a great discussion.

Scott Berry: Okay, so why don't
you, for our listeners, tell us what

Response Adaptive Randomization is?

Michael Proschan: Yeah, response.

Adaptive randomization is changing
the randomization probabilities, uh,

in response to, uh, outcome results.

So you, you, you're seeing the actual, um,
outcomes, um, uh, not just changing, uh.

Randomization probabilities in response
to covariates, baseline covariates,

but actually looking at outcomes.

And so the idea is that, uh, you know,
you can assign more people to the

better performing arm, uh, and uh,
in that way, perhaps be more ethical.

Scott Berry: Okay, so, so the, the, the
non-response adaptive randomization in a

trial might be that there's, there, there.

Two arms in the trial and you do
one-to-one randomization from the

start of the trial to the end of the
trial, regardless of what the data show

response, adaptive randomization, you have
planned interim analysis and the R and

which of the two arms is doing better.

You might assign differentially
with a higher probability.

It might go to 60%, 70%, 80%
in a multi-arm trial with.

Uh, either multiple doses, you could
change the randomization to favor those

arms during the course of the trial.

So that's the res response.

Adaptive randomization.

Michael Proschan: Right.

Scott Berry: Okay, so on the side of,
of, of response, adaptive randomization,

and so maybe you resist the temptation.

So what is it about response,
adaptive randomization that

you find to be a negative?

Michael Proschan: Yeah, so
one of the, one of the big

problems that I have with it is.

That what can happen is that as
you're changing these randomization

probabilities, let's suppose
you're in an extreme situation and

you're now assigning 90% of part
participants to the active arm.

Let's say there's just two, two arms.

Um, then, um.

What happens is if, if, you know, if
that happens later in the trial, there

could be temporal trends that occur in
clinical trials, and they do happen.

Um, and maybe the earlier patients
are different from the later patients,

and so all the later patients, or
most of the later patients are getting

assigned to the active treatment.

But the later patients are also
tend to be healthier than the

earlier patients, for example.

And so if you're not careful, uh,
you could, you know, uh, declare

a treatment benefit when it's
actually just a temporal trend.

And there are ways to
try and deal with that.

But that's, uh, one big concern.

Um, I would say that's
my, uh, main concern.

I, I also though.

Um, think like in an unblinded trial.

Um, you know, if patients are
seeing who's in the, uh, you know,

if, if patients are seeing these
randomization probabilities changing.

Then they go, they get good knowledge
of, you know, which, which arm is doing

better, that could cause patients who
are in the other arm to perhaps drop out,

um, or start taking the active drug, um,
and changing their behavior basically.

So that can influence the
results of the clinical trial.

So I would say, you know,
both of those are concerns.

Uh, my main one is the, uh, is, is the
first one because, you know, most of

our trials are blinded, although I must
say that in these multi-arm trials,

uh, there are several of them happening
during COVID in which, um, you were.

Uh, you, you figured out which arms
you were eligible for, and then you got

randomized to, you know, one of those
eligible arms and, uh, which, which of

those treatments you were eligible for
wasn't blinded, but whether you got the

active or the placebo, um, was blinded.

So, uh, sometimes there is some
level of unblinding in these trials.

Um,

Scott Berry: Okay.

Okay, so let's sort of,
let's unpack a little bit.

The first, the temporal trend.

So the, the issue could be
that in the first half of the

trial, it's largely sicker.

Patients, you're doing
one-to-one randomization, the

treatment is doing better.

You have two arms, you have a placebo in
a treatment, the treatment's doing better.

You change the randomization.

75%, 25%.

And in the second half of the trial,
the patients are relatively more

healthy when you get to the end.

If you just take the raw average
of the treatment, you have more

patients during the healthier part,
and the raw average is gonna look

better than it is on the placebo.

Uh, in that scenario, even if there's,
there's not a difference between them,

the temporal trends can be different.

Now, you, you alluded to,
you can correct for that.

So if I were involved in a trial where
we had that, you could put in a covariate

for the time and, and that covariates
gonna pick up because you have the same

arms in the first half and you have
in the second half, you know that the

patients are less healthy because they
do, I'm sorry, they're more healthy.

They do better in the second half.

'cause you have both the arms of
data and you make an adjustment

for your relative effect of that.

Uh, it would be a way to
potentially mitigate the

temporal trends in that scenario.

Um, now.

You, you, that, that means at the end of
the trial you need an adjusted estimate.

You can't just take the
raw averages of that.

So now you've got an adjusted estimate.

In many clinical trials, we
adjust for covariates, we

adjust for ma uh, uh, gender.

We adjust for age, we adjust for
severity, other aspects of this so that

that could be a way to mitigate it.

The unb blinding, I
agree, is an issue if, if.

The, if an investigator sees that all
of a sudden a particular treatment is

coming up more and more and they know it's
response, adaptive randomization, that

could potentially lead to maybe not even
enrolling a patient in the trial because,

boy, that that must be doing better.

I'll just give that outside of it.

In, in other aspects of it, we do
you, you can have trials that are

completely blinded and the randomization
can shift and nobody's the wiser.

Uh, uh, other than the people that are an
analyzing the unblinded, uh, aspect of it.

Now you didn't bring up the, the part
that I thought for sure you would

bring up, uh, which is the first
criticism, is the lowering of power.

Um, even in, even if we do that
adjustment, whether you're comfortable,

the adjustment in a two arm trial,
if I'm trying to compare the two

arms and say which one's better?

Now there are weird scenarios in terms
of the variation of the arms where

you want differential randomization,
uh, a 5% rate to a 25% rate.

You'd rather have more on the 25,
for example, but let's ignore those.

Almost always.

You want equal randomization of
the two arms, and so going in your

case of nine to one or three to
one, or four to one lowers power.

Michael Proschan: Right.

Yeah.

Uh, I was going to, yeah, I, I didn't
say that because I did say that,

you know, as, as you mentioned that
there are ways of dealing with this.

Um, I didn't mention their drawbacks,
but I, you know, like one of

the, one of the standard ways of
doing it would be to stratify.

By time.

And then within that second half, if
you've got nine to one randomization,

you're very inefficiently estimating
the treatment effect because

of that unequal randomization.

Yeah.

So I, I, um, I was gonna say
that in response to how you

try and fix these issues.

Scott Berry: Okay.

So you're setting me up is what
you're, what you're just saying?

Yes.

Michael Proschan: I would
never do that, Scott.

Scott Berry: Okay, so I think we,
we largely agree on these points.

I agree.

It, it lowers power and I think if you
are running an experiment comparing

two arms that you need a, you need
another reason besides statistical for

doing response, adaptive randomization,
uh, we'll come back to the ethical

issues and, and what you think
about that, uh, in this scenario.

Um, and, and largely.

It's, it's two armed trials there,
there needs to be a compelling reason

to do it under those circumstances.

Um, uh, on that, the unblinding
I think is an absolute issue.

If you're in a scenario where there's
unblinding and there are a number

of scenarios where patients and
and investigators are unblinded.

Uh, oncology trials, they're
frequently unblinded.

It's very hard to blind the chemotherapy
regimen or the, the intensive regimen

that they're getting in oncology.

They can be unblinded.

Uh, and there are some trials that it's,
it's very hard to blind patients and

investigators, uh, in those scenarios.

But, but there are certainly many
that, that, that can be blinded.

So the temporal trends, I also, uh.

Temporal trends, I think in your
life of being an infectious disease

is certainly a bigger issue than
in other non-infectious diseases.

Um, in a LS or Alzheimer's or oncology
or other scenarios where you might not

think time makes as much of a difference.

Now, it can make a difference in those
scenarios, uh, in severity, in the

trial, the behavior, the sites you open.

Uh, within this scenario, but certainly
I imagine in infectious disease,

temporal trends are a much bigger issue.

Michael Proschan: Yes.

Uh, and that's true, you know.

Especially, uh, with a new disease where
temporal trends, uh, could result from

you're just doing better background care.

You didn't know how to treat
them, you know, uh, uh, before

and now over time you're learning.

Or there could be a new vaccine
introduced that changes things.

Um, and, uh, you know, there are
a number of other things that, uh,

viral variants, you know, change.

And so really, I mean,
that, that's a problem.

Even if you don't want
to talk about response.

Adaptive randomization, that's a real
problem when you do a trial and then,

you know, couple years later everything
has changed and you're, you know,

you're wondering whether, you know.

Um, whether you can really
rely on, on the results.

So yeah, we're in a particularly tough
situation in infectious diseases,

Scott Berry: Yeah, so we brought
up the idea that if suppose we're

doing response adaptive randomization
and we're gonna adjust for strata,

Michael Proschan: right?

Scott Berry: adjustment is valid
and very reasonable as long

as the effects are additive.

Michael Proschan: Mm-hmm.

Scott Berry: So, uh, we're fitting a
logistic regression or a continuous

endpoint, uh, on that as long as that
he, you brought up the example that

they're healthier in the second half.

If we make an adjustment at the relative
treatment effect stays the same.

It's just patients go up and
down equally in the two arms.

That's a perfectly valid thing to do.

Interactions with time,
which you just brought up.

The potential that the disease is
almost different in the second half.

It's a new variant.

Is, is a, is clearly a problem,
but it's clearly a problem to

science that you might run a trial
and then wonder six months from

now, is our treatment still valid?

Uh uh, in this scenario of that.

Um, uh, in that, so that,
that's a huge problem.

Okay, now let's talk
about, so two armed trials.

I think we, we largely agree on that.

Let's talk about multi-arm trials.

So what, what's different
about multiple arm trials?

So, a, and it's not, it's not uniformly
that response-adaptive randomization

is better in multiple arm trials.

It really depends on the goal of that.

So we've done platform trials where there
are five companies that bring their drug

in and they would all like to understand
in a phase two trial does their drug work.

Michael Proschan: Yep.

Scott Berry: one potential platform
trial, and we'll come back as response.

Adaptive randomization.

Good there.

Another potential trial is where
you have multiple arms and you're

really interested in the best arm.

And the goal is to find the best, almost
a King of the Hill platform trial of that.

Now, one of those is perhaps
dose finding, so I may have,

we've done a response-adaptive
randomization trial where we had

seven doses in a diabetes trial.

Where are we looking at endpoints like
hba one C, blood pressure, heart rate,

weight loss, uh, over seven doses.

And in that case, the company, the
sponsor, they want to know the best arm.

So if we do response-adaptive
randomization in a multi-arm setting where

our goal is to find the best and probably
identify how good it is, you can imagine

by lowering the probability to arms that.

Are, are, are early on identifying
themselves as not as good.

I increase the randomization to the
better arms and, and in simulations

in certain scenarios, in certain,
certain ways to do response-adaptive

randomization that can improve your
ability to find the better dose.

And to identify the better dose.

Now that's at the negative to doses
you go away from, you learn less

about the doses you go away from.

So coming back to a platform trial where
five different sponsors put their drug in.

If I'm doing response adaptive
randomization, and I'm favoring company a.

It's to the detriment
of company B perhaps,

Michael Proschan: Right,

Scott Berry: and that might not be why
I'm doing the trial in the scenario,

but in a scenario that matches these
goals of King of the Hill, find the

best, identify the best, it can improve
power or, or is that reasonable?

Michael Proschan: Well, so I, uh, you
know, I will stipulate to the fact that,

Scott Berry: Okay.

Michael Proschan: That, um, there are
some situations where I, you know,

I find the idea attractive of, of,
uh, you know, trying to assign more

to the arms that are doing better.

Um, and like you said, um, that may help
you find, you know, a, a good drug fast.

Which, you know, might be especially
important in certain situations

where there's, there's nothing
that works and you know, it's

a very serious, uh, disease.

Uh, so I can see being attracted to it.

Um, uh, but as you say, it's
to the detriment of these

other ones, other drugs.

Um, and the problem is, you know, um, if
the early results are not necessarily.

Um, you know, that reliable and, uh,
you know, the late, uh, Richard Simon,

uh, once said, uh, Hey, we don't even
get good estimates of the treatment

effect at the end of the trial.

And now you're trying to respond
to, you know, estimates that

you're getting very early on.

So, I, I think one of the things that
I imagine that you and I both agree on,

I, I think we've already hit on one.

Um.

But I think another one that we both
agree on is, um, that if you're gonna

do response adaptive randomization,
you should have a burn in period of,

uh, uh, more traditional randomization
first, because those early results

could be, um, quite variable.

Scott Berry: Yeah, absolutely.

And, and for example.

Um, you know, fall and
whan presented response.

Adaptive randomization goes bad when you
lessen the probability on the control.

So my seven doses in a diabetes trial,
if I'm lessening control in that

scenario, 'cause it's doing worse.

My ability to identify those
doses goes, goes the wrong way.

It's kind of the two arm
trial scenario like that.

So most of the time we do
response adaptive randomization.

We're trying to fix the control and we're
altering the other ones in that scenario.

And yes, adequate burn, we're
doing lots of simulations to find

what's the likelihood I misled
and I put too many on that.

Uh uh, and that's part of this is I,
I'll, I'll agree absolutely that this.

You can do it poorly.

I've been involved in a couple trials
that have gone a bitter eye, but, um, you,

you can have algorithms that don't do it
well and you can have algorithms that do

it well that increase your probability
of finding the right dose and all that.

So we've done trials on Alzheimer's
where we've done had five different

doses, diabetes with seven,
even sepsis, uh, which I think.

I think it's an infectious disease.

Um, is that an infectious disease?

I know it's a, an infection of something
causes sepsis typically, but, um, that,

multiple treatments of that, now we
did it in COVID and I'll come to that.

Uh, I think that, that, I'm,
I'm teeing it up for you, uh,

in COVID, uh, in that scenario.

But first I wanna come back.

I, I did not hear Richard
Simon passed away.

Michael Proschan: Uh, yes.

Yes he did.

Scott Berry: Oh, that's, that's awful.

Uh, he's a, uh, certainly a hero to
statisticians and a, a brilliant guy.

And a wonderful guy.

Yeah.

Huh.

Uh, that's too bad.

Um, so I, I, in the, so in a scenario
where multiple arms, if the goals

of the trial, if you're okay.

Um, you know, lessening the lower
dose, lessening the higher dose because

it's got adverse events in this and
you're, you're, you're good with

the less precision on the, if done
well, that could potentially lead to

improvements in, in multi-arm trials.

You, so you're, you know, we're, we're,
we're at least in the same neighborhood.

Michael Proschan: Yeah.

Yeah.

I mean, I, I, I do, I do see the
potential for some, some benefit,

uh, that, you know, uh, of that yes.

Scott Berry: Yep.

So now let's talk about a multi-arm
trial that you were involved in.

And this was absolutely incredible trial.

The Palm trial was in a, in a, a
platform trial in Ebola, one of the

first platform trials in infectious
disease, if not the first perhaps

incredible effort by the NIH.

Um, and you investigated four different
therapies for the treatment of Ebola,

um, in, in this, and this was not the
sort of 2014, 15 go around where, where

this, but I think this was later sort
of 2018, 19 ish where you investigated.

Right.

So in investigating, and I should
probably, I, I wrote down to make

sure this, that you investigated
within this, um, zap rem Desi.

MA one 14 and a Regeneron,
uh, EB three, some,

Michael Proschan: Yes, that's right.

Scott Berry: so, um, within the, so
four of them, I think in the way you set

it up, zap played a role of a control.

It.

Michael Proschan: It did.

Yeah.

And there was some criticism about that
because Zumab in an earlier, uh, trial

in Western Africa, uh, that the trial
was stopped before the evidence was

sufficient to say that it's, you know,
uh, better than not giving anything.

Uh, but we felt pretty confident that,
um, if if something beat zap, then

it would be, it, it would have to
be better than, than giving nothing.

Scott Berry: Yep.

Yep.

Okay.

And in, so the results of this
trial and, and largely you

compared each of the other three.

To zap each of the other three drugs
were to zap, and at, at least in

the, the, the part that I have within
this, the REM DESI severe was stopped

because it didn't improve upon Zap.

But the other two arms,
approximately 50% mortality within

28 days on Zap and REM Desi.

And approximately 33, 30 5% mortality
at 28 days on the other two drugs.

Michael Proschan: Right, right.

Yeah.

So there was a clear separation between,
um, uh, zap, MDES pair, um, and the, and

the other two, the monoclonal antibody
one 14 and, uh, the Regeneron product.

Um, it was quite clear
that those two were better.

Then, uh, zap and, and
better than mdes Fair.

Scott Berry: Oh, okay.

So if, if we were to take those sort
of results and say, what if we would've

done response adaptive randomization,
and I'll come back and ask, was this ever

entertained or, you know, all of that.

Uh, reasons yes or no, that you could.

You if you simulated a scenario
where the control was 50 and you

had two arms that were 35 and one
another one that was 50, and you

simulate, you did interim analysis.

And you simulated it, uh, where
you, uh, it changed the response.

Adaptive randomization to those you
might find that you're quicker to

find one of those two were effective
and you maybe put more people on

there if you simulated that scenario.

Now, I don't know the goals
of this trial and all of this.

I wasn't there.

But was this entertained?

Are there reasons not to do it?

There reasons to do it
in the trial like that?

Michael Proschan: Yeah, so that's,
that's an interesting trial because.

Um, that was not, uh, a trial in which,
you know, oh, you're eligible for these

and so we're gonna randomize you to one
of these two instead of one of these four.

That was a trial where he had to
be eligible for all, all four.

And so, um, you know, that's a
situation where I, I think I would, um.

You know, I would object less to,
uh, response, adaptive randomization.

I still would not do it because
I, I think that there are, um, you

know, things that can go very wrong.

And, and I think you can achieve
similar results by, you know,

um, by monitoring and, and, uh,
perhaps responding, you know, um.

Uh, you know, stopping arms early.

Um, and they were, you know, they,
it, it was stopped, uh, early, um, you

know, not, not super early, but it was,
it did not go all the way to the end.

Um, and so, yeah, so I think you
can achieve similar things by having

equal randomization, and then you
would get better information on.

Um, you know, on both the, uh, the
two products that did well, the

Monoclonal and Body one 14 and the
Regeneron product, um, I don't know

that, you know, with response, adaptive
randomization, perhaps it would've given

more people to only one of those arms.

Um, you know, I didn't, I didn't, uh, do
any simulation after the fact to see what

would've happened if we had used response.

Adaptive randomization.

Um, I'm guessing that you
may have done that, Scott.

Scott Berry: Yeah, I, I, it's not
like I'm gonna spring them on you,

but it is actually the ca the case.

I think we, we did a number of these
during Ebola with a Gates trial

that never enrolled any patients,
uh, in this scenario, the sort of.

The, the somewhat optimal circumstance for
response adaptive organization would be

if one of those four was that effective
35 and the rest were 50 with two of them.

You do find advantages,
uh, in that scenario.

And it's, it's, it's sort of a question
of how important is time, and you

touched on that within a pandemic within.

Uh, uh, a pandemic where maybe
time means a huge thing, um, uh,

getting results out earlier on.

One of them, how important
that is relative to getting

results on all of them.

For example, um, uh, and, and I think
you're right, so about a hundred,

and I know you're right, about 175
patients each to these arms, roughly.

In that, and then you stop two of
them, which is kind of a brute force

response, adaptive randomization where
it goes 1, 1 1 to one to one, and then

it goes one to one to zero to zero.

'cause I think you kept
enrolling the other two.

Michael Proschan: Yeah, so we, well
we, we, we made the, um, decision

that DSMB made the recommendation
to, um, continue follow up because

not everyone had had the full 28 days
of follow up to continue follow up.

Um.

And, uh, and then the final results would
be after the people that had already been

randomized, um, you know, would, were, uh,
evaluated for that 28 day, uh, outcome.

That was the, the final analysis.

Scott Berry: Yep.

Yep.

Okay.

Um, so we did, we did the remap cap trial.

Does response adaptive
randomization, it's a little ba

Is it a multi-arm trial or not?

There are.

It has domains, so it actually does
multifactorial randomization to a patient

during, during the height of COVID.

Uh, I think one patient was
randomized to seven domains.

What, what are the domains where their
therapeutic anticoagulation arm in

immune modulation arm, there was a
vitamin C arm that many of these are.

Thought of as fairly, you need to
make a decision in routine care.

Do we give them, do you
give a steroid yes or no?

Uh, aspect of it.

And so, um, in that setting, some of
those domains are multi-arm domains.

The immune modulation had anakinra,
uh, uh, tocilizumab, it had interferon

and it had no IMU modulation.

And so that was, that was one where.

Actually Tocilizumab and Sarilumab
IL six receptor antagonists got

higher randomization faster.

Then we had domains that were two
and um, uh, particular one of them,

simvastatin ended up randomizing it
about eight to one or nine to one at

some course of it when COVID stopped.

Um, uh, in that setting.

In hindsight, we have changed
the design that we only allow.

I believe it's now 70%, so 7% to 30%.

It's a little bit different.

That part of it's a little bit
of a of multi-arm and two arm

where there's a question of do
you give a simvastatin yes or no?

Was the randomization.

So you could call that a two arm domain.

But at the same time, interactions
between the different domains

became really quite important.

Does a statin interact with IL six?

Does it interact with steroids?

Does it interact with antivirals?

Things like that.

And so putting more patients on the
things that we're doing better was

perceived to have benefits in that scale.

We have, we, we, we, we largely.

We didn't want that to happen, so
we've put governors on it in this way.

But we have done an infectious disease
that within COVID in that, now we're doing

it in commun community acquired pneumonia,
which is certainly an infectious disease

and can have seasonal variability as well.

Michael Proschan: Right.

Scott Berry: Okay.

Another, uh, a, a another trial.

And then I want to come back
to the, the question of ethics.

You, you brought up ethics
and, and your views on that.

So I SY two, are you
familiar with ipy two?

Michael Proschan: Uh, I,
I know something about it.

Scott Berry: Yeah.

Okay.

So this is, uh, neoadjuvant
Breast Cancer trials phase two.

It was a platform trial and.

What's a little bit different
than some other platform drivers?

Multiple agents come in, there's
a standard of care, and in breast

cancer, they really differentiate
breast cancer really well.

Baseline factors on estrogen receptor
status and hormone receptor status.

And so there's mammoprint status, but
it plays a lesser role, but largely.

You can think of the eight classifications
of plus minus for those that in the

trial, even though it was multiple
sponsors who brought a drug in Amgen,

Merck, uh, uh, brought drugs in that
we did response adaptive randomization

because it was believed efficacy was
likely to vary by the eight subgroups.

So if one drug was being favored within
hormone receptor positive, the other drug

might be favored in HER two positive.

And that drug's getting more patients
within subgroups that they're doing better

on, unless where they're doing worse
on if we keep up equal randomization.

So in a circumstance where there's a
platform trial and sponsors were good

with this plan, even though it kind of
goes back to this platform trial idea

that it's multiple sponsors, it's almost
this personalized setting in a case where

there's strong belief that these diseases
are largely related but different.

Okay.

Wildly shaking your head that
that's awesome and you love

response, adaptive randomization.

No.

Uh,

Michael Proschan: Um.

No, I, I, I didn't mean
to convey that Scott.

Scott Berry: Okay.

Okay.

Okay.

Um, so, um, I, so, so let me, I, I don't
want to, to this, so I, so I brought

several trials up, uh, within this.

Let me make sure you want, uh,
additional topics to bring up

before we touch on ethics of this.

Michael Proschan: Um, well, you know,
I do think one topic, and you, you, you

mentioned a, a little bit about this is
that, um, you know, the, the more, uh.

The more complicated the, the
randomization, the easier it

is to, you know, make an error.

And, you know, I certainly, I certainly,
um, you know, worry about, um.

You know, and, and the same thing
happened with, with, uh, with

Covariate, adaptive randomization.

There was a trial in which, um,
you know, they, they meant well.

Um, but the randomization
scheme that they, uh, used

didn't maintain the same, uh,

you know, different patients depending
on when you arrived in the trial, you

had different probabilities of being
assigned to, uh, you know, a given arm.

And that led to problems.

And so I worry that, um, you know,
if you have people who are, you

know, wanna go out and try this new
response, adaptive randomization, um,

you know, they might not be aware of
how careful you have to be because

it, it, you can, you can mess it up.

So just the mechanics of it, I
think are, are more difficult.

Um, so, um.

Scott Berry: Uh, critically
important that operationally.

It's done well.

You need data availability.

We didn't touch on the notion that
you, you need your endpoints proximal

enough that you can respond to it.

Uh, in the PALM trial, for example,
28 dorm Day mortality, I think

largely patients by day 10 have
probably determined not, could be a

Michael Proschan: About 90,

Scott Berry: in a, yeah.

Michael Proschan: yeah, about 95%.

If they died, they died within 10 days,

Scott Berry: Yeah.

Michael Proschan: so yeah.

Scott Berry: The, so the other side that
this has been presented is the ethical

side of this, that if you're a patient
in a trial and it's been phrased and,

and, uh, my father Don, whose original.

Introduction was multi-arm bandit
problems, where you phrase this as

there's a horizon of patients and you're
trying to have better outcomes for all

of them and the patients in the trial
and the patients outside the trial.

So part of this depends how big is the
horizon outside as opposed to inside.

If every patient in the world has
the disease and is in the trial.

You might behave differently than if
you have a very, very small fraction.

And what you learn is going to
be given to millions of people

afterwards, uh, within this.

But the idea is that during a trial,
if you're doing 50 50 randomization,

if there was a VIP who could see the
data and, and now had to be treated

and make a decision that, that that
person would be treated differently,

uh, than a patient in your trial.

And is there an ethical advantage?

Is there advantage to patients to
have the increased probability,

say in a multi-arm trial?

The two arm trial now two arm
trial is a little bit, uh, weird

because you're lowering power
and what that means ethically.

So it's a complicated story ethically.

Michael Proschan: Right.

Yeah.

Yeah, no, I, as you know, as many trials
are, it, it's a, it's a difficult issue

because, you know, you definitely, you
know, I feel, you know, it makes sense

that if I'm a patient, you know, I
wanna, I wanna get the best treatment.

Um, but, you know, I'm not sure I am
getting the best treatment if it's, if

it's made on the basis of interim results.

Uh, we've talked about the fact
that, you know, you, you wanna make

sure you have, um, uh, conventional
randomization, you know, up to a

point so that you don't have that.

But I mean, that's, you know, one
of the things every time I, I bring

up the e word, the ECMO trial,
you know, people go crazy, uh,

because, you know, ECMO was a trial.

Um.

Involving primary pulmonary hypertension.

And it was an early example of
response, adaptive randomization.

I, I grant that a lot has
been learned from ecmo.

Um, but that did lead to, uh, a severe
imbalance where 11 people were given the.

Um, the new treatment and one, one person
was given the standard treatment that,

that one, given the standard treatment
died and the conclusion was made, um,

that ECMO was better, which turned
out to be the correct decision after

another trial, uh, showed that, um.

But, uh, that caused a lot of problems
and, and some of the, some of the problems

it caused, we haven't touched on at
all because I don't think that these

are major concerns for large trials.

But, you know, we take for granted that
the sample size is, is a fixed quantity.

We, we, we condition on the sample size.

We never have to take into account
that that is a random variable that

needs to be taken into account.

But in that ECMO trial, you do
have to take that into account.

The fact that E ecmo, that, that
the sample sizes themselves are

giving you information about
whether the treatment works or not.

And in particular.

The lopsided results, um, you know, uh,
indicate that the treatment must have

been working because, you know, otherwise
it wouldn't have gotten lopsided.

And so, you know, it's not even
clear how to analyze the data.

Now, if you're a vasin.

Um, you know, then, then
you say, ah, no problem.

But if you're a non Bayesian, they, they,
you know, there were many debates about

how you could even analyze those data.

Very interesting debates.

A nice paper by Colin Begg.

Um.

You know, after, after that trial.

So it was a fascinating trial, but
I think those are some of the issues

that we haven't really touched on
that, that also give me some concern.

Uh, as I say, I think that
those are not particularly

problematic with large trials.

Um, I'm not sure how large they have
to be to not have them be a problem.

Um, but you know, who
would, yeah, go ahead.

Sorry.

Scott Berry: so I'm so sorry.

This is the Bartlett, um, uh,
play the winner design, um,

using EC ECMO as a treatment.

And, um, uh, which is, which I think,
I think at one point 35 different p

values had been created from that same
trial, and there was not, was not clear

how to analyze it as a frequentist.

Now we won't, we won't talk
about whether this is about.

The, the challenges of sample
space inference, uh, as

opposed to Bayesian inference.

But, uh, I've never done
a play the winner design.

I think it, it, it is
awkward and it, it, it, it.

I, I once got the benefit
of asking Bartlett.

I said, suppose somebody
would've simulated this trial

and showed you that result.

Would you have said, I don't like
that and done something different?

He said, yes.

If I.

If somebody would've simulated
that as an example and said

that's, that could possibly happen.

I would've done a different design.

Which, uh, you know, I use that
as an example of the value of

simulation and making sure within
an adaptive design that you've seen

lots of examples and what this does.

And a burn here probably would've
made a world of difference,

I suspect, in that trial.

Michael Proschan: Yes.

Yes.

Or even starting, you know, instead
of, instead of starting with one

ECMO ball in the urn and one.

Standard, uh, ball in the urn, you
know, if you started with 10 of

each or something, something like
that where it's not so volatile,

doesn't quickly devolve into.

You know, um, virtually everyone
getting only one treatment.

And by the way, that that one, uh,
standard treatment patient, um,

was objectively the sickest one, as
it, as luck would have it, so that

patient may have died no matter what
the treatment that that baby may

have died, regardless of treatment.

Scott Berry: yeah.

There's a whole separate issue about,
um, if you analyze that trial using

real world evidence, other things
about whether that rate was so,

so that that trial is probably the
most analyzed trial in the world,

Michael Proschan: Yes,

Scott Berry: Very interesting.

Yes.

Okay.

So, um, um, uh, within
that, so I think we.

We largely disagree in two arm trials.

Response, adaptive randomization
is, is challenging and there

you need other reasons why than
statistical generally not a good idea.

Play the winner designs.

We, we largely, I.

It sounds like a neat idea, but it needs
to be modified at the minimum to do it.

I think there are better ways to do it.

Uh, within the scenario
in multi-arm trials.

There are scenarios where as
long as it meets the goals of

the trial, it can improve some
operating characteristics like that.

You made the point, a great point
that operations need to be in order

to do this well, certainly within it.

Unblinding, which by the way was also an
issue in that Bartlett trial that, that

there was unblinding as to the ecmo.

Uh, certainly needs to be, to be, uh,
worried about, uh, in the scenario

if there's potentials for people
seeing one of the arms doing better.

We both are not experts
in the ethical side of it.

We think there may be some
potential advantages to response.

Adaptive randomization in that.

If, if I was a patient that would want
to go in a trial, that's a response.

Adaptive randomization trial.

But, you know, I'm not sure
we're gonna, we're gonna solve

the ethical issue to that.

Fair enough.

Michael Proschan: That sounds fair to me.

Scott Berry: All right.

Well, um, uh, Mike, I, I don't know if
you've ever been in the interim before,

but I appreciate you being in the interim.

Michael Proschan: Well, yeah, I have not.

And, and thank you for inviting me.

I,

Scott Berry: yeah.

No,

Michael Proschan: good discussion.

Scott Berry: Wonderful.

And, um, uh, clearly to point
out the, the people at the

NIH are doing fantastic work.

I know it's, it's, it's challenging
times and we all, we all, uh, uh, uh,

certainly think very highly of the work.

And you've had 19 great years at
the NIH and thanks for your efforts.

Keep 'em up and,

Michael Proschan: right.

Thank you, Scott.

Scott Berry: and, and thanks
for joining us in the interim.

Michael Proschan: All right.

Thank you.