Bayesian Clinical Trials with Frank Harrell

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Scott Berry: All right.

Welcome everybody.

Back to in the interim,
all things clinical trials,

statistics, statistical science.

Today we have Frank Harrell.

Frank is a professor in the
founding chair of the Department of

Biostatistics at Vanderbilt University.

Uh, helped found the
department back in 2003.

He is still there.

He was a 2014 WJ Dixon Award
for Excellence in Statistical

Consulting winner, an award of the
American Statistical Association.

And uh, what is wonderful, Frank is
very much a Bayesian and for people who

listen to this podcast, they know we.

We are Bayesian, we go about the
Bayesian Way, and Frank is Bayesian.

Bayesian.

Uh, and that's, that'll be part
of our discussion today, so, so

Frank, welcome to in the interim.

Frank Harrell: Scott, it's a privilege
to be here and, and it's great to

talk with you, especially who's been
a force and a pioneer and thought

leader and front lines, uh, commander
in the Bayesian clinical trial world.

Scott Berry: We, we are, we are
fighting this battle and we're

gonna talk about that today.

So, um, and, and, uh, a good bit of work,
and we may even get into things like your,

uh, modeling of daily ordinal values,
which, which has been really kind of neat.

Uh, Frank does quite a bit in
computational statistics, software,

uh, consulting, predictive modeling.

But let's talk about, uh, a
little bit of your career.

So you, initially you started
at Duke and DCRI, so that was

right out of graduate school.

And your, your PhD was at UNC,
and then you went to Duke.

So tell me a little bit about the,
and maybe tell me a little bit

about, uh, uh, Frank, the Bayesian.

When did, when did that come about?

Frank Harrell: Yeah.

So, uh, my first real job was
at Duke before the Duke Clinical

Research Institute existed, and
we were in a biometry division and

doing mainly cardiovascular um,
research as my whole career, uh, has

been, has involved cardiovascular
research, um, continuously, and, um.

It was a great job at a great institution
and, um, I was starting to get frustrated

with certain aspects of clinical
trials that I had just had trouble

learning about, namely, uh, group
sequential testing and the um, the real.

Like general way to deal with that
is you have to know about stochastic

processes and brownie and bridges of all
things, uh, to really get down into it.

I had a great deal of time, uh,
trouble understanding those things

and I was starting to think there's
a cleaner way to approach it.

It can't really be this complicated
and it can't really involve.

So many subjective intentions that have
to be factored into calculation of alpha.

And so I was really ripe for the picking
and I was influenced by uh, a paper that

came out from David Spiegel Alder and his
colleagues and JRSS, but something really

great, which is your dad, Don came to Duke
and we were both in the same building.

So he was in Cancer Biostat, I
was in cardiovascular biostat.

And, um, he told me one day, you
know, we really ought to meet.

I, I've got something you need to hear.

And I went downstairs to his
office and had a fantastic meeting

and I came out of Bayesian.

At the end of the meeting, I
told him I was very swayed by the

arguments and, um, I'm really going
to move in a different direction.

And to me that's really a
fun thing to change because.

Some people are just afraid of change.

That's one thing I was never afraid of.

I, I got invigorated by change,
even though one of my professors

at UNC said that base was evil
and should never be used at all.

Uh, and it, and they didn't
teach it, you know, it was

pathetic, uh, how little coverage.

We had one week of base in one course.

Um, that's just not gonna cut it.

Uh, but, but Don was
very influential to me.

Um.

And then I started reading more and
I started trying to get my clinical

trial colleagues interested in it.

And I found something you'll
probably identify with, which is

the statisticians I work with who
were great statisticians at Duke.

Uh, chief among them in my group
was Kerry Lee, who was, before he

retired a few years ago, was like
a, a powerhouse in cardiovascular

clinical trial design and execution.

And, um, but the statisticians were
less likely to change to a different

paradigm than the clinicians I work with.

They were much more open, but the
statisticians always were kind of

the, uh, not so much carry, but
uh, many others that I encountered

were more dragging their feet about
changing the statistical paradigm.

And then, and then to do Clinical
Research Institute was formed.

I was in on the formation of that.

I worked in DCRI for several
years in cardiovascular clinical

trials and loving all that.

Um, and the best I could do at
that time, I got them interested in

bootstrapping as a poor man's base.

And so we.

Take clinical trial results and we would
calculate how many bootstrap samples

came up with a positive treatment effect
as a poor man's posterior probability.

And that, that took, that
had some traction there.

And, uh, not really formal bays at all,

but my time at Duke was great.

I enjoyed all of it.

Um.

And then I had a chance to start a new
division at, um, university of Virginia.

Uh, and then went always there.

So I was Duke of 17 years and
I was at Virginia seven years.

And then I've been at
Vanderbilt since 2003.

I.

Scott Berry: Yeah, so, so going
back, interestingly, cardiovascular,

part of it is the enormous trials.

They run very large trials.

It's, it's somewhat, you know, the Pocock.

Uh, um, Janet Wit's brilliant
statisticians, but, but still

to this day, reasonably frequent
is there's the win ratio tests.

You know, the, the, the
Finkelstein Schoenfeld test

things that, almost proportional
odds type modeling, things that.

That that you've gotten into, I don't
know as though a little bit, maybe

daily ordinal, but the cardiovascular
word hasn't, world hasn't changed much

in terms of its statistical philosophy
since then, or do you think it has?

Frank Harrell: I think it's, it's
prematurely adopted win ratio and

it's leading to a lot of problems.

Uh, so it's interesting.

They were, they were willing to
adopt that in a, in a heartbeat,

I guess because of advocacy
from some great statisticians.

Um, and a lot of papers written
in cardiovascular journal.

But in terms of overall design,
it's been very frustrating.

And one of the most, um, strange things
about cardiology is those massive trials,

you know, the typical cardiovascular
trial, 6,000 patients to 10,000 patients,

they don't need to be that big, but
the cardiologists are stuck in their

way, so they do time until first event.

So if a patient has a heart attack
and then they die a month later, the

death is totally and utterly ignored.

I've been fussing at them about that for a
long time, but it, so far it hasn't taken.

Uh, but, but by using time until
a, a binary outcome, you know,

they have minimum power, so they're
gonna have maximum sample size.

And also they never do sequential studies.

They just grind it out until the end.

So their expected sample size is not
optimized in any, in any fashion.

Scott Berry: Okay, so now you,
you go to, you go to Vanderbilt,

and Vanderbilt has grown a, a
wonderful biostat department there.

Um, still multiple therapeutic
areas, uh, cardiovascular.

But in terms of critical care,
other, other areas at at Vanderbilt.

Frank Harrell: Yeah, a lot
of us dropped everything we

were doing when COVID started.

To launch this big, uh, active six, uh,
platform, uh, trials in repurposed drugs.

These were all drugs that none
of them should have worked for

any reason, and none of them did.

And there's another story there
that's really worth getting into

in the future, but I'll, I'll get
in trouble for telling that story.

So, um, but.

We really got into, uh, ordinal modeling,
ordinal clinical outcomes, longitudinal

ordinal outcomes, and Bayesian designs.

And we were gonna do
everything as Bayesian.

And then NIH gets involved
and they veto a lot of that.

So we ended up using Bayesian
analysis in, in one of our key papers.

Uh, to help interpret the results,
but we mainly use bays for

all of the interim monitoring.

So it was Bays interim using,
uh, full subjective bays.

And then it was, uh.

Mainly a defective outcome for the main
analysis, using things like the COX

model for time until recovery, which
recovery is not even a proper outcome in

the clinical sense of the work because
we have patients to un recover and

the UN recovery gets totally ignored.

So that's why I've been pushing
longitudinal outcomes to the degree.

Scott Berry: Yeah.

Yeah.

So, um, and, and we found ourselves in the
same situation where we may design a trial

that uses Bayesian machinery subjective
for decisions of continuing or not.

And when it gets to the end, it
does a traditional analysis and, um.

You know, a lot of it is the, the
restrictions built around us, whether

it's regulators, whether it's NIH,
whether it's, uh, some other restrictions.

And so we are, we're,
we're getting closer now.

Your, your view of that, so you,
you've done the same where you've,

in order to meet, re you know, to
make progress, you've gone less

than what you would like it to be.

Have you run, do you remember your first
trial that maybe was just full base.

The whole thing was just full base.

Frank Harrell: Uh, I think it
was with the, um, orbital group

at Imperial College London.

So there's this cardiovascular group
that is true state-of-the-art thinkers.

Who have no obligation to the past.

They embrace change.

They embrace innovation and,
um, Russia, alami, and, um.

Uh, Matthew Hin and others in
that group, um, they have adopted

longitudinal ordinal to the hilt.

They've adopted pure bays and my
most thrilling, thrilling moment,

um, and at some other point, I'll
tell you my lowest point in dealing

with New England Journal of Medicine.

My most thrilling moment was dealing
with Lancet when we submitted

an all Bayesian clinical trial.

But we did have one secondary.

Outcome that was analyzed with
frequentist methods and the reviewer for

Lancet said, this is a Bayesian study.

Uh, please remove that frequentist
analysis from the paper.

So then I knew that things were possible.

And, um, since then, Lancet's been sending
me some Bayesian studies to review, which

I'm really glad to return the favor.

Scott Berry: Yeah.

And the New England Journal of Medicine
does the same now where we've submitted

them, where secondaries were going
to be frequentist, but the primary

is Bayesian, where they want it all.

Bayesian,

Frank Harrell: I'm amazed at that.

I'm thrilled to hear that

Scott Berry: Yeah.

Um, let me go back to, to DCRI.

So you must have cardiovascular
DC you must have OOO overlapped

and worked with Rob Kali.

Frank Harrell: Well, he and I have
been collaborating since 1980 when

he was a cardiovascular fellow.

Uh, just having returned from
his residency at University

of California San Francisco.

So we've, we've been communicating
off and on since 1980 and written

a slew of papers together.

And when he wanted to form the
D-C-I-D-C-R-I, my, my experience in

clinical trials had been working as
a graduate student at UNC on the most

expensive clinical trial ever done,
which was the lipids research clinics

program, cholesterol reduction trial.

Working in the coordinating center and I
thought all the work was boring and it was

very, uh, not embracing any new designs.

Uh, I was kind of bored in that and I
told Rob, don't get into clinical trials.

It's boring.

So that was, that was because of
my, my exposure and it was the most

ridiculous advice I ever gave anyone.

And he totally ignored it, thank goodness.

But.

Uh, I've, I've worked under him at two
stents at FDA when, when Vanderbilt,

uh, puts me on detail, loaning me out
to FDA, so I got to work for him twice.

And also a thrill to work for Alyssa l

uh, at FDA, but working with Robert,
FDA was a special thrill, and, and I, I

never dreamed he would get fired twice
and how political things are, but when

he was there, it was just a total joy.

Scott Berry: Yeah.

Yeah.

So, uh, he's, he's not a statistician,
but is he, do you think he thinks Bayesian

and is receptive of Bayesian ideas?

Frank Harrell: He thinks, I
would say fairly Bayesian.

But the thing that distinguishes
Rob is that he always believes in

division of intellectual labor.

And so he says, he said,
I'm a cardiologist.

I'm gonna do my job.

Um, you are a statistician.

You're gonna do your job, and he has
just this infinite respect for people

of all the different disciplines.

And he also has this South Carolina
way about him such that there

are people I work with that when
they praise me, I feel like dirt.

And when Rob told me to go to
hell, I would say, you know, by

which route do you want me to take?

And so he, even when he's telling
you something you don't want to

hear, you feel really good about it.

There's an art to that, that I
could never achieve what he did,

but it is just the way he is.

Scott Berry: Yeah, I,
a huge respect for him.

I listened to him many times when he
was FDA commissioner, talk about things.

I thought he was fantastic.

Absolutely fantastic.

Um, okay, so now you're,
you are, so what is this?

You're a special advisor
for the FDA, what is that?

The title?

Frank Harrell: Yeah, I'm, I'm
now in my, uh, seventh year.

I actually, in my start of my eighth year
of doing this, I did four years as a,

um, what they call IPA or inter-agency
personnel agreement where you're.

You're paid by Vanderbilt, but
FDA pays part of your salary.

And, uh, I did that for the Office
of Biostatistics in center for

Drug Evaluation research for
four years, and then they make

you take a two year holiday.

Can't keep doing that continuously.

And then I became, um.

A little over three years ago, uh,
attached to CR uh, the, uh, director's

office and, um, instead of the Office
of Biostatistics, but I work extremely

heavily with the Office of Biostatistics
and, and with a lot of rare disease

initiatives from the overall Center
for drug Evaluation and research.

So that says a special
biostatistics advisor.

Scott Berry: Yeah, so it, it's
interesting, I mean the FDA,

uh, and it's hard to attach
any single thing to the FDA.

It's a big, big organization,
but over the last 20 years, have.

Been much more receptive to Bayesian
things, uh, doing Bayesian things, I think

on the world stage in comparison to other
regions, uh, are innovating in that space.

But relative to Frank Harrell, they,
they, they're probably not quite, uh,

uh, where you think Bayesian should
be clinical trials and all of this.

Uh, but it must be fascinating to,
to work with the, the FDA that.

Still has great amount of, of
frequentist flavor, two adequate

and well controlled trials.

We generally think of this as
5% errors and things like that.

How, how is that going at the FDA
and, and you don't have to name names

or tell stories, but do you feel
like you're able to have an impact?

There's reception at the FDA.

Frank Harrell: S there's so
many good things happening.

It's really exciting.

Uh, there's a story to be told for
another day about the bravery and the

sticktoitiveness of the FDA statisticians
in a political environment of great

upheaval and uncertainty and fear.

Uh, a ton of great people are still there
and they're working really, really hard.

There's multiple committees.

Uh, in the office of biostatistics
that I'm taking part in, and these

are just thrilling things to watch.

There's, there's a group dealing with
multiplicity, uh, guidance, um, uh,

supporting documents for how to, for bays.

Um, and it, it is just so much going
on and the dedication and the talent

level of the FDA statisticians
is really something to watch.

Uh, now not all of the statisticians
are completely supportive of Bay.

There are some that are motivated by other
things that I'm not totally understanding.

Uh, these are smart people, but they
are so much, uh, reliant on, uh, alpha.

It's really hard to shake them.

And so, um.

I think they'll eventually come
around, but I think many of the

statisticians are seeing, uh, a a and
we're trying to, I'm hoping someday

this is perfectly well defined.

The idea that there are Bayesian
operating characteristics that don't

have anything to do whatsoever with
as operating characteristics, and

that's a really strong idea to me that.

Could have a lot of impact in the future.

And so just getting people to think that
there's other design characteristics

that you would want to satisfy.

And in many cases it's better to
satisfy them than it is to satisfy,

satisfy the classical conditions, uh,
because they're more concordant with

how decisions are actually made and.

A really key thing that I'm
pushing hard at every instance is

they don't involve unobservables.

So I'm trying, in all of my
teaching, I'm trying to educate

people about, uh, you know, what,
what sort of things are observable

and what sort of things are not.

And if you start hanging your hat
on unobservables, there's typically

problems just waiting to happen.

Scott Berry: Yeah.

Yeah.

Um, so I is, is this the traditional
story that a p value analyzes the

probability of a range of things, many
of which didn't actually happen, um,

where a Bayesian conditions on what
happens makes inferences based on that.

Much of it's similar stories to
this, the likelihood principle.

You know, unbiased ness is even a
weird sort of category that involves

unobserved things within this,
these, these types of traditional

statistical, uh, uh, discussions.

Frank Harrell: It.

It's all of those things,
but the central thing is the

conditioning on a no hypothesis.

Scott Berry: Yeah.

Frank Harrell: So that leads to a
lot of, um, reliance on what, what

might have happened, sort of things.

And every time you condition on
something that it's impossible to know.

You're gonna be looking at the world
through a very strange lens, and if

you ask how often that occurs, outside
of no hypothesis testing, look at how

people make decisions in business,
in sports betting and whether to

cross a busy street as a pedestrian.

Uh, I don't think you'll find any other
field other than classical statistics.

Where people are trying to evaluate
the worthiness of a judge or a decision

maker by something that's conditional on
something, it's conditional on something

that the decision maker could never know.

It's just not done.

It's not done in the real world, and
it's only done in the small area.

So, uh, educating people about
what alpha actually is, is

the first order of business.

And.

Uh, I think it's time for us to take
off the gloves because, um, and,

and, and I get, I get very nitpicky
about this, but when, when Alpha was

invented, people, somebody, I don't
know if it's Fisher or who used the term

type 1 error, and it's not an error.

It's, uh, it's a.

Trigger, it's an assertion
trigger probability, and that

has nothing to do with an error.

So the, the rest of the world, when you
use the word error, they think of one

word comes to mind, which is mistake.

So if, if I wanna get a prob of error,
it's a probably making a mistake.

And so Alpha has nothing to do with that.

So when you talk about if
somebody makes a mistake.

The way that you would calculate
their probability of making a

mistake is you would not make use
of any conditioning about the truth

in calculating that probability.

So the probability of making a
mistake is strictly an unconditional

probability, except it's conditional
on what information they had

when they made the mistake.

And so when you condition on.

What's observable there?

You're asking to judge their accuracy
in their decision and the accuracy

of the decision can't condition on
what they're trying to find out.

It's just not the way anybody
would look at accuracy.

How do you judge who wins?

Uh, uh, you know, a, a bet on a horse race
or anything you can name, um, you don't

condition on, on something like that.

And so getting people to realize that
Type 1 alpha is not a probability of

making an error is the first step.

In other words, you could say that all
along before we even had our Bayesian

push in modern times, all along
the idea of Alpha was misconstrued.

It was not something that I
would say is a noble cause.

It's not a noble goal.

It's a convenient goal when you didn't
have computers and you could do stuff.

You needed to do something with
pencil and paper because assuming

HO makes the model so simple that in
some cases you can calculate P-values

by hand, that it was an expedient
thing, and so it was never noble.

It was never consistent with
how judges need to be judged or

decision makers needed to be judged.

Scott Berry: Yeah.

Yep.

Um, and now, um, so as, so you've.

You think your work at the
FDA it's having impact.

And by the way, I agree with you.

Um, the FDA does a phenomenal job and I
think statistics has actually moved quite

a long ways from Lisa's time forward.

Uh, in terms of it, the, um, uh,
we do have a guidance coming out.

I know John Scott talked about it.

At JSM, it's not out, so people can't
talk about, but the expectation is

this coming out highlights for this.

This has to be pretty exciting for you.

Frank Harrell: This is the
single most exciting thing.

So this is a dream of my career,
uh, to be involved in helping

with that guidance document.

I have to mention one
other important thing.

Jack Lee, the great Bayesian
statistician at MD Anderson Cancer

Center was involved in this too.

So they had two, two IPAs.

He, he did an IPA that just ended.

To the office of Biostatistics and I
was doing it to the Cedar Home Office.

We were both working on this so
to have to have, be able to work

with Jack Lee on this and for the
committee to hear from both of us.

Um, and we really agreed on everything.

And so we sort of reinforced and
backed up each other just by accident.

Uh, you know, that was a thrill, but
this, uh, being able to have input even

if, even if not all the input is accepted
and we don't know what the final document

will look like, you know, to have the
opportunity for input and suggesting edits

and multiple rounds of things, it was a,
was a goal I'd had for many, many years.

Scott Berry: Yeah, that is fantastic.

That is fantastic.

Um, and.

Do we know a sneak peek
of timing for this?

Maybe that's a hard

Frank Harrell: It.

I think the earliest it'll come
out for comment is September.

Uh, that's this month, isn't it?

I forgot.

We just, it may be this month.

Scott Berry: Yeah.

Okay.

Yep.

Frank Harrell: I, I hope that was
the original timetable and no one

has said anything to contradict that.

So that's what the hope is.

And I, I can say that the last
version I've seen, I'm very,

uh, very thrilled with it.

It it, it has something for everyone.

And of course, I don't speak for
FDA in any way, shape, and or form

for anything I'm saying today.

But I think it's a very balanced, uh,
way to look at it, and it has a lot of,

um, operationalizable, uh, aspects to

Scott Berry: Mm, that is awesome.

I'm, I'm excited to, to get there.

Oh, oh.

So you are also in, within the statistics
community, one of the more social

media, uh, active statisticians out
there, whether it's Twitter, x blue

sky, you have a really nice website
with a great deal of material and blogs.

Um, and now you're joining
a podcast, uh, a new thing.

Um, I, I, this has been a, a positive
thing in terms of making an impact.

I imagine your social media presence.

Do you ever get overly
frustrated with social media?

I imagine, but this is still
something that you do quite a bit.

Frank Harrell: Yeah.

Once I quit engaging with people in a.

Emotional way, which is the tendency,
uh, everything started working better.

It's been, it's had a huge impact for
me and I never would've dreamed it.

Even just one little example that
just reading and, and subscribing to

a good many people on X and blue sky.

The amount of stuff that I've
learned, uh, just by watching certain

people and then someone who will
post, okay, I have a new course.

Here's the handouts for it.

Um, and here is a new white paper about
this problem in Bayesian computation.

Whatever it is, the amount of
things people alert me to, uh, is

worth the, worth it all by itself.

But for me personally, was
just as big or bigger, is that

I, I got the urge to have a.

Um, blog and I put out a blog and I
got like a hundred people look at it,

and I, I contract contacted Andrew
Gelman and said, you know, I'd really

like to have more, uh, exposure.

And so he mentioned my blog on his blog,
and then I got a thousand hits that day.

Scott Berry: Right.

Frank Harrell: And then, then I discovered
that really it's all about triangulation.

So the, the reason I got into
Twitter, which I, I swore I would

never do, I still have never touched
Facebook, but that's another issue.

But, uh, or Instagram, but I
got into Twitter to triangulate.

And so what, what, um, x and blue sky
are really good is pointing people

to your more permanent, uh, things,
you know, your, your white papers.

Blog articles, opinions,
and uh, uh, course notes.

And so I find the social media
is just fantastic for pointing

people, uh, to the blogs and then
also for interacting with people.

And people will say, you know, you
missed something, and then I'll, maybe,

I'll maybe rewrite part of the blog.

Um, and then we created at Vanderbilt
a more in-depth discussion that's

modeled after the stand discourse.

Uh, which is called data methods to.org.

And in data methods at org, I get into,
and we get into extremely long discussion.

So there's, there's a very, uh,
emotional long discussion now about

whether frequentness methods have
any role in observational data

analysis, and we're sort of concluded.

And it does not, it doesn't, it doesn't,
it's not consistent with the sampling.

Uh, envisioning samples like
you need to envision and that

frequentness tests really need.

But, um, I, I love these discussion
boards and data methods org.

We've had discussions that
have like 350 replies.

Um, you get into like things like
odds ratios versus risk differences.

It's amazing who comes out to debate that.

It's incredible.

But I found that all of these, um, uh.

Social media opportunities are really,
um, uh, paying off for me because they

educate me and it gives me an opportunity
because at the heart of me, I'm a teacher.

So that's, that's what's
underneath everything else.

So when you're in social media
and you think of yourself as a

teacher, your classroom just gets.

Scott Berry: Yeah.

Yeah.

That's awesome.

That's awesome.

Okay, so we, uh, and it's interesting,
we, within, within the neighborhood of, of

statisticians, we, we, we think Bayesian,
we go about designing Bayesian trials.

Um, I, I always look up to you as
somebody who is very strongly in,

uh, Bayesian, um, thinks that way.

There have been times
where we may be at odds.

Um, now.

Compared to what, um, sort of thing.

Yeah.

You sort of thing.

But, uh, which is all part of
this learning and, and moving

forward and doing better.

So, you know us that we do
a number of these trials and

we simulate type one error.

So we design Bayesian trials and it
might even be, the trial is entirely

Bayesian, the final analysis of
Bayesian posterior probability.

The trials Bayesian, but we can simulate.

From a conditioning on a null
hypothesis that we've already been

through, uh, what you think of that.

But, uh, it traditionally, um, these
are addressed and we can simulate

the probability that that trial, if
repeated many times, would end up.

Determining a treatment that
has no benefit exactly equal to

placebo, say results in a a positive
trial, we can simulate other

characteristics, really simulate
frequentist characteristics of that.

Now we do that quite a bit
in presenting to the FDA,

presenting to other organizations.

We don't do that in every trial.

Our remap cap trial didn't do it,
uh, in, in all circumstances, but

you would be critical of that act.

Of a Bayesian trial.

Frank Harrell: Yeah, I think
once you give people Alpha,

all kinds of trouble happens.

And I have a blog article that if you
haven't seen it, I'm sure you'll find

it controversial because it's about.

Consulting I did with your company
many years ago where the simulations

caused major harm to the Bayesian
procedure, uh, because it created

a decision rule where the posterior
probability was, was only 0.975.

The decision rule was required a 0.986

to preserve alpha,

and so the study didn't meet its target
and luckily the advisory committee.

Overruled that and used a secondary
endpoint, which is emergency room

visits and acute asthma attacks.

But it is just an example where just
the act of doing that simulation

can cause untold damage to the
Bayesian decision making advantages.

And so once you give in and you
do alpha, you're inviting the

consumers of that output to, to.

Create a hybrid procedure that's very
hard to understand and it's not going

to result in optimum decisions, and
it's especially gonna give the wrong

correction for sequential testing.

It's gonna make sequential testing
far, far, far, far more conservative

than it needs to be, and losing
opportunities for really nice

futility, analysis and whatever else.

So this is the only place
you and I differ, I think.

And, and so every time we
clash, which is not often, it's,

it's on a really great topic.

So I I, I've written a lot
about how considering alpha.

Uh, you know, just, we
shouldn't call it an error.

It shouldn't be blessed.

It's at odds with base, and I think it's
time for bayesians to quit apologizing.

And so my, my message to the listeners is.

Uh, the reason you choose to be a
Bayesian is because you're interested

in different kinds of probabilities than
classical statistics is interested in.

You're not interested in
probabilities about data.

You're interested in
probabilities about unknowns.

'cause the whole thing here, just like
when you put a bet on a horse, you know,

you don't know if it's gonna win or not.

You're trying to predict whether,
when that sets the odds and, um.

Just the idea of, um,

dealing with observables and
dealing with the probabilities of

interest, which is always moving
from the known to the unknown.

It's always predictive.

Predictions are all about predicting
what hasn't happened yet or

what hasn't been revealed, like
a diagnosis of ovarian cancer.

It's a prediction problem, and so it's
conditional on what has been revealed and.

We just have to quit apologizing for that.

So if you want to be a Bayesian,
I think she'd be a Bayesian.

If somebody says, uh, I really want to
be also guided by Alpha, my main response

I would hope to give them without me
being on the front lines as you are is

that, uh, you know, you really have just
told me you don't want a Bayesian trial.

And so you're gonna jettison a
lot of the advantages of base.

You're gonna create tremendous complexity.

You're gonna create an
interpretation nightmare.

You don't want base.

I'm gonna advise you to save a lot of
money and do something the old way.

I.

Scott Berry: Okay.

And contrasting that with.

We, we, we may be able to build
a better trial doing that.

And by the way, the, the asthmatics
trials be really interesting to go back

if somebody wants to go look at that.

What was fascinating, it was actually
more complicated and more, um,

uh, on your side of that argument
that the adjustment, the 0.986,

they enrolled so fast, they never

actually did the interims,

so.

Frank Harrell: exactly right.

Scott Berry: Yeah.

So it was

alpha spent.

Yeah.

And it, and it, it kicked the
perfect field goal in there.

And you're right, the, the
emergency room visits and the other

endpoints were, were, were highly,
highly impactful at the advisory

committee meeting and all of that.

You gave examples of.

Cardiovascular trials were, or, or where
you're able to build Bayesian trials

at the end that did a Cox model in,
in all of that where you think you can

build something better than going back
and doing the old way and you're still,

you're moving the Bayesian field forward.

You're moving it forward with
better analyses and better things,

and you're trying to satisfy
those that want to see an alpha.

Um, now.

You're not advertising,
you may not be proud of it.

You know, that sort of thing.

And that's this interesting thing that
I always interested in your viewpoints

on that, uh, in that part of it.

But still feel like you're
designing a better trial.

Frank Harrell: And, and
where you can really unleash.

Uh, the power of Bayes is in two
directions, one of which you spend most

of your career, which is adaptive trials.

And the other one is, is non-adaptive
trials that are highly sequential.

And so I've been putting most of
my effort into, since I'm not.

Any expert on adaptive trials.

Most of my effort in terms of showing
advantages of highly sequential

trials with no limits on the number
of data looks, and that's where Bayes

has a very, very distinct advantage.

And then I also put a lot of emphasis
on what are the intervals that you want

to calculate posterior probabilities of?

And if you make those intervals,
bring into the picture

the clinical significance.

You actually can have futility
stopping that is far earlier

than people would ever dream.

And I think this is in, in
all fields, this is important.

It's even an ethical issue.

But in rare diseases, my estimate
is that of all the studies that end

without a positive result, which is
the vast majority, uh, they could have

had the same conclusion with only one
third of the duration of the trial.

Scott Berry: Yeah,

Frank Harrell: They could have stopped
on the average, about a third of the way

through, uh, with a Bay Ins sequential,
uh, uh, analysis that includes a threshold

for triviality of treatment effect.

So I'm spending a lot of effort on, on
having better clinically relevant decision

rules and with bays and thresholds and
that sort of thing, um, but as well

as really pushing the idea of, of.

Whenever we have a sample size,
we're usually, we're almost

always making it up anyway.

Scott Berry: Yeah.

Yeah.

Uh, a a another topic that we
have, uh, uh, have, have done a

good bit of researching in is, and
you've recently come out with, I'll

call it daily ordinal modeling.

It could.

Uh, time interval weekly, but that
an individual, a patient has a status

on a particular day that's ordinal.

One of five states, say one is a good
state, five's a bad state, and then the

next day they're an an ordinal value.

The next day they're ordinal value
and you're modeling that across

the, the length of the follow up.

Uh, hopefully eventually they move
on and they're outta the hospital.

On the other end, they could.

They could potentially die in
that, but you're modeling that with

proportional modeling, uh, which
was very relevant in COVID for a

number of trials and all of that.

We found it very valuable.

I'm interested in that in we have
found some cases where we actually

haven't liked the outcome of it,
and in part it's not the model.

But it's the, it's the
ordinal states themselves.

If you end up with somewhat less
clinically relevant parts, you

break up one into one A, B, C,
and D, those alterations can

really drive the power of it.

So it's still an approach that's driven by
a statistical assumption of proportional

effect across days and states, uh, which
I know you can investigate and all that.

I'm surprised that you as a Bayesian
and a, a, a subjective, have

you spent time trying to attach
relative utilities of those states?

Where I struggle with ordinal outcomes,
yes, this, you know, three is worse

than two, but it might be that four
is much worse than three, relative

to three to two, and two to one.

But yet the proportionality
doesn't reflect that.

Have has that bothered you As
a Bayesian and the modeling is

fantastic and we use it, but we've
run into that issue at times.

Frank Harrell: Yeah, that that particular
thing doesn't bother me in the least.

And the example you gave where
you have some fine grades and

those little variations and those
might drive the result, that's

exactly what you want to happen.

Uh, you want to get maximum statistical
information that doesn't tell you

how to interpret it clinically.

So the analogy is when you're
doing a blood pressure study, you

never round blood pressure to the
nearest five millimeters of mercury.

You use the, uh, use it in as an integer,
non rounded variable, and you're gonna

find changes of one millimeter mercury
that are not clinically relevant.

But, uh.

Those, those little bits of information
do add enough statistical power,

uh, to be worth it and to avoid any
arbitrariness of lumping categories.

And so, but the clinical imp
interpretation comes at the

end, like, what's the reduction
in mean blood pressure?

What's the posterior mean?

Delta?

That's where you put your,
your, your clinician hat on to

look at the clinical relevance.

So the way I think about it, in particular
for the, the ordinal longitudinal data,

which we find the markup model is.

Is the best fitting in terms of the
correlation structure most of the time.

And it's so easy to program
that with an ordinal model.

Uh, is that?

You're, you ultimately would like
to have patient utilities, and we

find that there's so little research
going on to elicit utilities.

It's really embarrassing the clinical
trial profession has dropped the

ball, except in certain cancer
areas where you're trading off

toxicities and stuff like that.

But in general, I find that utilities are
all important and almost never elicited.

So I think of the ordinal variable as.

Uh, an approximation to what
you would need to be doing.

And the question is, what about what you
mentioned with these little differences?

Well, when you're asking which
treatment is better, uh, you don't

want to, uh, combine any categories
you want to do, like the Wil coxin

sort of idea, concordance probability
or so-called probability index.

Which treatment is better
than than the other?

And if you had non-proportional
odds, the proportional odds model

still tells you the right answer
about which treatment is better.

Uh, so then the question is, is
it enough better to be worth it?

And so.

The only time where the spacings of
the categories are relevant are, are

definitely relevant is if you want to
calculate a mean or an expected utility.

If you wanna calculate a median
or the probability of having

an outcome greater than 10.

Uh, or you wanna calculate a concordance
probability, like vo coxin sort of thing.

Uh, the spacings are irrelevant, but if
you want to calculate, uh, a mean, uh, to

bring in something on a, on your outcome
scale that might be clinically relevant,

the spacings come back in to haunt you.

And if you wanted to calculate the
treatment, has the best expected utility.

You can't analyze utilities
with an ordinary linear model.

They have a distribution that
doesn't behave correctly for that,

so you have to model utilities with
an ordinal model to handle bimodal

distributions, floor and ceiling effects.

But once you get your ordinal model,
now you have order categories and

you assign the utilities to those,
and then your ordinal model will

give you the expected utilities.

And your Bayesian result would say,
what's the probability that the expected

utility for treatment B is more than 0.5

units better than the expected
utility for, for treatment A.

So the utilities come back in at
the scoring point, um, just as if

you were calculating a mean, uh,
which needs more than ordinal to.

Carry it off.

So I, I put all, I intertwine all these
things and I really agree with your

viewpoint about utilities and I think
that until we get the utilities, the

ordinal analysis is the approximation
That is the best we can do at the moment.

Scott Berry: Oh, okay.

Wonderful, wonderful.

I agree with that.

Alright, Frank.

Uh, one thing about in being in the
interim is it doesn't last very long.

Uh, we're very quick doing interim
analysis, so, uh, I very much

appreciate, appreciate you joining
us here in the interim is fabulous.

Uh, love to have you back on maybe
after the guidance comes out and

talk about the FDA Bayesian guidance,

uh, and your thoughts on That

Frank Harrell: would love it.

Scott Berry: Fantastic.

Well, thank you very much Frank.

Appreciate you joining us.

Frank Harrell: Scott, thank you for
having me and thanks for the stimulating

conversation that you generated.

Scott Berry: I enjoyed it.

Thank you.

Frank Harrell: Alright, take care.