Episode 14
· 41:09
Judith: Welcome to Berry's In the
Interim podcast, where we explore the
cutting edge of innovative clinical
trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott Berry: All right.
Welcome everybody to, in the Interim,
this is a podcast of Berry Consultants
on all things science of clinical trials,
medical decision making, drug development,
and we, we are statisticians typically
talking about the science of this.
Today we have a really.
Cool topic and a fun one for me
and I know a fun one for my guest.
Uh, for the first time on.
In the interim, my guest is Dr.
Nicholas Berry, who also
happens to be my son.
And the topic for today is Lessons for
drug Developers from the World of Sports.
So we're gonna talk about what clinical
trials drug developers could learn
from examples in the world of sports.
So Nick, welcome to in the Interim.
Nick Berry: Thanks.
Yeah, happy to be here.
Um, yeah, I'm happy to talk about this.
I'm, I'm, I'm glad to be talking
about sports because, you know,
a lot of our relationship when
I was young was based on sports.
You were my coach for, I dunno,
15 years playing baseball.
And a lot of how I started to like
perceive statistics was through
sports, uh, through watching twins
games, through watching baseball,
through watching other sports, and
sort of learning about all of the.
The weird things that would happen
and how a, a skeptical statistician
like you perceives sort of home
run races in the late nineties and
batting titles and things like that.
And so a lot of the way I learned to
infer about statistics came from sports.
So this is a sort of near and
dear topic, and I, I wanted
to be a sports statistician.
For, for a long time, um, sort of when
you were at Texas a and m, you wrote
this column, um, the statistician
reads the sports pages in chants.
And so I would read those, I would
look at those and sort of, uh.
Um, it, it got me interested in
that and I, I tr I sort of searched
out a sports path too, right?
I, I applied for some
internships in sports.
I worked with Hal Stern who wrote a lot
of sports stuff back in the day, and so
I, I, I was on a statistics path for a
while before I veered to clinical trials.
So, uh, this is a perfect sort of
merging of the two worlds for me.
Scott Berry: Yep.
Yep.
Okay.
So, and, and many of those experiences
were my same experiences with
my father as well, who's also a
statistician and, and loves sports.
Uh, and so I.
We're, we're gonna talk about
various concept in sports, and
this is the first of, of, uh, the
first topic we're gonna talk about.
And we have multiple other
topics that I think are really
valuable, uh, for drug developers.
And I think the examples will, will
bring home some of the concepts.
And so that's what we're also
gonna try to make clear here.
Uh, we're not gonna get too
deep in the sports, not too deep
in the drug development, but
make sure we tie 'em together.
So
Nick Berry: So.
Scott Berry: topic is
regression to the me.
And it was talking about the, the
family love of sports and the family,
of course the family love of statistics.
If, if somebody in our family
brings up an interesting thing that
happened in the world and says to my
mother, your grandmother, gee, what,
what, what do you think that is?
She will 90% just say regression to
the mean, not necessarily knowing
what it is, but knows that that's
the answer to most of the questions
are it's regression to the mean.
So, so what is regression to the mean?
So we'll talk about it within sports,
uh, and then we'll talk about how
it, it, it, uh, what, what it means
in the world of clinical trials
and statistics and, and science.
So, my first experience
in regression to the mean.
That I remember, um, it was
the 1977 baseball season.
I was 10 years old.
I'm a little bit older than
Nick, not surprisingly.
Um, and in 1977 being 10, I loved baseball
and I could read the box scores and I
understood how to extrapolate statistics
at one point in the season and say, what
is going to be the end of the season?
Within that.
And the first of those was home runs.
George Foster was on my
favorite team at the time.
The Cincinnati Reds, the, the
Minnesota Twins were terrible
at the time, by the way.
So, uh, the big red machine where,
where the Cincinnati Reds and
George Foster, uh, halfway through
the season had 31 home runs.
And he was leading the league in home
runs and he had 31 home runs, and I
was sophisticated enough to double
that and say, okay, that's a pace of
62 home runs by the end of this season.
At the time, the record was 61.
The famous story of Roger Mars's.
61 home runs he beat Babe Bruce.
Babe Ruth's record of 60 home runs.
But that's the best that had ever
been done in a season before that.
And George Foster was on pace to
break that, and I knew that halfway
through the season, players have.
Half of their, their,
their expected number.
And so was on pace to break it.
And as a 10-year-old, I
thought he was gonna break it.
And I even mentioned this to my
father, who's a statistician, said
he's gonna break Mars's record.
And we made a wager based on
that where bet, whether or
not he would break the record.
And I thought at the time my dad was nuts,
that he would, not only would he make
that bet, but he gave me, um, 54 or more.
He said, I'll bet he doesn't even hit
54 home runs by the end of the season.
at that point, you know, does he
hit another 23 home runs when he
hit 31 halfway through the season?
That seemed like a this, this
was a sucker bet for my father
that of course I'd make that
Nick Berry: Yeah.
Scott Berry: And of
course, he didn't hit 54.
He didn't hit 62.
He hit 52.
is actually very good number.
Um, from his 31, he hit 20 more
home runs, which is on pace.
That 21 is a 42 home run hitter, which
is an incredibly good home run hitter.
So
Nick Berry: Sure.
Scott Berry: did really, really
well, better than than average,
but not the 31 pace in that.
Okay, and this, this
happens baseball season.
After baseball season.
A really fun thing about that season
was my Minnesota twins, rod Caru, was
flirting with batting 400, a batting
average of 400, which is the number
of hits divided by the number of
attempts, and that's easy to project.
It's the batting average and
at, that point in the season,
he was hitting over 500.
And so we might, in this 1977 season
have a 400 hitter and break the
home run record all in one season.
On my two favorite teams at
Nick Berry: He was hitting over 400,
right?
And yeah, which is an astronomical number.
Like this has happened less
than, you know, a handful of
times in history of the game.
Scott Berry: The, the last time
before 1977 that it had happened,
which is still the last time it has
not happened even since 1977, was,
uh, Ted Williams hit 4 0 6 in 1941.
so this has become a, a, a, a
number that baseball fans know
and know this hasn't happened.
And so that's a very rare thing.
And he had the best batting average
in the league at the time, and I
didn't make a wager on this, but he
ended up hitting 3 88 actually, is
one of the highest numbers since 1941.
An incredibly high number
rod crew, a hall of fame.
Baseball player, uh, because
of his batting average.
Incredibly high number.
And we've had several people
since then that have done similar
flirting with that 400 number.
George Brett did.
Uh, actually Lenny Dykstra did Tony
Gwynn, uh, even Joe Mauer flirted
with it in another Minnesota twins,
and nobody has accomplished that.
So this is something we're very
familiar with in sports, uh, uh, of
this kind of phenomenon happening.
So
Nick Berry: So
what does it mean?
Scott Berry: Uh, in, in that
scenario were, was, was, was
Rod Caru, truly a 400 hitter?
was George Foster truly a 62
home run hitter at that point?
Nick Berry: Yeah.
Scott Berry: Uh, and they weren't.
within that scenario, and
they were the extremes.
And so if we were to say how good are
they really, we would never estimate them.
And my father knew that George Foster
was not a 31 home run hitter, which is
why he was very comfortable betting.
He knew he probably wasn't even a.
21 home run or a 23 home run
hitter, which is why he took that
bet that that's an extreme number.
His, his estimate was regressed.
Now, George Foster had previous
performances, but also regressed
towards the middle, uh, in that setting.
Nick Berry: Yeah, I was gonna
say, I'm sure that Don, your
dad looked up, George Foster's.
Previous three years and saw you hit
35 home runs to 40 home runs every
year, and, and, and made a deduction.
Scott Berry: Yep.
So we'll come back to sort of how we
might estimate, um, what they really are.
But this phenomenon
happens in every sport.
It, it happens in hockey,
it happens in baseball.
It happens in basketball.
We have, uh, people in teams.
We have teams halfway through a season
that are on pace to break the, the
record number of wins in basketball.
We've had some, some do that,
uh, and they fall short of that.
Uh, and we always talk about that.
We've actually recently had teams break.
Uh, and we might have a team,
uh, this season, break the record
for most losses in a season.
You know, these extreme sort of things,
uh, that, that we've talked about.
And then it doesn't happen.
And in the sports world, we
hear reasons why that happens.
We hear
Nick Berry: Yeah.
Scott Berry: the, the, the scrutiny
of the media, uh, everybody asking
about it day in and day, day out.
It's really hard to continue such a pace.
Nick Berry: Yeah, they got in their
heads and yeah, the whole, yeah.
Scott Berry: Yep.
let me pro, let me provide another
example of this in a, in a sport, Nick
and I both, uh, love, uh, which is golf.
Uh, we play golf, uh, and if you're
looking on video, we have golf shirts on.
Uh, so, within this we love to play golf.
So let me provide another example.
Moneymaking gig, uh,
for us potentially is.
a golf tournament, and I'm gonna
quote numbers from the 2017 US Open,
but this happens week in, week out.
Uh, very, very similar numbers within it.
The US Open might be a little bit
more extreme because there's more
of a variety of players in the
US Open than, than the run of the
mill Weekly PGGA tour tournament.
So in the 2017 US Open.
If I take the baseline score, and I'm
calling it baseline, but we're gonna
start talking about clinical trials.
But the first day, the golfers shot on
their first day in the US Open, and I'm
gonna break them up into four uh groups.
Uh, four Quantiles, the top
25% of golfers, second 25%
of what they shot on day one.
The third and the fourth, the bottom in
terms of the worst score within that.
So I'm gonna
Nick Berry: Gonna take those.
Scott Berry: and let's talk
about the worst quantile, uh, uh,
at first, uh, quartile, sorry.
The worst quartile of this, the
25% that shot the worst score,
they shot an average of 78.
On day one, US opens traditionally
very, very hard, and this
round was a hard round.
The average was 78.
I'm going to do an
intervention on that group.
gonna go around to each one of them
and I'm gonna think really good
thoughts about them, and I'm gonna
give them encouragement in that.
And so what happened to them
on day two when I intervened
on them is they averaged 3.04
shots better on day two
than they did day one.
They, their change from baseline from day
one to day two was three shots better.
I did that.
I caused them to shoot three shots better.
And is does three shots matter
in a PGA tour tournament?
Three shots around is multimillions of
Nick Berry: Yeah.
Scott Berry: It's the difference
between, uh, uh, being on, uh,
uh, uh, uh, one tour, another tour
winning major golf tournaments.
It's millions of dollars.
It's an enormous number
Nick Berry: Yeah.
Scott Berry: in that.
And think about a four round
golf tournament that's 12 shots.
That if I could bottle that and
have a three shot effect, I'm
making millions, uh, in this.
Okay?
Likewise, the people that shot the
golfers that shot in the top the
best, uh, quartile, they got three
shots worse on day two on average.
I didn't think good shots of them.
And in fact, I thought,
I thought, I thought.
Bad thoughts about them, and they
do three shots worse on day two.
This is repeatable.
This happens every single
week in the PGA tour.
happens all the time,
uh, in this circumstance.
Um, so what does it mean?
did, did, did my thinking about them
affect them in any way, shape or form?
No.
Nick Berry: Yeah.
Scott Berry: not, um, in that,
that doesn't mean other things are,
not understood whether they had an effect,
Nick Berry: Yeah.
The bottom quanti core quartile of people
didn't feel no pressure and go straight at
pins because they had a tough first day.
It wasn't some conscious, concerted
effort by the players to to,
to do better play freer, right?
This is variability, it's
randomness that led to it.
Scott Berry: So, uh,
announcers will tell you that,
Nick Berry: Yeah.
Scott Berry: I, I had a bad day and I
just, I was so super aggressive and I
wasn't worried about my positioning.
And you shoot much better when
you do that, uh, or vice versa.
Now you're near the lead.
You didn't sleep well.
Uh, you kept thinking about
hosting the trophy and, and, and
you shot worse with it, with it.
So people attach reasons to that.
Something that is nothing
other than randomness.
And this is
Nick Berry: Yeah.
Scott Berry: other than randomness.
Nick Berry: And this isn't
just like casual fans.
I mean, the players themselves say, this
announcers that played for 15 years say
this, and this is a, this is a, you know,
people who have spent their life playing
the game still contribute a lot of, you
know, variability to physical aspects
rather than the fact that there is just
a lot of variability in database scores.
Scott Berry: So what is, we have
this term regression to-the-mean
what do we mean by regression-to
the-mean Uh, within a setting like
this, the, there's the average score
on day one, which I think was about
73 and a half, something like that.
Uh, and we have the average of
those golfers, all these golfers are
participants, and the average is that
73 and a half and somebody shoots 78.
On day one, we, and let's ignore previous
tournaments that they come into the US
Open, we're interested in, in, in that.
Um, you want, if you estimate that
that individual's true average score
on that golf course is 78, you are
shocked when you find out the average
of the people in that quartile got
better by three shots on day two.
And there must be a reason for that.
You could think everybody is a 73.5
and all golfers are identical, and
there's no difference between them.
You'd estimate those golfers to shoot 73
and shoot five shots better on day two.
Uh, not all golfers are all the same.
they're different.
Some golfers are better
than the other ones.
We know that, and especially
the US open, there's more
heterogeneity in those players.
There's variability across players.
There's variability on
a day score a setting.
So what does it mean
if somebody shoots 78?
They, they, they, they've
shot a high number.
They're probably not as good
as the average, but they're
not as extreme as the 78.
If we were estimating them statistically,
we would take an average of their 78,
their, their, their score, and the
average of the average of all the golfers.
If this is all we knew, trying to
estimate them, which is 73, and we would
do some estimate midway between that.
Nick Berry: Yeah, and how far
you go from one to the other
depends how well you think.
You actually know how
good golfers are and how.
Maybe if a, if a golf round was a
thousand holes, you think you know
more about the players, so you know the
amount of data you have and the amount
of belief you have and how good the
players are goes into this weighting
of where you are between the two.
Scott Berry: Right.
So in, in only using the 2017 US Open,
Nick Berry: Yeah.
Scott Berry: The key things about
how much regression you would do
towards the mean is the between
subject variability, how variable are
golfers, and then the within subject
variability in a, in an 18 hole score.
And by the way, spent some time in this.
It's, it's a little less than three,
is a standard deviation of within a
professional golfer on any one day.
Uh, within that, the between
variability in this UO US Open
is quite a bit bigger than that.
Um, within the setting, this,
the, the, the, the bottom qua
Quantile was 78, the top was 69.
There's probably a five or six
standard deviation across golfers.
So we would use those, uh, in that setting
now to provide that estimate of that.
So we're not at all surprised
when the next day scores.
For all of them shrink in towards the mean
it's, it's inevitable because the first
day score variability pushes them outward
and their truth is somewhere in the middle
of that variability always goes outward.
That's what variability does.
So we would always say when somebody
does better on day two, they go from 78
to 75, we say, oh, that's regression-to
the-mean Our estimate, if I was betting
on somebody that shot 78 on day one, I
would probably estimate 75 on day two.
And I would bet and I would
behave that way, uh, in the
particular setting like that.
Meanwhile, somebody that shot 67,
I would probably guess 70 or 71.
It's a regression to the mean,
statistically a common way we do this
in Bayesian approach is something
called the hierarchical modeling.
And the hierarchies are,
the players is a hierarchy.
And within players, their different
scores is a hierarchy and that's
used to provide these estimates.
And we do it in sports, all the time.
We also do it in drug development.
Now, the is, this is a natural phenomenon.
this is a natural phenomenon.
If you're rolling dice, this
is a natural phenomenon.
If you're flipping coins, if you're
playing golf, if you're playing
baseball, if teams are having
outcomes, it's pretty well understood.
In sports, it's sometimes understood
in drug development, and this is the
part that drives us nuts, is people
attach to it that are not right,
wrong, then they behave that way.
To it, is, which creates bad decision
making, which, uh, bad betting, bad
decision making, bad paying of baseball
players, expecting them to do the same.
And there are some you may heard of.
So, uh, I, uh, you know, there's something
I don't even know if people talk about
anymore, the Sports Illustrated Jinx.
recognized it that players or people
or teams got on the cover of Sports
Illustrated, a weekly magazine, and what
happened is they tended to do worse and
people referred to it as a jinx of being
Nick Berry: Yeah.
Scott Berry: Illustrate,
Nick Berry: Sports Illustrated
doesn't exist anymore, but it's
now the Madden Jinx or whatever.
You know, the video game that comes out.
You're on the cover of Madden, and
then the next year you do worse.
But it also exists crazy coincidence
that there's also a Madden Jinx,
like a Sports Illustrated jinx.
Scott Berry: And people
believe that's a real thing,
Nick Berry: Yeah.
Scott Berry: is a real thing.
There's, you've heard the sophomore jinx.
People have a great
freshmore freshman season.
They don't do quite as well in their
sophomore year, and it's a jinx.
Nick Berry: Yeah, sophomore slump.
Scott Berry: you know, we don't hear
this anymore because if you have
a great freshman year, you go pro.
Nick Berry: Oh, and yeah.
Scott Berry: there's a
rookie of the ear jinx.
Nick Berry: Yep.
Scott Berry: you go look at players who
win rookie of the year in baseball, in
any sport, they do worse the next year.
They're complacent.
It's a jinx.
Uh, and and it's nothing but
regression to the mean largely.
Nick Berry: overperformed
in their first year.
Scott Berry: yeah.
Yeah.
And if you believe they're
gonna repeat that performance,
that's the mistake you made.
then you think, okay, we have to
attach a, a, a, an effect to this, uh,
Nick Berry: Yeah.
Scott Berry: Um, media pressure
scrutiny, uh, team chemistry.
Uh, the team chemistry
that great was seasoned.
The next, the next year was the chemistry.
Wasn't very good
Nick Berry: Yeah.
Scott Berry: means we, we
don't know how to explain it.
So we, we invent chemistry for it.
This, now this happens
in all kinds of things.
Golf equipment, who goes, you
know, you try a new putter when
you're not playing well, a new
driver and you're doing better.
Nick Berry: Yeah.
Scott Berry: Golf teachers
benefit from this greatly.
Who goes to see a golf professional?
Somebody
Nick Berry: Yeah.
Scott Berry: I've, I,
I'm on that left side.
I'm I, you know, shooting the higher
scores and I play better, and I
want to pay more money to that
golf professional in that setting.
Now, lots of people benefit from this.
Chiropractors benefit, your back hurts.
You go in, doctors benefit, I'm not
feeling well, and then I feel better.
In the setting, and we attribute
it to that rehab therapists.
We, we, we use trinkets, we put magnetic
bracelets on our, on our arms, uh, to
make pain go away and it goes away.
And you know that, this sort of thing.
So it's all kinds of, this, this
shows up in every walk of life.
Nick Berry: Yeah,
Scott Berry: let's the, so let's
Nick Berry: you've been so sorry.
Scott Berry: trials.
Yeah.
Nick Berry: You've been
really pessimistic.
Like everything is fake.
That's not necessarily what you're saying.
You're just saying that every time
you observe something extraordinary,
especially in the circumstance where the
population that underwent this procedure
or something like that was different
than the norm population, but But you're
just saying shrink back the results.
You're not saying that.
It's fake golf teachers don't work.
It's just that if you see immediate
benefits after you go see a golf
teacher, some of that benefit is
due to regression, to the mean.
There are benefits of some of
these things, but believing at face
value, the change you see right
away is naive when we know that
there's going to be some regression.
Yeah,
Scott Berry: Yeah, exactly.
And, and, um, doctors are
phenomenally important.
I'm not saying don't go
Nick Berry: yeah, I know exactly.
Yeah, exactly.
Scott Berry: Uh, your uncle and my brother
is a golf professional, so, uh, they,
they, they absolutely play a role and
they will help you shoot better scores.
No
Nick Berry: Yeah,
but
it's hard to figure out how
much of it goes to each thing.
Like it's really hard to, to
assign how much it should shrink.
Scott Berry: Okay, so let's
talk about this in clinical
trials and drug development
Nick Berry: Yeah.
Scott Berry: and now you can
start to imagine some of these
effects showing up within it.
A, a very, one that I, I think is
very misunderstood is something
called the placebo effect.
And people talk about clinical trials
that in a clinical trial, and when people
say the placebo effect, what they mean is
somebody takes a treatment that is inert.
But they, they're taking a
treatment and the, the act of taking
that makes them perform better.
And you can see where I'm going with this.
I was labeling the, the, my thinking
about golfers in that situation.
As kind of a placebo and the golfer
doesn't know I'm thinking about them
Nick Berry: Yeah.
Scott Berry: and they do better.
So let's take a clinical trial.
And what happens in uh, in most
clinical trials is people who
enter a clinical trial like that
golfer who shot 78, they're doing
poorly, relatively on their scale.
And it could be.
It could be high blood pressure, it
could be high weight, it could be pain.
I'm not sleeping well.
Um, in that scenario, and this happens
in Alzheimer's trials, that people, that
their memory has been particularly bad.
I'm, I'm, I'm, I'm doing worse.
They enter in trials and in many of those
scenarios, patients who get no treatment.
Or an inert treatment, they do better,
and everybody calls this a placebo effect.
Now, in many circumstances,
that's the exact same thing we
Nick Berry: Yeah.
Scott Berry: It's a
regression to the mean.
Now, there are certainly some cases
where I believe that the act of
taking an intervention may help.
you could imagine sleep, you could
imagine potentially pain trials is
where they've talked about it and
there's actually been trials where they
give placebo or they give nothing, and
there's a difference between those.
But most clinical trials,
this is what happens And yeah.
Nick Berry: Placebo effects.
Interesting.
It's like you take statistics in high
school and you don't know anything
about statistics, but they're teaching
you about placebo effects and it's
like a immediate thing, and so you,
you just assign sort of improvement.
Without, uh, getting an active drug to
the placebo effect without assessing
the population or thinking about
sort of the populations going in.
It's just, oh, getting the drug makes
you think you're gonna get better,
so you get better in the placebo.
Getting the placebo makes you think you're
gonna get better, so you get better,
which is, is usually not what's happening.
Scott Berry: And so, um, I, I, the
misunderstanding of this has an effect.
And, and let me give you an example
that happened to me a couple weeks.
A company collected single arm data.
And what that means is they don't have
a placebo, that they run a trial where
everybody gets the treatment and they
have a baseline marker of severity.
And by the way, that's the
outcome measure, but that's
also measured at baseline.
It's similar to their first
round in a golf tournament.
Within the setting and they collected
it across a wide range of patients and
they noticed, wow, patients that have
more higher baseline, they improved
more when we gave them our drug.
Now, now in, in, in the golf example
that's taking that top quant quartile.
noticing they regressed three points in
better, by the way, the people in that
golf tournament in the second to the
worst quartile got better by one shot.
The data from this company looks almost
identical to the golf tournament.
Remember, in a, in a clinical trial, we
record your baseline before you get the
intervention, and what's your score?
There's variability in those.
There's variability across time.
A single patient goes up and
down on almost every measure.
Weight, blood pressure,
uh, uh, cognitive scores.
Uh, this cardiovascular endpoint
they were looking at goes up and
down within a patient over time.
so they're now running a randomized
trial in that particular population.
They're gonna run a randomized
trial, so we get to find out,
do the placebo get better?
And I told them, you're gonna get a
very large placebo effect in your trial.
And they looked at me like.
What are you talking about?
And they, their perception was what the
investigator tells them about how good
the drug is or what we know about it.
All those things are what caused the
placebo effect and not the inclusion
exclusion criteria at the beginning of
the trial, which causes a huge amount of
regression-to-the mean, which we don't
understand and we call the placebo effect.
Nick Berry: Yeah, the placebo
effect's not a psychological.
Thing that messes with patients.
It's a statistical process
that you just described, right?
Yeah.
Scott Berry: regression to the mean
is the statistical process that we
confuse for placebo effect, which
in some cases placebo effect is
Nick Berry: real thing
Scott Berry: said, we don't
understand very well, which is which.
Nick Berry: Yeah.
Scott Berry: we don't understand
very well at all, which is
which, uh, within that setting.
So they made huge drug development
decisions to enroll that particular
population from a single arm trial
without a control, not understanding
that by the way, the people on the
good side might actually be the ones
that have the bigger benefit relative
to a control, and now they're jumping
into a very, very large trial.
Do they understand this particular
effect or are they making a bad decision?
The beauty of it is the randomized
trial is gonna tell them,
Nick Berry: Yeah.
Scott Berry: and we're
gonna figure that out.
But are, do they understand it enough?
Enough to make good
decisions at this point?
Other examples that show up
that regression to the mean
is critically important.
We run a lot of trials and we
look at subgroups of patients.
We have a trial.
We have a trial designed
now called a basket trial.
the baskets in a basket trial is,
they're different kinds of patients.
And a very common one of these is
we have a treatment for, uh, cancer
and we enroll, uh, different.
of cancer.
It might be head and neck cancer,
breast cancer, lung cancer, GI cancer,
uh uh, all of these different types.
Or they may be subsets of kinds of cancer.
And we run trials in them and
we go into eight different types
and we find out, aha, these two
types we had the best effect in.
And.
look at the, uh, estimate from the
two best outta eight, and we run
a trial after that, or we try to
estimate the effect in the eighth.
The best one of the eight and
the data were, um, uh, we, the
number of responses we got.
Suppose we're looking at a cancer
trial and we look at how many patients
responded, and we got 10 out of 20
patients, and the best responded.
And all the other types had
worse responses than 50%.
I, as a statistician, don't believe
the response rate's 50% that one.
It's exactly the same as
the golfer that shut 78.
I don't believe they're 78.
They're better than that.
This 10 out of 20.
I would estimate their response
rate to be closer to the mean
response across all tumor types.
Depending on the variability of that
response and the variability of 10 out of
20, which is statisticians we understand.
Nick Berry: Yeah, In your hierarchy
model, like we talked about with the golf
example now says there is some variability
across the cancer types and we know that
it might actually work better in some,
but there's also a lot of variability
in how many responses you're gonna
observe about 20, uh, on 20 subjects.
And so 20 is not that many.
We don't know a lot about how
well each of these cancers or how
bad each of these cancers are.
So we shrink back and we say a lot of
the variability in this was due to there
only being 20 patients per cancer type.
So just pull everything
back towards the middle.
And if you're predicting what's
gonna happen in phase three,
you're not gonna predict 50%.
And if you do, you're gonna cost yourself,
you know, millions of dollars potentially.
So you just shrink it all the way back.
Really close to that.
Mean probably if you only have 20.
Subjects, you, you probably don't
vary very much from the, the
overall mean of all the types.
Scott Berry: Yep.
Yep.
Uh, we see lots of this
happening where we have units.
Uh, in the case like this, we have
trials with multiple arms, many doses,
um, and the effect on one dose, should
never estimate the effect on one
dose without using the other doses.
be a hierarchical model, it
could be a dose response model.
We have endpoints.
We collect a lot of endpoints in clinical
trials, and we analyze 12 endpoints
in a particular disease where we
think a treatment may have an effect.
this one did the best out of the
12, and this is like the 1977.
Halfway through the
season we see this effect.
Now we're gonna run
the rest of the season.
Do I think that effect in
the 12th endpoint's gonna
continue at the effect I see.
Or is it gonna shrink towards the
other 11 endpoints within that?
It's very similar scenarios to sports
where it seems almost obvious in that
setting, but a lot of these things
aren't naturally done in clinical trials.
Publications provide this single
estimate, and it's up to the consumer
to do that themselves, which.
I think experienced people in this
industry do that, and they understand
that there's this whole d, there's
this whole, uh, I don't, it's almost
controversy about the failure for phase
two trials to replicate in phase three,
and a lot of the things we just talked
about are reasons why that doesn't happen.
But the phase two process itself.
Is relatively small sample
sizes, so the variability in that
measurement, it can be large.
And we run hundreds of phase two trials.
What phase three trials do we run?
Those ones that are doing really well.
And, and there's a natural part to this,
and there's many phase two trials, and
you see the extreme right tail of those
that do well, they run phase three.
Lo and behold, the effect is
not reproduced in phase three.
it's completely gone.
Nick Berry: Yeah.
Scott Berry: sometimes it's just smaller.
It's regressed into the,
the average of the effects.
And people think that this
is a con, controversy.
The irre, we're not, this
isn't reproducible science.
What's wrong with what we're doing?
It's regression to the mean.
Nick Berry: A huge problem with that.
Yeah, yeah, yeah.
Scott Berry: a regression to the mean.
Nick Berry: A huge problem with
that though is that now we're
powering our phase three studies
based on the effect observed on
the best dose in phase two, and.
If we're regressing our estimate, the
truth is probably not as good as that.
You have an underpowered study and
when you inevitably observe, you know,
80% of the phase two effect in phase
three, you have a p value of 0.07
or something like that.
And so it's, it's not just
a, oh, our estimate was off.
I mean, this can have huge
implications in the, this sort
of life cycle of a, of a drug.
Scott Berry: Oh, I, I hope there
were lessons there, but let me
turn it back to you, Nick, because
I know this is a topic we talk
about with a lot of people and, um.
Your friends, uh, heard this topic
and they're in personal finance
Nick Berry: Yeah.
Scott Berry: what does the
regression to the mean mean to them?
So what would you say to, to
people outside of drug development?
Uh, outside of sports,
what does regression to the
Nick Berry: Yeah,
Scott Berry: to somebody who's doing
personal finance and buying stocks?
Nick Berry: yeah.
Yeah.
I think the general lesson that I would
would say to this sort of lay person
is that anytime you observe something.
Extraordinary.
And you get, uh, something that's,
you know, you've never seen before,
like this is an amazing result.
You just smooth over that and you
realize that there were maybe a lot
of opportunities from other places
to observe this amazing result.
There's a lot of stocks that could
have performed really well, and when
you start to, to base decisions off of
recent amazing things that happened.
You are inevitably setting yourself up
to be disappointed if you expect that to
be reproduced and to happen over again.
So when you observe something amazing,
just smooth over it, realize that it
might be good, but it's probably not
actually a world beater in that case.
And so just temper your expectations
every time you, you, you start to
make predictions based on past data.
Scott Berry: So if, if there are 50 stock
brokers in a company and in a particular
year, one person is the best stockbroker
and they earn a per certain percent.
Nick Berry: Yeah.
Scott Berry: best would be.
30, 30% on
Nick Berry: Huge.
Scott Berry: over a course of a year.
And then the next year they don't do 30%.
Nick Berry: Yeah, they fell off.
Yeah.
Scott Berry: they lazy?
They, you know, the,
the media scrutiny got
Nick Berry: Yeah.
Scott Berry: Yeah.
It, that they actually were never
that good and they, the data got
them there through randomness.
Now they may be better than average.
How
Nick Berry: Yep.
Scott Berry: depends on the
variability in stock pickers and the
variability in a particular year.
Nick Berry: Yep.
Scott Berry: the lessons are in
drug development, the lessons are
in sports, in in picking stock.
So this was our lessons learned for drug
developers from the world of Sports.
Episode one we
Nick Berry: Yeah.
We'll see you later.
Scott Berry: Yeah, we have more to come.
So.
Uh, Nick, thanks for joining me.
And
Nick Berry: Thanks for having me.
Scott Berry: we are in the interim.
Listen to In the Interim... using one of many popular podcasting apps or directories.