1. art and entertainment
2. visual art and design
3. design

Why the MRC-BSU is a great place to
learn your trade!
Richard Nixon
Advanced Quantitative Sciences | Economic Modeling
David Ohlssen
Integrated Information Science | Statistical Methodology
Medical Research Council Conference on Biostatistics in
celebration of the MRC Biostatistics Unit's Centenary Year
Our backgrounds
From MRC-BSU to Novartis
Richard Nixon
David Ohlssen
 MRC-BSU: Ph.D. with Stephen
 MRC-BSU: Ph.D. with Linda
Duffy (3 years)
• Applying Bayesian Random
effects models to cluster
randomized trials and evidence
synthesis
 MRC-BSU: Post doc with Simon
Thompson (6 years)
• Statistical issues in health
economic assessment
 Novartis: Economic modeling (7
years)
• Decision Analysis
• Health Economics
• Structured Benefit-risk
Sharples (3 years)
• Applying Bayesian methods to
assessment of healthcare
providers
 MRC-BSU: Post doc with David
Spiegelhalter (3 years)
• Bayesian model diagnostics for
DAGs
 Novartis: Statistical methodology (7
years)
• Trial design (Bayesian and
frequentist)
• Dose response modeling
• Advisory committee preparation
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Role and impact of biostatistics in drug development
Shaping and influencing decision-making
 A successful statistician working in drug development is a
translational scientist who
• Understands a problem
• Translates it to a model
• Communicates back the results
 These skills go beyond the technical skills
• Ability to think strategically
• Ability to ask the right questions, probe, challenge, listen
• Ability to influence decision-making of partners
• Ability to work in a cross-functional team
• Ability to communicate concisely internally and externally
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Our time and the MRC-BSU gave us in these skills
The BSU sits at the sweet spot between methodology and application
 Generic methodology comes about from real world
examples
• Reoccurring issues are discovered when working on real problems
• Robust statistical methodology developed to address real problems
 Many of the translational modeler skills, needed to work
on real world application, are tacit
• We learned from season experts how to approach a problem, and
tools to solve it
• Reading a polished solution in a paper hides the process of getting
there
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Using Decision Analysis to choose
a strategy for a trial interim
adaptation
Drug development is complex and quantitative
There are many decisions which statistical methodology can support
 Drug development involves complex, high value decisions
• Decisions are made in the context of uncertainty
• Information comes from many different sources
• A decision making team needs to structure and synthesize all this
information
 People are good at “thinking fast”, and making decisions
in the moment, but there are many physiological pitfalls
when making complex decisions
 Decision Analysis gives a set of tools for structuring and
analyzing decisions
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Components of Decision Quality1
 What we can do?
• Define a set of doable alternatives
Decisions
to be made
 What do we know?
• Parameters describing the process we are modeling
• Observable outcomes
Uncertainties
• Statistically these are random variables
 What do we want?
• Clear values and trade-offs
Values
• Statistically these are utilities
1 Ron Howard. The Foundations of Decision Analysis Revisited, in Advances in Decision Analysis 2007
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Problem statement and action taken
Build a decision analysis model to assess protocol amendment
 Interim data suggests a post-marketing study is under powered.
• Study was powered to demonstrate non-inferiority of new drug to standard-of-care with
regard to an event rate.
• Power calculations assumed the event rate per patient per year is 1.1.
• Blinded interim data suggests event rate will 0.59 at the end of the study. This would
reduce the power of the study to 57%.
 Build a decision model to assess different strategies for a protocol amendment.
• Build a model to predict events from patient characteristics and seasonality.
• Integrate information from interim data, study cost data and market forecasting model.
• Assess the effect of different strategies on values.
Sample size
Exposure time
Flexible exposure
Patient enrichment
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Power
Cost
Time to LPLV
eNPV
Strategies
Decision Analysis
Begin with identifying representative alternative courses of action
Sample size
Exposure
(years)
Exposure
Inclusion criteria
Baseline 1
Baseline 2
Baseline CD
Baseline 4
3500
1
Fixed
50
All
All
All
4000
1.5
Flexible
60
Severe
(coded as 0)
Yes
Yes
4500
2
40
5000
5500
 Pick one option from each column, e.g.
• Increase sample size
• Increase exposure to 1.5 years
• Change inclusion criteria so must be treated with concomitant drug at
baseline
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Probabilistic
dependence
Influence diagram
Information
Decisions
Value
Forecasting
Patient share
increase given
success
Event prediction
model
Interim # events
(season, patient)
Events
Market size
(region)
Combined # events
(season, patient)
Inclusion criteria
(new patients)
Patient share
(region)
New event rate
(season, patient)
Recruitment
Current recruit
rate/site (region)
New recruit rate/site
(region)
Power
Interim person-years
(region, season, pat)
Person-years
Comb person-years
(region, season, pat)
Number of sites
(region)
New recruit rate
(region)
Time LPLV
New person-years
(region, season, pat)
Drop out rate
Total sample size
Exposure time
(all patients)
Cost/patient/month
(region)
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Cost
Incremental
eNPV
Event prediction model
Pooled Phase III data is fitted by random effects Poisson regression
i = patient, j=country, s = season
Number of events
Yis
~
Po(  is )
Season effect
Person-years
Treatment group
indicator
Covariates
m
p
k 1
l 1
log( is )
  i   j ( i )   s    k Zik    l X il  log(Tis )
i
j
~
N (0, 2 )
Patient random effects
~
N (0, c2 )
Country random effects
 Protocol plans to model data with a Negative Binomial distribution
• This can be written as a Poisson distribution with a different mean for each
patient, and a gamma random effect distribution between patients
• The above parameterization is an approximation of this as it uses normal
random effects
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
One way sensitivity analysis
How event rate and power are affected by
Decisions
 Increasing sample size by 70% has much more effect on power than doubling the exposure time
 Enrichment only increases power by a small amount
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Multi way sensitivity analysis
Full sensitivity analysis on the four most influential decisions
 The current design is the top left panel
• The power for a sample size of 3500 and
exposure of 1 year is 57%
 Strategies that bring the power to 80%
Sample
size
Exposure
(years)
Flexible
exposure
Inclusion
criteria
CD
6000
1
No
All
4400
2
No
All
4750
1
Yes
All
5500
1
No
Yes
4100
2
Yes
All
4200
2
No
Yes
3800
2
Yes
Yes
 Other strategies to consider
• Current design
• Stop the study at 2500 patients
• Just increasing exposure to 2 years
• Just using flexible design
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Evaluate each strategy
Value
Which strategies are on the efficient frontier?
 Cost and time to LPLV are show on the axes, and power is given in () brackets
 Consider all the strategies with 80% power
• Those in the red area are dominated by another strategy. i.e. another strategy is either cheaper for the
same time, or quicker for the same cost, or both cheaper and quicker.
 Evaluate the strategies not in the red in terms of eNPV
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Net sales and incremental expected net present
value (eNPV)
Product share over time for the UK
2348 fails
Product share
over time in UK
2348 succeeds
Treated patients
patients over time
for the
UK in UK
Treated
over
time
2200
2000
Product share
Treated patients (1000's)
0.08
1800
0.06
0.04
0.02
1600
0.00
2010
2015
2020
Label change time
2010
2015
Year
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
2020
Year
Sensitivity analysis
How does % increase in net sales given study succeeds (shows noninferiority) affect eNPV
 Incremental eNPV =
(Expected net sales – cost of study) –
net sales if don’t do study.
• If this is positive then the money spent on the
study is more than compensated for in increased
sales.
 The line at the top for a given (relative)
% product share increase is the
optimal strategy
% Product share
increase if
successful
Strategy
0 – 25 %
Stop study
25 – 45%
Flex
>45%
Sample size & flex
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Why the MRC-BSU is a great place to learn your
trade!
1. Working with leading experts
2. Focus on real problems
3. Understanding a simple method before moving to a
more complex one
4. Applying Bayesian statistics
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Exploring treatment effect heterogeneity using
Funnel plots
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Funnel plot by health authority reviewer
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Alternative version
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Prediction from Bayesian models
Thanks to Roland Fisch
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Introduction
Objective and Problem Statement
 Design a study with a control arm / treatment arm(s)
 Use historical control data in design and analysis
 Ideally:  smaller trial comparable to a standard trial
 Used in some of Novartis phase I and II trials
 Design options
• Standard Design:
“n vs. n”
• New Design:
“n*+(n-n*) vs. n” with n* = “prior sample size”
 How can the historical information be quantified?
 How much is it worth?
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
The Meta-Analytic-Predictive Approach
Framework and Notation
Y1
Y2
Y1,..,YH
Historical control data from
H trials
YH
1,…, H
Control “effects” (unknown)
2
?
‘Relationship/Similarity’
(unknown)
no relation… same effects
1
?
*
H
*
Effect in new trial (unknown)
Design objective: [ * | Y1,…,YH ]
Y*
Y*
Data in new study
(yet to be observed)
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Example – meta-analytic predictive approach to form priors
Application
Random-effect meta-analysis
prior information for control group
in new study, corresponding to
prior sample size n*
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Bayesian setup-using historical control data
Meta Analysis of Historical Data
Study Analysis
Drug
Placebo
Observed Control Response
Rates
Prior
Distribution
of Control
Response
Rate
Historical
Trial 1
Observed
Control
data
Prior
Distribution
of drug
response
rate
Observed
Drug
data
Historical
Trial 2
Historical
Trial 3
Historical
Trial 4
Historical
Trial 5
MetaAnalysis
Predictive
Distribution
of Control
Response
Rate in a
New Study
Bayesian
Analysis
Posterior Distribution of
Control Response Rate
Posterior Distribution of
Drug Response Rate
Historical
Trial 6
Historical
Trial 7
Posterior Distribution of
Difference in Response
Historical
Trial 8
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Utilization in a quick kill quick win PoC Design
... ≥ 70%
... ≥ 50%
... ≥ 50%
1st Interim
2nd Interim
Final analysis
Positive PoC if
P(d ≥ 0.2)...
Negative PoC if
P(d < 0.2)...
... ≥ 90%
... ≥ 90%
With N=60, 2:1 Active:Placebo, IA’s after 20 and 40 patients
First interim
Second interim
... > 50%
Final
Overall
power
Stop for
efficacy
Stop for
futility
Stop for
efficacy
Stop for
futility
Claim
efficacy
Fail
0
1.6%
49.0%
1.4%
26.0%
0.2%
21.9%
3.2%
d = 0.2
33.9%
5.1%
27.7%
3.0%
8.8%
21.6%
70.4%
d = 0.5
96.0%
0.0%
4.0%
0.0%
0.0%
0.0%
100.0%
Scenario
d=
With pPlacebo = 0.15, 10000 runs
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
General Background: EMA Concept Paper on
Extrapolation
 EMA produced a “Concept paper on extrapolation of efficacy
and safety in medicine development”:
 A specific focus on Pediatric Investigation Plans :
‘Extrapolation from adults to children is a typical
example ...’
 Bayesian methods mentioned:
• ‘could be supported by 'Bayesian' statistical approaches’
 Alternative Approaches:
- No extrapolation: full development program in the target population.
- Partial extrapolation: reduced study program in target population
depending on magnitude of expected differences and certainty of
assumptions.
- Full extrapolation: some supportive data to validate the extrapolation concept.
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Extrapolation and Validation
Based on ideas from Bayesian model diagnostics
 Extrapolation step
• Posterior predictive distributions (PPD): Predict outcomes of study
regimens matching the ones in the available paediatric studies
 Validation step: Posterior Predictive Check (Full cross
validation)
• “confirm the predicted degree of similarity … in clinical response
(efficacy, …)” (EMA Concept Paper on Extrapolation).
• Overlay PPDs with observed incidence rates
 Priors
 Sensitivity and exploratory analyses
28 | Presentation Title | Presenter Name | Date | Subject | Business Use Only
Adult data Bayesian meta-analytic predictive approach
Model
 Mixed effect logistic regression model
Yi ~ Binomial( Ni , πi )
logit( πi ) = μ +  i + xi β
Study i, Yi = number of events, Ni = number of
patients, πi = event rate
• μ: intercept
•  i ~ N(0, σ2): random study effect
• xi : design matrix (Study level covariates)
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
DAG
Adult data Bayesian meta-analytic predictive approach
σ
n
*
μ
i
x*
yobs
Yrep
β
ni
YH
yi
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
xi
Summary why the MRC-BSU is a great place to
learn your trade!
1. Working with leading experts
2. Focus on real problems
3. Understanding a simple method before moving to a
more complex one
4. Applying Bayesian statistics
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
BACK UP
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Divide and conquer
Break it down, understand it, and put it back together
“The spirit of decision analysis is divide and conquer:
decompose a complex problem into simpler problems,
get one’s thinking straight on these simpler problems,
paste these analyses together with logical glue, and
come out with a program of action for the complex problem”
-- Howard Raiffa
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
One way sensitivity analysis
How time to LPLV and cost are affected by
Decisions
 Increasing sample size has less effect on time to LPLV than increasing exposure, but costs more
 Changing the inclusion criteria to only recruit patients taking concomitant drug has the least
effect on time to LPLV
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014
Multi-way sensitivity analysis
Explore the interactions between the most influential decisions
 One way sensitivity analysis shows that increasing power while minimizing the
effect on time to LPLV and cost is best achieved by
• Increasing the sample size
• Increasing the exposure time
• Using a flexible exposure design
• Enriching by including patients taking concomitant drug at baseline rather than by
selecting patients on other baseline characteristics
 Explore the interactions between these four decisions in a multi-way sensitivity
analysis
| Why the MRC-BSU is a great place to learn your trade! | Richard Nixon & David Ohlssen | March 2014