More parameters to adapt to specific settings

Johannes Cepicka

Suppose we are planning a drug development program testing the superiority of an experimental treatment over a control treatment. Our drug development program consists of an exploratory phase II trial which is, in case of promising results, followed by a confirmatory phase III trial.

The drugdevelopR package enables us to optimally plan such programs using a utility-maximizing approach. To get a brief introduction, we presented a very basic example on how the package works in Introduction to planning phase II and phase III trials with drugdevelopR. However, there are many more parameters, which you can provide to adapt the program to your specific needs. In this article, we therefore want to present the options you have.

The example setting and the input parameters

We are in the same setting as in the introduction, i.e. we suppose we are developing a new tumor treatment, exper. The patient variable that we want to investigate is the difference in tumor width between the one-year visit and baseline. This is a normally distributed outcome variable.

The parameters we insert into the function optimum_normal are the same parameters we also inserted in the basic setting. Thus, for more information on the input values we refer to the introduction. We start by loading the drugdevelopR package.

library(drugdevelopR)
#> Loading required package: doParallel
#> Loading required package: foreach
#> Loading required package: iterators
#> Loading required package: parallel

Back then, we got the following output:

res
#> Optimization result:
#>  Utility: 2946.07
#>  Sample size:
#>    phase II: 92, phase III: 192, total: 284
#>  Probability to go to phase III: 1
#>  Total cost:
#>    phase II: 77, phase III: 158, cost constraint: Inf
#>  Fixed cost:
#>    phase II: 15, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0.675, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.85
#>  Success probability by effect size:
#>    small: 0.72, medium: 0.12, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: 0.06 [Kappa] 
#>  Assumed true effect: 0.625 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma]

We now want to see how different modifications, i.e. using more parameters change the outcome of the program.

Fixed treatment effect estimates or prior distributions

Setting the parameter fixed to be FALSE models the assumed true treatment effect using a prior distribution. This is explained in detail in the article Fixed effect estimates and prior distributions.

Constrained optimization

The are three options how to include constrained optimization, by setting maximum cost limits, by setting a maximum number of participants or by demanding a minimum probability of a successful program. These options can be applied separately or combined, as it is best suited to your specific needs.

Cost constraint – maximum total costs

The total costs are given as sum of the cost in phase II and phase III, i.e. as \(K2\) + \(K3\). In our example we get total costs of \(72+139=211\) (in 10^5 $). Let’s assume an investor in your drug or a pharmaceutical company is only willing to pay 20,000,000 $. You can now set a maximum cost constraint by providing an parameter value of K = 200 into the optimum_normal function. The default value for K is Inf.

 resK <- optimal_normal(Delta1 = 0.625, fixed = TRUE, # treatment effect
                       n2min = 20, n2max = 400, # sample size region
                       stepn2 = 4, # sample size step size
                       kappamin = 0.02, kappamax = 0.2, # threshold region
                       stepkappa = 0.02, # threshold step size
                       c2 = 0.675, c3 = 0.72, # maximal total trial costs
                       c02 = 15, c03 = 20, # maximal per-patient costs
                       b1 = 3000, b2 = 8000, b3 = 10000, # gains for patients
                       alpha = 0.025, # one-sided significance level
                       beta = 0.1, # 1 - power
                       Delta2 = NULL, w = NULL, in1 = NULL, in2 = NULL, 
                       a = NULL,b = NULL, # setting all unneeded parameters to NULL
                       K = 200 # cost constraint
                       ) 

We get the following results:

resK
#> Optimization result:
#>  Utility: 2846.69
#>  Sample size:
#>    phase II: 76, phase III: 158, total: 234
#>  Probability to go to phase III: 0.97
#>  Total cost:
#>    phase II: 66, phase III: 133, cost constraint: 200
#>  Fixed cost:
#>    phase II: 15, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0.675, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.82
#>  Success probability by effect size:
#>    small: 0.7, medium: 0.11, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: 0.18 [Kappa] 
#>  Assumed true effect: 0.625 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma]

We see that due to the cost constraint the cost in phase II decreases from 77 to 66 and the cost in phase III decreases from 158 to 133, so overall the constraint is met. Moreover, the expected utility at the optimum decreases from 294 607 000 to 284 669 000 dollars.

Sample size constraint – maximum number of participants

Very similarly to the cost constraint, we can set a constraint on the maximum number of participants in our trial. Without constraints, we need a total number of 250 participants in the optimum. However, one can certainly imagine a situation, where the only 200 persons willing to participate in a trial, for example because the incidence of the illness is not very high within the target population. To model this, we can easily set a sample size constraint, in this case of N = 200. This allows for an optimal allocation of the participants between phase II and III. The default value for N is Inf.

 resN <- optimal_normal(Delta1 = 0.625, fixed = TRUE, # treatment effect
                       n2min = 20, n2max = 400, # sample size region
                       stepn2 = 4, # sample size step size
                       kappamin = 0.02, kappamax = 0.2, # threshold region
                       stepkappa = 0.02, # threshold step size
                       c2 = 0.675, c3 = 0.72, # maximal total trial costs
                       c02 = 15, c03 = 20, # maximal per-patient costs
                       b1 = 3000, b2 = 8000, b3 = 10000, # gains for patients
                       alpha = 0.025, # significance level
                       beta = 0.1, # 1 - power
                       Delta2 = NULL, w = NULL, in1 = NULL, in2 = NULL, 
                       a = NULL,b = NULL, # setting all unneeded parameters to NULL
                       N = 200 # sample size constraint
                       ) 

We get the following results:

resN
#> Optimization result:
#>  Utility: 2658.9
#>  Sample size:
#>    phase II: 48, phase III: 150, total: 198
#>  Probability to go to phase III: 0.93
#>  Total cost:
#>    phase II: 47, phase III: 127, cost constraint: Inf
#>  Fixed cost:
#>    phase II: 15, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0.675, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.76
#>  Success probability by effect size:
#>    small: 0.65, medium: 0.11, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: 0.2 [Kappa] 
#>  Assumed true effect: 0.625 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma]

Due to our sample size constraint to optimal number of participants in phase II is 48 and 150 in phase III (down from 92 and 192 respectively) within this framework. The expected utility in the optimum decreases to 265 890 000$.

Success probability constraint - minimum probability of a successful program

The last option provided for constraint optimization is the minimum probability of a successful program. In our basic setting without constraints, we had a probability that the program was successful of 0.85, i.e. we had a probability of 0.85 that the program would proceed from phase II to phase III and that a positive treatment effect could be detected in phase III. Let’s say, you are willing to have higher costs and in return want this probability to be 0.87. You can include this within the framework, by including the parameter S and by setting it to S = 0.87. The default value for S is -Inf.

 resS <- optimal_normal(Delta1 = 0.625, fixed = TRUE, # treatment effect
                       n2min = 20, n2max = 400, # sample size region
                       stepn2 = 4, # sample size step size
                       kappamin = 0.02, kappamax = 0.2, # threshold region
                       stepkappa = 0.02, # threshold step size
                       c2 = 0.675, c3 = 0.72, # maximal total trial costs
                       c02 = 15, c03 = 20, # maximal per-patient costs
                       b1 = 3000, b2 = 8000, b3 = 10000, # gains for patients
                       alpha = 0.025, # significance level
                       beta = 0.1, # 1 - power
                       Delta2 = NULL, w = NULL, in1 = NULL, in2 = NULL, 
                       a = NULL,b = NULL, # setting all unneeded parameters to NULL
                       S = 0.87 #minimum success probability
                       ) 

We get the following results:

resS
#> Optimization result:
#>  Utility: 2906.94
#>  Sample size:
#>    phase II: 172, phase III: 138, total: 310
#>  Probability to go to phase III: 1
#>  Total cost:
#>    phase II: 131, phase III: 119, cost constraint: Inf
#>  Fixed cost:
#>    phase II: 15, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0.675, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.87
#>  Success probability by effect size:
#>    small: 0.76, medium: 0.11, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: 0.06 [Kappa] 
#>  Assumed true effect: 0.625 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma]

The probability of a successful program now increased from 0.85 to 0.87, mainly by increasing the number of participants in phase II. The expected utility decreased to 290 694 000 $.

Customized effect size categories

Up until now, the effect size categories for a small, medium or large treatment effect were predefined and set to \(0-0.5\) for a small treatment effect, \(0.5-0.8\) for a medium treatment effect and \(\geq 0.8\) for a large treatment effect (each in standardized differences in mean). These parameter values were taken from Cohen (1988). However, one can easily customize the effect size categories by inserting threshold values for each effect size. Let’s say you only consider a program successful if the treatment effect is above 0.1. Furthermore, you want to assign a medium benefit if the treatment effect is between 0.6 and 1 and a large benefit if the treatment effect is above 1. Note that there is a one-to-one relation between effect size categories and benefit categories, i.e. the large treatment effect corresponds to the large benefit and so on. If you want to implement this scenario you can do this by defining the parameters steps1, stepm1 and stepl1. Here, we set them to steps1 = 0.1, stepm1 = 0.6 and stepl1 = 1.

 res <- optimal_normal(Delta1 = 0.625, fixed = TRUE, # treatment effect
                       n2min = 20, n2max = 400, # sample size region
                       stepn2 = 4, # sample size step size
                       kappamin = 0.02, kappamax = 0.2, # threshold region
                       stepkappa = 0.02, # threshold step size
                       c2 = 0.675, c3 = 0.72, # maximal total trial costs
                       c02 = 15, c03 = 20, # maximal per-patient costs
                       b1 = 3000, b2 = 8000, b3 = 10000, # gains for patients
                       alpha = 0.025, # significance level
                       beta = 0.1, # 1 - power
                       Delta2 = NULL, w = NULL, in1 = NULL, in2 = NULL, 
                       a = NULL,b = NULL, # setting all unneeded parameters to NULL
                       steps1 = 0.1, stepm1 = 0.6, stepl1 = 1 # step sizes for effect size categories
                       ) 

Differences in the treatment effect

Due to different population structures in phase II and phase III, the assumed true treatment effect Delta1 may not be the same in both phases. Phase III trials are often conducted in populations that are more heterogeneous than those of the preceding phase II trials (Kirby et al., 2012), so the effect in phase III may differ from that in phase II. In order to model these differences, one can use the parameter gamma. The treatment effect in phase III is then calculated as follows: \(\Delta_3 = \Delta_2 + \gamma\). Values of gamma below zero indicate a more pessimistic view, values above zero indicate a more optimistic view about the treatment effect in the confirmatory phase. The default value of gamma is zero.

Skipping phase II

If enough information is available, one may decide to skip the phase II trial and move directly to a confirmatory trial. This can be integrated in the framework by using the parameter skipII. If we set the parameter TRUE, the program calculates the expected utility for the case when phase II is skipped and compares it to the situation when phase II is not skipped. When skipping phase II, the assumed treatment effect used for planning the phase III trial is calculated as the median of the prior distribution. The results are then returned as a two-row data frame, res[1, ] being the results when including phase II and res[2, ] when skipping phase II. The default value is skipII = FALSE.

 resII <- optimal_normal(Delta1 = 0.625, fixed = TRUE, # treatment effect
                       n2min = 20, n2max = 400, # sample size region
                       stepn2 = 4, # sample size step size
                       kappamin = 0.02, kappamax = 0.2, # threshold region
                       stepkappa = 0.02, # threshold step size
                       c2 = 0.675, c3 = 0.72, # maximal total trial costs
                       c02 = 15, c03 = 20, # maximal per-patient costs
                       b1 = 3000, b2 = 8000, b3 = 10000, # gains for patients
                       alpha = 0.025, # significance level
                       beta = 0.1, # 1 - power
                       Delta2 = NULL, w = NULL, in1 = NULL, in2 = NULL, 
                       a = NULL,b = NULL, # setting all unneeded parameters to NULL
                       skipII = TRUE #skipping phase II
                       ) 

We get the following results:

resII
#> Optimization result including phase II:
#>  Utility: 2946.07
#>  Sample size:
#>    phase II: 92, phase III: 192, total: 284
#>  Probability to go to phase III: 1
#>  Total cost:
#>    phase II: 77, phase III: 158, cost constraint: Inf
#>  Fixed cost:
#>    phase II: 15, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0.675, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.85
#>  Success probability by effect size:
#>    small: 0.72, medium: 0.12, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: 0.06 [Kappa] 
#>  Assumed true effect: 0.625 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma] 
#> 
#> Result when skipping phase II:
#>  Utility: 3080.46
#>  Sample size:
#>    phase II: 0, phase III: 108, total: 108
#>  Probability to go to phase III: 1
#>  Total cost:
#>    phase II: 0, phase III: 98, cost constraint: Inf
#>  Fixed cost:
#>    phase II: 0, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.9
#>  Success probability by effect size:
#>    small: 0.81, medium: 0.09, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: -Inf [Kappa] 
#>  Assumed true effect: 0.62 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma] 
#> 
#> Skipping phase II is the optimal option with respect to the maximal expected utility.

It can be seen that in this case skipping phase II yields a higher utility than including phase II. Skipping phase II automatically sets n2 = O which leads to costs K2 = 0. Hence, overall costs are smaller in our example. Furthermore, note that the optimal threshold value Kappa is also automatically set to -Inf and the probability to go to phase III pgo = 1, as we go to phase III every time, when skipping phase II.

Parallel computing

As some of the programs may have very long run-times, especial more complex ones, faster computing is enabled by parallel programming. The parameter num_cl allows you to set the number of cores for parallel computing. It is advisable to check the number of cores available by using the function detectCores(). The default value is num_cl = 1. Note that using this parameters does not change the results, but can decrease run-time.

Where to go from here

This tutorial presents more parameters, so you can adapt the program to your specific needs. For more information on how to use the package, see:

• Introduction to drugdevelopR: See how the package works in a basic example.
• Different outcomes: Apply it to binary endpoints and time-to-event endpoints.
• Interpreting the rest of the output: Obtain further details on your drug development program.
• Fixed or prior: Model the assumed treatment effect on a prior distribution.
• Complex drug development programs: Adapt to situations with biased effect estimators, multiple phase III trials, multi-arm trials, or multiple endpoints.

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences.

Kirby, S., Burke, J., Chuang-Stein, C., and Sin, C. (2012). Discounting phase 2 results when planning phase 3 clinical trials. Pharmaceutical Statistics, 11(5):373–385.