Skip to contents

Interpreting the rest of the output

Johannes Cepicka

Source: vignettes/Interpreting_Output.Rmd
Interpreting_Output.Rmd

Suppose we are planning a drug development program testing the superiority of an experimental treatment over a control treatment. Our drug development program consists of an exploratory phase II trial which is, in case of promising results, followed by a confirmatory phase III trial.

The drugdevelopR package enables us to optimally plan such programs using a utility-maximizing approach. To get a brief introduction, we presented a very basic example on how the package works in Introduction to planning phase II and phase III trials with drugdevelopR. As we only discussed the most important results of the function in the introduction, we now want to interpret the rest of the output.

The example setting and the input parameters

We are in the same setting as in the introduction, i.e. we suppose we are developing a new tumor treatment, exper. The patient variable that we want to investigate is the difference in tumor width between the one-year visit and baseline. This is a normally distributed outcome variable.

The parameters we insert into the function optimum_normal are the same parameters we also inserted in the basic setting. Thus, for more information on the input values we refer to the Introduction. We start by loading the drugdevelopR package.

library(drugdevelopR)
#> Loading required package: doParallel
#> Loading required package: foreach
#> Loading required package: iterators
#> Loading required package: parallel

Then we insert the input values into the function optimal_normal:

 res <- optimal_normal(Delta1 = 0.625, fixed = TRUE, # treatment effect
                       n2min = 20, n2max = 400, # sample size region
                       stepn2 = 4, # sample size step size
                       kappamin = 0.02, kappamax = 0.2, # threshold region
                       stepkappa = 0.02, # threshold step size
                       c2 = 0.675, c3 = 0.72, # maximal total trial costs
                       c02 = 15, c03 = 20, # maximal per-patient costs
                       b1 = 3000, b2 = 8000, b3 = 10000, # gains for patients
                       alpha = 0.025, # significance level
                       beta = 0.1, # 1 - power
                       Delta2 = NULL, w = NULL, in1 = NULL, in2 = NULL, 
                       a = NULL,b = NULL) # setting all unneeded parameters to NULL

Interpreting the output

After setting all these input parameters and running the function, let’s take a look at the output of the program.

res
#> Optimization result:
#>  Utility: 2946.07
#>  Sample size:
#>    phase II: 92, phase III: 192, total: 284
#>  Probability to go to phase III: 1
#>  Total cost:
#>    phase II: 77, phase III: 158, cost constraint: Inf
#>  Fixed cost:
#>    phase II: 15, phase III: 20
#>  Variable cost per patient:
#>    phase II: 0.675, phase III: 0.72
#>  Effect size categories (expected gains):
#>   small: 0 (3000), medium: 0.5 (8000), large: 0.8 (10000)
#>  Success probability: 0.85
#>  Joint probability of success and observed effect of size ... in phase III:
#>    small: 0.72, medium: 0.12, large: 0
#>  Significance level: 0.025
#>  Targeted power: 0.9
#>  Decision rule threshold: 0.06 [Kappa] 
#>  Assumed true effect: 0.625 [Delta] 
#>  Treatment effect offset between phase II and III: 0 [gamma]

The program returns a total of thirteen values and the input values. We will now take a closer look and explain all output values with the necessary detail:

  • res$u is the expected utility of the program for the optimal sample size and threshold value. In our case it amounts to 2946.07, i.e. we have an expected utility of 294 607 000$. As the function maximizes the expected utility, this is the highest expected utility one can receive given the input parameters.
  • res$Kappa is the optimal threshold value for the go/no-go decision rule. We see that we need a treatment effect of more than 0.06 in phase II in order to proceed to phase III. If in phase II we get a treatment effect that is below 0.06, the program is deemed unsuccessful and no phase III trial will be conducted. If we are above the threshold value, we proceed to phase III.
  • res$n2 is the optimal sample size for phase II and res$n3 the resulting sample size for phase III. We see that the optimal scenario requires 92 participants in phase II and 192 participants in phase III. res$n is the sum of res$n2 and res$n3 and refers to the total number of participants. Note, that one can set a maximum number of participants, leading to constrained optimization. For more information, see the article about further input parameters.
  • res$pgo is the probability to go to phase III, which coincides with the probability that the observed treatment effect is above the optimal threshold value. In this case, we get a probability which is close to one and hence rounded to 1. This means that we can expect a treatment effect of above 0.06 in phase II and hence, the trial will proceed to phase III almost certainly.
  • res$sProg is the probability that the whole program is successful, i.e. that we proceed to phase III and observe a positive treatment effect in phase III. In our case the probability of a successful program is 0.85, so a trial with these input parameters will be successful 85% of the time. The probability of a successful program is further subdivided in probabilities for small, medium or large treatment effect. The probability of a small treatment effect res$sProg1 is 0.72 in our case, for a medium treatment effect the probability res$sProg2 is 0.12 and for a large treatment effect the probability res&$sProg3 is 0. The treatment effects correspond to the benefit categories, so with a probability of 0.72 we receive a benefit of 300,000,000$ and so on. Due to rounding, the probabilities may not add up exactly to the overall probability of a successful program. Note that one can set a minimum probability of successful program, leading to constrained optimization. For more information, see the article about further input parameters.
  • The final output parameters are the costs res$K2 in phase II and res$K3 in phase III. The costs in phase II are calculated as c02+n2⋅c2c_{02} + n_2\cdot c_2 and the costs in phase II are calculated as (c03+n3⋅c3)⋅pgo(c_{03} + n_3\cdot c_3)\cdot p_{go}, i.e. in phase III the expected costs are returned. In our setting, we have costs of 77 in 10^5$ (i.e. 7,200,000$) in phase II and costs of 158 in 10^5$(i.e. 13,900,000) in phase III. Note, that one can set a maximum cost limit for K=K2+K3K=K_2+K_3 , leading to constrained optimization. For more information, see the article about further input parameters.

Where to go from here

This tutorial explains how we can interpret all of the output generated by our program.

For more information on how to use the package, see: