Start FACTS. Double Click: Core Design Dichotomous. Use the menu option File > Save As to create a location for saving this design. Save it as “shared_control” in a convenient location (e.g. in a folder called FACTS\shared-control-examples in your ‘My Documents’ folder).
Quantities of Interest
The Quantities of Interest (QOIs) are the criteria on which decisions, interim or final analyses, will be made. Numerous QOIs are available and you may select multiple quantities for evaluation. A basic set of QOIs is provided by default. Additional QOIs may be added by clicking the Add row in the table. A QOI can be removed by clicking the X on the right side of the cell. Each QOI may be edited by clicking on it, which opens a pop-up box.
The trial will be judged a success at the end of the trial (after 250 subjects are completed) if one of the treatments has a high probability of success in a future trial, i.e. a phase 3 trial. A default QOI already exists to monitor the predictive probability of a future trial. We edit the existing Pr(Succ. Future Trial) quantity by clicking on it and a pop-up box will open. In this box:
Select a Phase of “Future Trial”
Select Test Type of “Superiority”
Specify 100 Subjects Per Arm
Specify a One-sided Alpha of 0.025
Specify a Superiority Margin of 0
The probability of success in the future trial will be calculated for each treatment, assuming that treatment is the one carried forward into the future trial. Thus, this is a vector quantity. It is also important to note the Alternative Name box contains the name seen in the output files for the quantities. A file titled QOIMappingFile.csv will also be output and can be used to determine the definitions for the output fields.
We also are interested in the posterior probability of the treatment compared to the control. This default QOI is a comparison to Control, Pr(P_d>P(Control)). We also want to add comparisons to each active treatment arm. In the Posterior Probabilities box we click Add…:
For Probability of being Target, we only keep Pr(Max), a default QOI. We will not be using this target for decisions, but may want it in the future if we post-process FACTS output.
To make decisions in the trial, we must specify which treatment will be evaluated, thus reducing the vector quantity above to a scalar quantity. This information is captured in the Decision Quantities (Scalars) window at the bottom right. We click Add… and a popup window opens. We select:
QOI(Vector Quantity) of Pr(Succ.Future Trial): N=100; Sup. α=0.025; δ=0
Evaluation Dose: Trt1
Repeat the above 2 steps for Trt2, Trt3, and Trt4
The trial will be declared a success if any of the treatments are successful in a future trial.
We deleted all the other default QOIs to reduce the amount of output, however they can be left entered in which case they will be computed, and the values output, but they will have no impact on the simulation of the trial.
For the Standard Evaluation Variables at the bottom:
We will not be using the Clinically Significant Difference (CSD), but we set it at 0.2 to show a reference line on the Virtual Subject Response tab.
Specify the Default QOIs compare to: Control
Specify that the Trial type is Superiority
Simulation Results
We start by running a small number of simulations, 50, as an initial check, Select All, then press Simulate. If the results look reasonable, increase the number of simulations to run to 1,000, Select All, then press Simulate. When the design is calibrated you may want to run a higher number, such as, 10,000 simulations.
Check the number of simulations per scenario (note the order of the scenarios) and the mean # of subjects per scenario.
We check the type-1 error and the power for each scenario.
We also use the graphs to understand how the design performed. We select scenario win2.3 and click View Graph. Select the Per Scenario Graphs. We next click on Per Sim: Posterior Quantities and select Quantity: Pr(Succ. Future Trial). Click on the “Set Y axis” button in the “Controls” window, and select a Fixed Maximum of 1, a Fixed Minimum of 0, and 6 Tick Marks. This plot shows the observed and estimated response for a single simulated trial, overlaid on a barchart showing the predictive probability of future trial success for each arm. We see that the predictive probabilities are greatest for treatments 3 and then treatment 2. Both of these arms have predictive probabilities greater than 0.6, so they both are eligible for consideration in a future trial. We can scroll through the simulations to see how this varies through the simulations.
In addition to the numerous default summaries and graphs in FACTS we would like to do some post-processing of the simulation results. We will define how we will select a successful treatment (or treatments) to move forward to the future Phase 3 trial. A treatment arm will be selected for the future study, if its predictive probability of success in the future trial is > 0.60, AND no other arm has a posterior probability greater than or equal to 0.9 of being better than the arm.
To do this we will use the simulations.csv files provided in the results folders. We want to aggregate the results for all the scenarios, so we push the Aggregate button in the Results Options area of the Simulations page.
We take the aggregated files and use the following short R program to select out the successful trials. Below are screen shots of the aggregate .csv file and the R program.

R code:
# *****************************************************************************************************************
# Berry Consultants, LLC
# PROGRAM NAME: facts_training_shared_control.r
# PROJECT NAME: FACTS training example
# PROGRAM PURPOSE: Post-process FACTS output files
# SOFTWARE/VERSION #: 5.5.11
# INPUT: simulations.csv for various scenarios
# OUTPUT: TBD
# *****************************************************************************************************************
rm(list=ls())
##read in aggregate simulations files created in FACTS
##replace the definition of in.path by the location of the FACTS output
in.path='FACTS/shared_control_results/'
d1='shared_control'
F1=paste(in.path, d1,'_results','/agg_',d1,'_simulations','.csv',sep='')
scenarios=c('none','win1','win2.3','win1.3.4')
##read in aggregate simulation file and select appropriate header file
read.facts = function(File, design) {
cols = unlist(strsplit(readLines(File,n=3)[3],split=',')) # Get col names, here want names from row 3
dat = read.table(File,header=FALSE,sep=',',skip=2) # data
names(dat) = cols
return(dat)
}
sims.out=read.facts(File=F1, design=d1)
count.wins=function(dat1=data.in, success.thresh, dose.thresh){
##success pred prob success >success.thresh and No arm dose.thresh or better
dat1$win.trt1=
ifelse(dat1$' Pr(SS) 2'>success.thresh &
dat1$' QOI_1_1 3'<=dose.thresh & dat1$' QOI_1_1 4'<=dose.thresh & dat1$' QOI_1_1 5'<=dose.thresh,1,0)
dat1$win.trt2=
ifelse(dat1$' Pr(SS) 3'>success.thresh &
dat1$' QOI_1_2 2'<=dose.thresh & dat1$' QOI_1_2 4'<=dose.thresh & dat1$' QOI_1_2 5'<=dose.thresh,1,0)
dat1$win.trt3=
ifelse(dat1$' Pr(SS) 4'>success.thresh &
dat1$' QOI_1_3 2'<=dose.thresh & dat1$' QOI_1_3 3'<=dose.thresh & dat1$' QOI_1_3 5'<=dose.thresh,1,0)
dat1$win.trt4=
ifelse(dat1$' Pr(SS) 5'>success.thresh &
dat1$' QOI_1_4 2'<=dose.thresh & dat1$' QOI_1_4 3'<=dose.thresh & dat1$' QOI_1_4 4'<=dose.thresh,1,0)
out=list()
for(i in 1:length(scenarios)){
dat2=subset(dat1,dat1$'Response Profile'==scenarios[i])
numsim=length(dat2$win.trt1)
dat2.success=c(sum(dat2$win.trt1)/numsim,
sum(dat2$win.trt2)/numsim,
sum(dat2$win.trt3)/numsim,
sum(dat2$win.trt4)/numsim)
out[[i]]=(cbind(scenarios[i],numsim,success.thresh, dose.thresh,c('trt1', 'trt2', 'trt3', 'trt4'),dat2.success))
}
return(out)
}
summaries=count.wins(dat1=sims.out, success.thresh=0.60, dose.thresh=0.90)
Below is a table of the results from the post-processing. The final column shows, for each arm, the proportion of simulated trials in which that arm was selected to be carried forward into a future trial.
| none |
1000 |
0.6 |
0.9 |
trt1 |
0.044 |
| none |
1000 |
0.6 |
0.9 |
trt2 |
0.042 |
| none |
1000 |
0.6 |
0.9 |
trt3 |
0.036 |
| none |
1000 |
0.6 |
0.9 |
trt4 |
0.039 |
|
|
|
|
|
|
| win1 |
1000 |
0.6 |
0.9 |
trt1 |
0.832 |
| win1 |
1000 |
0.6 |
0.9 |
trt2 |
0.016 |
| win1 |
1000 |
0.6 |
0.9 |
trt3 |
0.012 |
| win1 |
1000 |
0.6 |
0.9 |
trt4 |
0.019 |
|
|
|
|
|
|
| win2.3 |
1000 |
0.6 |
0.9 |
trt1 |
0.003 |
| win2.3 |
1000 |
0.6 |
0.9 |
trt2 |
0.529 |
| win2.3 |
1000 |
0.6 |
0.9 |
trt3 |
0.978 |
| win2.3 |
1000 |
0.6 |
0.9 |
trt4 |
0.011 |
|
|
|
|
|
|
| win.1.3.4 |
1000 |
0.6 |
0.9 |
trt1 |
0.843 |
| win.1.3.4 |
1000 |
0.6 |
0.9 |
trt2 |
0.016 |
| win.1.3.4 |
1000 |
0.6 |
0.9 |
trt3 |
0.623 |
| win.1.3.4 |
1000 |
0.6 |
0.9 |
trt4 |
0.845 |
The above code can be used to evaluate additional predictive probability and other treatment arm comparison thresholds. It also could be expanded to create other summaries of interest. For example, we could determine the proportion of trials in which exactly one arm was selected, two arms were selected, etc. Finally, the code could be expanded so different trial success thresholds other than 0.60 could be examined. In the current simulations, a predictive probability future trial threshold of 0.6 results in a proportion of late successes of 0.128 for the none scenario. Additions could be made to the code so that the proportion of late successes could be determined for multiple higher thresholds across all of the scenarios.
Going back to the win2.3 scenario, the specific simulation number 17, the simulated trial shown in the graph earlier, the predictive probabilities of success in a future trial are 0.709 and 0.992 for treatments 2 and 3 respectively and both satisfy the predictive probability of success in a future trial threshold. The posterior probability that treatment 3 is greater than treatment 2 is 0.986. For treatment 3, none of the posterior probabilities comparing other active treatment arms are greater than 0.9. Thus, in this specific simulated trial, only treatment 3 would be selected to move forward to a future phase 3 study.