FACTS 7.1.0 Release Notes

December 10, 2024

Introduction

FACTS 7.1.0 is now available for download via App Center. FACTS users can now:

• Perform concurrent control analyses using posterior probabilities, predictive probabilities and p-values in Platform Trial designs.
• Make interim and final decisions based on conditional power as well as Bayesian predictive probabilities.
• Export the data associated with the in-built graphs FACTS provides into a CSV file.
• Explore, via integration with AIRSHIP, simulation results graphically in a much more generic, versatile way; namely, by allowing the user to view the impact of simulation input dimensions through dynamic filtering of the simulation results.
• Create designs which use an independent Beta Binomial dose response model for analyzing (simulated) data for dichotomous endpoints in Core and Staged designs.
  
• Create designs where frequentist (p-value) calculations are performed using the Fisher exact test rather than a normal approximation for dichotomous endpoints.
• Provide three separate patient queue lengths for Continual Reassessment Methods (CRM) designs, based on dose clearance and the current model estimate of the MTD. In addition, the ability to backfill to the current escalation dose (“frontfill”) is now available.
• View explanations of some of the most important inputs in Core Continuous/Dichotomous designs through informative tooltips.
• Set different random number seeds when simulating multiple design scenarios.
• Create designs where decision QOIs can be used in multiple stopping criteria simultaneously.

FACTS Core and Staged Improvements

• In Core and Staged designs making use of dichotomous endpoints, FACTS users can specify a beta binomial model to independently model the response rate on each arm. With this model, the user specifies a Beta distribution prior on the response rate.
• In Core and Staged designs which perform frequentist analyses on dichotomous endpoints, users can now specify whether p-value calculations (including Current Trial Predictive Probability QOIs) should be performed using a normal approximation as currently, or using a Fisher exact test. The latter is of particular use when sample sizes are small.
• In Staged designs, the reported Early Success Time will now correctly only be reported for simulations which have stopped for early success. Previously, simulations which have graduated early to stage 2 in stage 1 were being reported as having an Early Success Time.
• When running analyses in FACTS Core and Staged designs, the user can now specify the number of MCMC samples to generate per imputation of missing data modelled by a longitudinal model.
• In Core and Staged designs with a dichotomous endpoint, the 95% confidence interval of the raw treatment estimates will now be reported correctly in relevant graphs.
• In Core and Staged designs, a new Operating Characteristics graph displaying the cumulative proportion of simulations having stopped at a given duration, sample size or number of events (in Time-to-Event) has been added.
• In Core designs, two new summary columns “Ppn Overall Success” and “Ppn Overall Futility” have been added, which simply sum up the existing columns “Ppn Early Success” and “Ppn Late Success”, and “Ppn Early Futility” and “Ppn Late Futility”, respectively. This allows users to quickly view the estimated type I error and power of the design.
• In Core and Staged designs, the number of participants at the end of each simulated trial who have had outcomes (“completers”) will now be summarized. The interim subject information (reported in the weeks file “Complete Info” columns) will now also be computed at the final analysis, reported in the simulations file and summarized.
• In Staged designs, the Stage 2 “Explore” Final Success/Futility graphs will now have the option to include/exclude simulations which have stopped in Stage 1.
• In Core and Staged designs, the criteria for selecting a dose at the end of the trial have moved from the “Variants” tab to the Success/Futility tab. These criteria will be used when reporting the proportion of times the correct/incorrect arm was selected at the end of the trial (as reported in the “Ppn Correct/Incorrect Arm” columns, which are now reported in the summary file).
• In Core and Staged Multiple Endpoint designs, designs not using a control arm will no longer crash when adding a new endpoint, and no longer crash when adding a new treatment to a design which uses Virtual Subject Response external data files.
• In Core and Staged designs, the “Legacy Second Order NDLM” dose response model and the “Legacy Adaptation” allocation option have been removed. Older designs making use of these features will be migrated over to the “Second Order NDLM” dose response model and the “Fixed Allocation” allocation option, respectively, when loaded in FACTS 7.1. A warning prompt will appear for such designs.
• Users can now choose a decision QOI in multiple stopping criteria simultaneously. Previously, after a stopping rule was created based on a certain decision QOI, any additional stopping rule could not use this decision QOI.
• In Core and Staged Designs, a new class of QOI as a sub-category of Predictive Probabilities was introduced: Conditional Power. Contrary to the existing Bayesian Predictive Probabilities, Conditional Power assumes the observed treatment effect estimates to be the truth and then calculates the probability of being successful either: 1) at a later final analysis, 2) after the currently enrolled subjects are followed up or 3) in a future trial.
• FACTS will now correctly calculate the Bayesian Predictive Probabilities of success at the final analysis when a continuous endpoint and change from baseline as the final endpoint are used.
• In Core and Staged Time-to-Event designs, minimum information required to trigger an interim will now be available when interims are based on predictor events.
• In Core and Staged Time-to-Event designs, the complete information columns report at interims with the weeks files will now display the correct information when interims are based on predictor events.
• In Core and Staged Time-to-Event designs with a Time-to-Event predictor, when interim information events and the event max cap differ (e.g. max cap on final events but interims based on predictor events), simulations will correctly stop at the specified max event cap.
• In Core and Staged Time-to-Event designs with a Time-to-Event predictor, current trial predictive probability QOIs are now handled correctly when max event caps refer to predictor events.
• In Staged Time-to-Event designs with a Time-to-Event predictor, current trial predictive probability QOIs are now handled correctly when interim information events and the event max cap differ (e.g. max cap on final events but interims based on predictor events).
• In Staged Time-to-Event designs, the censoring and event time of subjects in Stage 2 which have not observed their event in Stage 1 will now be handled correctly for both their final and predictor endpoints.
• In Core and Staged design, (Continuous, Dichotomous, and TTE) when a trial stops for early success or early futility and the option to continue follow-up after that decision is not selected, the final analysis model is no longer run. The data at the final analysis would be identical to the data at the interim analysis that the stopping threshold was hit at, so the model output for the final is now identical to the interim.
• Time Course Hierarchical longitudinal model has improved performance with informative priors on the variance components.
• In Staged designs, FACTS will now correctly handle the Stage 1, Stage 2 and overall sample size caps (maximum number of subject caps and maximum number of event caps) specified in the Variants tab.
• In Core and Staged Time-to-Event designs, the Cox Proportional Hazards current trial predictive probability calculation has been corrected.
• In Core and Staged designs, when using predictive probabilities that predict success at trial maximum, but the trial is stopped at an interim, the predictive probability is now calculated based on the originally specified number of subjects at trial maximum rather that assigning the predictive probability a value of 0 or 1 depending on the p-value of the interim data.
• In Core and Staged designs, when using predictive probabilities, visit values are now correctly imputed when there is only baseline visit data available.
• In Core and Staged designs, when using a dichotomous endpoint and predictive probabilities of treatment versus control, FACTS now performs a Farrington and Manning Test when there is a superiority or non-inferiority margin.
• In Staged design, the arm selection logic has been updated to properly account for arm-dropping in stage 1 when selecting a single arm from each group.

FACTS Enrichment Design Improvements

• In Enrichment Dichotomous designs, users can now perform frequentist calculations using a normal approximation as currently, or using a Fisher exact test. The latter is of particular use when sample sizes are small.
• In Enrichment Time-to-Event designs, the GUI will now correctly calculate the mean frequentist estimated treatment effect and its standard error.
• When running analyses in Enrichment designs, the user can now specify the number of MCMC samples to generate per imputation of missing data modelled by a longitudinal model.
• In Enrichment Dichotomous designs, the 95% confidence interval of the raw treatment estimates will now be reported correctly in relevant graphs.
• In Enrichment designs, a new Operating Characteristics graph displaying the cumulative proportion of simulation having stopped at a given duration, sample size or number of events (in Time-to-Event) has been added.
• In Enrichment designs, two new summary columns “Ppn Overall Success” and “Ppn Overall Futility” have been added, which simply sum up the existing columns “Ppn Early Success” and “Ppn Late Success”, and “Ppn Early Futility” and “Ppn Late Futility”, respectively. This allows users to quickly view the estimated type I error and power of the design.
• In Enrichment designs, the number of participants at the end of each simulated trial who have had outcomes (“completers”) will now be summarized. The interim subject information (reported in the weeks file “Complete Info” columns) will now also be computed at the final analysis, reported in the simulations file and summarized.
• In Enrichment designs, the frequentist test reported in the frequentist simulations file will now make it clear whether the performed test is a one-sided or two-sided test. In the associated frequentist summary file, the proportion of frequentist results for each test type that are significant will now correctly take into account the correct alpha level depending on whether the test is one-sided or two-sided.

FACTS Platform Trial Improvements

• BREAKING CHANGE: in Platform trial designs, the numerical value representing the outcomes for late futility and early futility have been swapped in FACTS 7.1.0. Any older designs with futility criteria will need to be re-simulated. Decision numeric values in FACTS Platform Trials and FACTS Core now match.
• In Platform Trial designs, users can now specify whether Posterior Probability QOIs, Predictive Probability QOIs or p-value QOIs should be calculated based on the entire control population in the trial (as previously) or based on the given treatment’s concurrent control population. These concurrent control QOIs can be used as treatment stopping criteria like any other QOI.
• A time window allowing a treatment’s concurrent controls to additionally include control patients a certain number of weeks from the treatment entering the trial can also be specified.
• Pr(Max) Target Dose QOIs can now be calculated based on all arms in the trial (as previously), only active arms or only randomizing arms.
• In Platform Trial Dichotomous designs, the 95% confidence interval of the raw treatment estimates will now be reported correctly in relevant graphs.
• In Platform Trial designs, a new Operating Characteristics graph displaying the cumulative proportion of simulation having stopped at a given duration or sample size has been added.
• In Platform trial designs, users can now specify different variants of the design by modifying both the maximum number of participants per treatment as well as the maximum number of concurrent treatments. These variants will display as separate scenarios on the Simulations tab.
• In Platform trial designs, the number of participants at the end of each simulated trial who have had outcomes (“completers”) will now be summarized. The interim subject information (reported in the weeks file “Complete Info” columns) will now also be computed at the final analysis, reported in the simulations file and summarized.
• In Platform trial designs, the “Per Sim: Arm and Participant Arrival” graph will now correctly display the accrual period for the control arm to end when the last treatment’s accrual period ends.
• Users can now choose a decision QOI in multiple stopping criteria simultaneously. Previously, after a stopping rule was created based on a certain decision QOI, any additional stopping rule could not use this decision QOI.
• FACTS will now correctly calculate the Bayesian Predictive Probabilities of success at the final analysis when a continuous endpoint and change from baseline as the final endpoint are used.

FACTS Dose Escalation Improvements

• The Dose Escalation design types CRM(Toxicity), CRM(Ordinal), bCRM and CRM(Efficacy) have all been deprecated. Users can still create new designs or open existing designs with these design types. We recommend using the more general and versatile “Continual Reassessment Methods (CRM)” design type (formerly known as “N-CRM”) for any 1D model based CRM designs.
• In CRM with open enrollment, users can now specify whether they want to allow backfilling to the highest dose (“frontfilling”) and conditions under which to do so. Users can choose whether to count these patients towards the backfill subject cap or regular allocation cap.
• In CRM with open enrollment, previously two queues (maximum number of subjects on uncleared doses and maximum number of subjects on cleared doses) determined the allocation behavior. In order to give users more flexibility and control, there are now three queue concepts (maximum number of subjects on uncleared doses, maximum number of subjects on cleared doses at MTD and maximum number of subjects below MTD). These queues are now used in the same way in the MTD and MED phase of the trial. The concept of max cleared dose and the new queues are now harmonized.
• In CRM, the stopping rule checker design was updated. In case of regular dosing, a concept of near doses like that of fine spaced dosing was introduced and both concepts aligned for both stopping in the MTD and MED phase. The stopping rule checker now also correctly checks the hierarchies and join conditions.
• In CRM with open enrollment, backfill will now correctly evaluate all cleared doses for eligibility.
• In CRM with open enrollment and fine spaced dosing, near doses are now correctly evaluated in regular allocation, backfill and for the purposes of stopping.
• In CRM designs, FACTS will now correctly apply study size constraints and queue size constraints when the underlying toxicity model considers all doses to be toxic. This also includes preventing an expansion cohort from being allocated in this situation.
• In Dose Escalation designs, a new Operating Characteristics graph displaying the cumulative proportion of simulation having stopped at a given duration or sample size has been added.
• In 2D-CRM designs, the dose response model’s eta parameter can now be specified in log-normal space.
• In 2D-CRM designs with custom run-in, FACTS now proceeds to the chosen escalation scheme more promptly, without first allocating more than one full-size cohort unless they are specified in the run-in.
• The “Per Sim Allocation History Grouped” plot will now be displayed correctly if, previously, the “Per Sim Allocation History Group 1” plot had been set to space interims equally.
• In CRM with an initial run-in, escalation will now be performed correctly and maximum cleared dose tracked correctly in all circumstances.
• In CRM with small-cohort pre-escalation, we now only switch to regular escalation based on observed toxicities, not MTD estimate or overdose control rules.
• In CRM, introduce the notion of “Selected MTD”, “Selected OSD” and “Selected MED” at the end of trial as a function of the respective model estimates and the max cleared dose.
• In CRM with two groups that enroll consecutively, there are now several options regarding what dose level to start escalation at in group 2.
• In CRM with fine grained dosing, the concept of near doses is now correctly applied in the escalation phase, when calculating the max cleared dose and adjusted for the new queue concepts.
• In CRM, cohort expansion will now enroll the correct number of patients even when accrual is very fast.
• In CRM, when using a cohort expansion or a two group design, the trial state on which both of these concepts depends is now the final state of the previous trial, not the state of the previous trial when its stopping criteria were met.
• In CRM, when using two groups and expansion cohorts in both groups, the group 2 expansion cohort now correctly uses its own cap regarding maximum number of subjects. In the same setting, a rare circumstance led to the allocation in the group 1 cohort expansion to continue beyond the max cap – this is now resolved.
• In CRM, stopping based on the max cap of subjects for escalation now does not lead to an overrun in patients in rare circumstances.
• In CRM with both an MTD and an MED phase (toxicity and efficacy endpoints), the transition between the phases is now handled correctly.
• In CRM, improved labels and default values in the GUI, such as improved values for the overdose control and open enrollment queue lengths and clearer labels in the Allocation tab.
• Several GUI stability improvements in the CRM engine.
• In CRM, there are several improvements to the “Per Sim Allocation History”, “Alloc and Tox History”, “Cohort Band Probabilities” and “Cohort Response” graphs, including showing the cohort expansions subjects separately, showing the max cleared dose at any point in time and showing the selected MTD/MED at the end of the trial.

General Improvements

• FACTS has now been integrated with AIRSHIP, which allows simulation results to be explored graphically in a much more generic, versatile way. Once simulation have completed, results can be explored with AIRSHIP by clicking on “Explore Results…” > “Compare Scenarios in AIRSHIP”. Note that use of AIRSHIP requires at least two scenarios to have been simulated and their results aggregated.
• The process of making the FACTS inputs more intuitive has started as of FACTS 7.1.0. In FACTS Core Continuous/Dichotomous designs, tooltips will appear against many of the inputs (when hovering over the relevant input) with explanatory text about their use and impact on the design. Tooltips can be disabled by going to Settings > Options > Tooltip Configuration.
• The data associated with all graphs displayed in FACTS can now be exported into a CSV format. In addition, hovering over these graphs will display the associated data point value as a tooltip.
• When simulating multiple scenarios, each simulated scenario can now be simulated with a different random number seed.
• When running simulations for a directory of FACTS files in FACTS Command Line or FLFLL, a different base seed can be set for each design within the directory.
• The order of scenarios to simulate as displayed on the Simulation tab has been set to be consistently displayed in alphabetical order.
• When aggregating simulation results for a design using variants, the relevant variant number will now correctly be displayed in the aggregated files.
• Several bug fixes and improvements have been made to all design reports.
• The associated engine executable now provides a new -o output flag argument specifying the location where to output all files generated by the engine [Enterprise licensees only].

Please contact us regarding any questions.