Ordinal Endpoint QOIs

QOIs specific to ordinal endpoints.

The FACTS Ordinal Engine features several special quantities of interest.

Posterior probability QOIs

The posterior probability QOIs available depend on whether the user has chosen to model the ordinal endpoint with the cumulative logistic family or with independent Dirichlet distributions.

Cumulative logistic modeling

When the cumulative logistic family is in use, each arm other than control has an odds ratio parameter relative to control. Posterior probability QOIs are defined based on these odds ratios. By default, the following posterior probability QOIs are available, (here \(OR_d\) is \(\exp(\theta_d)\) where the \(\theta_d\)’s are the parameters defined in the dose-response model):

  • The probability of being better than control, which is Pr(\(OR_d > 1\)) when large values of the ordinal index are good, or Pr(\(OR_d < 1\)) when small values are good.

  • The probability of being better than control by a clinically significant difference, which is Pr(\(OR_d - 1 > CSD\)) when large values are good, or Pr(\(OR_d - 1 < -CSD\)) when small values are good. The CSD is entered by the user in the Standard Evaluation Variables section of the Quantities of Interest tab.

Dirichlet modeling

When the user has selected independent Dirichlet modeling for the ordinal endpoint, the posterior probability QOIs are based on expected utility. The default posterior probability QOIs are:

  • The probability of being better than control with respect to expected utility, which is Pr(\(EU_d > EU_{Control}\)). Here \(EU_d = \sum_{k=1}^K p^d_k U_k\), where \(p^d_k\) is the probability of ordinal outcome \(k\) for arm \(d\) and where \(U_k\) is the utility for outcome \(k\).

  • The probability of being better than control by a clinical significant difference, which is Pr(\(EU_d > EU_{Control} + CSD\)) regardless of whether large or small values of the ordinal endpoint are good (large values of utility are always good). The CSD is entered by the user in the Standard Evaluation Variables section of the Quantities of Interest tab.

\(p\)-values

The Ordinal Engine has the following different frequentist final analyses available. If there are multiple experimental arms, the \(p\)-values can optionally be adjusted using the Bonferroni correction. The \(p\)-values may handle missing values by ignoring them or by imputing a specific ordinal outcome chosen by the user.

Utility \(t\)-test

One option is to perform a \(t\)-test on the utility values observed, comparing each experimental arm to control, in the same way as with a continuous endpoint.

Wilcoxon-Mann-Whitney test

FACTS will also perform a Wilcoxon-Mann-Whitney (rank sum) test comparing each experimental arm to control. FACTS adjusts for ties in computing the \(p\)-value.

Proportional odds likelihood ratio test

FACTS will also perform a likelihood ratio test based on the proportional odds model, comparing each experimental arm to control. It fits a multinomial model to both arms together, and also fits a model where the experimental arm has a single proportional odds model deviation from control, and compares twice the logged ratio of maximized likelihoods to the chi-squared distribution with one degree of freedom. In practice, this test will often give similar results to the Wilcoxon test. (See, for example, Harrell.)

Dichotomized ordinal test

FACTS will also test whether the probability of a “good” outcome is larger for an experimental arm than it is for control. The user defines a “good” outcome using the “Definition of Success” selector in the “Standard Evaluation Variables” section. The test is the same as for a dichotomous endpoint.

Dichotomizing an ordinal outcome is rarely a good choice because much of the information in the ordinal outcome is wasted. See, for example, Podcast by Scott Berry. FACTS simulations can be useful in demonstrating the reduction in power when an ordinal endpoint is dichotomized.

Bayesian Predictive Probabilities

Bayesian predictive probabilities are available for any of the four \(p\)-value-based frequentist final analyses described above. As with other endpoints, these can be predictions of the result of the current trial, or a separate two-arm trial.
The current trial predictions can refer either to an analysis of the patients currently enrolled but after follow-up is complete, or at the maximum sample size for the trial.

Here are a few notes about the different predictive probabilities.

  • Predictive probabilities for the proportional odds likelihood ratio test are likely to be more time-consuming, and less accurate, to simulate than the other \(p\)-values. These predictive probabilities require simulating a large number of final data sets, computing the test statistic for each, and counting how many achieved the desired significance level. We recommend substituting the Wilcoxon test while the design is being developed (and maybe permanently); its results should be similar.

  • Predictive probabilities for the Wilcoxon test: Both future trial and current trial predictive probabilities for the Wilcoxon test are computed using formulas developed by Graves for use in simulating trials designed by Berry Consultants including SEPSIS-ACT. The approach is to write the Wilcoxon-Mann-Whitney as a sum, over pairs of patients, of indicators that an experimental arm patient has a better outcome than a control arm patients, and then correctly computes the predictive mean and variance of this sum as a function of the probabilities of each outcome for each arm. The results are then averaged over a large number of Markov chain Monte Carlo samples of outcome probabilities. The predictive probabilities computed in this way are very accurate, the only exception is when there are a very small number of future patients whose data are yet to be collected.

Current Trial Bayesian Predictive Probabilities

For general discussion of current trial Bayesian predictive probabilities, see the core QOIs page. In the present version of the FACTS Ordinal Engine, current trial predictive probabilities are only available for the full sample size intended at the end of the trial.

Future Trial Bayesian Predictive Probabilities

For general discussion of future trial Bayesian predictive probabilities, see the core QOIs page.

Conditional Power

Conditional power in trials with an ordinal endpoint will be added in a future release of FACTS Ordinal.

Decision Quantities

Decision quantities in FACTS Ordinal operate similarly to decision quantities with other FACTS endpoints.

Standard Evaluation Variables

Definition of Success

In some circumstances, one may wish to dichotomize an ordinal endpoint, and define an ordinal value such that achieving that value or better counts as a success, while reaching a worse value is a failure. This value can be chosen here, and it will be used in p-values comparing treatments to control with respect to the probability of a success. “Better” than a specified value here is taken to mean a more favorable ordinal index (a higher index if high values are good, or a lower index if low values are good), regardless of which utility values have been entered.

Clinically significant difference

Here the user may define a clinically significant difference (CSD) to use in posterior probabilities of being better than control by a CSD.

  • If the user has selected independent Dirichlet modeling, the CSD is on the scale of utility. For example if the CSD is 0.5, FACTS by default computes the probability that each treatment has an expected utility that is more than 0.5 points higher than the expected utility for the control arm. Regardless of whether the high values of the ordinal index are good, large utilities are good in FACTS Ordinal.

  • If the user has selected cumulative logistic modeling, the CSD is on the scale of an odds ratio. For example if the CSD is 0.3 and large values are good, the default posterior probability of an odds ratio of at least 1 + 0.3 = 1.3 is calculated. If the CSD is 0.2 and small values are good, the posterior probability of an odds ratio less than 1 - 0.2 = 0.8 is calculated.

Super-superiority and non-inferiority

Super-superiority and non-inferiority are complicated concepts in ordinal endpoints and they have different meanings, or no established meaning, depending on how the ordinal endpoint is analyzed. These will be enabled in a future version of FACTS Ordinal.