Changes to design and analysis elements of research plans during randomised controlled trials in Australia

The protocol for a randomised controlled trial (RCT) pre‐specifies all facets of the research plan and thereby promotes consistency of trial implementation, the reproducibility of its findings, and adherence to ethical standards and regulatory requirements.1 The Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement provides clear guidance regarding the elements of an appropriate protocol, and a summarised subset of this content is recorded when prospectively registering a trial.2 Trial registration is now a prerequisite for institutional ethics approval1 and for the publication of clinical trial findings in many journals.3

Adherence to the protocol is important for protecting the scientific integrity of a trial from decisions that may introduce bias if inappropriately informed by unblinded outcomes data; that is, with knowledge of how the treatment effect estimate is influenced by the change.4,5,6,7 In practice, adherence to protocols and the documentation of changes has been unsatisfactory. A recent systematic review of publications on the subject8 found that the prevalence of deviations from the protocols ranged from 14% to 100% for the primary outcome, 9% to 47% for the statistical analysis, 27% to 60% for the sample size, and 12% to 45% for the eligibility criteria. This systematic review and other studies9,10,11 have typically looked for changes in one or two aspects of trials or compared publications with the limited information available in clinical trial registry entries rather than with full protocols. Obtaining full protocols, however, can be difficult,12 as they are not always published or readily available from the investigators.13,14

Adherence by recent Australian clinical trials registered with the Australian New Zealand Clinical Trials Registry (ANZCTR) to original protocols has not been investigated with respect to a comprehensive set of methodological elements. We therefore assessed how well Australian RCTs adhere to original research plans and document changes to these plans. Our specific objectives were to estimate the frequency and types of substantive changes to Australian RCTs registered with the ANZCTR, and to determine whether unblinded treatment information informed such changes.

Methods

We searched the ANZCTR for phase 3 RCTs registered during 1 September 2007 – 31 December 2013 (ie, the trial phase was specified as 2/3, 3, or 3/4, or was unspecified but planned recruitment was at least 60 participants) with Australia listed as a country of recruitment. This interval was selected because it corresponded to the introduction of the ANZCTR history function, which allows tracking of changes in registration details since August 2007, and to allow time for trial completion and the publication of final outcomes. We excluded studies incorrectly registered as phase 3 trials (eg, pilot studies, pharmacokinetic or pharmacodynamic studies) and those that did not report specific primary efficacy outcomes, those without an Australian or New Zealand sponsor or Australian site, and studies without published primary results (to 31 December 2020) identified by searching PubMed, Scopus, ResearchGate (https://www.researchgate.net), PubFacts (https://www.pubfacts.com), Google Scholar (https://scholar.google.com), Google, and the websites of the pertinent host institutions and coordinating centres.

We defined “protocol documents” as full protocols, published protocol articles, statistical analysis plans (treated as extensions to the protocol), and surrogate protocols. We searched online for protocol documents for registered RCTs identified by our search. If no protocol was publicly available, we requested one directly from the trial leader listed in the ANZCTR or from the corresponding, lead, or senior author of the results publication. If no formal protocol was available, we used other documents (ethics approval applications, protocol manuals, grant applications) as surrogate protocols.

For the central audit (for RCTs with available protocols), author XC compared the protocol documents (original to final, when available) with the methods and analyses reported in the primary results publication. Six design and analysis elements were examined, based on SPIRIT,2 International Committee of Medical Journal Editors,3 World Health Organization,15 and Consolidated Standards of Reporting Trials (CONSORT)16 recommendations and guidelines, as methodological features that could significantly influence the primary conclusion of a trial if changed:

• the primary outcome;
• the composition of the analysis set;
• the eligibility criteria;
• the sample size (number of participants or clinical events, as applicable);
• the primary analysis method; and,
• for multi‐arm RCTs, the treatment arms included in the primary comparison.

We noted explanations for changes and assessed whether a change was made blinded to treatment outcomes. The repeatability of our method had previously been confirmed in a pilot study of 30 representative trials (included in the full audit) by XC and a second auditor.

The primary auditor (XC) sought external advice about ambiguities that could not be resolved by reference to the reviewed material (eg, statistical advice about possible method changes; clinical advice about possible changes to eligibility criteria), and consulted investigators directly about any remaining uncertainties.

We undertook limited central reviews of otherwise eligible RCTs for which protocol documents were not available, including RCTs for which the investigators declined or did not respond to the invitation to participate in our study. These reviews were limited to elements of the research plan corresponding to mandatory ANZCTR fields; we did not forward any queries to the trial group.

Statistical analysis

Our target sample size was 100 RCTs with protocol documents. This would allow an acceptably precise estimate of the frequency of changes to RCT research plans (95% confidence interval within ten percentage points of the proportion point estimate; Supporting Information, part 1).

We report summary statistics (counts, proportions with Wilson 95% confidence intervals [CIs], and means with 95% CIs), overall and by protocol availability, and by trial characteristics. We assessed the statistical significance of differences between trials with protocols and those without protocols in Pearson χ² tests (for categorical characteristics), Fisher exact tests (when at least one expected cell frequency was lower than 5), or Wilcoxon rank sum tests (continuous variables).

Characteristics associated with changes, both overall and of individual methodology components, were assessed by univariable and multivariable logistic regression using the least absolute shrinkage and selection operator (LASSO) method (results reported as odds ratios [ORs] with 95% CIs), and by recursive partitioning (results reported as proportions of node [cluster] homogeneity). Five variables were chosen a priori as candidate predictors: type of intervention (drug trial, other), protocol source (publicly available, supplied by trial group), significance level for the primary analysis result (P < 0.05, P ≥ 0.05), year of primary results publication, and planned sample size.

All analyses were conducted in R 4.1.0 for Windows (R Foundation for Statistical Computing).

Ethics approval

The University of Sydney Research Ethics Committee approved our study (2019/579). To comply with the confidentiality conditions of the ethics approval, the RCTs included in our analysis are not identified in this article.

Results

Of the 463 trials identified in the ANZCTR electronic search, 181 met our eligibility criteria (Box 1). Most were drug trials (103 trials, 57%); the most frequent disease areas were cancer (32 trials, 18%), mental health (18, 10%), respiratory disease (18, 10%), musculoskeletal disorders (16, 9%), cardiovascular disease (12, 7%), and renal or urogenital disease (11, 6%). Most trials were undertaken in hospitals (126 trials, 70%), had parallel arms designs (156, 86%), and tested hypotheses of intervention superiority (168, 93%); sixty trials (33%) reported statistically significant findings for their primary outcome (Box 2).

Protocols were available for 124 trials (69%); 46 were publicly available, 78 were provided by the trial groups after we requested them. The participants, assessors, statisticians, or investigators were blind to treatment allocation in 106 of 124 trials with protocols (85%) and for 37 of 57 trials without protocols (65%; P = 0.002); investigators were blinded in 67 trials with protocols (54%) and in 29 trials without protocols (51%; P = 0.69). Ninety‐three trials with protocols (75%) and 33 trials without protocols (58%) were undertaken in hospitals (P = 0.020). The proportion of trials with protocols available differed by disease type (Box 2).

The overall mean planned sample size was 708 participants (95% CI, 424–992 participants); the mean size was larger for trials with protocols (922 [95% CI, 514–1331] participants) than for those without protocols (241 [95% CI, 145–338] participants; P = 0.002) (Box 3).

Trials with available protocols

It could be established by reference to the published results that no important changes had been made to any element of the available protocol in six trials (5%; 95% CI, 2–11%) and that changes were clear or probable for 111 trials (90%; 95% CI, 85–95%); for seven trials (5%; 95% CI, 1–9%) we could not determine whether changes had been made. Changes were made blinded to treatment outcomes in six of 111 trials (5%) and unblinded in four (4%); this aspect was unclear for 101 of 111 trials in which changes had been made (91%).

Of the 118 trials in which it was clear or probable that changes had been made or we could not determine whether changes had been made, the primary analysis method had been changed in 83 trials (70%), the sample size in 60 (51%), the eligibility criteria in 59 (50%), the primary outcome in 52 (44%), the analysis set population in 29 (25%), and the primary comparison in eight of eleven multi‐arm trials. Examples of protocol changes are included in Box 4.

After receiving replies to all requests for clarifications from investigators in 103 trials with available protocols, 11 of 124 trials (9%; 95% CI, 4–14%) were deemed to have had no changes and 78 to have been changed (63%; 95% CI, 55–71%), including 61 (78%; 95% CI, 69–87%) in which the changes were made in an appropriately blinded manner. It remained unclear whether changes had been made for 35 trials (28%; 95% CI, 20–36%).

Trials with publicly available protocols were less likely to have been changed in any element (OR, 0.22; 95% CI, 0.06–0.77) or with respect to eligibility criteria (OR, 0.37; 95% CI, 0.17–0.80) than trials for which we requested protocols. Change to the primary analysis method was less for drug trials than for other trial types (OR, 0.35; 95% CI, 0.15–0.80). No other associations between the five factors assessed and changes to RCTs were statistically significant, and the multivariable analyses did not identify any additional significant associations not identified by the univariable analyses (Supporting Information, table 1).

In the recursive partitioning analysis, the best discriminators for any change were publicly available protocol and reporting of a statistically significant result. The 46 trials with publicly available protocols were less likely to have included changes than the 78 trials with requested protocols (probability, 0.83 v 0.95); in the requested protocols group, changes were also less likely for the 25 trials reporting statistically significant results than for the 53 that did not (0.88 v 0.98). Recursive partitioning applied to individual methodology and analysis elements did not identify any consistent pattern across these elements (Supporting Information, figure 1).

Trials without available protocols

We undertook limited reviews of 57 eligible RCTs for which protocol documents were not available, including ten for which the investigators declined to provide protocols (of 135 requests; 10%). On the basis of ANZCTR registration information, we identified changes in 42 of 57 trials without available protocols (74%) and no changes in five (9%); we could not determine whether there had been changes in ten trials (18%). Logistic regression analysis identified no consistent associations between the candidate covariates and any change (Supporting Information, table 2).

In the recursive partitioning analysis, the best discriminators for any change were planned sample size, and drug trial as trial type (Supporting Information, figure 2). The 43 trials with sample sizes of at least 76 participants were more likely to have included changes than the fourteen trials with fewer participants (probability, 0.81 v 0.50); of the trials with larger sample sizes, the 21 drug trials were less likely to have included changes than other trial types (0.67 v 0.95). Given the limited information available, recursive partitioning for individual methodology and analysis components was not possible.

Discussion

We found that changes to important design or analysis elements of original research plans and the publication of RCT findings are quite common. In our analysis of available protocols and research publications, we identified clear deviations from the original protocols for 153 of 181 trials (85%). Of the 118 trials with available protocols and clear or probable changes, modifications were most frequent for the primary analysis method (70% of trials), sample size (51%), and eligibility criteria (50%). Even when changes were clear, it was difficult to determine whether they were made in an appropriately blinded manner (without knowledge of outcomes by trial arm randomisation) based on the available documentation alone. After seeking clarification from the investigators in 124 trials with available protocols, the proportion deemed to include clear or probable changes was lower (63%); in 61 of these trials (78%), changes had been made in an appropriately blinded manner. Most RCTs were therefore not subject to illegitimate changes affecting the validity of their findings.

As we included only published RCTs in our sample, a large proportion (33%) reported statistically significant findings for their primary outcomes. A follow‐up analysis of currently unpublished trials excluded from our study (should they be subsequently published) could yield a different proportion.

The 124 trials with available protocols differed from the 57 without full protocols in certain respects. A larger proportion of trials without protocols were conducted outside hospitals, their mean planned sample size was smaller, and a larger proportion involved mental health interventions. The outcomes of such trials may require closer scrutiny. Clear changes to trial conduct were less frequently identified for trials with full protocols (78 of 124, 63%) than for those without available protocols (42 of 57, 74%), possibly because more documentation was available for review.

In contrast to an earlier review,8 we found that changes to the primary analysis method were more frequent than changes to the primary outcome. While changing the primary outcome could clearly affect the conclusions drawn from an RCT, a large number of combinations of individual decisions define the primary analysis method. Flexibility in reaching each decision post hoc, after the results are known, entails the risk of selective analysis reporting and thus bias.

Based on our findings, we make three major recommendations. First, investigators should adhere to the principles for implementing and documenting important trial modifications detailed in the recent CONSERVE (CONSORT and SPIRIT extension for RCTs revised in extenuating circumstances) 2021 statement.17 Second, investigators should include contingency plans in RCT protocols for accommodating foreseeable challenges to the original study design (eg, planned sample size not achieved, poor adherence to trial interventions, missing data). Third, all versions of an RCT protocol or analysis plan should be publicly accessible (eg, in a trial registry) to promote robust peer review. Such transparency could be encouraged by journals that publish RCT findings and by ethics committees considering protocols or their amendment.

Limitations

We prospectively sampled all eligible trials registered in the ANZCTR, reviewed all available protocol materials in depth, and consulted trial investigators when clarifications were needed. However, we relied on investigators providing documents and information if they were not publicly available, and we accepted their responses without question. We could undertake only limited reviews of 57 eligible trials (31%) for which protocols were not available.

Conclusion

Changes to the design and methodology of RCTs can be legitimate, or sources of bias. Based on document review alone, we could establish that changes to certain elements of RCTs were frequent (and often undeclared and unexplained), but it was difficult to determine whether the changes were made blinded to treatment outcomes. Changes were less frequent in trials with publicly available protocols and consequently more open to scrutiny. We advocate a three‐step approach to improving documentation and peer review to allow readers to assess the scientific validity of published research findings.

Box 1 – Selection of the clinical trials for inclusion in our study

---