The efficacy of medical student selection tools in Australia and New Zealand

Entry to a medical program in Australia is highly competitive. Medical schools must allocate fairly the limited number of places available by identifying applicants with the potential to become successful doctors. To do so, they employ selection tools that include aptitude tests (eg, the Undergraduate Medicine and Health Sciences Admission Test [UMAT]), assessment of academic achievements (eg, final secondary school score or rank; university grade point average [GPA]), selection interviews (panel interviews, multiple mini-interviews), simulations and situational judgement tests, psychological tests, and random selection.1-6 Most schools employ a combination of weighted or unweighted scores to prepare a list from which the most highly ranked applicants are selected.2,7,8

The predictive power (efficacy) of selection tools can be estimated in multiple linear regression models. In general, the ability of selection tools to predict outcomes such as course completion (binary) or academic achievement throughout the medical program (including final grade; either continuous or categorical) is limited; the exception is prior academic achievement, which accounts for up to 23% of variance in academic outcomes.9 The predictive value of a battery of selection tools is generally calculated with a single equation, and the impact of each tool is measured while controlling variability in the other tools.1,10 Determining the efficacy of a combination of tools requires multiple regression models or multivariate analyses.2,11,12 For continuous outcomes, the impact of the selection tool score is measured against a unit of change in the outcome score without special treatment of the critical range around the pass–fail threshold. For example, if the pass mark is 50%, the difference between achieving 51% and 98% is significant, but the consequences for the student (and perhaps the university and health care system) are more serious if the student achieves 48% rather than 51%. As the pass rate in medicine usually exceeds 80%, the overall predictive power of a selection tool that poorly distinguishes between a low and a high pass (despite accurately distinguishing between passing and failing) would appear to be low.13 Consequently, the conclusions about how best to apply selection tools that can be drawn from regression models and multivariate analyses are limited.

Another reason for the low predictive value of currently available tools is the difference between research models of student selection and how selection tools are actually applied.8 Medical schools rarely publish how the scores of selection tools are combined, but in research studies they are usually analysed as averages or cumulative scores. Further, few studies of combinations of selection tools have controlled for confounders, such as ethnic background or sex.

Building on earlier research,1,2,10,11,14-17 we proposed a different approach to evaluating the predictive value of selection tools. Specifically, our study focused on the efficacy of selection tools for predicting medical program success when the outcomes are binary (completing or not completing medical school; passing or failing a key examination) rather than continuous (eg, test scores, final GPA) or ordinal (eg, fail, pass, pass with distinction). We proposed that, although whether a student is good or very good at medical school is important, it is more critical to distinguish between their passing and failing. Our investigation thus differed from earlier studies by focusing on outcomes from a consequential rather than a metric perspective.

Our second aim was to investigate whether or not our approach would support alternative methods for assessing the quality of selection algorithms that do not allow low scores on one tool to be compensated by high scores on other tools, as do, for example, averages or cumulative scores.

Methods

Context and data

Five undergraduate medical schools in Australia and New Zealand provided both selection and outcome data for 3378 students from four consecutive cohorts of students (enrolled 2007–2010) (Box 1). The predictor (independent) variables were student scores on the selection tools employed: a measure of prior academic achievement (GPA or Australian Tertiary Admission Rank [ATAR]), UMAT, and selection interviews (as applicable). The outcome (dependent) variables were graduation from the program in a timely fashion, and passing the final clinical skills assessment at the first attempt (Box 2). Timely graduation was defined as graduating no later than one year beyond the minimum study period. Some students complete their degree later for academic or personal reasons, but this is relatively rare; access to relevant data in this regard was not approved by the institutions’ ethics committees for reasons of privacy. Methods for the final clinical skills assessment included objective structured clinical examinations and workplace-based assessments.

Statistical analysis

As we could not measure the probability of success of applicants not admitted to medical school, we employed a feasible model in which the probability of success in the main outcomes among the admitted applicants at each school was assessed optimistically; that is, an algorithm that maximises the estimated accuracy of the selection decision. We used discriminant function analysis (DFA)18,19 to identify the cut-score for each selection tool that best discriminated between students who achieved or failed to achieve an outcome. DFA predicts the classification of subjects into pre-defined categories of a dependent variable (outcome) according to one or more continuous or binary independent variables (predictors). DFA and logistic regression yield results that are almost identical when there are no nominal predictors and the distribution of the continuous predictors is approximately bell-shaped;20 these conditions applied to our study. DFA was preferred to logistic regression because its purpose is classification, whereas that of logistic regression is identifying the likelihood of observing a particular outcome following a one unit change in a continuous predictor.

For each outcome at each school, the cut-score was defined as the middle value between the two centroid points for each group yielded by DFA. To estimate statistical errors, expressed as a 95% confidence interval (CI), each DFA was calculated 100 times for random subsamples (50%) of the data.21 Confidence intervals for each cut-score could not be generated by bootstrapping (which employs 1000 random subsamples of different sizes) because bootstrapping provides 95% CIs for DFA coefficients but not for the cut-scores. Coe’s guideline22 was used to estimate effect sizes from the percentage of classifications that were correct. An effect size under 0.4 is considered small, and one of 0.6 or more as large. The association between the number of cut-scores met and timely graduation was measured by Goodman and Kruskal’s gamma.1

All analyses were conducted in SPSS 22 (IBM).

Alternative selection algorithms

After establishing cut-scores, we examined whether alternative algorithms might be useful for selection purposes. In contrast to a compensatory algorithm — in which a high score on one tool can compensate for a low score on another — a non-compensatory algorithm requires that students reach thresholds on each of several tools. A third option would be to apply a rule of “sufficient evidence”: that is, if the result for one selection tool indicates that an applicant is likely to succeed, they can be selected regardless of their scores on other tools.

Ethics approval

The research ethics committees of each institution endorsed the approval of this study by the Human Research Ethics Committee of the University of New South Wales (reference, HC15421, HREAP G: Health, Medical, Community and Social). To maintain the privacy of students and schools, we have not reported student outcomes, nor have we described the interview or assessment methods; further, schools may have since altered their selection models. In this article, results are reported anonymously for schools A to E; the order is unrelated to that of the schools in Box 2.

Results

The efficacy of each tool in predicting each of the two outcomes is summarised in Box 3 and Box 4. The efficacy of each selection tool was independent of that of the other tools. For both timely graduation and final clinical skills assessment, prior academic achievement (GPA or ATAR) was the most effective selection tool, with medium to very large effect sizes for all schools. The predictive power of UMAT scores was limited, with two exceptions: a medium effect size of mean UMAT score in predicting timely graduation at School A, and of individual UMAT components for the same outcome at School E. The effect sizes for selection interviews predicting timely graduation or passing the final clinical skills assessment were small; at School A, the estimated effect size for predicting timely graduation was negative, meaning that the cut-score predicted failure more often than it predicted success.

Having determined the optimal cut-scores, we explored the utility of other selection algorithms. The feasibility of a “sufficient evidence” algorithm can be tested only if it includes two or more selection tools with medium or large effect sizes (0.4 or more22), a condition that applied only to the outcome of timely graduation. The first tool selected for testing the “sufficient evidence” algorithm was the GPA/ATAR, as it consistently had the largest effect size; the second was the UMAT, because it was used by all schools.

The association between the number of cut-scores met (0, 1 or 2) and the likelihood of timely graduation is shown in Box 5. For School A, a second cut-score added little to the predictive power of the selection decision (increased from 95% to 97%), while only 81% of those who reached neither cut-score graduated in timely fashion. School A might thus benefit from using a “sufficient evidence” approach. Schools C and D might benefit from assessing only the GPA/ATAR cut-scores, as reaching the cut-score significantly increased the proportion who graduated on time compared with applicants who did not (from 78% to 88% and from 85% to 92% respectively). On the other hand, for Schools B and E, a cut-score-based selection algorithm would achieve little, as meeting even one cut-score did not significantly improve the timely graduation rate (Box 5).

Discussion

We have described a new approach to assessing the efficacy of medical student selection processes, after examining how well current selection tools predict two pragmatic binary outcomes: timely graduation and passing the final clinical skills assessment.

It has been reported that the predictive power of composite scores based on the weighted means of the results from multiple selection tools is low, explaining less than 15% of variance in student outcomes, regardless of the statistical models, selection tools, and outcomes used.1-4,9-11,13,14,16,23 We found similar effect sizes to previous studies, but the comprehensiveness and novel types of analyses are the major strengths of our study. We analysed large datasets from five medical schools that applied the student selection tools in different ways. As a result, our results may be more generalisable than those of studies that have focused on a single medical school, or on a particular tool in a particular jurisdiction.1,8,23 It is the broadest study of its type to date.

We employed techniques rarely used to investigate the efficacy of selection by medical schools. DFA is an effective tool for facilitating decision making,18,24 and our approach of 100 random re-samplings is more conservative (ie, yields broader 95% CIs) than bootstrapping.21 We estimated optimal cut-scores for each selection tool and school for predicting binary outcomes. The cut-scores and the proportions of correct classifications may be useful for evaluating current tools and policies and informing future selection decisions. This technique could also be applied to generating cut-scores that predict progression from earlier to later program phases.

The cut-scores for timely graduation and passing the final clinical skills assessment were similar for different selection tools employed by a school. This may not be surprising; the clinical skills assessment, required for graduation, is undertaken late in the medical program. We selected these two outcomes because they are understandable and were available for all schools. The outcome of subsequent workplace performance, while important, is moderated by influences beyond the undergraduate environment, including experience as interns, postgraduate training, and personal circumstances. Timely graduation and passing the final clinical skills assessment might, however, be considered surrogate measures of subsequent clinical practice, as medical schools are guided by graduate outcomes when designing summative clinical skills assessments and making decisions about progression, including about whether a student should graduate.

If all selection tools are regarded as being important for selection, our findings suggest that schools might consider non-compensatory algorithms, as students should not be selected if they perform poorly on a selection tool that reliably predicts outcomes.7,17 The “sufficient evidence” approach could be implemented in two ways. If the selection tools each measure a different, independent set of attributes (ie, correlations between scores on different tools are small), an applicant achieving a higher number of cut-scores has provided more evidence that they are suitable for medical training across a broader range of attributes than one who has achieved fewer, and should be ranked higher for selection. Alternatively, if a particular tool reliably predicts an outcome, and the other tools add little to predictive accuracy, achieving the cut-score on the reliable tool would suffice for selection (a “sufficient evidence” approach), while the results from multiple tools could be combined for applicants who did not achieve the cut-score on the reliable tool.

Our investigation confirms the findings of local studies that prior academic performance is the most effective tool for predicting academic outcomes, and that neither UMAT nor interview scores have a consistent impact.1,2,10,16,17 Most studies have assessed continuous outcomes, and admission GPA/ATAR explained no more than 10% of variance.1,2,10,16,17 Only one study examined the effectiveness of selection tools in predicting success v failure at the end of the program as a binary outcome.1 This binary approach may be increasingly important as schools seek to further diversify their student intake to better meet health and community needs. Schools may need to focus more on improving the accuracy and validity of selection tools close to the selection threshold mark rather than at the upper end of the scale.25

One limitation of our study is that we could not measure outcomes for unsuccessful applicants. Another is that cut-scores were based on outcomes for earlier student cohorts, and we assumed that these applicants were similar to the current admission cohort. Further, the failure rates for our two outcomes were low, as is typical for a study examining success and failure in medical programs. When one of the dependent variable categories includes only a small proportion of the study population, the probability of a type I error increases. Nonetheless, we analysed data from four consecutive cohorts at each of five medical schools in Australia and New Zealand, and the analysis included 100 random re-samplings, achieving narrow 95% CIs for the cut-scores. This suggests that our assumption of stability across cohorts is valid, and supports the generalisability of our findings. Finally, the low predictive value of a selection tool such as interview may indicate an inability to effectively select for outcome success, but the tool may still have a high level of effectiveness in “selecting out” unsuitable applicants whose data are not included in the study.

In conclusion, there is no gold standard for selecting medical students. Schools must choose fairly the applicants they believe have the aptitude to meet the community’s health care needs. Completing the medical training program is the first step. Medical schools periodically evaluate their selection tools and algorithms according to accumulated evidence derived from institutional data. Our study suggests that new methods may be useful for guiding selection policy and processes, including using meaningful cut-scores for selection based on important program outcomes, and applying non-compensatory selection algorithms or a “sufficient evidence” algorithm.