Predictive validity of the Undergraduate Medicine and Health Sciences Admission Test for medical students’ academic performance

Selection of students into medical degree programs is highly visible and competitive. On one hand, there are many more suitable candidates than there are places. This requires the selection process to manage supply and demand in a defensible and transparent manner, usually focusing on academic ability. On the other hand, there is an understandable desire to select medical students who are most likely to become good doctors. This entails considering non-academic factors.1

In Australia and New Zealand, selection of students into medical study directly from high school typically uses a combination of academic performance at school, performance on a standardised test (the Undergraduate Medicine and Health Sciences Admission Test [UMAT]) and an interview. In 2009, 14 universities in Australia and New Zealand used the UMAT as part of their selection processes.

Devised by the Australian Council for Educational Research, the UMAT comprises three parts: Section 1 (logical reasoning and problem solving), Section 2 (understanding people) and Section 3 (non-verbal reasoning).2 In Section 1, students are required to exercise reasoning and problem-solving skills. Section 2 assesses a student’s ability to understand and think about people. Items are based on passages of text representing specific interpersonal situations. Section 3 consists of abstract items that are designed to evaluate a student’s ability to exercise non-verbal reasoning skills.3

There are little published data on the predictive validity of UMAT for medical student academic performance,4 although the impact of coaching and the association with emotional intelligence have been explored.3,5 This lack of data is surprising given the widespread use of UMAT.

These are the first peer-reviewed, published data reporting predictive validity of UMAT for academic performance at university. Our study shows only weak correlation between UMAT overall score and program GPA. Further, the weak correlation did not persist beyond the first year of university study, and in multivariate analysis, correlation was limited to UMAT Section 1 score.

What might explain this lack of predictive validity? The students in our analysis are, like most students entering medical programs, highly selected and high performing. Predictably, these students perform very well on the UMAT. At university, most of our students also perform very well; the mean GPA in our study was 6.1 out of a possible 7. Similar factors occur when studying other aspects of selection,1,7 and range restriction is an important limitation to this type of analysis. Statistical manipulation to control for this is possible, but we found that the correlation coefficient increased by no more than 0.03. This suggests that range restriction is not a major limitation here. Statistical power in our study was limited by the number of students available, especially in the later years of the study program. Larger studies, including those combining data from several medical schools would be beneficial.

Most students in our study have not yet completed their medical program. It is important to continue and extend our analysis throughout the medical program as it is possible that correlations between UMAT and performance during clinical training may emerge. Further, we have only reported on overall GPA, and have not explored correlation between UMAT scores and individual components of assessment such as knowledge, clinical skills or professionalism. This further research is important and will be done. Long-term follow-up into specialty training and clinical practice would shed more light on the value of UMAT in selection, as its value may become more evident later.

Selection of students into medical programs continues to be controversial, with demand outstripping supply, and with a widespread desire to select those students deemed more likely to become better doctors. UMAT does not seem to have useful validity in terms of predicting academic performance at university. Further research is needed to determine whether UMAT is predictive of performance during clinical training at medical school, in postgraduate training environments, and in clinical practice.

1 Correlation between overall UMAT mean score and undergraduate GPA

---

GPA = grade point average.
R2 = coefficient of determination for multivariate analysis.
UMAT = Undergraduate Medicine and Health Sciences Admission Test.