Admission to medical school is highly competitive and generally based on a combination of academic achievement, written tests and an interview. In Australia, two tests are widely used: the Graduate Australian Medical School Admissions Test (GAMSAT) and the Undergraduate Medicine and Health Sciences Admission Test (UMAT). GAMSAT is designed to assess problem-solving and data interpretation in the social, physical and biological sciences, as well as critical thinking, reasoning and written communication.1 In 2005, GAMSAT contributed to the selection of about half the medical students in Australia. However, no systematic analysis of GAMSAT’s predictive validity has been published.
The predictive validity of admissions tests is contentious, with inconsistent results from studies evaluating the reliability of cognitive measures as indicators of academic or clinical performance.2-4 In the United States, Mitchell et al found that the Medical College Admission Test (MCAT)-predicted grades in medical school were only slightly better than performance at school.5 In contrast, a review of medical schools in the United Kingdom concluded that pure measures of reasoning ability are less predictive than measures of knowledge, such as A-levels.6 However, there is evidence that personality factors have been underexplored and that situational tests which measure particular constructs may be more useful than cognitive tests.7 Interviews, another popular measure, may in fact be poor predictors of behavioural and attitudinal aptitude. Although interviews attempt to assess personality characteristics, their inherently subjective nature may reduce reliability.8
A hallmark of expert clinical performance is skill in clinical reasoning. Although it has been shown that clinical reasoning develops throughout undergraduate medical training,9 studies investigating the predictive validity of students’ entry characteristics commonly focus on their relationship with overall academic performance or performance as interns.10-13 Two studies have investigated the association between pre-admission test scores and clinical performance, including clinical reasoning, at graduation,14,15 and they yielded conflicting results.
We sought to confirm and extend the findings of a previous study at one graduate-entry medical school.16 That study found a significant relationship between pre-admission interview performance and development of clinical reasoning skill, but no relationship between interview performance and GAMSAT results or between GAMSAT results and the development of clinical reasoning skill in medical school. In this study, we asked to what extent do students’ entry characteristics, particularly performance against specific admission criteria, predict clinical reasoning skill and academic performance?
Participants were students enrolled in Years 2–4 of the graduate-entry medical programs at the Universities of Queensland (UQ) and Sydney (USyd), recruited on a voluntary basis.
GAMSAT consists of three sections: Reasoning in Humanities and Social Sciences, Written Communication, and Reasoning in Biological and Physical Sciences. It has been designed to:
assess the capacity to undertake high level intellectual studies in a demanding course . . . GAMSAT evaluates the nature and extent of abilities and skills gained through prior experience and learning, including the mastery and use of concepts in basic science as well as the acquisition of more general skills in problem solving, critical thinking and writing.17
At UQ, enrolment offers are based equally on GAMSAT and interview score, while at USyd, initial offers are based on interview score only.
Two instruments were used to assess clinical reasoning: a set of Clinical Reasoning Problems (CRPs),18 and the Diagnostic Thinking Inventory (DTI).19 Although both are reliable and valid methods of assessing clinical reasoning,18 their emphases are different. The CRPs focus on the integration of knowledge (biomedical and clinical) with reasoning ability, while the DTI is a self-report questionnaire that explores the respondent’s cognitive approach to the clinical reasoning process, and is independent of knowledge. The DTI has been widely used to investigate the relationship between clinical reasoning skill and various student characteristics such as learning style, communication skill and preclinical background.20-23 Both instruments were used in the original study.16
Participants completed the DTI and the CRPs without supervision and without a time limit. Participants were asked not to refer to textbooks or consult colleagues.
The entry characteristics (sex, age at commencement of the program, nature of previous tertiary qualification, and scores on selection criteria [GAMSAT and interview]) for participants and non-participants were provided on request from each school and recorded, along with year of medical training and academic performance. For participants, second-year examination results were obtained and used as the measure of academic performance, as this is a common assessment point for both universities.
We used the same analytical approach as in the original work.16 For each medical school, descriptive statistics, contingency tables and t tests were performed for participants and their corresponding year group to determine whether the results derived from the study data were generalisable to the entire year group.
To maximise statistical power, Fisher’s method was used to calculate combined significance levels for each outcome variable.
A total of 189 students participated over the 2003 academic year (Box 1), representing an overall participation rate of 13.6%.
For both universities, no significant differences were found between participants and non-participants in their year group with regard to sex, academic background, GAMSAT or interview score. However, Year 3 participants were older than non-participants at both UQ (P < 0.01) and USyd (P = 0.04). Participating Year 3 students at UQ also scored higher on the written communication component of GAMSAT than their year group (69.5 compared with 64.1, P < 0.01). At USyd, Year 2 students who participated scored higher on this component of GAMSAT than their year group (65.9 compared with 63.4, P = 0.02).
UQ participants were generally younger and scored higher on GAMSAT, including two of the three subscales, than USyd participants. There were no differences in interview scores. Both groups showed a progressive increase in scores on the two clinical reasoning indicators (CRPs and DTI). In each case, the structure of knowledge subscale of the DTI was primarily responsible for this increase.
Analysis of three of the five entry characteristics (age, sex and academic background) found no age- or sex-related differences for clinical reasoning or academic performance. However, students with a non-biological science-based primary degree scored lower for both the CRPs (P < 0.01) and the Year 2 exams (P = 0.03). No differences were found in relation to DTI scores (Box 2).
Box 3 shows the correlation of the remaining characteristics, GAMSAT and interview scores, with scores for the CRPs, DTI and the second-year barrier examination. There was a weak negative correlation between scores for GAMSAT (total) and the DTI, which was significant when both universities’ data were combined (−0.05 > r > −0.31, P = 0.03), but no association with the CRPs. The correlation between GAMSAT and examination results was weak (r < 0.24, P = 0.02). The correlation between GAMSAT and interview scores was weakly negative for UQ (r = − 0.34, P < 0.01) and weakly positive for USyd (r = 0.11), with a combined P < 0.01. Neither school showed an association between interview scores and any of the outcome variables.
We found little evidence that students’ entry characteristics significantly predict performance during medical training. The minor differences in average entry characteristics between the two universities generally reflect differences in admission criteria or the secondary education system; for example, schools in New South Wales offer an extra year of pre-university education compared with Queensland, and applicants to the USyd’s medical program require a minimum grade point average (GPA) of 5, compared with 4.5 for UQ. These differences have a small effect on the threshold GAMSAT scores at each school.
We found no age- or sex-related differences in clinical reasoning or academic performance. Although a non-biological science-based primary degree was shown to have an impact, this is not large enough to cause a difference in pass rate, and in-house analysis of assessment data at UQ has consistently indicated that any differences between students based on academic background have disappeared by graduation. At USyd, there is a whole-year assessment in Year 3, but no final year assessment. However, it seems likely from the available evidence that we would not find differences at the time of graduation. Nor did we find any evidence to suggest that academic background contributes to the correlation of GAMSAT scores with Year 2 results.
Our results show that students from non-biological science-based academic backgrounds are not seriously disadvantaged either by the selection process or during the program; medical schools that actively seek a more diverse intake can be reassured that these students are just as likely to succeed.
The poor correlation between GAMSAT and interview scores suggests that they do assess different constructs, as intended. However, we found little evidence that either predicts clinical reasoning or academic performance to any substantial extent. Indeed, GAMSAT’s negative correlation with the DTI, a knowledge-independent measure of reasoning, and its small positive correlation with examination scores reinforce the argument that GAMSAT is primarily a measure of knowledge. Furthermore, we were unable to confirm previous findings of a significant relationship between pre-admission interview and subsequent clinical reasoning skill.16 Although this may be due to differences in the interview questions between the two universities, it could simply reflect differences in the sample size and statistical power of the two studies. Nevertheless, our findings suggest that interviews do not discriminate well between applicants,8 and support recent moves to other formats that are less subjective and produce more reliable and valid outcomes.24-26
The combined student response rate of 13.6% allows for sampling error and affects the study’s confidence level. Although detailed statistical analysis was beyond the scope of this study, our findings do confirm and extend the results of previous research16 with regard to GAMSAT.
Most medical schools try to select students who will develop into safe and effective medical practitioners. Research in this area usually proceeds in two steps: (1) do entrance exams predict performance in medical school? and (2) does performance in medical school predict performance in practice? We did not find a strong association between performance in the two most widely used selection measures and in medical school. Other studies have failed to find a strong association between performance in medical school and clinical performance after graduation. Medical school selection is such a high-stakes process that we need to clearly define the performance indicators, including such measures as intern assessments, to be targeted by the selection process. We then need to evaluate both GAMSAT and the interview against these indicators. Without such evidence, it is difficult to resist arguments for a process that is less expensive, elaborate, stressful and time-consuming, such as the GPA-weighted lottery method of selection used in the Netherlands.27
Despite the limitations of our study, the results challenge the predictive validity of GAMSAT and the reliability of admission interviews, even when they are semi-structured. Because entry to medical school is a highly valued goal and a well educated medical workforce such an important national asset, there is an urgent need for collaborative studies to explore these issues further. Many new medical schools are in development or recently accredited and would benefit from the experience of more established schools in avoiding error.
2 Relationship of clinical reasoning and academic achievement to medical school entry characteristics*
University of Queensland and University of Sydney combined P values (n = 189) |
|||||||||||||||
3 Correlation of medical school admissions criteria (GAMSAT, interview) with clinical reasoning (CRPs, DTI) and academic achievement (Year 2 results)
Abstract
Objective: To examine the relationship between medical school applicants’ performances in the Graduate Australian Medical School Admissions Test (GAMSAT) and structured interviews and their subsequent performance in medical school.
Design: Students in Years 2–4 of two graduate-entry medical programs were invited to complete two previously validated tests of clinical reasoning. These results and their Year 2 examination results were compared with their previous performance in GAMSAT and at interview.
Setting: The graduate-entry programs at the Universities of Queensland and Sydney.
Participants: 189 student volunteers (13.6% response rate).
Main outcome measures: Students’ test results on a set of Clinical Reasoning Problems (CRPs) and a Diagnostic Thinking Inventory (DTI) and their Year 2 examination results.
Results: There was no association between performance in GAMSAT and performance in the CRPs; there was a weak negative correlation between performance in GAMSAT and the DTI (− 0.05 > r > − 0.31, P = 0.03). The correlation between GAMSAT and examination results was weak (r < 0.24, P = 0.02). The correlation between GAMSAT and interview scores for each school was weakly negative for University of Queensland (r = − 0.34, P < 0.01) and weakly positive for University of Sydney (r = 0.11), with a combined significance level P < 0.01.
Conclusions: We did not find evidence that GAMSAT and structured interviews are good predictors of performance in medical school. Our study highlights a need for more rigorous evaluation of Australian medical school admissions tests.