Selecting medical students for academic and attitudinal outcomes in a Catholic medical school

Selection of medical students has been a topical subject for many years.1 In Australia, criterion-based approaches based on academic grades have traditionally been used in medical student selection. However, initiatives by the University of Newcastle, New South Wales, in the 1970s, followed by other Australian universities, set the scene for diversification in the criteria used to select medical students. Selection criteria went beyond simple grade levels, with the adoption of interviews designed to assess personal qualities of applicants.2,3 The broadening of the criteria for selection was intended to address perceived deficiencies in communication skills and emotional intelligence, which had been reported as criticisms of some medical practitioners in consumer surveys.4

In a Catholic medical school such as the University of Notre Dame Australia (UNDA), the outcomes of social justice rank equally with academic outcomes and formed a key part in planning a selection policy for medical students. In Australia, the health disparities in some communities mean there is a need for a committed workforce of doctors who will work all their lives in those communities.5 Delivering doctors to the health care workforce with a positive attitude towards underserved communities was identified as a strategy by which the Catholic mission of the medical school could be achieved.

To balance the competing needs of having graduates who achieve academic excellence and having graduates with positive attitudes towards serving underserved communities, UNDA implemented a selection process that involved two academic and two attitudinal criteria to generate a selection score. The aim of this study is to report on whether these academic and attitudinal criteria successfully improved the University’s selection of appropriate students.

GAMSAT score: The GAMSAT (http://www.gamsat.edu.au) was designed by the Australian Council for Educational Research with a consortium of graduate-entry medical schools, and has been used as a quantified measure of academic ability to undertake demanding academic courses. The GAMSAT test is divided into three sections: section 1 (humanities and social sciences), section 2 (written communication) and section 3 (reasoning in biological and physical sciences).

Interview score: The interview was a semi-structured vignette-based interview lasting 20 minutes and designed to assess non-cognitive personal attributes. It was based on existing interview styles designed to attract more generalist career students.6-8 The interview was conducted by a pair of interview-trained medical academic and/or community volunteer staff interviewers, who retrain every 2 years and are briefed on the day of the interviews by the Head of Selection. Interviewers individually scored students based on their responses to the vignettes, using a predefined and calibrated set of possible outcomes. They also scored candidates for general communication and group-working characteristics. At the conclusion of the interview, members of the interview panel finalised their marks and conferred with their co-interviewer. During this time, the candidates waited in a secure waiting area. If consensus between interviewers was not achieved, the candidate returned to the interview room and a further 10-minute interview was conducted, using additional vignettes.

Portfolio score: The portfolio score was derived from the UNDA supplementary information form.9 The form collected the following information: rural background; higher research degrees and research experience; work experience; sporting, music, church and other participation or leadership activities; experience working with charitable agencies or with underserved communities; languages spoken other than English; curriculum vitae; two written references; and a personal statement about why the applicant wished to study medicine at UNDA.

Staff members were trained in reading and evaluating the portfolios in a didactic session using cases as examples. Their first 20 portfolios were scored under supervision and the scores were not used in final assessments. Portfolios were scored using a criterion-referenced scale, and the scores were provided to the Head of Selection. The Head of Selection received at least two portfolio scores from separate reviewers, and in some cases a third evaluation was conducted for audit purposes.

A total of 223 students were enrolled in the Sydney medical school of UNDA: 111 (50%) in 2008 and 112 (50%) in 2009. The students’ demographic characteristics, selection scores and the key study outcomes are summarised in Box 1 and Box 2.

Results obtained from the bivariate analyses suggested some potential predictor variables for outcomes.

We were disappointed that we were unable to identify a significant association between the current University portfolio and interview system, and students who had positive attitudes towards serving underserved communities, as measured by the MSATU score. This outcome was selected by the study designers to fulfil the Catholic mission values of the University, as it encompassed the primary service mission of Catholic Health Australia (CHA). The foundation principle of service, as stated by CHA (http://www.cha.org.au/site.php?id=26), is:

Other universities trying to achieve non-academic and socially oriented outcomes in their medical graduates report similarly disappointing outcomes. A comparison study from Drew University in the United States that aimed to determine measurable factors for selecting students who would then work to service underserved communities found that these outcomes were driven by individual values and subjective external influences, and not by standard predictors.5 A second US study that compared a customary committee review selection process with an actuarial system found the two groups were ultimately not reliably differentiated on any outcome.12 In this case, the committee selection system used aptitude tests as well as basic demographic data, information on extracurricular activities, vocational interests, work experience, letters of recommendation, personality test profiles and interview impressions. This was very similar to the interview and supplementary information portfolio used in our study.

However, other studies have reported that these strategies are successful in selecting graduates who have a greater focus on serving the community. New York Medical College has had considerable success in meeting its targets since implementing a detailed portfolio and pre-admission questionnaire for students.6

Given the results of our trial, UNDA will need to review its selection system again, to attempt to identify students who meet both the academic and mission goals of the University. This process will start with community consultation and evaluation of the service goals to which the University would like its graduates to aspire. This process has now begun.