Medical schools provide an important service to society by creating a cadre of practitioners responsible for healthcare. It is thus important to ensure that medical education is of acceptable and consistent quality and produces practitioners who can meet society’s needs. While Australian medical schools take considerable care in selecting applicants and introducing innovative teaching methods, their programs vary substantially in both process and content.1 In addition, curriculum change is becoming more challenging because of intense competition for time and space in medical courses (see Box). While diversity of teaching methods and content may be desirable, it is important to ensure that the outcomes of the teaching process are of an appropriate standard, irrespective of methods. At present, there is no process to evaluate these outcomes.
A national exit examination could help monitor knowledge and skills of medical graduates. Is it time for Australia to consider it?
Most data on standardised assessment programs come from the United States and Canada, which have had such programs for many years. These data demonstrate that an exit examination can be valid7 and reliable, and can correlate well with clinical skills and future performance in multiple disciplines.8-10 Such an examination can serve as a monitor of performance of individual medical schools and as a selection tool for postgraduate training programs that may be fairer than letters of recommendation. It may also act as a monitor of curriculum change or an indicator of the need for change. Data on foreign medical graduates sitting the United States Medical Licensing Examination indicate that Australian graduates perform very well (Dr J Boulet, Director of Research, Foundation for Advancement of International Medical Education and Research, USA, personal communication, Dec 2004). However, as the cohort is small and highly selected, it may not be representative of Australian graduates overall and does not allow analysis of individual medical schools.
Some countries, notably Germany and France, have state-based medical examinations, while others, such as the United Kingdom, base eligibility to practise on evidence of graduation from an accredited medical school. With the increased globalisation of medical workforces, the lack of a standard approach has been identified as a barrier to improving the medical workforce in the European Union.11
Australia has no standardised national exit examination. While debate on its merits is not new, such an examination has been previously regarded as unnecessary. This was because it has been assumed that the quality of Australian medical graduates was high, and the process of accrediting medical schools (established to ensure appropriate standards) was excellent, and because no medical school has ever been singled out as of low quality. Why change if there is no evidence of a problem? We propose the following reasons.
Firstly, as we do not systematically evaluate outcomes of individual medical schools, we can not be certain that the above assumptions are correct. The challenge to curriculum delivery caused by the continuing expansion of medical knowledge is but one justification for such scrutiny.
Secondly, systematic monitoring and improvement of outcomes may be desirable, even if outcomes are acceptable, and may prevent future problems. While we recognise potential downsides to a national exit examination, we argue for such an examination as a way of monitoring outcomes to stimulate and inform further improvement.
Passing a national examination requires a minimum standard of performance. This standard can serve as a target for attainment, a measure for comparing outcomes between universities and for monitoring the effects of curriculum change, and a benchmark for new medical schools. A national exit examination would necessarily entail the explicit statement of professional values and expectations, a laudable process in itself.
The examination can also be used to measure the performance of overseas-trained doctors, allowing fairer assessment of these doctors. Standardisation of assessment might also facilitate the recognition of Australian-trained graduates overseas and the development of reciprocal arrangements with other countries that use standard exit examinations.
Finally, a change to the assessment process can influence curriculum change.12
The main disadvantage of an exit examination is that it might lock some medical schools into merely preparing students for the examination and thus restrict their ability and willingness to explore more innovative teaching. However, innovative teaching is not an end in itself but a means to obtain better outcomes. There is no way of knowing if outcomes are improved unless they are measured.
It is important to ensure that an exit examination does not become the sole source of evaluation of students and medical schools, and thus become an inflexible tool that ignores regional and other differences. An exit examination should be designed to complement rather than replace the range of evaluation methods used by individual medical schools (including objective structured clinical examinations [OSCEs], portfolios and logbooks). The content as well as the format of the examination would need to reflect the desired outcomes. Thus, it might need to include a clinical component and might not be solely a written and/or multiple-choice examination — although data from the United States suggest that performance on the written Medical Licensing Examination correlates well with performance in structured clinical examinations.13 As it is increasingly recognised that students need to demonstrate skills (eg, problem solving) as well as knowledge, these skills must also be evaluated in the examination.
An important challenge is the need for resources to create and maintain a quality assessment tool. Are the advantages of the examination sufficient to justify its expense? The answer may lie in the observation of the founding father of the study of management, Peter Drucker, that “If you can’t measure it, you can’t manage it”. Our ability to measure outcomes of Australian medical education is currently limited.
A concern about the exit examination is its use to rank medical schools, resulting in competition between them. This concern seems surprising, as ranking of Australian universities already occurs.14
Finally, it can be argued that an exit examination is superfluous, as a range of competency examinations are already performed at the national level in specialty training programs. However, these examinations do not allow monitoring of medical school outcomes or feedback on the curricula. Furthermore, it is worth noting that, as all specialist examinations are already national, it is the individual university assessment of medical students that is out of step.
Other ways of influencing medical school outcomes are already in place, including accreditation and the development of “ideal” curricula.5 The Australian Medical Council — the main accreditation body — sets out the principles and standards of medical education, including assessment. However, these relate more to process than to curriculum content, with the latter left to the judgement of individual medical schools.15 There are few data on whether these processes actually lead to desired outcomes.
It may be that defining the desired outcomes should be the intermediate step before an exit examination is considered. This was the strategy of the Scottish Deans’ Medical Curriculum Group, which developed an agreed set of outcomes defining qualities and abilities of graduates from any Scottish medical school.16 This was then used to provide a common approach to curriculum and assessment.
An alternative, indirect measure of medical school outcomes would be feedback from the specialty colleges on results of their examinations. However, this may be cumbersome to coordinate, and demand as many resources as a national examination.
While the common perception is that the general competence of Australian medical graduates is of a high standard, the unanswered question is whether it might be improved by better monitoring of the outcomes of medical student education.
With the worsening shortage of doctors and the need for appropriately trained medical graduates, the time has come to ask not only how many medical graduates do we need, but also what standards should they achieve? A national exit examination gives society a way to set standards for acceptable levels of competence, and to monitor and influence standards to match future needs and expectations.
There is no doubt that an exit examination is only as effective as the assessment tools it uses. However, if some of the inventiveness and attention currently dedicated to curriculum design were redirected to developing a national assessment process, it is possible that this process could reliably inform the outcomes of medical student education and serve as a platform for continuing improvement.
And if medical education is as good as we believe, what are we afraid of?
The Oncology Education Committee and curriculum change
For over 10 years, the Oncology Education Committee of the Cancer Council of Australia has been monitoring the quality of cancer education among medical students in Australia,2-4 working towards ensuring that the knowledge of medical students more closely meets the needs of society. Cancer affects one in three Australians, making it an important component of medical student education. The Committee has developed an Ideal Oncology Curriculum,5 which has been endorsed by the International Union Against Cancer. The Committee has also been gathering and promoting educational resources for medical students.
Despite these attempts to promote adequate education, data show that graduates’ knowledge of cancer remains variable and at times suboptimal across different medical schools, while their exposure to cancer in medical curricula is less than it was 10 years ago.6 Discussions with students and medical school representatives about how best to ensure the inclusion of cancer knowledge and skills in medical school curricula have revealed enormous pressures from many sources to add to already overfilled programs. The final outcome is often driven more by the preferences, interests and capabilities of individual staff and students and less by externally imposed targets. Without uniformly agreed outcomes to aspire to, and a system to monitor their achievement, curriculum change becomes nearly impossible.
- Bogda Koczwara1
- Martin H N Tattersall2
- Martin R Stockler3
- Michael B Barton4
- Brendon J Coventry5
- Ian N Olver6
- Joanna M Dewar7
- Jeremy L Millar8
- Max A Schwarz9
- Darren L Starmer10
- David R Turner11
- 1 Flinders Medical Centre, Adelaide, SA.
- 2 University of Sydney, Sydney, NSW.
- 3 Liverpool Health Service, Sydney, NSW.
- 4 Royal Adelaide Hospital, Adelaide, SA.
- 5 Sir Charles Gairdner Hospital, Perth, WA.
- 6 The Alfred Hospital, Melbourne, VIC.
- 7 University of Western Australia, Perth, WA.
- 8 Flinders University, Adelaide, SA.
None identified.
- 1. Barton MB, Simons RG, for the Oncology Education Committee of the Australian Cancer Society. A survey of cancer curricula in Australian and New Zealand medical schools in 1997. Med J Aust 1999; 170: 225-227.
- 2. Smith WT, Tattersall MHN, Irwig LM, Langlands AO. Undergraduate education about cancer. Eur J Cancer 1991; 27: 1448-1453.
- 3. Tattersall MHN, Langlands AO. Oncology curricula in Australia. Med J Aust 1993; 158: 224-225.
- 4. Tattersall MHN, Langlands AO, Smith W, Irwig L. Undergraduate education about cancer. A survey of clinical oncologists and clinicians responsible for cancer teaching in Australian Medical Schools. Eur J Cancer 1993; 29A: 1639-1642.
- 5. Cancer Council Australia. Ideal oncology curriculum for medical schools. Sydney: Australian Cancer Society, 1999. Available at: www.cancer.org.au/documents/Ideal_Oncology_Curric_HI.pdf (accessed Jan 2005).
- 6. Barton MB, Tattersall MH, Butow PN, et al. Cancer knowledge and skills of interns in Australia and New Zealand in 2001: comparison with 1990, and between course types. Med J Aust 2003; 178: 285–289. <MJA full text>
- 7. Van Zanten M, Boulet JR, McKinley DW. Correlates of performance of the ECFMG Clinical Skills Assessment: influences of candidate characteristics on performance. Acad Med 2003; 78 (Suppl): S72-S74.
- 8. Andriole DA, Jeffe DB, Whelan AJ. What predicts surgical internship performance? Am J Surg 2004; 188: 161-164.
- 9. Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA 2002; 288: 3019-3026.
- 10. Tamblyn R, Abrahamowicz M, Brailovsky C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA 1998; 280: 989-996.
- 11. Richards J. Medical issues within a European dimension. Europa 2000; 4 (1). Web journal available at: www.intellectbooks.com/europa/number9/medic.htm (accessed Jan 2005).
- 12. Newble DI, Jaeger K. The effect of assessments and examinations on the learning of medical students. Med Educ 1983; 17: 165-171.
- 13. Simon SR, Volkan K, Hamann C, et al. The relationship between second-year medical students’ OSCE scores and USMLE Step 1 scores. Med Teach 2002; 24: 535-539.
- 14. Australian Education Network. Rankings of Australian universities. Available at: www.australian-universities.com/rankings.php (accessed Jan 2005).
- 15. Australian Medical Council. Assessment and accreditation of medical schools: standards and procedures. Canberra: Australian Medical Council, 2002. Available at: www.amc.org.au/forms/AccredGuidelines.pdf (accessed Jan 2005).
- 16. Scottish Deans’ Medical Curriculum Group. The Scottish doctor. Undergraduate learning outcomes and their assessment: a foundation for competent and reflective practitioners. Scottish Deans’ Medical Curriculum Group, 2002. Available at: www.scottishdoctor.org/resources/scotdoc2.pdf (accessed Jan 2005).
Abstract
Although it is commonly assumed that the quality of medical school education in Australia is uniformly high, there is no national process for assessing its outcomes.
There is substantial variability in the content of medical school curricula, and the process of curriculum change is becoming more challenging because of intense competition for time and space in the course.
A national exit examination could provide a uniform standard of assessment for all medical school graduates in Australia, as well as foreign graduates applying to work in Australia.
Such an examination could assess medical school outcomes, monitor the effects of curriculum change, and provide a benchmark for new medical schools that would help medical curricula evolve to better meet society’s needs.