The potential of workplace-based assessment of international medical graduates

The concept is attractive — but capacity may limit its practicality

For many years, Australia has relied on supplementing its medical workforce with doctors who have qualified outside Australia. Each year, about 2500 of these medical practitioners, known as international medical graduates (IMGs), seek general registration with the Medical Board of Australia. For many IMGs, this has included sitting the clinical examinations conducted by the Australian Medical Council (AMC) as part of the Standard Pathway for IMGs. The eligibility standard for registration is set at the expected level of an Australian medical graduate at the time they complete their internship.1 Concerns have been expressed about the accessibility of these examinations and the ability of IMG candidates to pass them. Some of these problems were highlighted during an inquiry in 2011–2012 by the House of Representatives’ Standing Committee on Health and Ageing, Lost in the labyrinth.2

Workplace-based assessments (WBAs) have been developed as alternatives to the Objective Structured Clinical Examination (OSCE) and other approaches for assessing clinical competence.3 They have the perceived advantage of allowing a single set of high stakes (summative) assessments in an examination environment to be replaced by multiple low stakes (formative) assessments conducted by supervising clinicians over a period of time and in the workplace. These methods have been progressively adopted in recent years by medical specialist colleges. The AMC commissioned a trial of WBAs as an alternative to their clinical examination in 2010, and have subsequently incorporated WBAs into the Standard Pathway.

An article by Nair and colleagues in this issue of the MJA evaluates the reliability of WBAs in this context.4 The usefulness of an assessment method relies on a number of psychometric criteria being fulfilled. These include the concepts of validity (does the assessment method reflect performance in practice?), reliability (is it reproducible and consistent?), feasibility (is it an efficient use of resources?), acceptability, and educational impact.5 WBAs are generally recognised as having high validity, as they are conducted in the workplace where the doctor is practising and are modelled on and executed as part of normal clinical practice.

Reports have previously been published about the feasibility and acceptability of WBAs for assessing IMGs, and on the reliability of a single method, the mini-clinical evaluation exercise (mini-CEX).6-8 A combination of assessment methods, however, allows different aspects of clinical practice to be assessed.3 The article by Nair and colleagues examined the reliability of such a combination (“composite reliability”), and found that it was good (reliability coefficient greater than 0.8).

Why is this important? In an examination process, reliability can be ensured by standardised processes, similar formats, and controlling the examiners, candidates and the examination environment. However, this often compromises the validity of the examination. The increased validity associated with WBAs, on the other hand, can affect reliability because of variations and pressures in the clinical environment, the fact that examinees are dealing with real patients, differences in assessment formats, and the lesser control over the candidates and assessors. A high degree of reliability for a combination of assessments indicates that candidates are being assessed in a standard and reproducible manner and to an equivalent standard of competence, comparable with the standard AMC examination process. The results reported by Nair and colleagues are, however, for a single program; for these findings to be generalised to other WBA programs for IMGs, further research may be required.

What do these results mean for assessing IMGs in the future? In 2015, 84 candidates participated in the WBA program, and 76 completed it successfully. In comparison, changes to the AMC processes and the establishment of the Vernon C. Marshall National Test Centre in Melbourne have allowed 2000 candidates to attempt the clinical examination over the same period (with about 590 completing it successfully).9 While WBAs have been shown to be feasible, affordable and reliable, they require resourcing and a commitment from the host institution. Further, candidates need to be recruited to a training position in a hospital offering the program. Investing in the program has rewarded some hospitals with improved recruitment and retention of practitioners, which is important in regional areas where this can be difficult. However, competition for training positions has increased; there are more than 3000 domestic medical graduates each year who need pre-vocational training.10

The AMC clinical examination offers access to a greater number of IMGs (possibly reflected in the lower rate of successful completions than for WBA). Australia continues to be an attractive destination for medical graduates from other countries, and the demand for assessment will therefore continue. WBAs will be a useful part of that assessment, but there are limits to the number of candidates who can be accommodated by this approach, especially when compared with the AMC examination.