A systematic approach to workplace-based assessment for international medical graduates

Over the past few years, awareness of the need for workplace-based assessment (WBA) of junior doctors has increased. This has resulted in the development of many assessment tools. The most popular, researched and validated tools are mini-clinical evaluation exercises (mini-CEXs), case-based discussions (CBDs), in-training assessments (ITAs) and multisource feedback (MSF), such as 360º assessments.1-4

There are many reasons for WBA development:

• most trainees are not observed in practice, particularly in history taking or physical examination;5

• lack of feedback is a major gap in medical education generally and;

• constructive feedback is necessary to improve trainees’ learning and clinical performance.6

WBA should be conducted using multiple tools, since one tool cannot assess all domains, but there are many challenges to implemention. The teaching faculty may not engage in assessment, provide constructive feedback or monitor the impact of feedback on performance.7 WBA is also costly in time and resources.

Australia, like many other developed countries, relies on international medical graduates (IMGs) for service provision. In some locations up to 30% of doctors are IMGs.8 To gain general medical registration in Australia, the IMGs currently have to pass a multiple-choice examination, followed by a clinical skills examination, both of which are conducted by the Australian Medical Council (AMC). However, there is a long wait to undertake these exams. The AMC clinical exam is a 16-station, multidisciplinary, objective, structured exam.9 Each station is 8 minutes’ duration. One or two stations may include real patients with clinical signs, and the other stations use role players. A clear pass score is 12 or more stations passed, including at least one obstetrics and gynaecology station and one paediatrics station. Marginal performance is a pass in 10 or 11 stations; these candidates are offered a remedial examination. The average pass rate for the AMC clinical exam is around 50% (Ian Frank, Chief Executive Officer, AMC, personal communication, 24 January 2011). The AMC have been proactive in evaluating other assessment methods, and we were invited to submit an expression of interest to conduct WBA.

In 2010, the University of Newcastle School of Medicine and Public Health, in collaboration with Hunter New England Area Health Service, were accredited to conduct WBA for IMGs in place of the AMC clinical exam. The purpose was to test whether WBA is feasible in the workplace and whether it is acceptable to IMGs and assessors.

Ethics approval for the study was obtained from the Hunter New England Human Research Ethics Committee. All candidates gave written, informed consent before the study started.

WBAs were conducted over 6 months from June 2010 at four teaching hospitals in Newcastle. IMGs invited to participate were those who worked in these hospitals, had passed the English language requirements and the AMC multiple-choice exam, and were on the waiting list for the AMC clinical exam. The IMGs were working in medicine, surgery, emergency, paediatrics, mental health and obstetrics and gynaecology as junior medical officers. They were from India, Pakistan, Bangladesh and China, and had been working in Australia for 1–10 years. There were two calibration sessions attended by 65 assessors, and an information evening attended by all 27 candidates, to familiarise themselves with the protocols, assessment methods and tools.

The assessment tools used were mini-CEXs, CBDs, MSF and ITAs (Box 1). The mini-CEXs and CBDs were “blueprinted” (assessment tasks mapped out to align with course content and desired learning outcomes, so that each prescribed discipline and domain is represented) as outlined in Box 1. All assessors used well validated, individually labelled assessment forms (assessment forms are available from the authors). Candidates received immediate feedback from examiners on the mini-CEX and CBD assessments. Pass criteria for each assessment tool were predetermined by an expert panel and are summarised in Box 2.

Candidates had to pass eight of 12 mini-CEX cases, with a minimum of one pass in each of the six clinical disciplines. With a pass in seven mini-CEXs, candidates were deemed “borderline” and given a supplementary mini-CEX in the same discipline and domain by two assessors. Each mini-CEX lasted about 30 minutes, including time for feedback. The blueprinted items were marked “critical” on the assessment forms; this was emphasised to assessors and candidates.

Candidates participated in seven CBDs and had to pass a minimum of four. The two clinical case patients were selected from patients whose care the candidates had managed. On two occasions, the candidates had to choose three cases and the assessor chose one from these three cases for the assessment. Many candidates worked only in one specialty (eg, medical registrar, psychiatry registrar).

To provide sufficient breadth to the assessment, the five other CBDs were paper cases, developed by a clinical panel from real cases. The topics were selected from the AMC syllabus and were common clinical scenarios, pitched at the intern level. For the paper cases, candidates were given 10 minutes to read about the case before the assessment. Each CBD assessment, including the feedback, took 20–30 minutes.

For the MSF assessment, candidates were asked to nominate 12 colleagues from medical, nursing and other staff (eg, a ward clerk) who could provide feedback on the candidate’s communication, teamwork and professional skills. The project team chose 10 of the nominated colleagues at random from the list of 12 (to maintain anonymity) and these 10 were sent the MSF assessment form. Candidates also completed a self-assessment form. The feedback was reviewed by a committee of medical, nursing and allied health professionals.

The first MSF (360°) assessment was performed in month 1 and the second in month 6. The first was used formatively, and candidates were given guidance, remediation and mentoring as needed. To pass the second (summative) MSF assessment, a mean score of ≥ 3 was needed in all domains (Box 2).

The ITA form used is the regular end-of-term assessment tool used by postgraduate training bodies and colleges. Two ITAs were required during the 6-month assessment period, and candidates needed an overall performance score of at least “at expected” (or equivalent) on both assessments.

Before WBA started, all candidates and assessors were given resource kits with assessment criteria, blueprinted forms and other relevant information. Each mini-CEX and CBD assessment stipulated the discipline and clinical skills to be assessed. Assessors were responsible for selecting an appropriate patient for each mini-CEX assessment. The schedule of assessments was centrally programmed to ensure compliance and proper sampling (so that history, physical examination and counselling, etc, are well represented in all disciplines).

The project was staffed by a full-time project officer, a part-time educational consultant and a part-time project manager with the academic team. Assessors were recruited by inviting interested clinicians to participate, with an honorarium offered per session. The project officer facilitated scheduling of assessments, and assessment tasks had a completion deadline which allowed for clinical emergencies. This flexibility enabled completion of all assessments within designated timeframes.

Feedback was provided by assessors at a focus group and through emails, and candidates provided feedback at a focus group with semi-structured verbal questions followed by a questionnaire.

At the end of the assessment period, all 27 candidates were successful according to the predetermined criteria, even though some had been deemed “unsatisfactory” in individual assessments (Box 3).

Of 27 candidates, 22 responded to the feedback questionnaire, and indicated that WBA was more realistic than the current format of the AMC clinical examination. The candidates felt that the WBA did not interfere with their clinical duties. Most of the candidates reported that they had received “constructive feedback”. One candidate wrote that the “program was really good in stimulating learning and appropriate as a tool of assessment.” Some candidates thought specialists should not be assessing in their field, but noted that “80% of the assessors were very appropriate”. “Better standardisation of examiners [needed]” and “discussion pitched at senior level” were two of the problems identified.

The feedback process was a recurring theme with assessors; “direct feedback on the spot ... and it is now recognised in management and health care that we need to begin to do that”. One assessor said “we have to be prepared, in giving feedback, that it may be uncomfortable”. Another assessor thought “it is more painful if you do not give feedback because the person does not get the chance to improve”. The process “did not under- or overassess someone’s abilities” because of multiple assessors and multiple tools. Assessors felt the candidates had improved over the assessment period.

Some of the issues raised by the assessors included the need to widen the pool of assessors, medical emergencies interfering with assessments, and assessment fatigue. However, in a subsequent focus group, the assessors indicated they were prepared to do more assessments and appreciated the training received.