Entrustable professional activities for workplace assessment of general practice trainees

Entrustable professional activities (EPAs) were developed in response to the shift in medical education toward competency‐based assessment.1,2 Scientific knowledge has been the centre of medical training since the 1910 Flexner report on medical education in North America,3 and the dominant aim has subsequently been to integrate scientific practice and findings into the curricula of university medical schools. Toward the end of the 20th century, a change in thinking required doctors to possess not only technical expertise and medical knowledge, but also the ability to work effectively in teams and to navigate complex health care systems.4 This shift has been complemented in a variety of contexts by developing competency‐based frameworks for training and assessing medical students5 and by replacing traditional final and barrier testing with longitudinal and programmatic assessment.6

One of the challenges of competency‐based medical education has been assessing the competence of trainees in busy clinical settings. Competence cannot be observed directly; it must be inferred from what trainees say and do, creating a gap between the competencies to be assessed and the daily tasks of doctors.4 EPAs were designed as a practical response to this problem.7 An EPA is a task or unit of professional practice that can be fully entrusted to a trainee as soon as they have shown they can perform it unsupervised.8 Whereas competencies are characteristics of individuals, EPAs are trainee activities observable in the workplace,9 reflecting decision making by supervisors in normal clinical practice.10

Clinical supervisors entrust clinical tasks to trainees several times a day in almost every clinical setting, as they are generally not in the same room as the trainee.11 Trust requires interdependence between the truster (supervisor or patient) and the trustee (trainee) and creates supervisor vulnerability, as the supervisor is responsible for the quality of care delivered by the trainee.11 Entrustment decisions must reconcile the educational need to push learners to extend their scope of performance with the obligation to ensure safe, high quality care. Making entrustment decisions explicit can lead to more effective assessments and firmly return the patient to the centre of educational decisions.10

It has been reported that entrustment decisions which mimic everyday decision making by supervisors in clinical practice have positive effects on the psychometric qualities of workplace‐based assessment. For instance, in an examination of entrustment decision making by supervisors of anaesthesia registrars in Australia and New Zealand, a conventional mini‐clinical evaluation exercise (mini‐CEX) scoring system was modified to include an entrustment scale on which supervisors rated trainees’ levels of independence as “supervisor required in the theatre suite”, “supervisor required in the hospital”, or “supervisor not required”. The authors found that a reliability coefficient of 0.70 was achieved with only nine assessments per registrar when the scale was used, but was not reached even after 50 assessments with the conventional scoring system. Further, the modified scoring system identified more trainees performing below expectations.12

While the rationale for using EPAs is clear, testing of their construct validity has been limited,9 although some international and hospital‐based studies have found marked educational benefits.13,14,15 Construct validation is an extensive process, but its central component is testing whether scores reflect expectations about the construct the test purports to measure.16

Employing EPAs in a diverse Australian community setting, such as general practice training, has not been examined. We therefore assessed the construct validity of assessment with EPAs in a postgraduate training environment by determining the sensitivity of EPA assessment processes for assessing competency development in trainees. Prior to our study, there had been no EPAs for assessing general practice trainees in Australia, although EPAs had been developed for psychiatry, surgical, adult internal, and paediatric medicine training,17,18,19,20 and in North America for family and palliative medicine training.9,21,22 We tested three hypotheses:

• Entrustment levels for junior trainees increase over time;
• Entrustment levels for senior trainees are higher than those for junior trainees; and
• Self‐assessment of entrustment levels by senior trainees more closely matches supervisor assessment than self‐assessment by junior trainees.

Methods

Study cohort

Australian general practice training includes one year of hospital‐based training followed by at least 2 years’ community‐based training for urban trainees, or 3 years for rural trainees. EPA assessments replaced mid‐ and end semester trainee assessments for general practice trainees in South Australia from January 2017. We analysed data from 2017 for all general practice trainees in South Australia. Hospital‐based trainees were not included. Trainees were defined as junior trainees (first 12 months of community‐based training) or senior trainees (more than 12 months’ experience).

Developing the entrustable professional activities

Published literature and guidelines informed the development of the 13 EPAs in the process summarised in Box 1.23 Overall competence comprises a number of specific competencies applied repeatedly and contemporaneously.24,25 By limiting the number of EPAs to 13 that encompassed broad responsibilities, we ensured their use would be practicable for supervisors and trainees, while still providing opportunities for specific feedback.8,24 EPA content headings were developed after a comprehensive review of the relevant literature and of domains and competencies of the curricula of the Royal Australian College of General Practitioners26 and the Australian College of Rural and Remote Medicine.27 Throughout the development process, key stakeholders, including supervisors, trainees, medical educators and college representatives, were consulted to ensure that published standards for content validation were met.23

Prior to introducing EPA assessment of trainees, a systematic change management approach was adopted to ensure engagement with the process. Short videos on the EPA concept were presented at education events for supervisors and placed online, and personalised communications with key messages advocating the change were forwarded to supervisors.

Data collection

Entrustment levels for the 13 EPAs were assessed by supervisors and by trainee self‐assessment at 3‐month intervals (May, July, October, January) during 2017–18. EPAs 1–8 (and for rural trainees only: EPA 13) were clinical in nature, EPAs 9–12 were non‐clinical (Box 2, Box 3). Assessments were uploaded to the usual online learning management system, and data on entrustment level were extracted from the system for analysis. The entire dataset was used for each analysis, except that data for trainees were excluded when any data for a particular analysis were missing.

Statistical analysis

Statistical analyses were conducted in Prism (GraphPad). For each EPA, the frequency distributions of levels of entrustment (1–4) at 3 for junior and senior trainees (scored by supervisors) were compared in Mann–Whitney U tests; after Bonferroni correction for multiple comparisons, P < 0.004 (one‐tailed) was deemed statistically significant. The ratio of the proportions of senior and junior trainees entrusted by supervisors with unsupervised practice at 3 months was calculated, with 95% confidence intervals (CIs) estimated with the Koopman asymptotic method.

Changes over 9 months in the proportions of EPAs junior trainees entrusted with unsupervised practice, separately for pooled clinical and non‐clinical EPAs, are expressed as ratios with respect to the 3‐month levels (with 95% CIs estimated with the Koopman asymptotic method).

To assess concordance of assessments of levels of entrustment by supervisors and trainees, the differences for each EPA were summed to obtain a net difference for each trainee (0 = trainee and supervisor overall scores agreed; negative difference: higher net score by trainee; positive difference: higher net score by supervisor). The spread of net differences was visualised in a frequency chart, and compared for junior and senior trainees in an independent samples t test.

Ethics approval

Ethics approval for the study was granted by the Flinders University Human Research Ethics Committee (reference, 7882).

Results

Demographic information

A total of 283 trainees completed at least one EPA assessment in 2017, including 168 urban and 115 rural trainees; 130 were junior trainees (48 men, 82 women), 153 were senior trainees (71 men, 82 women). Data were missing for seven trainees (2.5%) at 3 months.

Levels of entrustment: junior v senior trainees

The number of trainees entrusted with performing individual clinical EPAs without supervision at 3 months ranged from 24 (19%; EPAs 3 and 5) to 53 (41%; EPA 1) for junior trainees, and from 94 (64%; EPA 3) to 136 (93%; EPA 8) for senior trainees (Box 4). Senior trainees were 2.1 (95% CI, 1.66–2.58; EPA 1) to 3.7 times (95% CI, 2.60–5.28; EPA 4) as likely as junior trainees to be entrusted with performing clinical EPAs without supervision (Box 5). The differences between junior and senior trainees with regard to non‐clinical EPAs were also statistically significant (Box 5, Box 6).

Change in proportion of junior trainees judged able to practise unsupervised

Complete data for all four assessments were available for 70 junior trainees. The proportion of clinical EPAs with which these trainees were entrusted to perform unsupervised increased from 26% at 3 months to 35% at 6 months (rate ratio [RR], 1.37; 95% CI; 1.15–1.63), to 50% at 9 months (RR, 1.92; 95% CI, 1.64–2.26), and 69% at 12 months (RR, 2.68; 95% CI; 2.32–3.12). For non‐clinical EPAs, the proportion increased from 50% at 3 months to 56% at 6 months (RR, 1.13; 95% CI, 0.97–1.32), 72% at 9 months (RR, 1.44; 95% CI, 1.26–1.66), and 83% at 12 months (RR, 1.66; 95% CI, 1.47–1.90) (Box 7).

Concordance of supervisor and trainee assessments of ability to practise unsupervised

Self‐assessments of entrustment level at 3 months were available for 121 junior and 136 senior trainees. The mean differences in overall entrustment ratings between supervisor and trainee were 5.5 points (SD, 6.6) for junior trainees and 2.9 points (SD, 2.8) for senior trainees (P < 0.001) (Box 8).

Discussion

Our results confirmed all three hypotheses: entrustment levels for both clinical and non‐clinical tasks were higher for senior than for junior trainees; levels of entrustment for junior trainees increased over 12 months of training; and concordance of supervisor assessments with self‐assessments was greater for senior trainees than junior trainees.

The goal of medical education is to teach trainees how to provide safe professional care.10 In our study, the trust of supervisors in the competence of their trainees increased with time; for each EPA, at least twice as many senior as junior trainees were entrusted with performing tasks independently. Further, junior trainees’ entrustment levels increased fourfold over 9 months in their supervised training environment. Balanced, distant supervision that monitors safe practice while also providing trainees an authentic experience of responsibility is essential for learning to practise independently.10 From a risk management perspective, EPA assessments provide a method for maintaining the safety of health care while enhancing the training of the future health care workforce.

The closer concordance of self‐assessments by senior trainees than self‐assessments by junior trainees with those of their respective supervisors suggests that trainees were initially not aware of what they did or did not know. As trainees develop expertise, unconscious gaps in competency awareness close. Self‐regulated learning, including evaluation of one's own learning, is essential for becoming a safe practitioner.28 Identifying overconfident trainees, so that support and learning strategies can be instigated, is also an important aspect of workplace‐based assessment.

Despite differing contexts, our results are similar to those of other studies. For example, entrustment levels for paediatric medicine residents in an American study increased across each year of training and from year to year.13 Similarly, a study of medical students in Canada noted that entrustment scores increased over time during a surgical rotation.14 A study of general surgical residents found that residents scored their global EPA performance higher than faculty physicians, but their operative EPA performance ratings were similar.15 Each of these studies was undertaken in the slightly more controlled educational setting of hospitals; our study found that their findings were replicable in less structured community‐based settings.

With increasing evidence for the utility of EPAs, it is likely that postgraduate governing bodies will include EPAs as an integral part of trainee assessment. However, it is important that EPAs are carefully designed for their intended context rather than simply extrapolated from one environment to another.5,29

Our results indicate that EPAs are a robust assessment tool in workplace‐based training in a carefully designed setting. Although we did not seek to assess how EPA assessment achieved its aims, key factors probably included significant engagement by stakeholders (supervisors, medical educators, trainees, college representatives) during the development process, which focused on content validation to ensure that the EPAs were aligned with the core activities of general practice.8 Further, construct validity was enhanced by reviewing EPA content during the development process.23 Thirdly, EPAs were acceptable to trainees and supervisors because they reflected activities of daily clinical practice, they were described in normal clinical language that was clear for trainees and supervisors, and they allowed them to express feelings of trustworthiness.9,23 Finally, we intentionally applied co‐design principles to facilitate the change management process.

Limitations

As is often the case in education research,29 the generalisability of our findings is limited, as we examined a specific set of EPAs in a single postgraduate setting. Anchoring bias (that is, an assessment being influenced by an initial piece of information) may have resulted from the supervisors knowing the level of training of trainees. We attempted to minimise anchoring by ensuring that supervisors did not have easy access to earlier EPA ratings during the 6‐, 9‐ and 12‐month assessments. As we preferred ecological validity to being able to draw causal conclusions, we did not include a control group in our study. The differences we found were of a magnitude rarely seen in naturalistic field studies, in which ecological validity is more important than optimising the comparability of groups and isolating individual factors.

Conclusion

In addition to having a strong theoretical basis, EPAs are useful practical assessment tools in a workplace‐based training environment, even in less structured educational settings than, for example, hospitals. Establishing how best to adapt our findings to other settings and how supervisors make decisions about entrustment levels for clinical and non‐clinical EPAs should be further investigated. Our study will help advance the efficiency and effectivity of EPAs as assessment tools. Given the importance of patient safety in medical education and of the trust relationship between trainees and patients, the role of the patient in entrustment decisions should also be examined.

Box 1 – Process for developing entrustable professional activities (EPAs) for the assessment of general practice trainees

---

 

Box 4 – Supervisor‐determined levels of entrustment* for junior (A) and senior trainees (B) for clinical entrustable professional activities (EPAs) at 3‐month assessment

---

* 1. Needs frequent direct in‐room review by supervisor. 2. Needs onsite supervisor in the practice most of the time. 3. Safe to practise with phone contact with supervisor. 4. Safe to practise unsupervised. For each EPA, the distribution of entrustment levels for junior and senior trainees was significantly different (Mann–Whitney, P < 0.001). ◆

Box 6 – Supervisor‐determined levels of entrustment for junior (A) and senior trainees (B) for clinical entrustable professional activities (EPAs) at 3‐month assessment

---

For each EPA, the distribution of entrustment levels (1–4; Box 3) for junior and senior trainees was significantly different (Mann–Whitney, P < 0.001). ◆

Box 7 – Entrustment levels for 70 junior trainees during first 12 months of training, by clinical and non‐clinical entrustable professional activities (EPAs)*

---

* Depicted are the proportions of clinical EPAs junior trainees were trusted to perform at the four entrustment levels. Total number of clinical EPAs per assessment point, 584–588; of non‐clinical EPAs, 276–280. Complete data for all four assessments were available only for 70 trainees. ◆

Box 8 – Concordance of levels of entrustment with supervisor assessments for 121 junior and 136 senior trainees

---