The importance of better understanding error and safety in primary care is widely accepted, with recent calls to promote efforts and improve resources in this area of research.1 A few studies have described errors in general practice using the incident monitoring technique,2-10 but none have quantified the problem.
A study of family medicine physicians in Cincinatti, USA, showed errors were identified in almost a quarter of clinical encounters.2 An error rate of 75.6 per 1000 general practice appointments (7.6%) was reported in one UK study.3 Reviewing incident reports from an outpatient primary care setting, a US study reported a prevalence of adverse events of 3.7 per 100 000 clinic visits.11
Approval was obtained from the University of Sydney Human Research Ethics Committee. The investigating committee was granted qualified privilege by the NSW Minister for Health, providing confidentiality of meeting minutes and protection from their use in legal actions.
We adopted the definition of “error” used in previous work.5,6,12,13 Errors may have been attributable to the reporter’s actions or other unwanted occurrences:
The General Practice Programs Branch of the Commonwealth Department of Health and Ageing (GP Branch) provided a stratified random sample of 320 names from the population of 4666 vocationally registered GPs in NSW whose Medicare billings in the previous quarter indicated that they were engaged in full-time work.14
A sample size of 80 was calculated from pilot study results of 134 reports made from up to 23 GPs over 4 months, which had a 25% participation rate.5 Allowing 1 week of absence each and estimating 132 consultations per week,15 we hypothesised a reported error rate of 0.27% per consultation. To estimate this with a 95% confidence interval of 0.25%–0.29%, we required 205 465 patient encounters. We anticipated that 80 GP participants would have about 480 000 patient encounters, with the excess allowing for any effect of clustered samples.
The sample was drawn from Rural, Remote and Metropolitan Area (RRMA) groupings of RRMA 1, RRMA 2–3, and RRMA 4–7.14 The GP Branch stratified the sample by gender and age, dichotomised around 45 years. One hundred and sixty (50%) of the sample were drawn from RRMA 1, 80 (25%) from RRMA 2–3 and 80 (25%) from RRMA 4–7 regions. The proportions of the source population in these regional groupings were 68%, 16% and 16%, respectively. The smaller groups were over-sampled to ensure adequate counts of Medicare items to allow future subgroup analysis.
A secure website and reporting process were developed to house the error questionnaire, shown in Box 1, which was piloted in 2001.5 Data transmission was protected by 128-bit encryption and the website was hosted by a secure server behind a firewall. The website allowed participants to contact investigators and view linked medical resources. Participants were asked about any website problems in monthly contact via email, telephone or facsimile.
Total numbers of GPs, their sex and average age were obtained from the GP Branch by RRMA groupings, so that the study participants could be compared with the source population. We also compared the number of Medicare items billed relating to a patient encounter over the study duration.
Numbers of errors reported per PIN, the numbers of PINs obtained per RRMA group, and total numbers of errors from the group were obtained from website data. At the end of the study, participants were asked whether they had sent any reports electronically. To protect anonymity, we did not ask the number of reports submitted.
Total numbers of errors from participants, counts of the number of Medicare items billed that related to a patient encounter and the number of individual patients that were seen during the study were used in incidence calculations. Patients who were seen on more than one occasion during the year by a single participant were only counted once per GP.
The percentage of female participants was compared with the source population using Fisher’s exact test. The average participant age and number of Medicare items relating to patient encounters were compared using unpaired Student’s t tests. A non-parametric method adjusting for clustering was used to compare the median number of reports per PIN. Incidence calculations included sampling weights for the three RRMA groupings to adjust for the relative over-sampling of RRMA 2–3 and RRMA 4–7 GPs. All statistical analysis was performed using Stata 8.0 software (StataCorp, College Station, Tex, USA).
From the initial sample of 320, we were unable to contact 52 GPs (16.3%) who had moved away, retired, or were completely unknown to the practice. A further 43 GPs (13.4%) did not return researchers’ calls or gave no specific reason for declining. Of those remaining, 91 (40.4%) were too busy or not interested, 20 (8.9%) had difficulties with computer use or access, 18 (8.0%) were retiring or taking extended leave during the study, eight (3.6%) did not participate in research, three (1.3%) had concerns about privacy, and one (0.4%) had concerns about bad publicity. There were 84 GPs (26.3% of the original sample) who agreed to commence the study, with 41 from RRMA 1, 22 from RRMA 2–3, and 21 from RRMA 4–7 regions. One RRMA 1 GP and one RRMA 4–7 GP left the study. As reporting was anonymous, their results were included in further calculations.
The participants’ average age, percentage of female GPs, and average number of Medicare patient encounter items were assessed. There was no difference in comparison to the source population within each RRMA grouping, or for all combined, using any of these measures at the 5% significance level.
There were 418 TAPS reports from 85 PINs describing an error during the study. The mean number of error reports per month was 34.8 (range, 15–60). At feedback interview, 79 of the 82 completing participants had submitted a report.
Box 2 shows RRMA grouping comparisons of the number of reports submitted, PINs submitted, GPs who reported at interview sending a report, average number of reports per GP, and the median number of reports per PIN over the study duration.
The distribution of reporting frequencies among RRMA groupings was assessed by looking at the number of reports per PIN, shown in Box 3. This varied from one to 25, with a median of three for all participants combined. There was no significant difference between RRMA groupings in the median number of reports submitted per PIN (P = 0.40) (for this analysis, values that were equal to the median were evenly split between the above and below groups).
The total numbers of Medicare items relating to a patient encounter and total numbers of individual patients seen by RRMA grouping are shown in Box 4. Each report within an RRMA grouping was matched with the group’s average number of Medicare items and average number of patients seen, and assigned a sampling weight to adjust for the larger representation of RRMA 2–3 and RRMA 4–7 GPs in the sample design.
Original research on errors and adverse events in primary care consists of relatively limited numbers of studies, focusing on qualitative descriptions.16 The TAPS study provides the first calculated incidence of reported error from a representative random sample of GPs. Despite the small sample size and participation rate of 26%, these findings are likely to be generalisable to GPs in NSW, as we detected no statistically significant differences between the age, sex, and Medicare billings when comparing participants to the source population of 4666 GPs. Comparative data were not available from the GP Branch for the non-participants due to consent issues.
It has been suggested previously that an error database in general practice should be established.17,18 Our results indicate that using a secure website with an anonymous reporting method is a practical way of collecting error information from GPs, as 94% of doctors who enrolled in the study reported at interview that they sent an error report.
Our findings provide an estimation of the incidence of errors in general practice that would be reported if an appropriate reporting system were in place, but this should not be interpreted as an estimate of the total number of errors that participants noticed, or of the underlying number of errors occurring in the GP community. It is very difficult to assess the proportion of errors that would go unreported even when a reporting system such as TAPS is available. A GP may not be aware that an error has occurred. GPs have been found to under-report adverse drug events,19 and so other patient safety threats may also be under-reported.
Anonymity was of major importance in our methodological design, to encourage reporting. There has previously been a reluctance of health professionals to address the problem of errors due to feelings of guilt and a desire to avoid disapproval from colleagues.20 However, using a PIN that was unknown to investigators led to some difficulties in interpreting reports per GP. Researchers were unable to assist if a PIN was forgotten, and there was no way of knowing how many GPs changed PINs during the study. Seventy-nine GPs gave feedback that they submitted an error report, but 85 PINs were used for error reports, which suggests that on at least six occasions GPs changed their PINs. Our estimate of report numbers per participant, which is based on PINs, could therefore be an underestimate.
To our knowledge, there are no studies directly comparable with our incidence results, which are lower than hypothesised. The authors of a US study that found an error rate close to 25% state that their results were not generalisable due to limitations of the study.2 Only a small group of 15 doctors in seven practices made reports, completing a review form after every consultation during three half-day sessions. The UK finding of a 7.6% error rate was based on a 2-week collection of errors from 10 practices in a single city.3 However, a major difference is that this result was based upon a collection of reports from a large cross-section of staff working in general practice, with 163 people able to submit reports, and a high proportion of participants who had administrative rather than clinical roles.
Our method of prospective self-reporting appears to have returned a higher rate of error detection than that in a US study using retrospective record review in eight primary health care clinics.11 Again, direct comparison is difficult, as medical adverse events were counted in this work, rather than errors in the sense that we defined.
1 Error questionnaire used in the Threats to Australian Patient Safety (TAPS) study
Abstract
Objective: To determine the incidence of errors anonymously reported by general practitioners in NSW.
Design: The Threats to Australian Patient Safety (TAPS) study used anonymous reporting of errors by GPs via a secure web-based questionnaire for 12 months from October 2003.
Setting: General practices in NSW from three groupings: major urban centres (RRMA 1), large regional areas (RRMA 2–3), and rural and remote areas (RRMA 4–7).
Participants: 84 GPs from a stratified random sample of the population of 4666 NSW GPs — 41 (49%) from RRMA 1, 22 (26%) from RRMA 2–3, and 21 (25%) from RRMA 4–7. Participants were representative of the GP source population of 4666 doctors in NSW (Medicare items billed, participant age and sex).
Main outcome measures: Total number of error reports and incidence of reported errors per Medicare patient encounter item and per patient seen per year.
Results: 84 GPs submitted 418 error reports, claimed 490 864 Medicare patient encounter items, and saw 166 569 individual patients over 12 months. The incidence of reported error per Medicare patient encounter item per year was 0.078% (95% CI, 0.076%–0.080%). The incidence of reported errors per patient seen per year was 0.240% (95% CI, 0.235%–0.245%). No significant difference was seen in error reporting frequency between RRMA groupings.
Conclusions: This is the first study describing the incidence of GP-reported errors in a representative sample. When an anonymous reporting system is provided, about one error is reported for every 1000 Medicare items related to patient encounters billed, and about two errors are reported for every 1000 individual patients seen by a GP.