The comparability of emergency department waiting time performance data

One of the key goals of health reform is to drive hospital quality improvement by creating greater transparency in health services performance.1 With the launch of the MyHospitals website in December 2010, quality indicators for individual hospitals across Australia were publicly reported for the first time. The MyHospitals website began by detailing emergency department (ED) and elective surgery waiting time performance data for public hospitals, and has since added hospital rates of staphylococcal infection and waiting times for cancer surgery. Additional quality indicators will be added in the future. Our study focused on ED waiting times, which the website presents as the proportion of patients in each of five ED triage categories whose treatment began within recommended time frames. Previously, these statistics were only available at the state or territory level, and not for all jurisdictions.

Critics of the MyHospitals website have questioned the data quality and comparability.2,3 Concern over the comparability of ED waiting time data is not a new issue. Reports by the Productivity Commission and the Australian Institute of Health and Welfare (AIHW) have identified jurisdictional-level differences in the allocation of patients across the five ED triage categories, and have suggested that these differences may influence ED waiting time performance.4,5 If, in fact, reported ED patient urgency mix is associated with waiting time performance, then comparing performance of EDs that have different patient urgency mixes may be unfair.

We aimed to investigate the relationship between ED patient urgency mix and ED waiting time performance at the hospital level, using MyHospitals data. We also aimed to assess the variation among Australian hospitals in the assignment of ED patients to triage categories, and whether differences in the proportion of patients assigned to each category are associated with ED performance. Lastly, we aimed to determine the degree to which ED performance scores change when patient urgency mix and hospital size are taken into account.

Our study was a cross-sectional study using publicly reported Australian hospital-level data on the MyHospitals website. We recorded the number of patients assigned to each ED triage category and the proportion of patients in each ED triage category whose treatment began within the recommended time frame. Five triage categories are used by the MyHospitals website: resuscitation, emergency, urgent, semi-urgent, and non-urgent. These are the same categories used by the AIHW, and are based on the National Triage Scale (NTS).6,7 The recommended time frames used by MyHospitals (emergency, ≤ 10 min; urgent, ≤ 30 min; semi-urgent, ≤ 60 min; non-urgent, ≤ 2 h) are the same as the NTS and the updated Australasian Triage Scale8 except for the resuscitation category, to which MyHospitals assigned a measurable time frame of “receipt of care within 2 minutes” rather than “immediate”.6 We obtained data for July 2009 to June 2010 from the MyHospitals website.

Our analysis included all hospitals reporting ED performance data, with the exception of 25 hospitals with too few resuscitation cases to report full statistics.

Our analysis included 158 hospitals. These were all public hospitals, except for two private hospitals in Queensland that provided services to public patients. Most hospitals (113) were peer group A or B hospitals, while the remainder (45) were smaller hospitals.

The results of our study suggest that better performance by EDs in meeting waiting time criteria is related to the reported urgency mix of the EDs’ patients. The data for 158 Australian EDs in 2009–10 showed that those reporting a disproportionately large percentage of emergency patients had poorer performance than those reporting smaller proportions. We also found that EDs reporting proportionally more non-urgent patients had better performance than those reporting fewer non-urgent patients; and, related to this, smaller hospitals, which have more non-urgent patients, performed better than larger hospitals in the three less urgent triage categories. These results raise questions about the comparability of the current Australia-wide performance reporting methods.

The policy goals of publicly reporting hospital quality indicators are to provide greater public accountability, spark hospital quality improvement efforts that would lift the standards of hospitals across the country, and provide consumers with information for making informed choices about their health care.11,12 To achieve these goals, hospital performance data must be accurate and comparable.

One explanation for the patterns shown in the study is that it may be more difficult operationally to ensure that ED patients are treated within the recommended time frames when patients need treatment very quickly. Since patients allocated to higher urgency categories are more likely to be admitted to the hospital, one potential cause of lower waiting time performance among EDs treating highly urgent patients may be access block.13 Conversely, it may be easier for an ED to meet the recommended guidelines for initiating treatment when its patients need treatment to commence within a longer time frame. If true, then hospitals do not face a level playing field when being assessed on ED performance, and EDs with higher proportions of more urgent patients are disadvantaged under the current reporting system.

Another potential explanation for our findings relates to assigning patients to a less urgent triage category than is appropriate. Such “undertriaging” gives EDs a longer recommended time frame for initiating treatment, which would be likely to translate into better performance. If there are EDs that routinely allocate “true” emergency patients to the urgent category, their performance scores would probably be inflated. Our study does not provide any evidence that undertriaging is taking place in EDs. It is a possibility worthy of consideration, however, since there has been documented gaming of other hospital performance indicators in Australia, and of ED performance overseas.14,15

Performance scores that have been adjusted according to the urgency mix of patients and the size of the hospital are fair comparisons, and they are not dependent on which of the above explanations is driving the observed relationship between urgency mix and performance. Our findings suggest that, while on average such adjustments would be modest in size, they could have a substantive impact on hospital rankings. For example, of the 10 hospitals with the highest performance scores for the emergency triage category, only six would remain in the top 10 when adjusted performance scores were used. Fewer than half (4 and 3, respectively) would remain in the top 10 for performance in the urgent and semi-urgent triage categories if adjusted scores were used. It is important to ensure, however, that any adjustment of scores is limited to accounting for factors that causally affect performance and are not the result of confounding variables. If, for example, EDs with more emergency patients had worse performance because they attracted managers who were less skilled in managing high-demand situations and accepted long patient waiting times as immutable, then fully adjusting performance scores would in essence excuse poorer management. More investigation of the factors that affect ED performance is required.

Future research on the relationship between patient urgency and waiting time would benefit from analysis of patient-level data, which would allow controlling for differences in patient demographics and health status. This was not possible with the hospital-level data used in this study. It is also noteworthy that the measures currently used to assess ED performance are not validated.16 In other words, it is unknown whether patient health outcomes are worse in EDs that less consistently meet the recommended time frames for initiating patients’ treatment. Developing this evidence base or identifying alternative evidence-based performance metrics is important for creating a public reporting scheme that is trusted and respected by the medical profession.

Our study also contributes to the literature on equity in health care in Australia. Overall, the results suggest that hospitals have similar ED waiting time performance in areas of high compared with low socioeconomic status, and in urban compared with rural areas. There was, however, one exception. Performance in the emergency triage category was worse in areas of lower socioeconomic status. Future research should monitor these trends and further examine whether there are differences in ED waiting times for patients of differing socioeconomic status within hospitals.

In conclusion, our study highlights the challenge of publicly reporting hospital quality data. We found that the current ED performance metrics may be biased in favour of EDs that report fewer urgent patients. Adjusting performance scores for variation in patient and hospital characteristics could ameliorate this bias. This type of adjustment is considered crucial for the viability of patient-outcome performance measures,17-19 and our findings suggest it may be important for waiting time measures too. Data audits, however, would still be necessary to ensure the comparability of data collection processes across hospitals, particularly since inconsistencies have been documented in the past.20 As more quality indicators are publicly reported in the future, it will be increasingly important to consider when and how adjustment of quality indicators is applied.