Public reporting of hospital outcomes: a challenging road ahead

The outcomes for patients following admission to an Australian hospital, notwithstanding occasional medical misadventures hitting the mainstream media, remain largely unknown, even to those working within the sector. The Australian Government, along with all the states, has been moving toward addressing this gap in knowledge with recent policy documents and reports.1,2 This intent has been made concrete with the recent Council of Australian Governments agreement to establish a national performance authority that will be charged with publicly reporting hospital and community health outcomes. In a challenging area, where Australia has lagged behind many other countries, learning from the international experience may smooth the development of an Australian reporting program.

The United States has extensive recent experience with public reporting of health outcomes, starting with the Health Care Financing Administration’s publication of hospital-specific mortality outcomes in 1986. While there has been a proliferation of reporting since that time, arising from both government and the private sector, the federal government reporting of hospital outcomes for myocardial infarction, heart failure and pneumonia in US Medicare patients is perhaps the most relevant to Australia (available at http://www.hospitalcompare.hhs.gov).

Both the United Kingdom and Canada publicly report hospital standardised mortality ratios (HSMRs) using government websites; in the UK, a private company (Dr Foster) also provides HSMR data to the public. The methodology employed for calculating these values in the UK has recently been questioned,3 and some National Health Service (NHS) trusts have been “extremely concerned” about the Dr Foster quality measures and have been considering legal action.4 The UK Care Quality Commission also publicly reports a measure of hospital processes using a rating system that is tailored for health consumers.

State departments of health across Australia report on various aspects of the hospital system, mostly regarding access, but there is no coordinated reporting of outcomes for individual hospitals. Queensland has the most experience of hospital outcomes reporting using a variable life adjusted display to report various locally developed clinical indicators,5 following recommendations from the 2005 Queensland Health Systems Review.6

National reporting of sentinel events has been in place since 2003. This reporting has generated mainstream media coverage, but is likely to reflect outcomes from only a small proportion of health system processes.

In 2009, the Australian Commission on Safety and Quality in Health Care outlined a series of measures of hospital outcomes,2 since endorsed by Australian health ministers, with the intent that these data would be publicly reported by the planned National Health Performance Authority. One of these measures, the HSMR, was the subject of a 2009 Australian Institute of Health and Welfare report illustrating its feasibility for Australian hospitals using existing coding data.7

The drive for greater public reporting in health care has come from three major perceived deficiencies in the existing systems — those of failures in accountability, improvement and information to inform patient choice.8

The need for greater accountability in health systems has been highlighted by numerous government reports, both in Australia and overseas,6,9,10 examining failures of safety and quality. Marshall has perhaps best summarised the public’s desire for such information to be available: “they are dissatisfied with what they perceive as the veil of secrecy and professional protectionism currently seen in health care”.11 Recent comments that “the mice [are] in charge of the cheese” by the father of a young golfer who died unnecessarily in a Sydney hospital12 would tend to support this view. From a public policy standpoint, public reporting of hospital outcomes is an overdue step in enhancing accountability.

Public reporting as a tool for improving the quality of health care has some influential advocates, including the National Committee for Quality Assurance and the Institute for Healthcare Improvement13 in the US, and the NHS14 and King’s Fund15 in the UK. All of these organisations appear to be suggesting that reporting such outcomes is necessary, although not sufficient, for improvement in health quality. However, the medical evidence to support public reporting as a means of improving outcomes is not robust. A systematic review published in 200816 concluded that “Evidence is scant” and “Rigorous evaluation of many major public reporting systems is lacking”. A recently reported trial17 that examined public feedback of cardiac indicators (using measures of both processes and outcomes) showed no difference in the primary end point of a range of process measures. There were, however, statistically significant reductions in some secondary mortality outcomes for myocardial infarction in the publicly reported hospitals. Given the difficulties and expense of such studies, and the diversity of global health systems, it would seem unlikely that compelling scientific literature on public reporting will arise in the near future.

The third driver of public reporting is that of providing information to allow informed choice by health consumers. This has been most prominent as a justification for the early public reporting in the US18 and remains important in a country where so much health care is purchased by employers. However, the evidence suggests that consumers who use this information to guide health purchases are in a very small minority,19 and early public reporting in the US has been focused on acute conditions where choice is likely to be less valuable to consumers. A requirement for a nascent Australian program to primarily guide consumers in their hospital choices would make the task significantly more demanding.

Our key recommendations for public reporting in Australia are set out in the Box.

Although reporting of hospital outcomes may appear a simple exercise of accounting, there are significant issues concerning the data and its processing, its utility to the various stakeholders, and the possible effects of such reports.

All public reporting of hospital outcomes uses the World Health Organization’s International Classification of Diseases (ICD) system. All admissions to Australian public hospitals are coded, including the comorbidities that can be used in the standardisation of risk across different hospital populations. Although precision and consistency in coding will be a prerequisite for any public reporting system, the coding data are the best available data barring a significant investment in manual abstraction of medical records or rapid adoption of electronic health records.

Perhaps the major concern about ICD-based models for reporting outcomes is that the coding data lacks the clinical detail of medical records and will give rise to inappropriate classification of outcomes.20 This issue has driven the medical record validation of risk-adjustment models used in the US,21-23 where administrative models have been shown to be good surrogates for those derived from medical records.

The statistical methods used in public reporting of health outcomes are often inscrutable to many and make presentation and understanding of data more difficult. This complexity must be balanced against the potential of simpler measures and methods to be statistically challenged, as seen in a recent challenge to publicly reported HSMR measures in the UK.3,24 The US uses hierarchical logistic regression models, which take into account the clustering of outcomes within facilities, rather than the general logistic regression models used by some in the UK, with others recommending the use of funnel plots and variations on statistical process control.5 The greater danger in any new national program would appear to be the inappropriate classification of outcomes as aberrant from the norm, rather than the classification of aberrant outcomes as “normal”. Therefore, hierarchical logistic regression has an important advantage over other techniques that label a predetermined proportion of facilities as aberrant.25

One of the challenges in Australia will be reporting outcomes in smaller hospitals, where the caseloads are likely to be small, resulting in less precision of outcome measures and inability to draw meaningful conclusions. In the US, outcomes in smaller hospitals tend to be corrected toward the national mean, which reduces their likelihood of classification outside national norms, and hospitals with very low caseloads (less than 25 cases per year) are not reported. Larger hospitals, because of greater precision in the outcome estimates, tend to undergo less correction of their raw values toward a national mean. Many have raised concerns about the risk of “gaming” of any outcomes reporting system15,20 or even outright fraud.26 These issues have been best explored in the NHS, where such activity was not seen in emergency department initiatives to reduce waiting times.27 However, the criticism has mainly been directed at the excessive focus on administrative process measures and targets28 and the subsequent failure to develop a culture of improvement, rather than the public reporting per se. A further potential adverse consequence of public reporting is that of minimising risk in the pool of patients being treated at a given facility. This has been highlighted principally in cardiology procedures and postulated as a factor in the reduction in mortality after coronary artery bypass grafting in New York in the early 1990s,29 as driving differences in percutaneous coronary intervention (PCI) casemix between Michigan and New York,30 and for reductions in the proportion of high-risk patients treated with PCI in Massachusetts.31 Such risk avoidance may reduce patient harm but also has the potential to deny care to high-risk patients, who often have the largest absolute benefit from an intervention. Among the means of mitigating this risk are sound baseline measures of casemix, inclusion of measures of appropriateness of care, and development of risk-standardisation models that clinicians can trust to account for differences in baseline risk.

A recent UK report15 highlights the differing audiences for public reports of health outcomes, including the public, the media, industry, researchers, clinicians and government. These groups each have differing expectations and abilities to interpret such data. Attempting to reach or influence all these audiences as a first step in public reporting would be overly ambitious, so targeting specific groups (such as those working in the clinical delivery system) who are most able to use the data to drive change is likely to be most fruitful.

One of the risks of public reporting is the possibility of raising fear in the community about the health care system and eroding trust in important public investments. A Royal Statistical Society report on performance monitoring in the public services32 outlines many elements of such performance management programs that enhance their success. Prominent among these is the need for a “wide-ranging educational effort about the role and interpretation” of data.

There is also a risk that the selection of publicly reported outcomes will draw investment and resources away from other service areas that are not being measured. Inevitably, health services will have local priorities to address, and any mandatory reporting requirements may not coincide with these exigencies. Good design of the measures to minimise the burden of data collection and perverse behaviour (such as gaming and risk manipulation), along with a modest data burden that is supported by appropriate resourcing, should minimise this risk.

Public reporting of patient outcomes following hospitalisation in Australia has lagged behind that of other countries, but there is now clear commitment to its implementation. There remain numerous challenges in the design and implementation of such a program, including strategic decisions, statistical methods, and the prevention of risk aversion and perverse behaviour. While the international experience of public reporting can provide invaluable lessons, the solutions to these challenges and the shape of the Australian program will need to suit the local context.