Public reporting of hospital outcomes based on administrative data: risks and opportunities

In response to repeated disclosures of poor quality care in hospitals throughout the Western world, calls have been made for regular public reporting of quality and safety indicators from individual hospitals.1 In the United Kingdom, mortality data for selected surgical procedures from each hospital trust are posted on the Internet together with overall quality ratings for each hospital.2 In the United States, mortality and readmission data for cardiovascular diagnoses and surgical procedures for individual hospitals are issued as annual reports freely available to the public.3,4

The recent Bundaberg Hospital inquiry5 unearthed quality reports produced in 2004 for every Queensland public hospital (Box 1),6 but these were denied public disclosure. This prompted the Forster inquiry to recommend that such reports be made public on an annual basis (Recommendation 13.3).7 The 2004 reports were subsequently released by the Queensland Government and summary results were published in Brisbane newspapers in September 2005.

The current Queensland reports (Box 1) use routinely collected administrative data from each of 78 hospitals to report risk-adjusted in-hospital mortality rates, length of stay (LOS) and complication rates for all admissions over a 12-month period for each of 12 common medical and surgical conditions. Ratings of patient satisfaction with care are also provided. By comparing outcome data for each hospital with the average for all hospitals combined, and the average for other hospitals that belong to the same “peer group”, an attempt is made to identify “outlier” hospitals that demonstrate significantly better or worse outcomes. The intention of profiling hospitals in this way is to help patients and the public to select better performing hospitals, while affording an opportunity for managerial and clinical staff of poorly performing hospitals to investigate and optimise care processes using insights gained from better performing hospitals.

While intuitively appealing, such public reporting has been questioned with regard to its scientific rigour, and there remains uncertainty about its use by, and impact on, patients and providers.8,9 In the promotion of public reporting, several assumptions are being made that deserve empirical testing. The aim of our article is to raise awareness of potential pitfalls of public reporting and to suggest ways by which public reports, which focus on outcomes derived from administrative data, can be rendered more beneficial and less harmful in improving hospital practice.

Several considerations are pertinent.

How accurate are administrative data? The Queensland hospital reports rely on routinely coded hospital discharge data pertaining to specific principal diagnoses, comorbidities and complications, defined by International statistical classification of diseases and related health problems (ICD) codes.10 For quality assessment, such data are often inaccurate, incomplete, or provide insufficient clinical detail. While death, LOS and readmission are reliable descriptors, accuracy of diagnosis coding is variable. For example, Australian studies show that about 20% of cases of acute myocardial infarction (AMI) as primary diagnosis are not detected,11 and coding of comorbidities is associated with miss rates ranging from 11% for diabetes to as high as 100% for dementia.12 In-hospital complications may be difficult to distinguish from presenting diagnoses, may be variably documented according to specialty, or may be coded inappropriately.13

Systematic bias in coding error at the level of individual hospitals can invalidate the process of risk-adjusting outcome data (see below), which is necessary for fair comparisons between hospitals. The extent to which coding error varies between hospitals has not been formally assessed.

Have analyses controlled for differences in casemix and other relevant factors? Comparisons must be controlled for differences between hospitals in age, sex, and risk profiles of patients relevant to the outcomes of interest. Risk-adjusted mortality rates can differ markedly from crude rates. For example, in AMI, crude in-hospital mortality rates in the 2004 Queensland reports were 11.2% and 15.2% for tertiary and small hospitals, respectively, whereas the respective risk-adjusted rates were 8.2% and 20.5%.6 Risk adjustment models applied to administrative data must classify patient risk with a level of accuracy close to that achieved by validated prediction rules based on clinical data. While this has been confirmed for AMI care,14 other disease conditions require further study, with many clinicians15 and statisticians16 claiming such models inadequately account for variation in patient characteristics.

In addition, factors over which hospitals have no control may influence outcomes, as noted by higher readmission rates in geographic areas where appropriate levels of medical, nursing or social care are not available after discharge.17

Have the effects of random error been minimised? The smaller the sample size, or the more times data are analysed, the more likely it is that extreme results will occur simply by the play of chance. Data from small hospitals are more likely to yield spurious “outlier” results that categorise them as high- or low-performing hospitals when, in reality, on the basis of process-of-care reviews, the opposite may be true. In one analysis of reports similar to the Queensland reports, fewer than 12% of truly poor quality hospitals emerged as high-mortality “outliers”, while over 60% of seemingly poor quality “outlier” hospitals were in fact good quality institutions.18

The Queensland hospital reports attempt to deal with this problem by using more stringent tests of statistical significance in the form of 99.9% confidence intervals. Other statistical tools, such as “shrinkage” techniques,19 which minimise chance results, have been employed. Another option is to complement outcome reports based on relatively few events with “snapshot” audits of the care of samples of randomly selected or consecutive patients. Processes of care, compared with outcomes, are more frequent, more sensitive in detecting suboptimal quality, more easily controlled by hospitals, and more directly linked to remedial action.20

Can true variation in outcome be reliably detected for hospitals that are similar? As hospitals differ in service capacity and ability to care for different types of patients, it can be argued that comparisons should be performed within peer groups: tertiary, community or district hospitals. However, as the number of hospitals and total number of events are reduced, the magnitude of detectable “special-cause” variation between hospitals (ie, variation due to specific causal factors, such as quality of care, as opposed to random variation) decreases. Of 12 diagnosis-specific outcome indicators measured across 180 Queensland hospitals over a 12-month period, significant special-cause variation within peer hospitals was detected for only two indicators (17%).21 One solution is to restrict analysis to patients with lower baseline risk, for whom special-cause variation between hospitals is of greater magnitude.22

Do differences in outcomes based on administrative data reflect real differences in quality of care? It is assumed that worse outcomes in specific hospitals will correlate with less-than-optimal care as determined by the gold standard of reviewing individual patient records and applying explicit quality criteria. Using this method, up to 90% of variance in risk-adjusted mortality rates for AMI in NSW hospitals was attributed to variance in quality of care, and other studies report similar findings for heart failure, pneumonia, stroke and hip fracture.12 However, other investigations demonstrate little or no correlation between specific outcome and process-of-care measures, despite rigorous risk adjustment.23

Applying statistical process control techniques such as risk-adjusted cusum analysis to routine hospital outcome data may be better for detecting trends towards excess adverse outcomes at an earlier stage.19

Do outcome data for selected diagnoses predict overall quality of care? Hospitals may perform well on some indicators and poorly on others. Overall quality of care within an institution cannot be deduced from results of a selected sample of indicators, despite the inclusion of as many as 22 different clinical conditions.24

Expecting the lay public to use public reports when choosing which hospital to attend assumes that reports exist that the public are aware of and can access, understand, accept and act on.

In the US, where public report cards for hospitals have existed for almost two decades, surveys reveal that, as of 2004, only a third of the public had sighted them, and only 19% reported making use of such information, with most distrusting it or viewing it as unhelpful.25 Common sources of misunderstanding, more prevalent among patients of lower socioeconomic status, were language and terms used, the meaning of an indicator in relation to quality of care, and whether low or high rates of an indicator implied good performance.26

Interestingly, limited data suggest many doctors are also distrustful of hospital reports when making patient referrals because of concerns about adequacy of adjustment methods and immunity of ratings to the effects of manipulation (“gaming” [see below]).27

Moreover, public reports will only allow patients and clinicians to exercise discretion in their choice of hospital if there is more than one hospital in a given locale, and will only apply to care of elective conditions, as the nearest hospital will usually be preferred for urgent care of acute, life-threatening conditions. The lay public may also see hospital comparisons among peer groups as being unhelpful, given their desire to know which hospital, among several of mixed type (tertiary and community) in their area, provides the best care.

Even if patients do not use reports directly, they may still receive higher quality care indirectly if such reports prompt global change in hospital practice or oblige hospitals to limit the services they provide to those they deliver well.

A study that randomly assigned hospitals in Wisconsin to generating public, internal, or no quality reports saw more quality improvement activities undertaken by the first group — more so for hospitals with low quality results.28 Other “before and after” studies have claimed report-led improvement in coding accuracy, medical record documentation and selected clinical processes.29 Reductions in risk-adjusted mortality rates for cardiac surgery30 and for other medical conditions and surgical procedures31 have been reported.

Several dysfunctional consequences can arise from public reporting of performance data.

Gaming. Hospitals can be made to look better than they really are by, for example, up-coding the principal diagnosis (to a more severe condition or procedure) and coding every possible comorbidity on every patient to lower risk-adjusted death rates.

Early discharge. Discharging patients earlier, and possibly prematurely, reduces LOS and shifts into community settings a proportion of deaths and complications that would otherwise have occurred, and been measured, as in-hospital events.32

Avoidance of high-risk patients. Investigations in several US states suggest that lower cardiac surgery mortality did not result from better care induced by reports, but rather from increased refusal by operators and hospitals to operate on sicker patients who could have benefited.33

Outsourcing of high-risk patients. Hospitals may transfer more of their sickest patients to other institutions, as suggested by a 30% increase in transfer of patients, mostly high risk, to other states following release of hospital reports for cardiac surgery in New York state.34

Adoption of defensive medicine. Hospitals may be encouraged to overtreat all patients in chasing better outcomes, leading instead to more unnecessary interventions and unintended adverse events.

Withdrawal or disengagement. Unless public reporting is centrally mandated, hospitals reporting worse outcomes may simply cease reporting.35 If they do continue reporting, but the staff within these institutions feel, or are, powerless to influence poorly performing indicators, such measurement may lead to demoralisation and disengagement.36

Tunnel vision. Remedial efforts may concentrate on the services being measured, with neglect of equally important aspects of care that are not being measured.37

Several studies show that internal feedback of quality indicators, combined with other quality improvement interventions, raises standards of care, particularly within multi-hospital collaborations.38,39 While keeping reporting internal may engender conspiracy theories and lower public trust, professional peer comparisons probably drive most service improvement. This can be achieved just as well with internal as with public reporting, with less propensity to trigger dysfunctional responses.

There is limited research into the methods and effects of public disclosure of performance data,9 with claims of favourable effects on cardiac surgery mortality in several US states being the most frequently cited evidence supporting public reporting.30 On balance, we believe that public reporting is worthwhile, but needs to be performed in a circumspect manner based on our summary findings (Box 2). If public reports of hospital outcomes based on administrative data are to become permanent, we recommend mandatory reporting by all hospitals according to the standards listed in Box 3.

In particular, design formats and information presentation must ensure that public reports can be interpreted and used in ways that do not create anxiety or false notions about a hospital’s capacity to provide suitable care.40 Incentives and safeguards must also exist to guarantee appropriate care and counter any tendency for hospitals to discriminate against high-risk patients. The methods for producing public reports should be transparent and scientifically rigorous, the reporting and dissemination process should actively involve lay groups and the media, and the whole activity must be free of political and bureaucratic interference.

In February 2006, the Queensland Minister for Health established an independent Health Public Reporting Advisory Panel comprising clinical, academic, community and media representatives. This panel will advise the Minister and the public on the significance of key performance measurements of the public health care system by attempting to answer three sets of questions relating to such data:

• What is the statistical meaning of any variations? Are variations significant, and are there trends towards improvement or deterioration?

• What is the clinical meaning of the information? How should results be interpreted in the context of the data source, the relevant disease characteristics, current best clinical practice and current resource levels, including clinical workforce?

• What is the public meaning of the information? How can data be presented in a form that ensures accuracy, easy comprehension and linkage with information about clinical practice improvement efforts that are being, or could be, used to address any deficits in care?

In light of the political and professional sensitivities surrounding public reporting of performance data, it is imperative, if this experiment is to succeed, that comparative data are presented in their proper context and are used to define system problems for which remedial options are clearly articulated.36 In this way, thought and action are directed at the problem, not at an indicator; feedback is given and local targets agreed; action plans are formulated; and support for change is provided, together with additional resources where required. If this course of action is followed, public reports are likely to attract the trust of both health care staff and the public and to demonstrate that performance data collected from administrative sources are credible and transparent and are being used to improve health care outcomes.