Routine use of administrative data for safety and quality purposes — hospital mortality

In 2009, the Australian Institute of Health and Welfare (AIHW) published a report, Measuring and reporting mortality in hospital patients.1 The report followed a request from the Australian Commission on Safety and Quality in Health Care to determine whether it was possible to produce accurate, valid indicators of inhospital mortality using currently available Australian administrative data.

The report stated the affirmative. Here, we describe some of the issues that need to be considered if inhospital mortality rates generated from routine administrative data are to be used for safety and quality purposes.

It is common practice for hospital mortality to be reported as a hospital standardised mortality ratio (HSMR).2 The HSMR is an indirectly standardised mortality rate, generated by comparing observed mortality within a hospital against the expected mortality for the same patients had they been treated in a “standard” hospital, defined as comprising all hospitals of interest in the population. In this way, the observed mortality in one institution can be compared with what might be expected from overall outcomes in the population of hospitals as a whole. The expected probability of death for a given patient is computed using the risks (of death) of all patients in the combined standard hospital, with adjustment for various patient level characteristics that might influence mortality risk. The HSMR for a given hospital = observed number of deaths/expected number of deaths × 100. An HSMR above 100 indicates an unfavourable outcome, whereas an HSMR below 100 is favourable relative to the standard or total population.

Recent critiques of HSMRs have been based on the premise that HSMRs measure avoidable or preventable mortality.3,4 The critiques are ill founded. HSMRs are a measure of mortality, and mortality alone. This was explained very clearly by Professor Sir Brian Jarman, who provided evidence to the Independent Inquiry into Care Provided by Mid-Staffordshire NHS Foundation Trust:

The role of HSMRs as a measure of safety and quality of hospital care must be discussed in this context.

HSMRs based on information contained in routinely collected administrative datasets have been published for several countries.6-8 Such datasets provide primary and comorbid diagnoses and a variety of patient and episode characteristics, but not physiological data or clinical observations. The absence of patient-level clinical or physiological data may be seen as a weakness in HSMR presentations.9,10

Over the years, several studies have been undertaken to test the impact of such information on the discriminatory power of mortality risk adjustment models. Recently, researchers in the United Kingdom compared the discriminatory capacity of risk adjustment models derived from an administrative dataset with models based on databases compiled by professionals and held within specialist registries.11 Their results clearly demonstrated that models based on administrative data were as successful in discriminating cases as those derived from the more detailed clinical information held in specialist registries. Models derived from administrative data systems were also found to be adequately discriminatory in a study of postsurgical outcomes in the United States Department of Veterans Affairs surgical clinical improvement program.12,13

Contemporary administrative data systems — professionally extracted and coded, with a wide variety of primary and secondary diagnoses — are acceptable resources for generating HSMRs, and there is little difference in terms of discriminatory power between risk adjustment models derived from them and models derived from clinical databases.14 This is reassuring, because the cost and complexity of extracting clinical, or even simple laboratory, information on a large scale from existing record systems are substantial. This is true even in countries such as the US, where:

Although at times concerns have been expressed as to the accuracy of coding of diagnostic information within administrative datasets,16,17 the extent of such disagreements in Australia, at least, is modest. They certainly appear no greater than those found in the daily interactions between colleagues within the same team or discipline.18,19 Apart from diagnoses, data elements in administrative datasets have generally been chosen because they are robust, straightforward to collect and enumerate and, in the Australian National Hospital Morbidity Database, come with very explicit rules for their definition and tabulation.20 Coding audits constitute the test of inter-rater reliability relevant to assessing the utility of risk-adjusted measures of hospital mortality. Audits commonly lead to no more than a small proportion of cases being re-coded, implying an acceptable level of inter-rater reliability.21

In indicator work, reliability is considered before validity. Indicator reliability is conveniently considered in relation to two domains: inter-rater reliability and test-retest reliability. Inter-rater reliability relates to the likelihood that two raters, presented with the same information, will nominate similar values for a measure of interest. One of the advantages of using mortality as an indicator is that inter-rater reliability for the fact of death is high. Test-retest reliability relates to the broader issue of measurement stability, and this is a matter of relevance in relation to HSMRs.

There are two levels of influence on HSMR outcomes: patient-level influences, present at the point of admission; and institutional and jurisdiction-wide influences. There is a concern that the random play of chance on factors at and above the patient level may be so great as to render HSMRs uninterpretable. In this regard, the AIHW report was reassuring.1 The report contained a 3-year, longitudinal study of Australian hospital outcomes, and its results confirmed the findings of a Dutch study7 that hospital HSMRs are mostly stable over time, and that the effect of random variation is modest (the main exception being small hospitals, where numbers of deaths are small).

Although concerns about reliability can now be largely laid to rest, interpretation of HSMRs remains controversial,17,19 and the report concluded that HSMR reports should be seen as a safety and quality screening tool, rather than as being diagnostic.1 In other words, they should be seen as general-purpose indicators that provide a spur to further investigation, rather than as a definitive report on existing practice in any one institution or clinical service. Further, although HSMRs do seem applicable to a wide range of hospitals, they are less relevant to small hospitals and specialised hospitals with an atypical casemix, such as specialised women’s or children’s hospitals.

One abiding concern with HSMR measures is the interaction between acute and palliative care services. Hospitals that provide extensive palliative care services could appear to have an elevated HSMR.

Palliation as an intention of care at some point during an acute admission must be separated from management of a patient by a designated palliative care service. The latter are currently excluded from Australian HSMRs. Patients treated by a designated palliative care team are identified as patients treated with a palliative care type, and are excluded from HSMR calculations. Current coding standards strongly discourage the use of palliation as a secondary diagnosis for acute care, and palliation as a secondary diagnosis does not currently lead to exclusion from the HSMR measure. Research is continuing on this issue, and baseline data now exist against which to assess any abrupt changes in the use of the palliative care type in the future.

The methodology now uses an updated risk adjustment methodology that allows for interactions between the independent variables, which has also been tested for variations in the impact of age over the lifespan.

The Box shows the distribution of HSMRs across Australia for patients in one peer group of hospitals, reported using a visual presentation system known as a caterpillar plot.22 Other formats, such as funnel plots, locate hospitals against predetermined statistical parameters, and allow for identification of hospitals whose results are statistical outliers in terms of higher and lower than expected mortality.1,23 There is a concern that caterpillar plots can be turned “on their side” and used to rank hospitals. No visual plot is immune from being used as a ranking resource. If hospitals are named, and their position relative to each other indicated, they can be ranked. A sustained education campaign will be necessary to minimise the inappropriate use of HSMR information, if that information is to be placed in the public domain.

Peer-reviewed research demonstrating that the provision of HSMRs to hospitals acts as a spur to efforts to improve hospital safety and quality is limited, but it does confirm that, when taken at face value, HSMR reports can provide an impetus to hospital-level efforts to improve the safety of care that is provided, and be used to monitor the impact of efforts to reduce inhospital mortality.24,25 Experience indicates that for HSMRs to have an effect, they have to be provided regularly, and in a format that clinicians can understand and relate to. Concerns about methodology among those who feel their reputations might be prejudiced have to be able to be worked through, and may influence the nature and extent of data provision to jurisdictions, individual institutions, and the community at large.

Important opportunities exist to further enhance the robustness of mortality measures by means of data linkage processes creating patient-level, rather than separation-level, analyses of hospital activity. Including deaths occurring soon after discharge from hospital in mortality calculations may resolve many of the concerns about the influence of palliative care on HSMRs, and allow for assessing the extent to which existing risk adjustment for the impact of inter-hospital transfers on HSMRs do, or do not, require further elaboration. Also, from the beginning of the 2008–09 financial year, clinical coders have been required to indicate whether a secondary diagnosis was “present on admission” or occurred after admission. Risk adjustment that only adjusts patient-level problems present at the point of admission holds out a promise of avoiding the “moral hazard problem” of risk adjusting out the impact of secondary conditions that may have been the result of problems that occurred as a result of suboptimal care, rather than identifying them.26,27 Further work will be required before the real impact of the “present on admission” flag can be ascertained. But overall, it appears that Australia is well placed to make good use of the information provided in its administrative datasets to generate mortality measures, and has made solid progress towards the routine production of these important measures.

Caterpillar plot of hospital standardised mortality ratios for A1 hospitals22

A1 = major city hospitals with > 20 000 acute casemix-adjusted separations and regional hospitals with > 16 000 acute casemix-adjusted separations per annum.