Using hospital standardised mortality ratios to assess quality of care — proceed with extreme caution

Death occurs in 5%–10% of all hospitalised patients. Most of these deaths (between 95%11 and 98%12) reflect the natural history of disease, not poor-quality care. Conversely, most quality problems, while associated with injury and prolonged hospital stays, do not cause death. Consequently, the HSMR, as a screening instrument for quality, is limited by low sensitivity (most quality problems do not cause death) and low specificity (most deaths do not reflect poor-quality care). Past simulation studies using mortality data from 190 US hospitals reveal that almost two-thirds of poor-quality hospitals (in which 25% of deaths were classified as preventable) demonstrated no significant increase in HSMR.13 A further limitation is that, in the absence of data linkage systems, the HSMR will not encompass deaths that occur after discharge or interhospital transfer, some of which may reflect poor-quality inpatient care.14

The available evidence indicates a weak and inconsistent association between HSMR and other measures of quality of care when performed as between-hospital comparisons. In one review of 378 patients who died from stroke, myocardial infarction or pneumonia in 11 outlier hospitals with substantially higher HSMRs, no differences were seen, compared with hospitals with lower HSMRs, in adherence rates for 31 recommended processes of care.15 Even differences in risk-adjusted mortality rates for three high-risk clinical conditions — acute myocardial infarction, heart failure and pneumonia — correlated very poorly with variations in 10 condition-specific process measures of quality reported across 3657 US acute-care hospitals.16 A recent systematic review of 31 studies concluded that risk-adjusted mortality is a poor predictor of preventable complications and quality problems among hospitals.17

In rendering the HSMR more able to distinguish low-quality from high-quality hospitals, much attention is given to optimising statistical models that adjust for differences between hospitals in patient characteristics that increase the risk of death, but which are independent of the quality of care.18 While risk-adjustment models based on administrative data appear to be equivalent to those based on clinical data in predicting risk for selected, well defined primary diagnoses within large datasets,19 both types of model may not sufficiently adjust for differences in casemix between individual hospitals. Certain comorbidities relevant to risk adjustment — such as obesity, dementia and heart failure — are inconsistently recorded in hospital statistics,20 as is overall functional status.21 This means that diagnosis coding may not distinguish full recovery from persistent disability. Accordingly, tertiary hospitals receiving patients with severe, complicated or end-stage disease may be disadvantaged compared with secondary hospitals that retain patients with less severe, uncomplicated disease.22 The HSMR may be falsely elevated or reduced according to variations in coding of present-on-admission comorbidities versus new diagnoses arising during admission, which may reflect avoidable complications.23,24 If differences in coding are non-randomly distributed among hospitals, differences in HSMR will reflect these biases rather than quality of care. In a recent US study involving more than 2.5 million discharges from 83 acute care hospitals,25 four common risk-adjustment models — including the method advocated by researchers at Flinders University6 — produced substantially different mortality rates. In 2006, 12 out of 28 hospitals with higher than expected hospital-wide mortality as classified by one method had lower than expected mortality when classified by one or more of the other methods. Explanations included disparate statistical methods, differences between hospitals in eligibility and exclusion criteria regarding admissions, and flaws in the hypothesised association between HSMR and quality of care.25

Choosing which diagnoses or patient groups should be included in the reference populations used to generate risk-adjustment models is particularly problematic. Admission practices for unplanned presentations vary across hospitals, with some institutions coding patients who die in emergency departments shortly after presentation as non-admissions. In Canada8 and the UK,9 even among coded admissions, hospitals exclude vague or undetermined diagnoses, which account for 20% of all deaths. While this maximises the accuracy of risk prediction, it further limits HSMR sensitivity for detecting quality problems. Conversely, specificity is reduced if models include patients who are receiving palliative or end-of-life care, or who have severe illness with dismal prognosis despite best care. Many of these patients will not be admitted to designated (and easily coded) inpatient palliative care units. In one UK hospital, 37% of admissions were found on case review to be for end-of-life care, compared with the reported coded figure of 22%, resulting in a fall of the HSMR from 105 to 68.21 In a Canadian study, the HSMR for one hospital over 2 financial years dropped from 148 to 55 after removal of palliative care patients.12 The HSMR may also vary according to whether a palliative care classification is present-on-admission or applied at various times during an admission that may in fact have involved a quality event.18

The HSMR becomes more imprecise for small hospitals with fewer deaths as a function of fewer admissions. The resultant wide confidence intervals preclude detection of statistically significant outliers that warrant more detailed investigation. Statistical techniques such as regression to mean, shrinkage estimators and enlarged sample sizes (using multiyear data)26 can be used to improve precision. However, their application requires further validation, and those dependent on data gathered over several years in the past are less relevant to current practice.

At the hospital level, HSMRs should be relatively stable over time, and immune to seasonal and annual fluctuations. Although some studies confirm this,6,27 others show substantial short-term changes in HSMR beyond those reasonably attributable to clinical advances or quality-of-care improvements.2,12

In the absence of wide awareness of its limitations and careful clinical interpretation, the hospital-wide HSMR can mislead. Unfavourable HSMRs based on incorrect data or analyses can trigger external inquiries that stigmatise individual hospitals, lower morale and public confidence, and encourage “gaming” of data — eg, by upgrading risk assessments28 — or the pursuit of inappropriately aggressive care.29 The HSMR also overlooks factors that may account for seemingly better performance, such as greater access to step-down facilities, hospice or residential care, or community health services that reduce length of stay and risk of inhospital death.30 Compared with HSMRs specific to clinical condition, hospital-wide HSMRs do not enable hospital clinicians or administrators to easily pinpoint correctable processes of care at the level of the individual departments or units that account for most quality problems (eg, surgery at Bundaberg Base Hospital in 2003–2005 and paediatric cardiac surgery at Bristol Royal Infirmary in 1984–1995). Even within a hospital whose overall HSMR is 100 or less, individual diagnoses and procedures may demonstrate higher than expected mortality.