“Death in low-mortality diagnosis-related groups”: frequency, and the impact of patient and hospital characteristics

Monitoring and reporting of patient safety indicators (PSIs) at a national level has been identified as one of the major components of health care reform in Australia.1 Australia is following the trend of other Organisation for Economic Co-operation and Development (OECD) countries, such as the United States, which first introduced national PSIs in 20032 under the premise that collection and analysis of patient safety incident data would facilitate learning and the development of solutions that result in improved quality of care. “Death in low-mortality diagnosis-related groups” (LM-DRGs) is a quality and safety indicator used in health care services worldwide. The indicator identifies inhospital deaths that occur among patients assigned to a DRG with a low associated risk of death, and that are therefore more likely to be attributable to a safety or quality issue.3 In 2009, the indicator was included in the 25 national safety and quality in health care indicators proposed for Australian hospitals, and was recommended to be publicly reported at a national and individual hospital level.4 Since these recommendations were made, a comprehensive review has concluded that “higher quality, prospective, analytic studies are required before death in LM-DRG is used as an indicator of quality and safety in health care”.5

PSIs are designed to provide information about the relative quality-of-care performance of health care services. A recent article describes a framework to drive quality-indicator development and evaluation.6 The authors of this article state that, for an indicator to be scientifically acceptable, it must have several attributes. Important ones are:

• validity (it measures the intended aspect of quality, and accurately represents the concept being evaluated);

• sensitivity (sufficient variation can be explained by provider performance after patients’ characteristics are taken into account); and

• event frequency (the sample size is large enough to detect actual differences, and risk adjustment is adequate to address confounding bias).

Current use of the death-in-LM-DRG indicator does not include any risk adjustment. This assumes the indicator is unaffected by patient or hospital factors unrelated to quality of care. Given the heterogeneity of hospital service provision across Australia, it is likely is that these factors could be associated with LM-DRG deaths, and that failure to adjust for their influence in analysing rates may lead to inaccurate identification of hospital outliers. Meaningful hospital performance monitoring using PSIs also requires a sufficiently large number of events to be able to differentiate random noise from special-cause variation. A genuine rate increase (special-cause variation) should be identified promptly so that its cause can be investigated and eliminated.

In Australia, there is no published data on the frequency of deaths flagged by the death-in-LM-DRG indicator or on patient and hospital characteristics that influence their occurrence. In this study, we sought to identify the incidence of LM-DRG deaths, the impact of patient and hospital characteristics on their occurrence and the demographic, admission and clinical profile of patients in LM-DRGs who died.

We undertook a retrospective cohort study of discharge episodes from 122 public hospitals in Victoria. The analysis sample included all 2 400 089 discharge episodes for people aged 18 years or older recorded in the Victorian Admitted Episodes Dataset (VAED) from 1 July 2006 to 30 June 2008. The VAED is the administrative data set for Victorian hospitals that includes demographic, administrative and clinical information. Since 1998, clinical information in the VAED has been coded using the International statistical classification of diseases and related health problems, 10th revision, Australian modification (ICD-10-AM). Good to excellent coding quality of ICD-10-AM has been demonstrated for coding of diagnoses.7

There were 1 008 816 LM-DRG episodes over the 2 years. Box 1 outlines the patient discharge episodes included in the analysis. Box 2 presents the demographic characteristics of each financial-year cohort. More than 40% of patients with LM-DRGs who died were aged 83 years or older, and 39% or more had a length of stay of 1 day or less. In both cohorts, many LM-DRG deaths occurred in patients whose admissions were unplanned and classified as emergency admissions (> 74%) and as medical DRGs. Almost 20% of LM-DRG deaths in the 2006–07 financial year occurred in patients transferred from other hospitals. This proportion was lower (less than 12%) in the 2007–08 financial year. The DRG, primary diagnoses, procedures and complications recorded for LM-DRG deaths in the VAED were highly variable with no single DRG, diagnosis, procedure or complication being reported in more than 10% of cases.

Although the death-in-LM-DRG indicator has good face validity and is easy to generate from hospital administrative datasets, it is premature to infer differences in the safety of hospital patient care based on information yielded by this indicator. Our study shows that there are several patient-episode characteristics unrelated to quality of care that influence the likelihood of death in LM-DRG episodes. One main methodological challenge — the low frequency of LM-DRG deaths — suggests that this indicator is likely to be insensitive for true variations in quality of care in the hospital setting. The findings of this study highlight that the death-in-LM-DRG indicator requires further refinement before it can be employed broadly as a quality and safety metric.

An important consideration in the use of quality and safety indicators like death in LM-DRG is that their benefit must outweigh their burden. This study raises questions about the benefit of this indicator. Less than 50% of adult patient episodes are included in its generation (Box 1). Consequently, the indicator provides no information about the quality of hospital care for more than 50% of patient episodes. Patient age, sex, level of comorbidity and admission source and type were significantly associated with the risk of death in LM-DRG episodes in our sample. This indicates that several factors unrelated to the quality of care provided by a hospital may be the source of variations in this indicator.

The finding that more than 40% of LM-DRG deaths occurred in patients aged 83 years and over, and that these patients were also more likely to be medical patients admitted from residential aged care facilities and to die shortly after being admitted, suggests that there may be many false positives with this indicator. The indicator may be biased towards detecting deaths in older hospitalised patients for whom death may be an expected outcome. The finding that LM-DRG patients who were transferred from a residential aged care facility were more likely to die requires further investigation, especially given that this was independent of age and level of comorbidity. Our findings align with a study evaluating the impact of living in a nursing home on mortality within two hospital internal medicine services. The study found inhospital mortality was significantly associated with living in a nursing home, independent of age, sex, condition, level of comorbidity and hospital of admission.13 As highlighted above, death may be an expected outcome for many of these patients, and not an outcome of poor quality of care.

Another interesting finding of our study was that LM-DRG patients who were transferred from another hospital were three to six times more likely to die than those admitted from other sources. There are several reasons for patients to be transferred from one hospital to another. For example, a patient may require procedures or services not available at the hospital that they were first admitted to, such as dialysis or intensive care, or the transfer may be a natural care transition from acute to subacute care for rehabilitation purposes. Consequently, explaining the finding of increased likelihood of death in LM-DRG patients transferred from another hospital requires more detailed investigation of cases and interhospital transfer practices. Use of 30-day mortality, as opposed to inhospital mortality, should be investigated as this may show different associations.

We also explored the application of a risk-adjusted death-in-LM-DRG indicator, and found a very different picture of outlier hospitals than found when considering unadjusted LM-DRG deaths. Risk-adjusted death-in-LM-DRG results flagged substantially fewer hospitals as potential high outliers than the unadjusted death-in-LM-DRG indicator. This analysis shows that failure to adjust for patient and hospital characteristics that are unrelated to quality of care may result in unfair and inaccurate identification of outlier hospitals using the death-in-LM-DRG indicator. However, Box 4 shows that many hospitals have very wide SMR confidence intervals (hence the presentation on a log scale) suggesting that even the risk-adjusted death-in-LM-DRG indicator results are imprecise. The imprecision may be partially explained by the low event rates creating the situation of low “signal-to-noise” ratios, and smaller hospitals displaying greater variability in mortality performance indicators.14,15

We found that admission to a low-volume hospital (< 1000 episodes per year) significantly increased the risk of death in LM-DRG episodes. This risk was independent of age, sex, level of comorbidity and admission from residential aged care. Hospital type, classified as major provider/teaching hospital, and metropolitan location were found to have little association with death in LM-DRG episodes. These findings are consistent with a US study that also found teaching and rural/urban status were not associated with death-in-LM-DRG episodes.16 The finding of increased risk of death in low-volume hospitals is novel in the LM-DRG literature, but consistent with studies in several other populations. One systematic review, of studies across 33 diagnoses and interventions, investigated associations between hospital volume and risk-adjusted mortality.17 This review concluded that higher hospital volume is associated with higher survival. The relationship we found between volume and outcome is potentially important, but further investigation is required as differences in operational aspects of hospitals and casemix that we did not measure may have confounded the results.

Our findings, in conjunction with those of a growing number of other studies, raise questions about the validity of the death-in-LM-DRG indicator as a quality-of-care metric. A recent review provides a comprehensive synthesis of findings of 12 previous studies evaluating the indicator;5 only three of these studies provided some evidence that there were greater quality-of-care deficiencies in death-in-LM-DRG cases than other cases. A large-scale US study in 4504 hospitals found the indicator had a weak and sometimes inverse relationship with other measures of hospital quality, such as risk-adjusted mortality, Hospital Quality Alliance scores and US News and World Reports ratings systems.18 The indicator has also been found to have no association with hospital accreditation scores.19 Another US study reported on a medical record audit of 110 paediatric death-in-LM-DRG cases occurring in 14 hospitals.20 This study found that the indicator was not suitable for use in paediatric populations as 71.8% of the deaths were categorised as unpreventable and 13.6% as unable to be determined.20 A United Kingdom study found no association between hospital SMRs and the number of deaths in low-mortality health care resource groups.21

In consideration of the burden of the use of the death-in-LM-DRG indicator, we identify several issues. In Victoria, this indicator has been used as a screening tool to flag deaths that should undergo medical-record review to determine if there were deficiencies in the quality of care. Medical-record review by hospitals is a resource-intensive process. It has been recommended that to achieve higher levels of reliability in determining quality-of-care issues, the review should be undertaken by one registered nurse and at least two physicians.22 Our findings suggest that further validation of the indicator should be undertaken before valuable and often limited hospital health care staff resources are directed to the medical-record review of LM-DRG deaths.

Past literature suggests that differences in documentation, length of stay (time at risk) and coding practice (resulting from how ICD codes are used to set hospital payments) may cause variability in indicator events derived from administrative datasets.23,24 Also to be considered is that the death-in-LM-DRG indicator may have limited ability to detect quality-of-care deficiencies because of a low signal-to-noise ratio.

There are several limitations of our study that should be considered. First, analysis was based on episode-level and not patient-level data, so adjustment for clustering of observations using the hierarchical regression models may not have been optimal. However, these models are commonly applied in situations such as this where analysis is undertaken on de-identified administrative data. Second, there may be deficiencies in the quality of the underlying data source. There may have been differences in coding practices between hospitals.23 Finally, other factors, such as disease severity, ethnic disparities and socio-demographic characteristics, may also affect the likelihood of death-in-LM-DRG episodes, but we did not measure them.

The focus of our investigation was simply to establish characteristics of patient discharge episodes and of hospitals that influence the likelihood of death in LM-DRG episodes. To date, there is no proven link between deaths flagged by this indicator and the existence of quality-of-care problems in the Australian setting. In addition, it has not been proven, using either unadjusted or risk-adjusted methods, that outlier hospitals (those with higher rates of deaths in LM-DRGs) have higher proportions of cases with quality-of-care problems than non-outlier hospitals.

Further analyses that focus on determining the criterion validity of the indicator should be undertaken. These studies should employ medical-record review of LM-DRG and non-LM-DRG deaths to ascertain if deficiencies in the quality of care are more frequent in LM-DRG deaths. Risk adjustment models should be considered to prevent hospitals being wrongly being identified as outliers. In addition, the indicator should be evaluated using 30-day mortality to assess whether this provides a different profile and is a more precise indicator of quality of care. However, an important caveat in considering the further development of the death-in-LM-DRG indicator is that these events are rare. This suggests the indicator has limited utility as a quality-of-care metric. It is important to note that rare or sentinel events should not be ignored at a hospital level — local hospital quality systems should be developed to identify and investigate these events. It is using indicators of these events to measure and benchmark quality of care that is problematic. Alternative indicators that: (i) include a larger proportion of patient episodes; (ii) for which there are sufficient data to enable meaningful measurement and monitoring; and (iii) most importantly, that have demonstrated validity for measuring quality of care, should ultimately be the focus of future research.

1 Flow diagram of episodes of patient discharge from 122 hospitals in Victoria that were included in the analysis by year, for the financial years 2006–07 and 2007–08

DRGs = diagnosis-related groups. * Episodes may have had more than one exclusion criterion.

3 Odds ratios (and 95% CIs) for estimated associations between risk of death in low-mortality diagnosis-related groups (LM-DRGs) and selected patient and hospital characteristics in122 hospitals in Victoria for the financial years 2006–07 and 2007–08

RACF = residential aged care facility. * All episodes per year (pale blue shading distinguishes this as the only categorical predictor). Interpretation: If the 95% CI bars of the odds ratio cross the value of 1, there is no statistically significant association between the characteristic and risk of death in LM-DRG episodes. 95% CIs lower than 1 indicate a decreased risk of death, and above 1 an increased risk of death. For the predictor “Annual volume”, the category ≥ 6000 is the reference category against which the other two categories are compared. Thus, for example, a hospital with an annual volume of < 1000 episodes in 2006–07 had almost 10 times the risk of death in LM-DRGs than another hospital that had ≥ 6000 episodes in the same year. In 2007–08, this relative risk had dropped to less than five times in the smaller (lower volume) hospital compared with a larger one.

4 Hospital standardised mortality ratio results by year

Interpretation: If the 95% confidence interval bars of a hospital’s standardised mortality ratio (SMR) cross the value of 1 there is no significant difference in LM-DRG mortality for that hospital compared with the state benchmark. If the SMR confidence interval bars are below 1, the mortality is below the benchmark, and if they are above 1, the mortality is above the benchmark.