Risk-adjusted hospital mortality rates for stroke: evidence from the Australian Stroke Clinical Registry (AuSCR)

Stroke imposes a major health care burden, but the adoption of effective interventions varies widely.1 Efforts to improve the quality of stroke management rely on rigorous outcomes data2 for avoiding misleading comparisons of hospitals. To identify potentially modifiable factors, analyses must account for casemix differences and random error.3 In particular, analyses must take stroke severity into consideration, as it is one of the strongest predictors of stroke mortality.2,4,5

Although the methodology is still evolving, standardised risk adjustment2 is highly relevant to health care consumers and policy makers. In a recent report of routinely collected hospital admissions data, significant variation in 30-day stroke mortality was found after adjusting for age, sex and comorbidities (including hypertension and diabetes), but there was no adjustment for stroke severity.6 The National Health Performance Authority (NHPA) has identified stroke as a condition for which inter-hospital differences in models of care (eg, patterns of patient transfers) and inconsistent recording of clinical information and procedures (eg, palliative care coding) may distort comparisons of mortality.7 Because hospital data must be complete, accurate and consistent, the NHPA is currently unable to support public reporting of inter-hospital disease mortality rates, as such comparisons could be unreliable.7 In contrast, the ability to reliably compare hospital performance with respect to patient outcomes has rapidly accelerated improvements in health care overseas.7

Our aim was to describe variance in 30-day stroke mortality between hospitals using risk-adjusted mortality rates (RAMRs), as part of our trialling a recently recommended new statistical method that includes stroke severity as a covariate.2

Methods

Study design

The Australian Stroke Clinical Registry (AuSCR) is a voluntary, prospective, clinical quality registry that captures standardised data for nationally agreed variables for all patients admitted to participating hospitals with acute stroke or transient ischaemic attack (TIA).8 AuSCR includes personal information (eg, name, address), clinical characteristics (eg, type of stroke), quality of care indicators (eg stroke unit treatment), and outcomes measured at discharge and at 90–180 days (eg, survival and quality of life).8 Stroke severity is captured using a simple, validated prognostic measure, the “ability to walk unaided at the time of hospital admission”.9 In the original statistical modelling by Counsell and colleagues,9 this criterion was associated with a relative risk for 30-day survival of 1.63 (95% confidence interval [CI], 1.15–2.31). In our earlier work, the strongest predictor of independence at time of hospital discharge was the ability to walk on admission (odds ratio [OR], 2.84; 95% CI, 2.18–3.71).10

Data from participating hospitals were obtained for the period from 15 June 2009 (six participating hospitals) until 31 December 2014 (40 participating hospitals). We included all stroke types (ischaemic, intracerebral haemorrhage, and undetermined) in our analyses, as well as demographic variables, as stroke mortality is higher at all ages for Indigenous than for non-Indigenous Australians,11 varies according to country of birth,12 and is greater for people of lower socio-economic status.13 Socio-economic status was assessed by matching patients’ addresses with the corresponding Index of Relative Socio-economic Advantage and Disadvantage (IRSAD) score,14 collated as quintiles. Whether the stroke was the first or subsequent stroke experienced by the patient was included as a covariate, as the risk of death is greater for recurrent events.15 Age was included as a continuous measure in all models. All episodes occurring within 30 days of admission were included. Harrell’s concordance statistic (C-statistic) was used to determine how well the variable “ability to walk on admission” predicted 30-day mortality in our models.

We excluded patients who experienced a stroke while in hospital for another condition or when transferred from another hospital, as the different patterns of care may distort mortality ratios.7,16 Data for stroke care in a paediatric hospital were also excluded because of the small sample size (fewer than 50 care episodes).

Mortality data

Survival status at 30 days was obtained by probabilistic matching of AuSCR registrant identifiers with the National Death Index (NDI) by the Australian Institute of Health and Welfare. AuSCR staff undertook the review of non-exact matches; for discordant dates, we used NDI data as the reference. Based on this linkage method, in-hospital death reporting in the AuSCR had 98.8% sensitivity and 99.6% specificity, compared with an in-hospital death determined with the NDI date of death.

Outcome measures and analyses

The primary outcome was the risk-adjusted mortality rate (RAMR) at 30 days after admission, using the method recommended by the American Heart (AHA) and Stroke Associations (ASA).2 To maximise the reliability of our estimates,17 analyses were conducted for individual hospitals that provided data on at least 200 episodes of care for stroke between 2009 and 2014. The stages in deriving each model were:

• entering the observed values;

• generating estimates from generalised linear latent and mixed models (GLLAMM) by maximum likelihood;

• generating expected probabilities;

• generating predicted probabilities;

• generating ratios predicted as expected; and

• generating the RAMR.

The RAMR for each hospital was calculated by dividing the overall RAMR by the risk-adjusted average hospital mortality, and then multiplying by the overall crude (unadjusted) proportion of deaths in the whole sample.

The results were compared in models with covariates corresponding to those available in hospital admissions data (the hospital admissions model) and after also including covariates corresponding to those available only in the AuSCR (the Registry model). A model adjusted only for age and sex was also estimated. The hospital admissions model was adjusted for age, sex, year of data, stroke subtype, IRSAD quintile, Indigenous status, and place of birth (Australia v elsewhere). The Registry model was adjusted for the same variables, as well as for stroke history and severity. Differences between the models in the ranking of individual hospitals were explored.

Data are provided on the calibration and discrimination of the models,2 using the likelihood ratio test, the Akaike information criterion (AIC), the Bayesian information criterion (BIC), and the C-statistic. A smaller AIC or BIC indicates a better fitting model; a C-statistic of 1 indicates a perfect fit model, while a C-statistic of 0.5 indicates that fit is no better than chance. Multilevel models were used, with one level defined as the hospital unit, to account for correlations between patients who were managed in the same hospital, and the other representing patients as individual units.

P < 0.001 (two-sided) was deemed statistically significant because of the large sample size. Analyses were performed in Stata 12.1 (StataCorp).

Identifying significant mortality variation and differences in mortality outcomes

According to standard practice, hospitals within two standard deviations (SDs) of the overall average RAMR were deemed to lie within normal variation, and those outside three SDs were deemed to vary significantly from the other hospitals in the sample.18 Funnel plots were used to investigate deviations from the average hospital mortality rate.19 The direction of the change was also explored by graphing the difference between the age- and sex-adjusted hospital admissions and Registry RAMR estimates.

Ethics approval

Appropriate ethics and governance approvals were obtained for all participating hospitals in AuSCR, and from the Human Research Ethics Committee of Monash University (reference, CF11/3537–2011001884). Ethics approval was obtained from the Australian Institute of Health and Welfare to conduct data linkage to the National Death Index (reference, EO 2013/2/16).

Results

Between 2009 and 2014, 26 302 episodes of care for 24 806 individual patients from 45 hospitals were recorded; 3151 patients (12%) died within 30 days of admission (excluding TIAs, 14%). The concordance of ability to walk on admission as an indicator of stroke severity with 30-day mortality was excellent (C-statistic, 0.97). Patients with intracerebral haemorrhage (29%) were more likely to die within 30 days than those with other stroke types (ischaemic, 12%; undetermined stroke, 14%; TIAs, < 1%) (online Appendix 1).

Data from hospitals reporting at least 200 episodes of stroke care

Eighteen hospitals located in metropolitan areas and ten in rural and regional areas, each with a stroke unit, provided data for at least 200 episodes of care. Hospitals in the eastern states contributed most data (Victoria, 40% of episodes; New South Wales, 17%; Queensland, 34%; Tasmania, 4%); in Western Australia (5% of episodes), only two hospitals participated. We excluded from our analysis 7509 patients who had a TIA or in-hospital stroke, or who were transferred from another hospital (online Appendix 1).

In total, 16 218 episodes of care were provided to 15 951 individual patients (median age, 77 years; women, 46%; ischaemic stroke, 79%). Compared with patients who were alive 30 days after admission, the proportion of women among those who died was greater; they were also older, fewer were able to walk on admission, and more had a history of stroke or TIA (Box 1). The characteristics of patients with stroke were similar across the 28 hospitals with respect to age, sex, and ability to walk on admission (online Appendix 2). The proportions of patients with severe strokes were similar for hospitals with more or fewer episodes of care (data not shown; P = 0.59).

Comparison of hospital 30-day mortality outcomes

The unadjusted (crude) mortality rates for the 28 hospitals with at least 200 episodes of care ranged between 7% and 23%. Excluding the 7509 patients who had a TIA or in-hospital stroke, the unadjusted mortality rates for hospitals ranged between 5% and 20%, and the age- and sex-adjusted mortality rates ranged between 8% and 20%. The RAMRs estimated by the hospital admissions model ranged between 9% and 20%, and those by the Registry model between 9% and 21% (Box 2). The overall RAMRs adjusted for different combinations of Indigenous status, country of birth and history of stroke are reported in online Appendix 3. According to the model fit statistics (BIC, AIC, likelihood ratio test, C-statistic), the Registry model had the best fit (Box 3, online Appendix 3). Correlations between the number of episodes contributed by a hospital and the differences between age- and sex-adjusted RAMRs and the Registry RAMR estimates (R² = 0.021) or hospital admissions RAMRs (R² = 0.001) were low. When the results of the hospital admissions and Registry models were compared, the variance ranged between 0% and 3%.

Although the ranges of estimates by the adjusted models were similar, the rank order of hospitals changed according to the initial crude estimate and simple age- and sex-adjusted models (Box 2; online Appendix 4). The change in ranks in the hospital admissions and full Registry models illustrates the possibility of a hospital attaining very different results. The models with the best fit were those that included stroke severity as a covariate (Box 3). Based on the funnel plot distribution, the estimated mortality for only two hospitals was more than three SDs from the mean, one with low mortality, and the other with borderline excess mortality relative to other hospitals (Box 4).

Quality of care and correlations with mortality rates

Adherence to processes of care was similar for all hospitals (online Appendix 2). Stroke unit admissions ranged from 99% for the hospital with lowest RAMR to 80% for the hospital with the highest RAMR (weak positive correlation between increased stroke unit access and lower RAMR: R² = 0.138). A negligible positive correlation was noted between increased prescription of antihypertensive drugs at discharge and lower RAMR (R² = 0.021).

Discussion

Assessing the quality of health care delivered by different health care providers is complicated by the variable quality of routinely collected hospital data.7 For burdensome conditions such as stroke, this problem is exacerbated by the inability to account for differences in stroke severity and by inaccuracies in the coding of diagnosis or cause of death.20 Clinical quality registries have emerged as important tools for resolving these problems, but support from government agencies is not as consistent in Australia as in comparable countries.

We have provided an important illustration of the value of a national clinical quality registry for stroke, using a new method for calculating mortality statistics. The models with the best fit for standardising mortality were those that included adjustment for stroke severity, a covariate routinely available only in AuSCR. The change in rank position according to different RAMRs was clearest for hospital 20, which was ranked number 12 in the full Registry model, but number 25 in the hospital admissions model. Rankings that are not based on models adequately adjusted for relevant risks can lead to interpretations that suggest that some hospitals provide substandard care, and thereby impugn their reputations and that of their clinicians. The funnel plot approach provides an alternative method for assessing performance, but the control limits associated with the assumption of a normal distribution of the data makes caution advisable, particularly if the data are skewed, as in our sample of only 28 hospitals. For the hospital with the lowest mortality in each of the models (Hospital 1), selection bias may have arisen because 99% of its patients were treated in a stroke unit (online Appendix 2), and other unmeasured factors may have also contributed to its better outcome.

Our findings differ from a previous investigation of hospital stroke mortality rates in NSW that applied more conventional modelling methods, without adjusting for stroke severity.6 Standardised 30-day mortality rates varied significantly, from 15% to 30%, and several hospitals were categorised as “poor performers”.6 Cases were sampled across different timeframes and with varying sample sizes, but there was a greater diversity of hospitals than in our study; for example, hospitals without stroke units were included.

Registry data that include disease severity risk-adjustment variables that supplement hospital data can be used to ensure that performance comparisons are more reliable. Given the growth in public reporting of hospital performance and the recognition of its potentially driving improvement of quality of care,21 it is essential that appropriate methods are employed. We estimated RAMRs using a new approach recommended by the AHA/ASA,2 replacing the observed number of deaths with a prediction of numbers of deaths estimated from the average number of deaths for hospitals in a risk-adjusted model. This reduces the influence of chance on the variation in RAMRs (predicted v expected). Our study is the first report on the application of this new approach, and our models predicted numbers of deaths within 0–9% of the actual number.

Our investigation has broader implications for Australia, in that it advances methods for hospital-level comparisons of risk-adjusted mortality, particularly on the basis of routinely collected registry data. We acknowledge, as a limitation of our study, that not all hospitals contribute data to AuSCR, and our findings may consequently not be generalisable to all hospitals. Further, our sample was restricted to hospitals reporting at least 200 episodes of care (10 084 episodes from 17 hospitals were therefore excluded from our analyses). Including all hospitals may have led to greater variance in our results, but our sample was broadly representative of the entire cohort (online Appendix 1). The overall crude 30-day death rate for eligible hospitals (with at least 200 episodes of care) was 15% (range, 7–23%), similar to reports from other countries (13–15%).4,17

Critical predictors of stroke mortality include age, sex, stroke severity, and comorbidities,18 and a further limitation of our study was the inability to adjust for comorbidities, but inconsistent reporting of International Classification of Diseases (ICD-10) coding of comorbidities is recognised.22 Future linking of AuSCR data with hospital admissions data will enable a greater range of variables to be explored. Several International stroke registries incorporate National Institutes of Health Stroke Scale (NIHSS) data that can be used for adjusting for stroke severity, but training is required to administer the NIHSS.23 The ability to collect NIHSS data was introduced in AuSCR in 2015, but the level of missing data currently undermines its usefulness, whereas “ability to walk on admission” information was available for 90% of episodes. In recent validation work,9 a model based on simple variables (including ability to walk) performed as well as one employing NIHSS and age data; the choice of measure should therefore be based on practical considerations.24 Because there were very few episodes of intracerebral haemorrhage, we included stroke type as a covariate rather than stratifying the dataset, as has been previously recommended by other authors.25

In conclusion, we highlight the importance of using appropriate risk adjustment variables and methods for comparing hospital outcomes for stroke, with particular emphasis on the need to account for stroke severity. Moreover, we have shown the value of clinical quality disease registry data for refining outcome performance measurement in health care. As this is an evolving field, further research into risk adjustment variables and comparison of mortality rates is encouraged.

Box 2 – Comparison of ranking of hospitals according to 30-day mortality for stroke by the hospital admission and full registry models*

---

* The full data for this figure are included in online Appendix 4.

Box 4 – Funnel plot of risk-adjusted mortality rates for hospitals (Registry model)*

---

SD = standard deviation. * The numbers for the hospitals indicate their rank according to crude mortality rates (lowest to highest). Registry model was adjusted for age, sex, stroke type, index of relative socio-economic advantage and disadvantage, Indigenous status, country of birth, year of admission, history of previous stroke, and stroke severity.