Comparing risk-prediction methods using administrative or clinical data in assessing excess in-hospital mortality in patients with acute myocardial infarction

Study participants were 467 consecutive patients admitted to a tertiary hospital in Queensland, between 1 July 2003 and 31 March 2006, with a primary discharge diagnosis of AMI (International classification of diseases, version 10.3, Australian modification codes I121 and I122) and who met the following criteria: age 30–89 years; hospital stay < 30 days; Queensland resident; acute admission via the emergency department (ED); and not transferred to another hospital. Coded diagnoses were ascertained by review of medical records and, in the case of deaths, of death certificates. Administrative data were collected, retrospectively, by coders on all cases within 3 months of discharge, and clinical data were abstracted by a single clinical investigator (S N) between 1 October and 22 December 2006. Coders and the investigator were blind to study objectives and collected data on standard forms. The study was approved by the chair of the district quality and safety council.

The graphical tool used in this study was an adaptation of the VLAD, which incorporated control limits derived from the risk-adjusted cumulative sum.16,17 Briefly, the VLAD shows a plot, for a series of consecutive patients over time, of the cumulative risk-adjusted difference between observed and expected deaths, expressed as the statiscal estimate of excess lives gained or lost. Upper and lower control limits (based mathematically on the sequential probability ratio test2 applied to 10 000 iterations of specified datasets of 10 000 patients) were added which, when breached by the curve, corresponded, respectively, to a real 30% decrease or increase in mortality with 95% confidence. Each time the control limit was breached, the limits were reset, with the breach point taken as the new baseline. More in-depth explanations of the mathematics involved can be found elsewhere.4,5,17

Associations between patient variables and mortality were identified by using χ2 methods and were expressed as odds ratios (ORs) with 95% confidence intervals. Independent predictors were determined by multivariate logistic regression models. Two VLAD curves plotted from the administrative and clinical models were compared, and the models were assessed for their discriminatory power (C statistic11) and goodness of fit (Hosmer–Lemeshow χ2 test, a measure of calibration),18 as applied to the study sample. The curve was also replotted by using an altered administrative risk-prediction model whose reference population included only patients admitted to tertiary hospitals.

In sensitivity analyses, curves for both models were replotted, after sequential exclusion of the previously defined patient groups, and compared. The incidence of non-fatal adverse events related to health care occurring during admission in patients who survived more than 24 hours after presentation was determined.

Of the 467 patients included in the original administrative dataset (Box 3), most were male, and the mean age was 65 years. Of these patients, 416 (89%) were acute admissions to the ED from the community, and 447 (96%) had a confirmed diagnosis of AMI. Of 67 patients who died in hospital, nine represented either misdiagnosis (two with alternative diagnoses of septic shock and pulmonary thromboembolism) or unsubstantiated diagnoses (seven with AMI listed as an unconfirmed cause of death on the death certificate), 14 had out-of-hospital, out-of-ambulance cardiac arrests in which circulation was lost for at least 15 minutes, and 23 were deemed as receiving care with palliative or conservative intent. Of 400 survivors, 11 cases were misdiagnosed cases of unstable angina.

Variables associated with a significantly increased risk of death on univariate analysis were out-of-hospital arrest, misclassification, palliative or conservative care intent and care received from non-cardiology teams, while being male was associated with less risk (Box 3). After multivariate regression analysis, misclassification (OR = 4.52; 95% CI, 1.49–13.68; P = 0.008), transfers from other hospitals (OR = 3.20; 95% CI, 1.30–0.92; P = 0.01) and care from a non-cardiology team (OR = 3.56; 95% CI, 1.50–8.44; P = 0.004) remained significant as independent associations.

When applied to the total patient sample, the clinical model showed slightly better discrimination (C = 0.89; 95% CI, 0.85–0.94) than the administrative model (C = 0.84; 95% CI, 0.81–0.88). Goodness of fit of both models was low (Hosmer–Lemeshow χ2 = 22.69 and χ2 = 25.57, respectively; P ≤ 0.004 for both), although the clinical model generated significantly higher risk estimates for patients who died than did the administrative model: median (interquartile range) values were, respectively, 40.0% (34.0%) and 21.5% (29.8%), P < 0.001.

The original VLAD for the study hospital based on the administrative risk-prediction model (Box 4) showed a steadily increasing excess mortality over the 33-month study period, culminating in an estimated 11 excess deaths, with 30% excess mortality signalled with 95% confidence at Patient 134. A replotted administrative curve using tertiary hospital patients only as the reference population (not shown) showed a nearly identical pattern, with signalling of excess mortality delayed marginally to Patient 139.

Applying the clinical prediction model to the original VLAD also produced a nearly identical curve (Box 5 [A]), with significant correlation between models in risk estimates for individual patients (r = 0.46, P < 0.001) and similar ranges for risk estimates: administrative 1.6% to 63.9%, clinical 0.2% to 52.0%.

Replotting of data from both models, after exclusion of misclassified cases, out-of-hospital, out-of-ambulance cardiac arrests and deaths in ED within 30 minutes of presentation (n = 37), yielded two upwardly directed curves (Box 5 [B]) which, depending on the model, represented a net gain of three or seven lives at the end of the study period, with no breaching of control limits (not shown). This upward pattern persisted as previously defined patient groups were sequentially excluded (data not shown), with the final plot after deselection of 156 patients showing a net gain of seven or 10 lives (Box 5 [C]).

There were three health care-related serious events (two episodes of major bleeding requiring transfusion and one of reversible renal failure requiring temporary dialysis) among 400 survivors (0.6%) and none among the 67 who died.

The first limitation of our study relates to incomplete ascertainment of all cases of AMI in the original administrative dataset, because of misdiagnosis by clinicians or error by coders. During the study period, the hospital pathology laboratory reported 2970 instances of elevated troponin I, a sensitive biomarker for AMI, but only 447 correctly coded cases of AMI. While a detailed audit has not been done, we suspect that at least half of these “positive” troponin assays represent non-AMI elevations (≤ 0.1 ng/mL),20 and another third comprise serial assays performed on the same patient. This reduces the number of cases with probable AMI to 990, corresponding to a sampling fraction in our study of 45%, which we contend constitutes a representative sample.

Second, unlike some investigators,21 we excluded patients transferred in from other hospitals, because performance within a single tertiary hospital was being assessed, not the performance of a regional health care system in which timely interhospital transfer of patients requiring tertiary care is a quality indicator. Moreover, initial management of transferred patients at the referring hospital, which may increase mortality risk, is not within the control of the study hospital.

Third, while some of the deaths we labelled as being misclassified may in fact have constituted cases of AMI, we argue that diagnostically uncertain cases lacking confirmatory autopsy or investigation results should be excluded from analysis.

Fourth, the mortality risk of patients presenting with out-of-hospital cardiac arrest or dying shortly after presentation may have been underestimated, despite assignment of the highest possible mortality risk in both models.

Fifth, we suspect the association of increased death risk with non-cardiologist care reflects selection bias, as virtually all patients presenting with AMI to this hospital are referred in the first instance to cardiologists, who may decline care on the basis of age and comorbidity burden.

Finally, while goodness of fit for both risk-prediction models was low for the total patient sample, this significantly improved after patient deselection (Hosmer–Lemeshow test P > 0.10 for both), with no decrease in discrimination (C = 0.92 and C = 0.85).

VLAD analyses have the ability to detect runs of favourable or adverse outcomes and provide timely, regular updates as “real-time” reporting. Such analyses are feasible on a large scale and at low cost when they use readily available administrative data routinely collected from all hospitals.22 However, the accuracy of such data has been questioned because of variations in the results of same-condition mortality analyses that use different administrative datasets23 and between those using administrative versus clinical data.24,25

In response, Queensland Health has adopted a multitiered flagging mechanism which begins with clinicians first examining the validity of the administrative data and risk-adjustment models used for specific indicators.7 The accuracy of risk adjustment of administrative data may also be improved by adding a small number of readily available laboratory results26,27 or clinical variables28 to administrative datasets.

This study suggests that a-priori determination of appropriate patient selection criteria may be an even more important consideration than risk adjustment in preventing VLAD analyses unfairly labelling individual institutions as being poor performers. Coding practices in regard to admission status, diagnosis or complication, and verifiable cause of death need to be reviewed and standardised with input from clinicians. Audit-based coding error rates of 17% for principal discharge diagnoses and missing-diagnosis rates of 27% across Queensland hospitals (KPMG Consulting. Casemix coding audit and process review, 2002) need to be corrected if VLAD analyses are to adequately account for differences in risk between hospitals29 and be acceptable to clinicians.30 Mandated use of clinician-verified primary data contained within regularly audited structured discharge summaries may be one possible solution.

The early experience of Queensland Health with VLADs has shown that many flags, indicating that review is required, relate to data coding problems and inadequate risk adjustment; correction of these has resulted in flag cancellation. Data collected between 1 July 2004 and 30 September 2007 at the study hospital have not given rise to any further flags (Kirstine Sketcher-Baker, Senior Analyst, Quality Management Statistical Unit, personal communication, 2007), which may reflect improved coding.

Finally, VLADs and related tools do not, in themselves, provide definitive proof of, or explanations for, lower quality care. Their results should not be used in interhospital comparisons for purposes of ranking, but to monitor outcomes within single institutions over time. If excess mortality is found, then in-depth, clinician-led investigations should be initiated to identify and remedy system-of-care problems (including inadequate resourcing) or impaired professional performance.