Sepsis incidence and mortality are underestimated in Australian intensive care unit administrative data

The World Health Organization has highlighted sepsis as a major threat to patient safety and a global health priority.1 At its 70th World Health Assembly (May 2017), it passed a resolution, co-sponsored by Australia, that recommended measures for “improving the prevention, diagnosis and clinical management of sepsis”.2 The resolution was prompted by the conservative estimate that 30 million cases of sepsis each year cause 6 million deaths worldwide, many of them preventable.2 The WHO noted that data on the epidemiology of sepsis are incomplete for many countries, and almost totally lacking for low and middle income countries.3,4 The resolution therefore specifically urged WHO member states to investigate the national prevalence and profile of sepsis, and to develop and foster epidemiologic surveillance systems.1

Accurately quantifying the incidence and disease burden of sepsis is difficult.5 As there are no definitive tissue or serological tests for sepsis, the gold standard for diagnosis is clinical identification of organ dysfunction caused by infection.6 At the population level, this would require either prospective cohort studies or retrospective medical record reviews on a scale impractical for routine disease surveillance. Consequently, routinely collected data are analysed to estimate the incidence and burden of sepsis, generally based on International Classification of Disease (ICD) coding of cases.7,8 A major United States study9 recently found that estimates of the incidence of sepsis based on ICD coding ranged between half and twice the actual clinical rate, depending on the methods applied. The lower estimate included only explicit diagnoses of sepsis; that is, with an ICD-9, clinical modification (ICD-9-CM) code for severe sepsis (995.92) or septic shock (785.52). The higher estimate also included “implicit” diagnoses of sepsis, in which patients had codes for both an infection and an organ dysfunction.9

Most estimates of the incidence and mortality rates of sepsis in Australia have been based on studies of adult patients in intensive care units (ICUs). A prospective cohort study estimated that the annual incidence of sepsis among adult patients in Australian and New Zealand ICUs was 77 cases per 100 000 population, with in-hospital mortality of 37.5%; that is, 17 000 cases and more than 6000 deaths.10

More recently, a retrospective analysis of the Australian and New Zealand Intensive Care Society Centre for Outcome and Resource Evaluation (ANZICS CORE) adult ICU patient database applied methodology similar to that used to generate implicit diagnoses from ICD coding. The study found that the proportion of ICU patients with sepsis had increased from 7.2% in 2000 to 11.1% in 2012, with in-hospital mortality of 18.4% for patients with sepsis and 22.0% for those with septic shock.11

As the outcomes of applying the ANZICS CORE methodology had not been compared with the reference standard, prospective clinical diagnosis, we conducted an inception cohort study to do so.

Methods

Our study was conducted in the 60-bed multidisciplinary ICU at the Royal North Shore Hospital in Sydney. All patients 18 years old or more admitted to the ICU during 1 October – 31 December 2016 were screened daily by two members of the research team (MH, MS) for clinical diagnoses of sepsis or septic shock. They were also assessed with the ANZICS CORE criteria for sepsis and septic shock.

Clinical definitions of sepsis and septic shock

The criteria for a clinical diagnosis of sepsis were based on the 1991 consensus criteria for severe sepsis, current in 2016; specifically, organ dysfunction attributable to an infection (Box 1).12 The criteria for clinical organ dysfunction were derived from those used in the Recombinant Human Activated Protein C Worldwide Evaluation in Severe Sepsis (PROWESS) study.13 To confirm that treating clinicians regarded a patient as having a clinically significant infection, we further required that the patient was being treated with intravenous antibiotics. When it was not clear from the medical record whether organ dysfunction was related to the infection, the treating intensive care specialist was asked to make the judgement. For a diagnosis of septic shock, cardiovascular organ failure after completion of fluid resuscitation that lasted more than one hour was required.

Database (ANZICS CORE) definitions of sepsis and septic shock

We applied the methodology of Kaukonen and colleagues,11 also based on the 1991 consensus criteria,12 to assess data for the first 24 hours of the ICU admission. Diagnosis of sepsis required meeting two or more criteria for systemic inflammatory response syndrome (SIRS),12 an Acute Physiology and Chronic Health Evaluation (APACHE) III14 admission diagnosis consistent with sepsis/septic shock or an APACHE III admission diagnosis consistent with infection, and organ failure (defined as a Sequential Organ Failure Assessment [SOFA] score15 of 3 or more in at least one domain). For a diagnosis of septic shock, lowest mean arterial blood pressure under 65 mmHg or lowest mean systolic blood pressure under 90 mmHg was required. Evidence of fluid resuscitation was not required for a database diagnosis of septic shock (Box 1).

Data collection

All patients were screened daily for the presence of documented or suspected infection; when the presence of infection was unclear, the treating intensive care specialist was asked to make a judgement. APACHE III diagnostic codes were separately assigned to each patient by a member of the research team and by an independent, trained data collector. Discrepancies were arbitrated by a member of the research team not involved in screening for clinical sepsis. For each admission with a documented or strong suspicion of infection or an APACHE III code consistent with sepsis/septic shock or infection, additional data were collected: date and time of sepsis episode, site of infection, causative organism, SIRS criteria, presence and type of organ dysfunction by clinical definition, and SOFA score. Missing values were assumed to be normal. Patients with clinical sepsis were screened daily for septic shock. Daily screening was discontinued once a patient met the criteria for septic shock, as no further hierarchical diagnosis was possible.

If a patient had several ICU admissions during the study period, they were analysed as separate admissions. For patients without sepsis, the baseline characteristics were drawn from data for the first admission; for patients with sepsis, they were drawn from data for the first admission with sepsis. For estimating the mortality rate, patients were followed up to hospital discharge or for 60 days, whichever period was shorter.

Data management and statistical analysis

Continuous, normally distributed data are presented as means with standard deviations (SDs), non-normally distributed data as medians with interquartile ranges (IQRs). Categorical data are presented as numbers and proportions. Estimates of incidence are presented with 95% confidence intervals (CIs). The significance of differences between clinically and database-derived estimates was assessed in Fisher exact tests, Mann–Whitney tests, or McNemar tests as appropriate; P < 0.05 was deemed statistically significant. Data were analysed in Stata 14.2 (StataCorp).

Ethics approval

The Northern Sydney Local Health District Human Research Ethics Committee provided ethics and scientific approval for the study and waived the requirement for individual patient consent (reference, RESP/16/239).

Results

During the study period, 909 ICU admissions were recorded for 864 patients; their mean age was 62.1 years (SD, 17.6 years), 484 (56.0%) were men, and 380 admissions (44.0%) followed surgery. Overall, 64 patients (7.4%) died in the ICU, and a total of 83 (9.6%) died in hospital (Box 2).

Incidence of sepsis and septic shock

A total of 146 patients (16.9%; 95% CI, 14.5–19.6%) received clinical diagnoses and 98 (11%; 95% CI, 9.3–14%) CORE-based diagnoses of sepsis (P < 0.001). Of the 146 patients with clinical sepsis, 76 (52%) satisfied the CORE criteria; 22 of 96 patients (23%) who satisfied the CORE criteria did not receive a clinical diagnosis of sepsis (Box 2, Box 3). For the CORE criteria, sensitivity was 52%, specificity 97%, the positive predictive value 78%, and the negative predictive value 91% (Box 4, online Appendix, table 1).

A total of 49 patients (5.7%; 95% CI, 4.1–7.2%) received clinical diagnoses and 83 (9.6%; 95% CI, 7.6–12%) CORE-based diagnoses of septic shock (P < 0.001); of the 83 patients with a CORE-based diagnosis, 32 (39%) also received a clinical diagnosis of septic shock (Box 2, Box 3). For the CORE criteria, sensitivity was 65%, specificity 94%, the positive predictive value 39%, and the negative predictive value 98% (Box 4, online Appendix, table 1).

In a post hoc analysis we excluded patients diagnosed with clinical sepsis more than 24 hours after ICU admission (25 of 146 [17%] patients with diagnoses of clinical sepsis), as data for the CORE database are collected only during the first 24 hours. In this analysis, the sensitivity of the database criteria was 63%, the specificity 97%, the positive predictive value 78%, and the negative predictive value 94%. In a post hoc analysis excluding patients diagnosed with clinical septic shock more than 24 hours after ICU admission (16 of 49 patients, 32%), the sensitivity of the database criteria was 85%, specificity 93%, the positive predictive value 34%, and the negative predictive value 99% (online Appendix, table 1).

Reasons for false positive and false negative diagnoses of sepsis using database criteria

Seventy patients had a false negative diagnosis of sepsis with the database method; the most frequent reasons were absence of an APACHE III diagnostic code for sepsis or infection (57 of 70 patients, 81%) and sepsis diagnosed more than 24 hours after admission (25 of 70 patients, 36%) (Box 5).

Twenty-two patients had a false positive diagnosis of sepsis with the database method; the most frequent reasons were organ dysfunction not being related to infection (11 of 22 patients, 50%) and patients not meeting the clinical criteria for organ dysfunction (8 of 22 patients, 36%) (Box 5).

Source of infection and causative organisms

The sources of infection and the organisms identified in cases of clinical and CORE database sepsis are summarised in the online Appendix, tables 2 and 3. The infection was confirmed by positive cultures in 107 of 156 episodes (68.6%; 95% CI, 60.6–75.7%) of clinical sepsis and for 73 of 105 episodes (70%; 95% CI, 60–78%) of sepsis diagnosed by CORE criteria (P = 0.88).

Type of organ dysfunction

Type of organ system dysfunction for the clinical and the CORE database sepsis groups are listed in Box 6. Cardiovascular organ failure was more frequent in patients diagnosed with sepsis by database criteria (85 of 105, 81%; 95% CI, 72–88%) than in patients with a clinical diagnosis of sepsis (54 of 156, 35%; 95% CI, 27–43%; P < 0.001).

Patient outcomes

The median length of ICU stay for the 864 patients was 2 days (IQR, 1–4 days); in-hospital mortality, censored at 60 days, was 9.6% (83 of 864 patients; 95% CI, 7.8–12%). Length of stay was greater for patients with a clinical diagnosis of sepsis (median, 4.5 days; IQR, 2–8 days) than for patients with a database diagnosis (median, 3 days; IQR, 2–7 days; P = 0.020). Of 146 patients diagnosed with sepsis by clinical criteria, 39 (27%; 95% CI, 20–35%) died in hospital within 60 days of admission, as did 17 of 98 patients (17.3%; 95% CI, 11–27%) diagnosed by CORE criteria (P = 0.12). Eighteen of 49 patients (37%; 95% CI, 24–52%) diagnosed with septic shock by clinical criteria died in hospital, and 13 of 83 patients (16%; 95% CI, 8.9–26%) diagnosed by database criteria (P = 0.006) (Box 2). During the first 24 hours after ICU admission, 32 of 121 patients (26%; 95% CI, 19–35%) with a clinical diagnosis of sepsis and 17 of 98 patients (17%; 95% CI, 11–27%) diagnosed by database criteria died (P = 0.14), as did 8 of 33 patients (24%; 95% CI, 12–43%) with a clinical diagnosis of septic shock and 13 of 83 patients (16%; 95% CI, 8.9–26%) with a database diagnosis of septic shock (P = 0.29) (online Appendix, table 4).

Discussion

Key findings

In this prospective study, we found that the estimated incidence of sepsis among adult ICU patients was significantly higher when based upon the reference standard — prospective clinical diagnosis — than when applying the ANZICS CORE database criteria, as was the proportion of patients with sepsis who died in hospital. The estimated incidence of septic shock was higher when using the ANZICS CORE methodology.

Comparison with earlier reports

Few studies have compared estimates of the incidence and mortality of sepsis based on clinical assessment with estimates based on ICD coding or the analysis of administrative data. A whole of hospital study (nearly 1000 patients) in the United States indicated that the sensitivity of a surveillance system for septic shock based on administrative data was 74.8% with respect to retrospective chart review and that of ICD-9 coding 48.3%,16 suggesting that both methods underestimated the incidence of sepsis and septic shock.

In a recently published US study, the incidence and outcomes of sepsis during 2009–2014 were estimated according to an Electronic Health Record (EHR)-based surveillance definition, and also according to explicit or combined explicit and implicit ICD-9-CM diagnostic coding.9 None of the three methods identified all cases detected by retrospective clinician chart review: EHR surveillance and combined explicit and implicit diagnosis coding missed about 30% of sepsis cases, while including only cases with explicit ICD-9-CM coding missed 68%. False discovery rates were 20% for the EHR method and 70% for combined explicit and implicit diagnosis coding.9

The differences we found between estimates for sepsis and septic shock based on clinical and database criteria-based diagnoses may have several explanations. First, the ANZICS CORE includes data only for the first 24 hours in the ICU, and the estimate therefore applies only to sepsis apparent on admission to the ICU. Second, the APACHE III system allows only one diagnostic code to be allocated, and patients with sepsis may be coded for another reason for ICU admission. Inter-observer variability in the allocation of APACHE III codes is well recognised.17,18 Third, it is not possible to indicate a causal or temporal relationship between organ dysfunction and infection when coding, which could lead to misdiagnosis of sepsis in some patients. Fourth, the CORE database collects physiological data that are used to generate APACHE III scores, but these data are not aligned with clinical criteria for organ dysfunction, leading to inaccuracies in estimates of its incidence. This is particularly evident with regard to septic shock; its incidence was higher using CORE criteria, as they require only mild transient hypotension rather than the recognised clinical criterion of hypotension after restoration of intravascular volume.

Strengths and limitations

The main strength of our study was its prospective nature and the daily screening of all patients in the ICU. In addition, we faithfully and prospectively replicated the ANZICS CORE methodology. Limitations include the fact that it was a single centre study conducted over a short time period, which may limit the generalisability of our findings. The study was conducted during summer, and seasonal differences may have affected our findings, although this would apply to both diagnostic methods.19

Conclusion

Our study indicates that the incidence of sepsis among adult patients in an ICU is significantly underestimated and that of septic shock overestimated by ANZICS CORE criteria. This method also underestimates the proportions of patients with sepsis or septic shock who die in hospital.

Prospective cohort studies are resource-intensive and time-consuming, and are therefore impractical for ongoing disease surveillance. Calibration of methods based on ICD coding and routinely collected data (such as those of ANZICS CORE) in cohort studies similar to ours could increase the accuracy of data on the incidence and burden of sepsis, a major public health challenge and a threat to patient safety.

Box 3 – Relationship between groups of patients satisfying clinical and database criteria for sepsis and septic shock*

---

* The numbers in the overlapping areas indicate the numbers of patients satisfying both sets of criteria for each condition.

Box 4 – Classification of 909 admissions of 864 patients to the intensive care unit at Royal North Shore Hospital, Sydney, 1 October – 31 December 2016

---

APACHE = Acute Physiology and Chronic Health Evaluation; SIRS = systemic inflammatory response syndrome. * APACHE III admission diagnosis consistent with sepsis/septic shock or an APACHE III admission diagnosis consistent with infection.