Pneumonia risk stratification in tropical Australia: does the SMART-COP score apply?

Risk stratification systems are useful for determining hospital discharge policies and antimicrobial therapy in patients with pneumonia, and identifying patients at risk of adverse outcomes. The best known of these, the Pneumonia Severity Index, was primarily designed to assess the suitability of patients for discharge from hospital.1 This has been shown to perform similarly to or better than other scoring systems, such as the CURB-65.2-4

A recently proposed scoring system, SMART-COP, was developed in Australia and validated in a variety of settings, including patients from North America, temperate Australia and Scotland, but few from tropical areas.5,6 In contrast to previously developed tools, SMART-COP was designed to identify patients requiring intensive supportive care, and its use is being considered for inclusion in version 14 of the Australian Therapeutic guidelines: antibiotic (Jenny Johnstone, Editor, Therapeutic Guidelines Limited, personal communication).

The epidemiology of pneumonia is significantly different in tropical northern Australia than in temperate regions — a high proportion of patients are Indigenous and the average age of patients is lower.7 Furthermore, the pathogens implicated in pneumonia are significantly different to those identified in temperate latitudes.8-10

We examined the performance of the SMART-COP scoring system in patients with community-acquired pneumonia presenting to our referral centre in tropical northern Australia.

Royal Darwin Hospital (RDH) is the only tertiary referral hospital for the tropical Northern Territory and serves a population of about 150 000 spread across an area of over 500 000 km2. We conducted a prospective observational study of adult patients with sepsis (infection plus at least two criteria for systemic inflammatory response syndrome [SIRS]11) admitted to RDH between August 2007 and May 2008. The study was approved by the Human Research Ethics Committee of the NT Department of Health and Families and Menzies School of Health Research.

Inclusion criteria were at least two symptoms suggestive of pneumonia (new cough, fever, rigors, chest discomfort, new-onset dyspnoea) and a chest radiograph or computed tomography scan taken within 24 hours of admission demonstrating acute pneumonia. All chest radiographs were reviewed by a radiologist and, where the report was inconclusive or ambiguous, radiographs were also viewed by an infectious diseases physician. Exclusion criteria were: immunosuppression,5 active orders limiting life-sustaining treatment, and direct admission to the intensive care unit (ICU).

We defined intensive respiratory support as the need for non-invasive ventilatory support or intubation and mechanical ventilation. Vasopressor support was the use of dopamine, noradrenaline, adrenaline or vasopressin for treatment of hypotension.

We calculated SMART-COP scores as defined in the Australian Community-Acquired Pneumonia Study (ACAPS) (Box 1).5 We used univariate logistic regression analysis to examine the individual components of the SMART-COP score against the need for intensive respiratory or vasopressor support (IRVS). We then assessed the performance of the SMART-COP score and evaluated several variations of it, primarily on the basis of its negative predictive value (NPV), because of the need to identify patients who did not require IRVS and could thus be safely managed in a general ward. Statistical analysis was performed using Intercooled Stata, version 10 (StataCorp, College Station, Tex, USA). A significance level of 0.05 was used.

During the study period, 246 patients were admitted with sepsis and a clinical diagnosis of pneumonia. Of these, 40 did not have radiological evidence of pneumonia and 22 met exclusion criteria (immunosuppression, 10; active orders limiting life-sustaining treatment, 9; direct admission to the ICU, 3), leaving 184 eligible patients (Box 2).

Of the total group, 111 patients (60%) were in the low-risk SMART-COP group (score ≤ 2), and 11 of these (10%) required IRVS (Box 3, Box 4). As a predictor of need for IRVS, a SMART-COP score of ≥ 3 had a significantly lower sensitivity, NPV and area under the receiver operator characteristic curve (AUROC) in our RDH group than in the ACAPS cohort (P ≤ 0.05, Box 5).

Of the SMART-COP score components, new-onset confusion (odds ratio [OR], 22.0; 95% CI, 2.5–194.4) and serum albumin level < 35 g/L (OR, 6.8; 95% CI, 2.9–15.9) were the strongest predictors of the need for IRVS (Box 6). However, confusion was present in only six of the 184 patients (3%), making it less clinically useful in predicting those at risk of needing IRVS. Indigenous status was also associated with the need for IRVS (OR, 2.3; 95% CI, 1.0–5.5). Contrary to expectations, none of the comorbidities that were assessed were significantly associated with the need for IRVS (Box 6).

Given that the SMART-COP score performed poorly at the low-risk end in the RDH cohort, we designed and tested a modified score, SMARTACOP, which increased the weighting for albumin to 2 points and included Indigenous status as a variable (Box 1). These changes improved the performance of the scoring system, largely due to better discrimination of patients in the low-risk group. Of the 111 patients in this group according to the SMART-COP score, 43 were recategorised as moderate risk, of whom 10 (23%) required IRVS. The specificity of both the SMART-COP and SMARTACOP scores was poor (Box 5), and thus they should not be used for their positive predictive value.

The SMART-COP score also had poor sensitivity for predicting 30-day mortality: six patients (5%) with a low-risk SMART-COP score died (sensitivity, 67%), compared with none with a low-risk SMARTACOP score (sensitivity, 100%). The AUROC for 30-day mortality was 0.74 for SMART-COP and 0.77 for SMARTACOP.

In this study, we have demonstrated that the SMART-COP scoring system underestimates the need for intensive supportive treatment and the risk of death in a tropical Australian population hospitalised for pneumonia.

Our study population differs substantially from those involved in previous studies of pneumonia risk. Compared with patients in the ACAPS cohort used to derive the SMART-COP score, patients in this study were younger and more likely to be Indigenous, drink hazardous amounts of alcohol, smoke, and have chronic liver disease. In addition, the causative organism in those with an identified pathogen was a gram-negative bacillus in more than half the patients in this study, compared with 18% in the ACAPS study. More than 20% of our patients required IRVS, compared with 11% in the ACAPS study. This may account for a proportion of the decrease in NPV for the SMART-COP score in the RDH cohort compared with the ACAPS cohort, but is unlikely to be solely responsible for the observed difference in NPV.

Low serum albumin level appears to be more strongly associated with the requirement for IRVS in this study than in the original derivation study. Albumin decreases in response to acute inflammation13,14 and thus may be a surrogate marker for late presentation, as well as severity of inflammation. Chronic liver disease and poor nutrition associated with hazardous alcohol intake may also be contributors to low serum albumin levels.

We found Indigenous status to be associated with the requirement for IRVS and with mortality. It is likely that Indigenous status is a surrogate marker for poor health and social disadvantage,15 and these factors are likely to contribute more to poor outcomes than any possible genetic susceptibility. Hazardous levels of alcohol use were present in the majority of Indigenous patients in this study; however, this was not a risk factor for IRVS or mortality. For pneumonia generally, it is likely that an interaction between severe infection and decreased physiological reserve due to multiple underlying comorbidities is what puts an individual patient at risk.

An alternative strategy to improve the performance of the SMART-COP scoring system would be to change the cut-off value used to define low risk. However, changing this cut-off score from 3 to 2 is not as effective as modifying the score; the sensitivity for IRVS changes from 71% to 89% with a cut-off of 2 (compared with 97% at a cut-off of 3 for the modified score), and four of 38 patients who required IRVS would be misclassified using this strategy.

There are several limitations to our study. We only considered patients admitted to hospital; however, it is unlikely that patients not admitted would require IRVS. We also included only patients meeting SIRS criteria, and cannot exclude the possibility that patients without SIRS initially may develop worsening pneumonia later. This study was limited to a single centre, albeit one with the only ICU servicing northern Australia between the Kimberley region in Western Australia and the Queensland border. The number of patients in this study meant that we had limited statistical power to make comparisons or perform multiple logistic regression analysis. The high proportions of Indigenous Australians and of patients with hazardous levels of alcohol use mean that the results of this study may not be generalisable to tropical regions in other countries. We did not collect information on antibiotic use and thus could not control for this in our analysis; however, considering our hospital uses established antibiotic protocols with a high level of staff compliance,7 we would not expect this to vary between groups.

We have demonstrated that the current SMART-COP scoring system does not adequately identify patients requiring IRVS in this tropical setting. We propose some minor modifications that improve its performance, particularly its NPV. We intend to validate this modified scoring system prospectively in patients presenting to the emergency department at RDH.

4 Risk of needing IRVS according to SMART-COP score in the RDH cohort compared with (A) ACAPS cohort and (B) SMARTACOP score in the RDH cohort

Vertical dashed line indicates cut-off score for low risk (0–2). IRVS = intensive respiratory or vasopressor support. RDH = Royal Darwin Hospital. ACAPS = Australian Community-Acquired Pneumonia Study.5