Risk assessment can help predict future cardiovascular disease (CVD), which should prevent undertreatment of high-risk patients and inappropriate treatment of those at low risk.1 To simplify multivariate risk models, point-scoring systems have been developed that allow several risk factors to be considered simultaneously.2,3 Multidimensional charts, based mostly on Framingham Heart Study data4 and displaying risk by categories of age, sex, smoking status, total serum cholesterol level and systolic blood pressure, have also been produced.5,6 Computer or calculator implementations of the Framingham7 and United Kingdom Prospective Diabetes Study (UKPDS)8 risk equations are available.
The validity of applying risk models developed in one population to another is debatable.9 As well as genetic and environmental variation, there may be differences in variables such as definition and ascertainment of outcomes, observation period, and selection of risk variables. Models derived from general population data that include presence/absence of diabetes but not prognostically important variables, such as glycated haemoglobin level (HbA1c) and diabetes duration, do not adequately discriminate the spectrum of CVD risk in diabetes.10 To date, the UKPDS risk engine, developed from a randomised intervention trial, is the only peer-reviewed and readily accessible model for use in diabetes.8
Because of the need to determine whether available risk equations can be appropriately applied to Australian patients with diabetes, we assessed the performance of the Framingham and UKPDS risk equations in predicting the risk of a first CVD event over 5 years in a community-based cohort of patients with type 2 diabetes who were initially CVD-free.
We analysed data from the Fremantle Diabetes Study (FDS), a longitudinal observational study of a representative sample of patients from a population of 120 097 in Fremantle, Western Australia.11 The FDS was approved by the Human Rights Committee, Fremantle Hospital, and all participants gave informed consent. A description of recruitment, sample characteristics and identified but non-recruited patients has been published elsewhere.11 The FDS recruited 1294 patients with type 2 diabetes between 1993 and 1996, 815 (63%) of whom were CVD-free. Of these, 791 (97%) were diagnosed at ≥ 20 years of age and had complete risk factor and outcome data for the UKPDS coronary heart disease (CHD) and stroke equations. Similarly, 697 (86%) were aged 30–74 years at baseline and had complete data for the Framingham CHD equation. Most (99.7%) of the latter participants were also eligible for UKPDS assessment.
At recruitment, demographic and clinical information was documented, a standard clinical examination carried out, and biochemical tests performed on fasting blood and first morning urine samples. FDS data were linked with those from the Western Australian hospital morbidity database and mortality register12 from January 1993 to the end of June 2006.
Prior CVD was defined as hospitalisation with or self-reported myocardial infarction (MI), angina, coronary artery bypass grafting, coronary angioplasty, stroke or transient ischaemic attack, and/or definite MI on a Minnesota-coded electrocardiogram (ECG).13
Peripheral arterial disease was defined as an ankle : brachial index < 0.90 (either side) or non-traumatic lower extremity amputation.13
CVD during follow-up was defined as hospitalisation for or with MI, angina pectoris or stroke; death from MI, cardiac or cerebrovascular causes, or sudden death; definite MI on a Minnesota-coded ECG; or uptake of anti-anginal medication.
For the UKPDS risk engine analyses, outcomes were classified as fatal MI/sudden death (fatal CHD); hospitalisation for/with or death from MI/sudden death (all CHD);14 fatal stroke; and hospitalisation for/with or death from stroke (all stroke).15
For Framingham risk calculation, “total” CHD was taken as all (including silent) MI, cardiac/sudden death or angina pectoris.2 Causes of death were reviewed independently by two blinded FDS investigators and classified under the UKPDS system.13 As the Framingham risk calculator defines 10-year CHD risk,2 the 5-year risk was estimated as 1 − √ (1–10-year CHD risk).
The computer package SPSS for Windows, version 14.0 (SPSS Inc, Chicago, Ill, USA) was used. Model performance was gauged from:
Ability to identify those at high risk (discrimination). This was assessed from the area under the receiver operating characteristic curve (AUC; c statistic), with discrimination considered perfect if AUC = 1, good if AUC > 0.8, moderate if 0.6–0.8, poor if < 0.6, and no better than chance if AUC = 0.5;16
Ability to quantify risk (calibration or goodness-of-fit — how close the predicted probabilities are to the observed outcome). This was assessed by the Hosmer–Lemeshow Ĉ-test, which divides the cohort into deciles of predicted risk and compares these with actual outcomes.16 A P < 0.05 indicates a significant discrepancy between observed and predicted numbers. Because Framingham risk is based on categorisation of factors, we sorted cases by risk and then divided them into deciles; and
Accuracy of the models (how well the model predicts the likelihood of an outcome in an individual patient). This was assessed by the Brier score (mean squared error, range 0–1, with greater accuracy at lower values). The positive predictive value, negative predictive value, sensitivity and specificity of the models were determined for a 5-year CVD risk of 10%.
At FDS entry, the 791 CVD-free people with type 2 diabetes included in the assessment using the UKPDS risk equations had a mean age of 62.2 years (95% CI, 37.9–82.3 years), 46.4% were male, and median diabetes duration was 3.0 years (interquartile range [IQR], 0.8–7.0 years). The 697 participants eligible for assessment using the Framingham CHD equation had a mean age of 59.9 years (95% CI, 37.8–74.0), 46.6% were male, and median diabetes duration was 3.0 years (IQR, 0.7–6.0 years). Additional baseline characteristics are summarised in Box 1.
During 3731 patient-years of follow-up (patient mean, 4.7 [95% CI, 1.3–5.0] years) until first MI, death or 5 years after study entry, 38 participants (4.8%) from the UKPDS-assessable cohort experienced at least one MI (11 first MIs were fatal). Similarly, during 3774 patient-years of follow-up (patient mean, 4.8 [95% CI, 1.7–5.0] years) until first stroke, death or 5 years after study entry, 23 (2.9%) experienced at least one stroke (13 first strokes were fatal). For the Framingham-assessable cohort, 130 patients (18.7%) had at least one CHD event during 3111 patient-years of follow-up (patient mean, 4.5 [95% CI, 0.9–5.0] years) until first CHD event, death or 5 years from baseline.
Box 2 summarises the performance of the Framingham and UKPDS models in predicting 5-year risk in initially CVD-free FDS patients with type 2 diabetes. The UKPDS equations for all CHD, fatal CHD, and all stroke overestimated the number of events by 6.5, 2.8 and 1.8 times, respectively, while the risk equation for fatal stroke underestimated the number of events by 38%. The UKPDS CHD risk equations showed modest discrimination (AUC = 0.68 for both) and poor calibration (Ĉ-test, P < 0.001), while the stroke risk equations showed good discrimination (AUC ≥ 0.86) and calibration (Ĉ-test, P > 0.05). The Framingham equation predicted 28% fewer CHD events than occurred (93 v 130) and had poor discrimination (AUC = 0.59) and calibration (Ĉ-test, P < 0.001).
Accuracy was best for the UKPDS fatal stroke risk equation and worst for the Framingham equation. The 10% 5-year CVD-risk threshold for the UKPDS and Framingham equations had sensitivities of 56.5% (UKPDS all stroke) to 78.9% (UKPDS all CHD), specificities of 36.5% (Framingham CHD) to 99.5% (UKPDS fatal stroke), and positive predictive values of 2.8% (UKPDS fatal CHD) to 22.1% (Framingham CHD).
This study provides the first validation of the UKPDS and Framingham risk equations in an Australian, community-based cohort of people with type 2 diabetes who were initially CVD-free. The Framingham CHD equation proved poor at identifying individuals with high CHD risk and quantifying the risk. The UKPDS CHD risk equations were only marginally better as identifiers of high-risk patients but were also poor at risk quantifications. Although the UKPDS stroke risk equations performed satisfactorily, our overall conclusion is that both the Framingham and UKPDS equations have limited value in guiding vascular risk factor management in Australian patients with type 2 diabetes.
The relatively poor performance of the two risk calculators may be related to differences between the patient cohorts providing the source data and the FDS sample. These include epidemiological setting, age of eligible participants, other demographic features such as race or ethnicity, presence of comorbidities, and temporal changes in management.
The FDS cohort was from an urban Australian setting, and no patients were excluded.11 The Framingham study was conducted from 1948 onwards in a semi-urban community in the United States, comprising predominantly white “middle-class” people.2 Baseline measurements for the risk-equation cohort were made between 1968 and 1975 in those aged 30–74 years who were cancer- and CVD-free, with follow-up over 12 years.4,25 At baseline, US life expectancies at age 65 years were 13 years for white men and 17 years for white women.17 In contrast, the FDS began in 1993, when 65-year-old Australian men and women had life expectancies of 16 and 20 years, respectively.18 The longer life expectancy in Australia reflects lower CVD risk, which is likely to apply to people with diabetes. However, more events were observed in the FDS cohort than predicted by the Framingham equation, confirming that the effect of diabetes on CVD risk is not well represented by the equation and that, by implication, the two populations were significantly different at the time of data acquisition.
The UKPDS risk equations were derived from a randomised trial of 5102 patients aged 25–65 years newly diagnosed with type 2 diabetes.14,15 In addition to macrovascular complications, exclusion criteria comprised ketonuria, nephropathy, severe retinopathy, malignant hypertension, uncorrected endocrinopathy, or severe concurrent illness.19 The UKPDS sample was recruited between 1977 and 1991 and followed up until 1997. When the UKPDS began, life expectancies of white men and women at age 65 years in the UK were the same as those in the US when the Framingham study commenced (13 and 17 years, respectively).20 Overall, the UKPDS equations overestimated CVD risk in the FDS cohort, suggesting that, despite the range of exclusion criteria, the UKPDS patients were generally high-risk.
An explanation for the discrepancy between actual events in the FDS cohort and those predicted using the UKPDS engine is the uptake of cardiovascular therapies. Of the CVD-free FDS patients at study entry, few (7%) were taking lipid-lowering medications, < 50% were taking antihypertensive medication, and 8% took aspirin regularly. Reflecting the subsequent publication of trials confirming the benefits of intensive non-glycaemic vascular risk factor management in type 2 diabetes,19,21,22 these proportions increased to 41%, 54% and 28%, respectively, after 5 years. Although therapeutic intensification was more likely in people who experienced an event during follow-up compared with those who did not (eg, Framingham CHD endpoint: aspirin, 40.3% v 24.8%; antihypertensive medication, 69.4% v 50.0%; lipid-lowering medication, 54.2% v 38.9%; P ≤ 0.02), it was also seen in the latter group. The UKPDS and Framingham studies were conducted at a time when fewer people took such medications. For example, < 2% of UKPDS patients took lipid-lowering therapy at any stage.23 Additionally, only 15% of FDS patients were current smokers at baseline, compared with 30% in the UKPDS and 39% in the Framingham studies.
The UKPDS sample comprised predominantly white people of European background (82.7%), but also Afro–Caribbean and Asian–Indian people.23 Consideration of southern European and Indigenous ethnicity is more appropriate in Australia. Indeed, the UKPDS equations significantly underestimated CHD and stroke risk for Indigenous Australians compared with Anglo-Celts (UK or Ireland ethnic background) (data not shown). In contrast, the Framingham equation correctly estimated CHD risk for southern Europeans, but underestimated it for Anglo-Celts (data not shown).
Unlike the Framingham equation, which utilises diabetes as a binary risk variable but has no other diabetes-specific factors, the UKPDS risk engine includes age at diagnosis, diabetes duration and HbA1c.8 CVD risk in the FDS cohort might be better characterised with a different set of diabetes-specific and other variables, including those that relate to microangiopathy, such as albuminuria. A formal analysis of this question was beyond the scope of our study. The Framingham equation defines “total” CHD (combined macrovascular endpoints),2 while the UKPDS engine defines CHD more narrowly.14 This is underscored by the 695 common FDS participants, in whom 130 Framingham versus 29 UKPDS CHD events occurred over 5 years. The Framingham calculator is limited to age 30–74 years, while the UKPDS risk engine allows non-validated extrapolation beyond the 25–65-year inclusion criterion. Of the 791 FDS participants eligible for the UKPDS risk assessment, 298 (38%) were outside this age range, and their predicted risk should be interpreted cautiously.
Given the urban location and demographic profile of the FDS sample,11 it is likely that its outcome data can be extrapolated to patient groups in other major Australian centres. A further strength of the study is that there is a low rate of migration out of WA.24 Ongoing linkage with the WA hospital morbidity and mortality register means that “hard” events (hospitalisation and death) have excellent ascertainment. To the end of June 2007, only two eligible participants in this study had not been linked, and both are known to continue to reside in the study area. A limiting factor for ascertainment of “soft” outcomes (eg, silent MI and anti-anginal medication uptake) was that only 45% of the UKPDS-assessable and 49% of the Framingham-assessable FDS cohorts returned for annual reviews at 5 years and later. Self-reported MI, stroke and angina were not included in the definition of outcome. This may partly explain the inconsistency between the UKPDS predicted risk of all stroke and the actual number, as strokes that did not lead to hospitalisation or death were not counted. For the UKPDS, the definition of stroke was more inclusive (neurological deficit lasting ≥ 1 month). Conversely, the higher number of fatal strokes observed than predicted may reflect differences in health care between the UK and Australia, or differences in death coding.
The UKPDS risk equations are limited to an age at diagnosis of diabetes ≥ 20 years which, with the inclusion age range, has implications for assessing the increasing number of young patients with type 2 diabetes. Both sets of equations can be used only in a primary prevention setting. Of the FDS type 2 diabetes baseline cohort, 479 patients (37%) already had CVD. These patients are at high risk of further events, but there is no diabetes-specific risk equation available to quantify that risk. Although guidelines commonly recommend that all such patients be treated intensively, risk quantification may still be useful in patient education and in ensuring compliance with appropriate lifestyle changes and pharmacotherapy.
The UKPDS and original Framingham25 10-year CHD risk equations were previously compared in a primary-care cohort of newly diagnosed, CVD-free patients aged 30–74 years with type 2 diabetes from Poole in south-western England.16 Both equations showed only modest discrimination and poor calibration. Although the Framingham equation underestimated CHD risk (in agreement with our finding), the Poole study suggested that the UKPDS also underestimated CHD risk,16 perhaps because of the addition of angina to the UKPDS CHD endpoint. In a study from the UKPDS group, 5- and 10-year fatal CVD risk estimates from the Framingham equation25 were compared with actual events observed in UKPDS patients.10 The Framingham equation underestimated both 5- and 10-year fatal CVD risk by 56% and 32%, respectively. By comparison, the observed 5-year fatal CVD rate in the FDS UKPDS cohort was 69% lower than in the actual UKPDS cohort, underlining the lower risk of FDS patients.
In conclusion, the Framingham CHD risk equation is poor at identifying FDS patients at high risk (discrimination) and quantifying that risk (calibration). The UKPDS CHD risk equations showed modest discrimination but poor calibration. The UKPDS stroke risk equations performed relatively well. However, taken together, our analyses suggest that these two widely used CHD risk equations should not be used to predict risk in Australians with type 2 diabetes.
1 Baseline characteristics of the cohort with type 2 diabetes and complete risk-factor data, by risk equation*
UKPDS8 |
|||||||||||||||
UKPDS = United Kingdom Prospective Diabetes Study. HbA1c = glycated haemoglobin level. HDL = high-density lipoprotein. GFR = glomerular filtration rate. |
|||||||||||||||
2 Performance of the UKPDS and Framingham equations in predicting 5-year cardiovascular risk in Fremantle Diabetes Study participants with type 2 diabetes who were initially free of cardiovascular disease
Framingham2 |
UKPDS8 |
UKPDS8 |
UKPDS8 |
UKPDS8 |
|||||||||||
UKPDS = United Kingdom Prospective Diabetes Study. CHD = coronary heart disease. AUC = area under the receiver operating characteristic curve.
PPV = positive predictive value. NPV = negative predictive value. |
|||||||||||||||
Received 29 December 2007, accepted 31 August 2008
- Wendy A Davis1
- Stephen Colagiuri2
- Timothy M E Davis1
- 1 School of Medicine and Pharmacology, Fremantle Hospital, Fremantle, WA.
- 2 Institute of Obesity, Nutrition and Exercise, University of Sydney, Sydney, NSW.
We thank the FDS patients for their participation, FDS staff for valuable support, the Fremantle Hospital and Health Service Biochemistry Department for laboratory tests, and the Diabetic Education, Podiatry and Dietetic departments for help with patient recruitment.
None identified.
- 1. EUROASPIRE I and II Group. European Action on Secondary Prevention by Intervention to Reduce Events. Clinical reality of coronary prevention guidelines: a comparison of EUROASPIRE I and II in nine countries. Lancet 2001; 357: 995-1001.
- 2. Wilson PWF, D’Agostino RB, Levy D, et al. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97: 1837-1847.
- 3. Assmann G, Cullen P, Schulte H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the prospective cardiovascular Munster (PROCAM) study. Circulation 2002; 105: 310-315.
- 4. Anderson KM, Wilson PW, Odell PM, Kannel WB. An updated coronary risk profile. A statement for health professionals. Circulation 1991; 83: 356-362.
- 5. Second Joint Task Force of European and other Societies. Prevention of coronary heart disease in clinical practice: recommendations of the Second Joint Task Force of European and other Societies on Coronary Prevention. Eur Heart J 1998; 19: 1434-1503.
- 6. Jackson R, Barham P, Bills J, et al. Management of raised blood pressure in New Zealand: a discussion document. BMJ 1993; 307: 107-110.
- 7. British Hypertension Society. Cardiovascular risk charts and calculators [online]. http://www.bhsoc.org/Cardiovascular_Risk_Charts_and_Calculators.stm (accessed Aug 2008).
- 8. Diabetes Trials Unit, The Oxford Centre for Diabetes, Endocrinology and Metabolism. UKPDS risk engine [online]. http://www.dtu.ox.ac.uk/index.php?maindoc=/riskengine/ (accessed Aug 2007, link updated February 2009).
- 9. Chambless LE, Dobson AJ, Patterson CC, Raines B. On the use of a logistic risk score in predicting risk of coronary heart disease. Stat Med 1990; 9: 385-396.
- 10. Coleman RL, Stevens RJ, Retnakaren R, Holman RR. Framingham, SCORE, and DECODE risk equations do not provide reliable cardiovascular risk estimates in type 2 diabetes. Diabetes Care 2007; 30: 1292-1294.
- 11. Davis TM, Zimmet P, Davis WA, et al. Autoantibodies to glutamic acid decarboxylase in diabetic patients from a multi-ethnic Australian community: the Fremantle Diabetes Study. Diabet Med 2000; 17: 667-674.
- 12. Holman CD, Bass AJ, Rouse IL, Hobbs MS. Population-based linkage of health records in Western Australia: development of a health services research linked database. Aust N Z J Public Health 1999; 23: 453-459.
- 13. Davis WA, Norman PE, Bruce DG, Davis TM. Predictors, consequences and costs of diabetes-related lower extremity amputation complicating type 2 diabetes: the Fremantle Diabetes Study. Diabetologia 2006; 49: 2634-2641.
- 14. Stevens RJ, Kothari V, Adler AI, Stratton IM; United Kingdom Prospective Diabetes Study (UKPDS) Group. The UKPDS risk engine. A model for the risk of coronary heart disease in type II diabetes (UKPDS 56). Clin Sci 2001; 101: 671-679.
- 15. Kothari V, Stevens RJ, Adler AI, et al. UKPDS 60: risk of stroke in type 2 diabetes estimated by the UK Prospective Diabetes Study Risk Engine. Stroke 2002; 33: 1776-1781.
- 16. Guzder RN, Gatling W, Mullee MA, et al. Prognostic value of the Framingham cardiovascular risk equation and the UKPDS risk engine in patients with newly diagnosed type 2 diabetes. Diabet Med 2005; 22: 554-562.
- 17. Centers for Disease Control and Prevention. Table 27. Life expectancy at birth, at 65 years of age, and at 75 years of age, according to race and sex: United States, selected years 1900-2001 [online]. http://www.cdc.gov/nchs/data/hus/hus07.pdf (accessed Aug 2007, link updated February 2009).
- 18. Australian Institute of Health and Welfare. Life expectancy [online]. http://www.aihw.gov.au/mortality/life_expectancy/index.cfm (accessed Jul 2007, link updated February 2009).
- 19. UKPDS Study Group. Tight blood pressure control and risk of macrovascular and microvascular complications in type 2 diabetes: UKPDS 38. BMJ 1998; 317: 703-713.
- 20. Office of National Statistics and Government Actuary’s Department. Life expectancy [online]. http://www.statistics.gov.uk/cci/nugget.asp?id=168 (accessed Aug 2007).
- 21. Collins R, Armitage J, Parish S, et al; Heart Protection Study Collaborative Group. MRC/BHF Heart Protection Study of cholesterol-lowering with simvastatin in 5963 people with diabetes: a randomised placebo-controlled trial. Lancet 2003; 361: 2005-2016.
- 22. Heart Outcomes Prevention Evaluation Study Investigators. Effects of ramipril on cardiovascular and microvascular outcomes in people with diabetes mellitus: results of the HOPE study and MICRO-HOPE substudy. Lancet 2000; 355: 253-259.
- 23. UK Prospective Diabetes Study Group. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet 1998; 352: 837-853.
- 24. Bradshaw PJ, Jamrozik K, Jelfs P, Le M. Mobile Australians: a moving target for epidemiologists [letter]. Med J Aust 2000; 172: 566.
- 25. Anderson KM, Odell PM, Wilson PW, Kannel WB. Cardiovascular disease risk profiles. Am Heart J 1991; 121: 293-298.
Abstract
Objective: To assess the performance of the Framingham and United Kingdom Prospective Diabetes Study (UKPDS) cardiovascular risk equations in Australian patients with type 2 diabetes who were initially free of cardiovascular disease (CVD).
Design and setting: The Fremantle Diabetes Study (FDS), a community-based longitudinal observational study; data for the period 1993–2006 were used.
Patients: Of the 815 FDS participants with type 2 diabetes who were initially CVD-free, 791 (97%) were eligible for assessment using the UKPDS equations, and 697 (86%) using the Framingham equation.
Main outcome measures: CVD endpoints during 5 years of follow-up. For the UKPDS equations, these were fatal myocardial infarction (MI) or sudden death (fatal coronary heart disease [CHD]); hospitalisation for/with or death from MI or sudden death (all CHD); fatal stroke; and all stroke. For the Framingham equation, they were all MI, sudden death or angina pectoris (CHD).
Results: During follow-up to first CVD event, death or 5 years, there were 38 MIs (11 fatal) and 23 strokes (13 fatal) in the UKPDS-assessable cohort of FDS participants. The UKPDS risk equations for all CHD, fatal CHD, and all stroke overestimated the number of events by 6.5, 2.8 and 1.8 times, respectively. The risk equation for fatal stroke underestimated the number of events by 38%. The UKPDS CHD risk equations showed modest discrimination and poor calibration, while the stroke risk equations showed good discrimination and calibration. The Framingham equation predicted 28% fewer CHD events than occurred (93 v 130), and discrimination and calibration were poor.
Conclusions: While the UKPDS stroke risk equations performed relatively well, the UKPDS and Framingham CHD risk equations are not suitable for predicting risk in Australians with type 2 diabetes.