Public reporting of individual surgeon performance information: United Kingdom developments and Australian issues

There is increasing evidence that public reporting of comparative performance data improves quality of healthcare.1 However, the most controversial question is whether patients should have access to performance data for individual clinicians. In the United States, outcomes data for individual cardiac surgeons have been publicly available in New York State since 1991, in Pennsylvania since 1992, and in New Jersey since 1994.2-4 In the United Kingdom, performance information for individual cardiac surgeons will be made public this year for the first time: the Society of Cardiothoracic Surgeons will be releasing mortality information for the approximately 180 surgeons who perform coronary artery bypass surgery.5,6

Performance reporting overseas: In the US, the key comparator reported is the surgeon’s risk-adjusted mortality rate. However, in the UK, the report will not be in the form of a “league table”, at least initially. This is primarily because thoroughly validated risk adjustment of each surgeon’s patients is not yet available. Instead, a three-star scale will be reported, indicating that a surgeon either “fails”, “meets” or “exceeds” the standards of the Society of Cardiothoracic Surgeons. The index procedure is first-time, isolated coronary artery bypass graft (CABG), with inpatient mortality as the outcome indicator. Initially, the required standard is that an individual surgeon’s mortality rate, taken as a 3-year rolling average, must be within 99.9% confidence limits of the mean mortality rate for the nation. The intention is that risk stratification will be phased in over the next few years.

To understand this initiative it helps to set it against the background of the huge structural reforms under way in the UK National Health Service, which aim to transform it from a centralised monolith into a more flexible, “patient-centred” system, with more autonomy for individual trusts.7 While these reforms devolve more managerial authority to hospitals, clinical audit has been standardised and centralised. The Commission for Healthcare Audit and Inspection (which replaced the Commission for Healthcare Improvement in April 2004) is the independent authority that undertakes clinical-governance audits for British hospitals and other healthcare trusts. All acute, specialist, mental health, ambulance and primary care trusts are audited against a range of indicators and given a rating on a three-star scale,8 which is published on the Internet (Healthcare Commission, www.chai.org.uk/Homepage/fs/en).

The principal driver for these audit and governance reforms was the Bristol Royal Infirmary Inquiry,9 which announced its findings in July 2001. That inquiry, chaired by Sir Ian Kennedy, was concerned with understanding how unacceptable mortality rates for certain surgical procedures in the Bristol paediatric cardiac unit had gone unchecked from 1984 to 1995. Here was a case where self-monitoring had manifestly failed. The report found the roots of the Bristol tragedy in the insular and conformist “club culture” of the NHS. It argued that the goals of the reform process demanded a sea change in the ethos of the hospital system.

In January 2002, the UK government’s response to the Inquiry was tabled in the House of Commons.10 Agreeing in full or in part to 187 of the Inquiry’s 198 recommendations, the government committed to far-reaching reforms aimed at developing a culture of public disclosure and accountability in healthcare. At that time, the then Secretary of State for Health, Alan Milburn, announced an agreement with the Society of Cardiothoracic Surgeons to publish mortality rates for every cardiac surgeon in Britain from April 2004. He stated that “this is just the first step to publishing more information on individual consultant outcomes over time”.11

In 1998, in the aftermath of the events at Bristol, the Society had begun collecting surgeon-specific data for certain marker operations (primarily, first-time CABG). These data are independently analysed, and the results scrutinised within the Society to provide early warning of potential problems. Surgeons whose performance lies outside predetermined limits are notified and required to respond.

Cardiothoracic surgeons were “not comfortable” with public release of individual performance information but accepted that it was inevitable. The surgeons’ main concern is that it will lead to defensive medicine and the avoidance of high-risk patients.12 If individual mortality rates are published, risk adjustment for each surgeon’s casemix is essential, not least because the sickest patients are usually assigned to more experienced consultants. Whether or not high-risk patients are disadvantaged then depends crucially on whether surgeons have confidence in the risk-adjustment process.

Reporting in Australia: There have been moves to publish comparative surgical performance data in Australia, but only at the level of cardiac units. Outcome data for individual surgeons are collected by many hospitals and by surgeons themselves, but they are not centrally coordinated into a comprehensive database, and no surgeon-specific data are available to the public.

The 1999 Victorian Health Services Policy Review recommended that the Victorian Department of Human Services (DHS) publish annual data on the comparative performance of hospitals for specific procedures, and that the Commonwealth and the States collaboratively develop a set of comprehensive, consumer-oriented performance indicators.13 In 2003, the DHS began publishing annual outcome reports for the six Victorian public hospital cardiac units, drawing on a database established by the Australasian Society of Cardiac and Thoracic Surgeons with the help of a $200 000 grant from the DHS.14-17 However, in these reports the unit-level data are de-identified. These reports show that overall cardiac surgery outcomes compare favourably with those in the US and UK. Also, there has been an overall reduction in mortality rates for CABG surgery in Victorian public hospitals from 2002 to 2003, which may have been partly due to the monitoring of unit performance in these public reports.

The Australasian Society of Cardiac and Thoracic Surgeons supports a national scheme of cardiac-surgery reporting based on the Victorian model. The Australian Council for Safety and Quality in Health Care is also developing the National Cardiac Procedures Register, which will record de-identified, risk-adjusted information on the outcomes of various procedures, such as CABG surgery and interventional cardiology procedures.18

Improving quality of healthcare: Good clinical governance requires data collection at the individual level. One of the clearest lessons from Bristol is the need for a continuous audit process to identify outliers in mortality and complication rates. When public reporting systems were developed in the US, an explicit aim was to try to improve clinical quality by changing consumer behaviour. It was hoped that better-informed purchasers would demand quality, and that poor performers would be disciplined by the market.19 CABG outcomes in New York have improved markedly since “report cards” were introduced, but not in fact because of an effect on the market share of performance outliers. Instead, healthcare quality improved as poorly performing hospitals used the data to identify problems with their processes.20

The question, therefore, is not whether these data should be collected, but whether they should be made public. The answer depends on what we think public reporting of individual performance is for — patient choice, or patient safety and professional accountability? We argue that, independent of considerations of patient choice, the arguments for both safety and accountability strongly justify public reporting of outcomes. The legitimate public interest in healthcare quality makes it very difficult to argue against the public release of performance data.21

Reporting of surgical outcomes at both unit and individual levels can have a powerful positive effect on public confidence and trust. However, overseas experience shows just how challenging an undertaking this is. Meaningful performance assessment requires the development of guidelines and standards against which performance is to be measured. Half of the surgeons on any league table will be, by definition, worse than average. For public reporting what matters most is not a ranking, but rather that surgeons are shown to meet acceptable performance standards. The development of national standards depends on centralised data collection, to build an evidence base and to identify best practice.

Before individual performance data can be publicly released, a standardised minimum dataset must be established, with uniform reporting requirements in all centres and an audit system to validate data against case notes. The data must also be robust, averaged over several years to counter statistical volatility in individual casemix, and adequately risk-adjusted.

If professional reticence about reporting surgeon-specific data is to be overcome, surgeons must have confidence in the data. This means that the surgical associations must take the lead in developing data standards, and processes of collection, validation, analysis and publication. In addition, the possible impact on access to surgical care should be considered, as well as the form and context in which the data should be made available to patients. All of this will take considerable resources, and any policy initiatives for comprehensive public reporting of outcomes in Australia must be backed by a political commitment to proper funding.

We believe that, for surgeons to meet their obligations for public accountability, public reporting of surgeon-specific performance information is ultimately needed. In the next few years, it will be important to watch how this reporting develops in the UK, whether fears of defensive medicine prove justified, and how it affects both surgeons’ and the public’s perception of the surgical profession. We hope that Australia will not need a scandal like Bristol for state and federal governments to fund the development of the databases and audit systems needed for continuous, standards-based performance management. As the Bristol example illustrates, public trust in the healthcare system is too important to be jeopardised. Once lost, it is both difficult and very expensive to regain.