Measurement of the safety and quality of health care

Measurement is vital in all areas of clinical medicine. To fully understand any disease or therapeutic process, it is essential to describe it quantitatively and qualitatively. However, as identified by Florence Nightingale over 140 years ago, the use of rigorous measures to describe the quality, safety and effectiveness of our health care system has lagged behind the science of clinical measurement.

Indeed, it was not until the 1990s that landmark studies confronted all of us with comprehensive data describing what we already knew as clinicians — that health care sometimes does harm.1,2 To better understand the size of the problem and to detect changes, it is essential to measure the safety and quality of health care.

A variety of strategically chosen measures are needed to understand the quality and safety of health. Quantitative measures, such as mortality, will always be integral to this measurement, but, for clinicians, semi-quantitative and qualitative assessments can highlight broad areas or issues that require scrutiny (Box 1).

Useful quantitative measures may require large databases and powerful statistical analyses, such as those developed in the United States to highlight areas of unexpectedly high mortality in cardiac surgery.3 However, measures can also be as simple as local audits of practice to determine whether benchmarks are being met.4

A variety of tools are becoming available to help health professionals approach particular issues of health care safety or quality and to choose an appropriate measurement. The Australian Council for Safety and Quality in Health Care (ACSQHC) has produced a “Measurement for Improvement Toolkit”, which is a practical, evidence-based guide for use by clinicians in both the public and private sectors in Australia4 (Box 2).

Just as in clinical medicine, any measurement of safety and quality is useful only if it actually measures what it is supposed to, and is used and interpreted correctly. The National Health Performance Committee is a committee of the Australian Health Ministers’ Advisory Council, whose role is to develop and maintain a national framework of performance measurement for the health system; to establish and maintain national performance indicators within the national performance measurement framework; to facilitate benchmarking for health system improvement; and to report on these to the annual Australian Health Ministers’ Conference. The Committee has developed criteria for selecting health performance indicators (Box 3).5 To avoid duplication of effort, indicators should use existing datasets whenever possible.

Nevertheless, it is important to remember that many measures of safety and quality of health care are relatively inexact, and so should not be interpreted as a conclusive picture of an individual’s, an agency’s, or even a system’s performance. An indicator is not an absolute measure of quality or safety, but rather can act as a screen to determine or identify areas for further local analysis. While data can be collated, analysed and fed back centrally, it is only at a local level that the underlying reasons for a particular result (eg, rate of surgical-site infections) can be truly explained, and changes made to improve practice.6 Thus, indicators are a tool to encourage performance improvement and to identify areas worthy of further study; they are typically hypothesis-generating rather than hypothesis-proving.

In clinical terms, measurements of health care safety and quality may be useful for screening and ruling out a problem, for diagnosing a problem, and for monitoring progress.7 However, use of a screening measure to diagnose poor quality will produce “false positives”; equally, use of a highly specific diagnostic indicator to rule out problems will produce “false negatives”. Either way, the measure is not useful. False positives with screening tools such as raw mortality cause much anxiety when interpreted to indicate a health system problem, rather than a need to focus more closely to determine whether a problem really exists. For these reasons, a variety of risk adjustments have been developed to make raw data more meaningful to the types of patients seen and to adjust for factors over which clinicians have no control, including sociodemographic and clinical characteristics (eg, age, sex, socioeconomic status, comorbidities, physiological variables, and emergency versus planned status). These adjustments make the raw statistic more specific and meaningful when deciding whether there is a problem. However, as risk adjustment has limitations and can adjust only for known confounders, it seems highly unlikely that it can ever fully compensate for the effects of casemix variables, so that remaining variation reflects quality of care alone.8,9

Other problems arise in understanding when a change in an event rate is a real improvement or deterioration, especially when the event is uncommon. Frequency charts of the raw number of events occurring over time (time series or “saw tooth” charts) are rarely helpful, because of underlying background variation and small numbers of adverse events. Statistical process control methods (eg, exponential weighted moving averages, process control limits, and cusum analyses) developed in laboratory science for quality control make these fluctuations more interpretable. For example, cusum analysis was used recently to better understand bed occupancy and to plan medical and surgical admissions with the aim of improving access to health care,10 while statistical process control methods have been used to identify outliers more reliably.11,12 Such methods are now routinely used for reporting safety, quality and administrative data at Flinders Medical Centre and the Repatriation General Hospital, Adelaide.

The final problem with measures of health care quality and safety is to ensure that they are timely and repeatable. All measurement and clinical practice change is ultimately individual and local. If results are to support change, they must be reported to those who use them in a way that is relevant to current practice. While system-wide measures might be ideal to ensure equity of safety and quality, and to monitor effects of broader-scale or longer-term initiatives (eg, via national agencies such as the Australian Institute of Health and Welfare), the coordination and standardisation of their collection, submission, analysis and publication makes timeliness difficult and decreases their usefulness for local clinicians. Similarly, measures are most useful when they can be repeated after practice change, to determine its effects. The development of such key performance indicators is fundamental to any clinical practice improvement or innovation. This has been highlighted by recent studies which demonstrate evidence-to-practice gaps in virtually every area of health care.13-16 By developing measures that are timely, can be replicated, and inform understanding of the quality of care, local change initiatives can lead to dramatic improvements in care.17-19

Recently, Wilson and Van Der Weyden called for better systems by which we can understand how our health system is performing.20 Ways of measuring processes, outcomes and the culture of health care are well described and freely available.4 However, the most fundamental barrier to better measurement seems to be our failure to invest in these systems as part of the health care structure, in the way we have invested in, for example, financial management systems. Gathering data on measures of safety and quality of health care systems that are structural, valid, reliable, accurate, timely, collectable, meaningful, relevant and important requires resources, which are still lacking. However, the situation is changing rapidly with the introduction of nationally agreed requirements for health care incident reporting systems, sentinel event reporting, and a variety of morbidity and procedure registries, and with development of a national minimum dataset for safety and quality through the ACSQHC and its successor, the Australian Commission for Safety and Quality in Health Care.

Similarly, professional bodies, such as the Royal Australasian College of Surgeons (RACS) Section of Breast Surgery and the Australasian Society of Cardiac and Thoracic Surgeons, have introduced systems of performance reporting, feedback and improvement for their members. Indeed, participation in the process is required for membership of the Section of Breast Surgery. The Cardiac and Thoracic Surgeons’ national reporting system is not mandatory, with eight hospitals participating in 2005, and another six to join in 2006. A major issue is the $15 000–$20 000 recurrent cost per hospital needed for data collection. Clearly, such initiatives are resource intensive and, like financial management systems, require structural investment.

The UK National Health Service (NHS) has also realised the need for a systematic approach to improving patient safety and has established the National Patient Safety Agency (NPSA) to bring together information to quantify, characterise and prioritise patient safety issues. A core function of the NPSA is the development of the National Reporting and Learning System to collect reports of patient safety incidents from all service settings across England and Wales, and to learn from these reports, including developing solutions to enhance safety.21,22

However, it is recognised that incident reporting on its own cannot reveal a complete picture of what does, or could, lead to patient harm. Incident reporting systems are not comprehensive, because of under-reporting, biases in what types of incident are reported,23 and the multiplicity of reporting systems. For example, in addition to the National Reporting and Learning System, the UK has separate reporting systems for medical device incidents, adverse drug reactions, health care-associated infections, and maternal and infant deaths. Furthermore, as serious events are rare, and information on them is distributed across the health care system, better use needs to be made of data collections already in existence, even if such collections were designed for different purposes.

Recognition of the need to access a range of data sources led the NPSA in 2004 to set up a Patient Safety Observatory in collaboration with partners from both within and outside the NHS. These include key national organisations, such as the Healthcare Commission (an independent body set up to improve health services in England), the Office for National Statistics, the Medicines and Healthcare products Regulatory Agency, which regulates medicines and medical devices in the UK, patient organisations such as Action against Medical Accidents, the NHS Litigation Authority, and medical defence organisations.24,25 The Observatory enables the NPSA to draw on a wide range of data and intelligence, including clinical negligence claims, complaints, and routine data from a range of sources about complications of clinical care. These form the basis for identifying and monitoring patient safety incident trends, highlighting areas for action, and setting priorities. Examples of NPSA Observatory activities are shown in Box 4.

This type of approach is well established in the UK for public health, with a network of regional public health observatories tasked with providing health intelligence to support the monitoring and assessment of population health.29 Setting up similar networks in Australia, with its smaller population base, would be relatively more costly, and would need to be done efficiently and, where possible, within expanded and strengthened existing organisations.

The biases within incident reporting systems also provide challenges for their use to compare or evaluate safety across institutions. Thus, a hospital may have more reports as a result of a better developed reporting culture: for example, incident reporting rates for acute trusts in England vary by a factor of seven, from 1.8 to 12.4 incidents per 100 admissions,24 but this is likely to reflect differences in completeness of reporting and artefacts of the reporting process, rather than differences in the occurrence of incidents. Using comparative data on reporting rates is thus highly problematic and even counterproductive, if external judgements about safety are crudely made on the basis of reporting rates.

The NPSA Observatory faces other challenges to integrating data from a range of sources. Many of the data sources with potential for assessing patient safety are collected for other purposes, and there may be limitations to their use. For example, a study of the value of clinical negligence data to assess safety encountered issues of confidentiality, data quality and completeness, and the resources needed to extract relevant information.30 The NPSA is working with the relevant organisations in England and Wales to develop a more consistent approach to collecting data about clinical negligence that will support patient safety.

Clinicians in practice today are generally well educated in the basics of clinical data assessment, with many having participated in clinical research activities. Especially in teaching hospitals, there is a sophisticated appreciation of the science of clinical measurement and its strengths and weaknesses. In the past, a major barrier to quality improvement activities has been the poor quality of data presented to clinicians purporting to represent indicators of performance.

In the future, the potential to engage clinicians in quality improvement activities will require information that is respected for its accuracy, relevance and impartiality. Clinicians will need training in the use of measurement to improve health care safety and quality, just as they require training in the use of clinical diagnostic tests. Already many examples exist, ranging from local initiatives7,12,13 to national procedure registries and disease databases, which demonstrate that clinicians are interested in this issue and that they respond positively to trusted performance data that are methodologically sound, risk-adjusted and timely. Clearly, in the short term, we could all do more to understand and improve what we do with the measures, techniques and skills that are already available. However, for the longer term, investment is needed to extend the required measures and skills widely and systematically through our health care system, especially where the financial and human costs and consequences of variable performance are high. It is hoped that we will be able to redress these deficiencies in a much shorter time than has elapsed since Florence Nightingale identified them.