Mapping the limits of safety reporting systems in health care —what lessons can we actually learn?

The collection of data relating to incidents and near-miss events has become an entrenched and critical component of safety management across high-risk industries worldwide.1 According to the quality and safety axiom that “every defect should lead to improvement”, reporting systems exist to provide the raw data for continual improvement processes,2 as well as serving critical functions with respect to the local management of incidents.

In the quest towards enhancing patient safety, health care has also embraced the collection of near-miss and incident data, with all health services in Australia collecting some form of data. Across most health systems, voluntary reporting systems exist for near-miss and minor incidents, with the primary aim of collecting information about vulnerabilities in the health care system, so that remedies can be applied before an actual adverse event takes place.3 According to the World Health Organization, the primary role of such reporting systems is to enable learning across large health systems:

Collecting data is only the first step in a process of organisational learning, through which lessons can be drawn from incidents, modifications made to practice, and the risk of adverse events reduced. At the organisational level, aggregate data are available and more sophisticated monitoring and analysis processes can take place, using data from multiple facilities or multiple units within the organisation. This process of using safety-related data has been termed quadruple-loop learning (personal, local, national and international), emphasising the wide-reaching potential for harnessing lessons from incident report data.5

Recently, a disparity between the number of reports being made and the rate of meaningful evidence-based change to practice has been identified.6 This in turn has suggested that the ineffective use of data, and a lack of published learnings from such systems, may be responsible for the deficit of evidence-based change.

In addition to the primary purpose of systemic quality improvement, secondary administrative roles such as local incident management functions, as well as political and organisational reasons for reporting also exist. As reporting systems become embedded in the day-to-day process of health care, these administrative functions (which demand less detailed reports and analysis) may have come to outweigh the primary purpose. Furthermore, the effective use of increasing volumes of data itself becomes increasingly difficult.

Our study explored the utility of incident reporting systems currently used in most Australian health services. Our primary aim was to determine the depth of information available within a typical incident reporting system and whether more sophisticated “human factors” classification schemes relating to incident aetiology and systemic defences can be applied to incident reporting data.

The principal role of incident reporting systems is to ensure a consistent and coordinated approach to the identification and analysis of incidents so that lessons can be learned and shared across the whole health system.13,14 Reporting of near-miss events has been shown to offer numerous benefits compared with retrospective investigation of adverse events. Perhaps the most important of these benefits are the greater frequency of near-miss events, thus allowing quantitative analysis; and the ability to identify and analyse the recovery functions that enabled the accident trajectory to be stopped before an actual adverse event took place.15

Efficacy of current incident reporting in Australia. Our study shows that the information contained in the current incident reports is frequently inadequate to allow sophisticated analyses of incident aetiology and, more importantly, the recovery functions involved in averting or ameliorating adverse events. The results suggest that incident reporting may not be meeting the primary objective of enabling continuous improvement of safety and quality.

These results echo a report of the Institute of Medicine in the United States, which suggested that the value of near-miss reporting required further investigation.16 Although there is general agreement that near-miss reports contain relevant information about identifiable hazards that cannot be collected by other means,15 most organisations only take action on serious adverse events, which diminishes the value of reporting large numbers of near misses.14,17 Indeed, the sheer volume of incidents reported means that health care organisations tend to investigate most events superficially. For the few incidents that receive thorough investigation, the principal method is root cause analysis, which is seen as the gold standard for gaining deeper insights into the causal features of an adverse event.14,18

Incident reporting systems are also subject to a number of other limitations, including international suggestions of significant underreporting of incidents,19 and of medical error, as doctors rarely report.20 Any reporting is highly affected by hindsight bias (based on the degree of harm the patient has suffered, especially considering that doctors tend to disproportionally report the more severe events). Incident reports are thus “a nonrandom sample of identified hazards from a larger unknown universe of hazards”.14 In light of these limitations, it is critical that we explore alternative ways to learn from incidents in health care.15-18

Doing more with less — category saturation. Currently, public health systems with established incident reporting systems (such as South Australia, Queensland and New South Wales) receive on average between 30 000 and 100 000 incident reports annually. While there are significant amounts of data available, our study demonstrates that effective learning from incidents requires much less data.

The category saturation results suggest that a critical threshold exists, beyond which quantitative analyses of existing forms of incident data yield limited new information. For health care incident types related to patient misidentification, this was found to be no more than 400 incidents. In any case, a substantial amount of information was available from only 200 incidents.

Human factors classification — measuring the resilience of the system. Early studies on adverse events in health care focused on the origins of errors and on the mechanisms involved in their production. Accordingly, most effort was dedicated to the field of error prevention and focused on addressing the contextual and organisational issues in the production of errors.1 However, in a more enlightened world where we accept error to be both inevitable and ubiquitous, current safety science now focuses on the processes of error detection and error recovery.21-23

Our results indicate that incident reporting systems as they are currently designed and administered do not always elicit sufficient detail to enable a systematic analysis. While these current systems may not be designed to collect details relating to incident aetiology from a human factors perspective, the study highlights that existing forms of data collection focus predominantly on a description of the incident as it unfolded. They contain limited information about causal or contributory factors and whether there were any features in the system available or needed to prevent an actual adverse event. However, given the ubiquity of error, a renewed focus on reporting the aetiology and recovery from error may place a massive burden on the time taken to report.

We suggest that the goal of improving learning from patient safety incident data will be best served by exploring innovative approaches to sampling incidents and eliciting more detailed data for each event. It is apparent that we are lacking quality, not quantity, of incident reports. We should be seeking data that will inform a deeper understanding of the system features that will promptly detect and mitigate the inevitable failures that occur in a complex system such as health care.

Innovative approaches to maximise the efficacy of incident reporting. To facilitate collection of better-quality incident reports, we propose testing a modified incident reporting system in which new incident reports are selected at random and followed up with the individuals involved, to seek more detailed information pertaining to the incident. We envisage that this type of system would complement the existing approaches of incident reporting and root cause analysis, but with better return on investment.

A novel approach to more effective use of incident report data in industries such as health care would first involve risk-based sampling of incidents from traditional incident reporting systems. The aim of this sampling would be the proactive identification of emergent high-risk safety concerns, rather than the analysis of sentinel events after they have occurred. Risk-based sampling would involve introducing risk assessment into the initial data-collection process within the incident reporting system. This approach already exists within other health care safety and quality methods. Health care failure mode and effects analysis, for example, risk assesses each failure event in terms of potential severity and probability of occurrence, producing a hazard score that in turn enables prioritisation of safety management activities.18

Second, this new approach would involve eliciting more detailed data on these high-risk incident types. This process could involve targeting specific incident types and using telephone interview or survey techniques to elicit further information. More detailed information on the aetiology of the event could be obtained in this manner, enabling more sophisticated approaches to analysis. Perhaps more importantly, this approach would enable collection of information on the features of near-miss events that relate to the error tolerance or resilience of the system. In this way, the focus of improving learning from patient safety incidents would shift from the negative analytic frame of “what went wrong” to the more positive analytic frame of “what enabled us to maintain the integrity of the system”. This more positive focus on the systemic defences that enable the detection and timely mitigation of incident trajectories could hold an important key for the renewed vigour of quality and safety interventions in health care. Further research is required to examine the resource implications and practicalities of such changes to incident reporting systems.

2 Number of categories of incident type, by number of incidents recorded in Australian incident reporting systems, 2004–2008 (n = 487)