Professional monitoring and critical incident reporting using personal digital assistants

The recent failure of professional monitoring systems in the United Kingdom, particularly in Bristol, has highlighted the need for improved monitoring of training and performance.1,2 The culture of medicine needs to change to embrace the feedback of data on personal and unit performance.3 External assessment of competency is a difficult task and agreement about what clinical outcomes are acceptable is open to debate.4 Self-assessment can include logbooks, maintenance of professional standards programs and self-audit.5

The recent introduction of handheld computers, or personal digital assistants (PDAs), has given medical practitioners the ability to accurately collect data at the point of care. The potential uses for medical practitioners have recently been highlighted.6,7 We investigated the feasibility of using a PDA software package custom-designed for anaesthesiology trainees. It incorporates a bedside logbook, a self-assessment tool using the cumulative summation (cusum) technique for particular anaesthetic procedures, and the ability to collect data on anaesthetic adverse events.

The PDA software was developed from a handwritten blueprint, with the emphasis of the program being on ease of use. In association with a software developer (Sync International, Victoria, Australia) over a period of four months and five revisions, a definitive version of the program was produced which included a logbook (Box 1) and cusum self-assessment tool. The software allowed users to review all past data entries, filter this information to obtain a summary of caseload or procedural exposure, annotate interesting cases and make handwritten notes on the PDA.

Six first- or second-year registrars from two tertiary referral centres (in Victoria and New Zealand) and a large district hospital in Queensland and their training supervisors were recruited for data collection over a four- to seven-month period, from March to October 2000. Each trainee was supplied with a Palm Ille PDA (Palm Computing Incorporated, California, USA) loaded with the experimental software. Registrars and their supervisors had a one-hour educational session before beginning data collection. Ongoing technical support via the training supervisors was provided as required.

The registrars recorded logbook entries for each anaesthetic procedure undertaken. Eight anaesthetic critical event categories (Box 2) were included as drop-down menu items in the software. If a defined anaesthetic incident occurred it was recorded using the menu along with the outcome of the event (eg, uneventful, minor morbidity, major morbidity or death). In this way the true incidence of adverse events could be determined.

The data from the PDAs were uploaded onto a secure departmental desktop computer for storage and the generation of cusum graphs with Microsoft Access and Microsoft Excel software.

During the study period 1690 anaesthetic cases were logged: 62% were performed with level 1 supervision as defined by the Australian and New Zealand College of Anaesthetists10 (Box 5), 22% with level 2 supervision, 15% with level 3 supervision and 1.3% with level 4 supervision. Twenty-seven per cent of procedures were performed out of hours. The main operation classes were general surgery (22%), orthopaedics (17%), plastic surgery (15%) and obstetrics (10%). Eleven per cent of procedures were performed in remote locations outside the operating suite (general ward, radiology, cardiology, and magnetic resonance imaging).

Four hundred and eighty procedures were logged for cusum analysis. Ninety of these were excluded from analysis, as the supervisor of the case performed them (Box 6). The cusum plots showed variation in the failure rates between registrars, but all were within the acceptable range according to the predefined parameters.

Forty-two critical incidents were recorded, giving a critical incident rate of 2.5%. Twenty-one critical incidents were airway/respiratory events, 17 were cardiovascular, and one each of pharmacological, severe hypothermia, equipment, and regional-procedural (dural tap). Of these 42 events, 19 had uneventful outcomes, eight had minor morbidity, 14 had major morbidity and there was one death.

An estimated 10% of data were lost owing to operator and equipment errors. These problems related to PDA breakage or battery failure. Data loss occurred because desktop computer software for backing up data was not introduced until midway into the project.

PDAs are a technology in their infancy. Released in 1995, their adoption has been remarkable, with over 15 million units sold worldwide (Palm Inc Australia, personal communication). The medical profession has embraced this new technology, with an estimated 26% of practitioners using PDAs, a number estimated to grow to 50% by 2005.6

We encountered significant practical problems resulting in major data loss with the model of PDA used in our trial. Our PDAs are now protected physically with a durable external case, which increases their robustness for theatre use and has prevented further breakages. We now have a reliable backup system using an Internet-based program to upload data from the PDA to a secure server, which can be accessed using any modem or compatible mobile phone. This also gives us the ability to download updated software to the PDA. Information and analysis of the data can be viewed via a website that is accessible only with a password.

The use of logbooks does not provide a qualitative analysis of performance, and indeed this has been one of their major criticisms.10 Logbooks do provide evidence of exposure to cases and techniques, and one might hope that the greater the exposure the higher the level of performance. The Australian and New Zealand College of Anaesthetists training guidelines stipulate supervision requirements for various levels of training,11 as well as exposure parameters for practices such as labour ward epidurals, neurosurgical cases, double-lumen tubes and "out of hours" procedures.12 This portable logbook can easily provide this information, as well as details on exposure to subtypes of operation, age ranges of patients, American Society of Anesthesiologists scores, amount of out-of-hours work, and types and numbers of critical incidents.

The desirable depth of information recorded in a logbook is a contentious issue, and we have found opinions vary widely. Our logbook has more depth than that of the UK Royal College of Anaesthetists, but, despite this, data entry by an experienced user took less than 45 seconds for an average case.

While exposure to procedures may lead to improved performance, this may not always be the case, as shown by the cusum graphs (Box 4). From the first graph it can be seen that the registrar has shown a continuous improvement in performance in the study period. The second graph reveals an initial unacceptable performance, followed by a continued stable level of performance. This is indicative of a learning curve and one would hope that, with continued experience, an improvement in performance would be indicated by the graph trending downwards.

The advantage of serially reviewing performance is that even a small deviation (especially deleterious) is revealed early, allowing timely intervention. Likewise, excellent performance can identify practitioners capable of training more junior trainees, and this can enhance training programs.

Although our pilot study limited cusum analysis to four basic procedures, our software has been designed to allow cusum audit of any other procedure designated by the operator. All one need do is define success and failure for a procedure and commence data collection.

Objectively defining success or failure for a task is probably the most difficult part of the process. Our objective criteria (Box 3) were obtained by consensus of the working group. This is the aspect of cusum analysis that is highly interpretive and does warrant further debate. Previous studies have used different criteria for determining success and failure.8

Procedural analysis is only one aspect of training and will only ever be one component in competency assessment. Throughout training, academic knowledge is tested rigorously, but assessment of procedural competency, professionalism and crisis-management skills may be neglected. Competency assessment would be aided by cusum analysis of these less rigorously analysed areas.

The recording of adverse incidents in anaesthesia is at present a generally haphazard affair. The major advantages of data entry at the point of care are that events can be recorded soon afterwards, and the incidence of given events can be rapidly tabulated with both a numerator and denominator. In our study we found an adverse incident rate of 2.5%. Having both a numerator and denominator can only improve the relevance of data for daily clinical practice.

There are few published prospective studies of the rate of anaesthetic incidents, with reported incident rates ranging from 0.76% to 6.70%.13,14 One of the difficulties of tabulating an incident rate is the lack of universal definitions for a critical incident.15 Our aim was to collect data on incidents that resulted in, or had the potential to result in, patient morbidity or mortality. While, in theory, this list could be quite exhaustive, we restricted our drop-down menu to major items, with the ability to write notes for unusual or rare events.

Finally, two philosophical issues need to be addressed. Firstly, the ownership of the data collected is important. As the primary aim is to allow a practitioner to independently review his or her own performance, we believe the data should remain confidential to the individual anaesthetist. Our main aim is for self-analysis with secure data. Secondly, there is the problem of reliability of the data. Any self-assessment tool relies on the integrity of the person entering data. This is the case with the ANZCA Maintenance of Professional Standards program. The ability to obtain maximal value from a system such as this is highly dependent on the operator. We adhere to the concept that Australasian anaesthetists want to do the very best for their patients, and we have aimed to give them a tool to help them achieve this.

1: Logbook data fields

• Date and time stamp

• Out-of-hours work

• Unit record number and date of birth

• American Society of Anesthesiologists (ASA) status

• Supervision level

• Type and subtype of surgery

• Anaesthesia technique

• Regional techniques

• Procedures

A screenshot of the PDA logbook.

4: Cusum plots for two registrars

Note horizontal boundary lines. A cusum plot which spans the gap between two boundary lines in the downward direction is proof of acceptable performance according to the defined parameters. Conversely, a plot which spans the gap between two boundary lines in the upward direction is proof of unacceptable performance according to the defined parameters for the intervening series of procedure attempts.The first plot (of a registrar performing epidural anaesthesia) shows a steady trend of improving performance; the second (a registrar inserting arterial lines) shows a more difficult progress, with alternating stages of acceptable and unacceptable performance.