Agreement between diagnoses of otitis media by audiologists and otolaryngologists in Aboriginal Australian children

Otitis media (OM) is very common in children,1 but is more severe and disabling in indigenous populations. The onset of OM in Australian Aboriginal and Torres Strait Islander children is frequently earlier and the disorder has a more prolonged and complicated course than in non-Indigenous children.

Chronic or recurrent OM is the leading cause of hearing loss in children, with long term educational and psychosocial impacts.2 Early detection is critical, but there is no single reference diagnostic test, and expert clinical judgement is required. Such expertise may be limited in the primary health care setting, where most diagnoses of OM are made, particularly in the rural and remote settings in which the OM burden is greatest.3,4 Despite OM being one of the most common childhood illnesses and having important, long term health consequences,3 diagnostic inaccuracy is common,5-7 leading to delayed treatment, under- or overtreatment, and an increased risk of complications, including hearing loss8,9 and antibiotic resistance.3

In many countries, general practitioners manage children with OM, some of whom are referred to audiologists. Otolaryngologists are specialists in the diagnosis and treatment of OM, but access to their services is limited, especially outside major urban centres.4 Audiologists are usually more readily available, and could provide an initial assessment before the child is referred to an otolaryngologist. Whether audiologists can reliably triage at risk children for specialist otolaryngology management, however, is unclear.

The Study of Environment on Aboriginal Resilience and Child Health (SEARCH)10 is a cohort study of urban and regional Australian Aboriginal children in New South Wales designed to determine predictors of health and wellbeing and the critical time points for intervention. SEARCH evolved from extensive engagement with Aboriginal communities, during which ear health was identified as a health care priority. In the investigation reported in this article, we sought to determine the level of agreement between OM diagnoses by audiologists and otolaryngologists, and to evaluate whether the triage of children by audiologists could be an effective referral and care strategy in settings with limited access to otolaryngology services.

Methods

Study design

We applied the Standards for Reporting of Diagnostic Accuracy Studies (STARD).11 The entire SEARCH cohort of 1669 children (0–17 years of age) from four participating Aboriginal Community Controlled Health Services (ACCHSs) across New South Wales (three metropolitan, one regional centre) were eligible to participate.10 Between 2008 and 2012, the four ACCHSs cooperated with investigators to recruit and enrol children in SEARCH. Each child was included in this study only once. All children and their caregivers were interviewed by Aboriginal research officers; a range of clinical measures were recorded as part of baseline assessment, including results of ear assessments. The main reason for missing ear assessments (165 children) was refusal of permission by caregivers.

Ear assessment

Each participating child was assessed by one of five audiologists using age-appropriate audiometry (Interacoustics AD226/AC30 audiometer), tympanometry (Earscan tympanometers), and standard and pneumatic video-otoscopy (Inline Systems otoscopes). No manufacturer was involved in the assessments or analyses. Audiologists recorded diagnoses for each ear. The otoscopy images, pneumatic videos, and tympanometry results, but not the diagnosis or clinical history, were then forwarded electronically to one of three otolaryngologists for review at their convenience, sometimes several months after the assessment (online Appendix, figure 1). We used the otolaryngologists’ diagnoses as the reference standard, both to be consistent with earlier studies,12-15 and because it is usually otolaryngologists who determine how children will be managed.

At the beginning of this study, the audiologist who conducted 59% of the assessments had 26 years’ experience; the other four had 1–14 years’ experience. The three otolaryngologists, including the first Aboriginal surgeon in Australia, had extensive experience in treating ear disease in Aboriginal Australians.

Statistical analysis

As we deemed tympanic membrane perforations to be the most important clinical ear-related problem, the nine possible audiologist diagnoses were reduced to three categories for analysis at the child and ear levels (online Appendix, table 1): OM with perforation (acute OM with perforation, dry perforation, or chronic suppurative OM); OM without perforation (acute OM without perforation, recurrent acute OM, OM with effusion, chronic OM with effusion, or undifferentiated OM); and no middle ear pathology (“normal”). Child-level diagnoses were based on the highest order finding in either ear; if one ear was classified as normal and a finding for the other was not recorded, the child-level diagnosis was classified as “missing” data. We excluded 13 assessments performed by audiometrists or training audiologists.

Differences between the characteristics of those included in or excluded from the analysis, and between audiologist diagnoses for children with and without an otolaryngologist review were assessed in χ² tests (categorical variables) or Wilcoxon rank sum test (ordinal variables). We calculated the observed agreement of ear- and child-level diagnoses by audiologists and otolaryngologists. We performed Stuart–Maxwell tests of marginal homogeneity to assess whether there were systematic differences between audiologists’ and otolaryngologists’ diagnoses.

Inter-rater agreement of diagnoses (OM with perforation, OM without perforation, normal) by audiologists and otolaryngologists was expressed as linearly weighted κ values (proportion of agreement beyond chance). Comparisons were made between all audiologists and all otolaryngologists, and for each audiologist–otolaryngologist pair with a sufficient sample size (at least 32 diagnosis pairs for 80% power, 2-tailed).16 Prevalence- and bias-adjusted linearly weighted κ values were calculated to assess the impact on the level of agreement of imbalances in the prevalence of the diagnoses and bias (ie, difference in the frequency of diagnoses in audiologists’ and otolaryngologists’ assessments);17 low or high proportions of responses for one or more diagnoses can reduce the magnitude of κ. We adjusted for bias by calculating chance agreement, assuming that the marginal distributions of audiologists’ and otolaryngologists’ diagnoses were equivalent. We applied standard κ interpretations of observer agreement (< 0.21, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; > 0.80, almost perfect agreement).18

To assess accuracy, we calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the audiologists’ diagnoses at the ear and child levels (normal v OM).12-15 To assess whether there was a learning effect for use of study equipment, we compared agreement between audiologists and otolaryngologists for the first 100 ear assessments by each audiologist, and separately for all subsequent assessments. We also compared agreement between diagnoses for left and right ears.

We performed sensitivity analyses by excluding ears or children if there was any uncertainty about the matching of the audiologist assessments to the otolaryngologist review. This arose when more than one audiology assessment was conducted for a child (ear level, 84; child level, 43) or any doubts about the identity of the child for the audiology assessment or the otolaryngologist review (ear level, 8, child level, 4). For all analyses, we used bootstrap methods with 1000 replications to estimate standard errors, accounting for ears within children and children within families as appropriate. All analyses were conducted in Stata 14 (StataCorp).

Ethics approval

The Aboriginal Health and Medical Research Council of New South Wales (reference, 568/06) and the University of Sydney Human Research Ethics Committee (reference, 8506) provided ethics approval for this study.

Results

Participant characteristics

Of the 1669 children enrolled in SEARCH, 1310 (78.5%) received an ear health assessment leading to an audiologist diagnosis (Box 1). Our sample included 990 children (59.3%) with paired audiologist and otolaryngologist diagnoses available at the ear or child levels; 55.0% were boys, 49.4% were under 6 years of age (mean, 7.0 years; standard deviation [SD], 4.4 years). The sex distribution of children included in the analysis was similar to that of those not included, but their age distribution was different (mean, 7.7 years [SD, 4.3] v 6.0 years [SD, 4.4]) and they had a higher proportion of normal audiology findings (69.8% vs 58.1% of children) (Box 2). The distributions of audiology diagnoses for children with and without an otolaryngologist diagnosis were similar (online Appendix, table 2).

Otitis media findings: diagnostic agreement between audiologists and otolaryngologists

The diagnoses by otolaryngologists by ears (n = 1775) were 12 of OM with perforation (0.7%), 384 of OM without perforation (21.6%) and 1379 normal (77.7%); their diagnoses by child (n = 863) were 11 of OM with perforation (1.3%), 240 of OM without perforation (27.8%); and 612 normal (70.9%) (Box 3).

There was 92.2% agreement between all audiologists and all otolaryngologists at the ear level and 91.7% at the child level; the false negative rates were 3.0% (child level) and 3.7% (ear level), the false positive rates were 4.5% (child level) and 3.6% (ear level) (Box 3). For individual audiologist–otolaryngologist pairs, concordance ranged between 88.8% and 98.3% at the ear level, and between 85.5% and 98.2% at the child level (online Appendix, tables 3 and 4). Marginal homogeneity analysis indicated that otolaryngologists were less likely to diagnose OM with perforation than audiologists (1.3% v 2.2% at the child level, 0.7% v 1.3% at the ear level).

The overall agreement between all audiologists and all otolaryngologists was substantial at the ear level (κ = 0.78; 95% confidence interval [CI], 0.74–0.82; Box 4), and almost perfect (κ = 0.81; 95% CI, 0.77–0.85) at the child level (Box 5). There was no difference between agreement for right (κ = 0.79; 95% CI, 0.74–0.84) and left ears (κ = 0.78; 95% CI, 0.73–0.82). Agreement for individual audiologist–otolaryngologist pairs ranged from substantial to almost perfect at both the ear and child levels; κ values ranged from 0.69 (95% CI, 0.54–0.85) to 0.95 (95% CI, 0.85–1.00) at the ear level, and from 0.67 (95% CI, 0.49–0.84) to 0.96 (95% CI, 0.88–1.00) at the child level (online Appendix, tables 3 and 4).

The prevalence- and bias-adjusted κ values for all audiologists and all otolaryngologists were 0.94 (95% CI, 0.93–0.95) at the ear level and 0.94 (95% CI, 0.92–0.95) at the child level (Box 4, Box 5). Adjusted κ values for individual audiologist–otolaryngologist pairs are included in the online Appendix, tables 3 and 4. Agreement at the child level for audiologist–otolaryngologist pairs is included in the online Appendix, table 5.

Sensitivity and specificity

For comparisons of normal and abnormal diagnoses at the ear level for all audiologists and all otolaryngologists, sensitivity was 83.3% (95% CI, 78.9–87.8%), specificity 95.4% (95% CI, 94.1–96.6%), PPV 83.8% (95% CI, 79.5–88.0%), and NPV 95.2% (95% CI, 93.8–96.6%). For comparisons of normal and abnormal diagnoses at the child level, sensitivity was 89.6% (95% CI, 85.8–93.5%), specificity 93.6% (95% CI, 91.7–95.5%), PPV 85.2% (95% CI, 80.9 to 89.5%), and NPV 95.7% (95% CI, 94.0–97.3%) (online Appendix, table 6).

Learning effect

We found a lower level of crude agreement at the ear level for the first 100 audiology assessments by each audiologist compared with the crude agreement for subsequent assessments (first 100 [n = 422 ears]: κ = 0.66; 95% CI, 0.54–0.78; subsequent [n = 1353 ears]: κ = 0.81; 95% CI, 0.77–0.86; κ difference, 0.15 [95% CI, 0.03–0.28]; P = 0.016). However, after adjusting for prevalence and bias, there was no difference (first 100: adjusted κ = 0.92; 95% CI, 0.88–0.96; subsequent: adjusted κ = 0.95; 95% CI, 0.93–0.96; adjusted κ difference, 0.03 [95% CI, 0.02–0.07]; P = 0.23). Results were similar at the child level (not shown).

Sensitivity analysis

Excluding ears or children with any uncertainty about the matching of the audiologist’s assessment to the otolaryngologist’s review did not alter our findings: at the ear level, κ = 0.79 (95% CI, 0.75–0.83) and adjusted κ = 0.94 (95% CI, 0.93–0.96); at the child level, κ = 0.82 (95% CI, 0.77–0.86) and adjusted κ = 0.93 (95% CI, 0.91–0.96).

Discussion

For almost 1000 NSW Aboriginal children living in metropolitan and regional centres and with a high prevalence of OM (29%), we found substantial agreement between diagnoses by audiologists and otolaryngologists; agreement was almost complete after adjusting for prevalence and bias, and was consistent across audiologist–otolaryngologist pairings. Overall, OM would have been missed in only 3% of children were an audiologist triage strategy in place; no perforations would have been missed.

Although the prevalence of OM in this sample of urban Aboriginal children was higher than in other populations of children,19 we expect that our finding of diagnostic agreement between audiologists and otolaryngologists would apply in other settings of comparable diagnostic expertise. Differences in diagnostic expertise, seasonal fluctuations, and child characteristics may have affected prevalence estimates, but are unlikely to have substantially affected diagnostic agreement. As the distributions of audiologist diagnoses for children with and without an otolaryngologist diagnosis were similar, selection bias in the included assessments is unlikely. We used the otolaryngologist diagnosis as our reference standard, but, given the possibility that not everything the audiologist saw was clearly captured in the images and videos they received, audiologists may have been correct in the cases in which they diagnosed tympanic membrane perforation and the otolaryngologist did not. A minor degree of overdiagnosis of perforation would be clinically preferable to underdiagnosis.

This is the first published study to assess agreement between audiologists and otolaryngologists in the diagnosis of OM. A few studies have compared diagnoses of OM by other specialists, with sample sizes ranging from 23 to 171.6,20,21 With our larger sample size, we could perform analyses of agreement by individual clinician pairs, at both the ear and child levels, and adjust for OM prevalence and bias. Our agreement level (κ = 0.81) was comparable with the level found by a Finnish study20 that examined diagnoses for 58 children by one audiologist and ten doctors (for 242 tympanograms: κ = 0.77). We found higher levels of agreement than investigators who compared diagnoses by paediatric residents and otolaryngologists (κ = 0.38)21 or by primary health care practitioners and otologists κ = 0.68–0.74).22 Researchers who compared different diagnostic techniques applied by different health care professionals found much lower levels of agreement, including Roberts and colleagues23 (κ = 0.09–0.46) and Steinbach and his co-authors (κ = 0.32).21 Others have reported that agreement between audiologist and otolaryngologist diagnoses was validated, but without providing supporting data.24,25 The sensitivity we found was lower than in two earlier, smaller studies — one comparing diagnoses by an audiologist and a nurse practitioner (171 children)26 and another comparing acoustic admittance by audiologists with tympanometry (50 children)27 — but our calculated specificity (95.4% at the ear level, 93.6% at the child level) was higher than reported for those studies.

Limitations

We analysed agreement at both the ear and child levels because referrals are based on the child level diagnosis, and an error in transcribing which ear had middle ear disease may affect ear-level, but not child-level, diagnoses. On the basis of our findings, and assuming 29% prevalence of OM, about 925 of 1000 children screened by an audiologist would be correctly triaged, 45 would receive a false positive diagnosis, and 30 would have OM but be incorrectly diagnosed as normal (false negative); no perforations, however, would be missed. We found lower diagnostic agreement for the initial 100 assessments by individual audiologists than for subsequent assessments (crude but not adjusted κ). Although we found high levels of diagnostic agreement, it may have been higher had we provided detailed clinical information to the otolaryngologist, as would happen were this work translated into clinical practice. It was beyond the scope of our study for the otolaryngologist to independently examine all children personally, but had they done so they may have identified perforations they missed in the supplied material. It is also possible that an audiologist and otolaryngologist both made incorrect diagnoses, but, given the clinical experience of the participants, this should have been rare. The degree of diagnostic agreement we found may not have been seen with less experienced assessors.

A key unanswered problem is how to appropriately weight the results of individual tests used for diagnosing OM. SEARCH will investigate this question by providing limited information (eg, tympanometry results) to otolaryngologists before they make their diagnoses, then serially providing further information to determine whether they change their diagnoses.

Conclusion

When access to otolaryngologists is limited, review by an audiologist in the primary health care setting may facilitate the identification of OM and subsequent referral to an otolaryngologist. Our findings could inform investigation of the value of tele-otology and tele-health in regions with limited access to otolaryngologists.

Box 1 – Flowchart of participants through the study

Box 4 – Agreement of otitis media diagnoses (with or without perforation) for each audiologist–otolaryngologist pair and for all audiologists and otolaryngologists, at the ear level*

---

CI = confidence interval. * Unadjusted and adjusted κ values.

Box 5 – Agreement of otitis media diagnoses (with or without perforation) for each audiologist–otolaryngologist pair and for all audiologists and otolaryngologists, at the child level*

---

CI = confidence interval. * Unadjusted and adjusted κ values.