Sensitivity and specificity of Aboriginal‐developed items to supplement the adapted PHQ‐9 screening measure for depression: results from the Getting it Right study

Depression is a syndrome diagnosed after a particular constellation of signs and symptoms has been elicited through a semi‐structured interview. These assessments focus on two cardinal and seven associated symptoms of depression as identified in diagnostic manuals.1,2 It is possible to meet the diagnostic criteria for depression through a diverse combination of symptoms. One study of more than 3700 individuals found 1030 distinct depression symptom profiles.3 Similarly, there is substantial diversity in the symptoms and signs used to indicate the degree of depressed affect being experienced,4 making it challenging to compare prevalence estimates across studies.5

Different cultures have common and culturally specific components of the aetiology, meaning, experience and expression of depression. The World Health Organization's International Study of Psychological Problems in Primary Care demonstrated that the same measurement tool has variable measurement properties and discriminant thresholds for depressive disorder across cultures.6 This is generally ignored in large surveys, where one measure and one threshold are applied, regardless of cultural background. Consistent with this perspective, research that was initiated and led by an Aboriginal man, and involved men from five Aboriginal language groups in Central Australia, identified the Patient Health Questionnaire 9 (PHQ‐9) as potentially useful but requiring modification for use in their communities. The PHQ‐9 was adapted and expressed in simplified English for use across Aboriginal languages in Central Australia with Aboriginal community members.7,8 The adaptations used agreed and consistent local language and expressions of distress; two bidirectional items (poor appetite or overeating; moving slowly or restless) were separated but sleep disturbance remained a single item. This adapted Patient Health Questionnaire 9 (aPHQ‐9) version was first tested with 186 Aboriginal men from Central Australia and was found to be acceptable and appropriate. It was then compared with a semi‐structured diagnostic interview in a community sample of 78 Aboriginal men,8 and this indicated that the aPHQ‐9 had promising psychometrics.

However, thematic analysis, by an Aboriginal‐led team of researchers, of semi‐structured interviews with Aboriginal men, Ngangkari Tjuta (traditional healers) and cross‐cultural mental health experts, identified several affective and cognitive phenomena indicative of Aboriginal people's distinct experience of depression.7 These phenomena were developed, discussed with expert informants, and collated into a list of questions. The draft questions were translated and back‐translated by three bilingual experts (from the Pitjantjatjara, Anmatyerr and Luritja language groups) and simplified English questions were finalised by consensus.9 The completed questionnaire was piloted, resulting in seven items reflecting the experiences of depression in Central Australian Aboriginal men (Box 1). The items centred around perceptions of a weakened spirit, the use of illicit drugs (particularly change over time), and feelings of homesickness, anger, irritability, worry and thinking about worry. Some symptoms were consistent with the dominant Western allopathic diagnostic model of depression, whereas others laid outside core components of depression under International statistical classification of diseases and related health problems (ICD) or Diagnostic and statistical manual of mental disorders (DSM) models.1,2 It is important to understand how these Aboriginal‐specific experiences of depression relate to the allopathic model of depression.

Culturally developed depressive symptom screening is one strategy to support culturally safe clinical practice. We explored the utility of the seven Aboriginal culturally specific depression items. Participants of a study that validated the aPHQ‐9¹⁰ also responded to the seven Aboriginal culturally specific depression items. Here, we report on the psychometric properties of these items and their relationship to the aPHQ‐9,11 and determine their sensitivity and specificity in relation to a diagnostic psychiatric interview.

Methods

Ethics

The study was approved by the University of Sydney Human Research Ethics Committee (HREC) (2014/361), Aboriginal Health and Medical Research Council of NSW HREC (1044/14), ACT Health HREC (ETH.8.14.207), Metro South Health (Queensland Health) HREC (HREC/14/QPAH/503), Central Australian HREC (HREC‐15‐287), Menzies School of Health Research HREC (2014‐2289), Aboriginal Health Council of South Australia Aboriginal Health Research Ethics Committee (04‐15‐622), and Western Australian Aboriginal Health Ethics Committee (607).

Study design and participants

Getting it Right was a prospective diagnostic accuracy study conducted in ten Aboriginal and Torres Strait Islander primary health care services across the Australian Capital Territory (Narrabundah), Queensland (Inala), South Australia (Adelaide), Western Australia (East Perth), New South Wales (Airds, Broken Hill, Chittaway Point, Forster) and the Northern Territory (Alice Springs, Darwin).11 The protocol was developed in accordance with the principles of reciprocity, respect, equality, responsibility, survival and protection, and spirit and integrity.10,12 This article reflects a long‐standing program of work initiated by an Aboriginal Australian researcher.7,8,9 Two Aboriginal Australian researchers were senior investigators on this project and provided oversight and governance for all aspects of the study and publications. Both are authors of this article, and the investigator who developed the Aboriginal developed items and adapted the PHQ‐9 was invited to be the first author.

Individuals were eligible to participate if they were ≥ 18 years of age, self‐identified as Aboriginal and/or Torres Strait Islander, and were able to provide informed consent and answer questionnaire and interview questions. People with a diagnosis of psychosis or bipolar disorder were excluded. Trained staff at each study service screened everyone attending on recruitment days and approached eligible people to invite them to participate. If they were willing, staff obtained written or verbal informed consent. Recruitment for the study occurred between 25 March 2015 and 2 November 2016. The study was coordinated from The George Institute for Global Health (Sydney, Australia). This study is reported according to the STARD guideline for diagnostic accuracy studies (Supporting Information, table 1).

Procedures

The study protocol has been reported elsewhere.10 In short, trained, culturally competent staff from each health service (Supporting Information, appendix 1) completed all assessments using face‐to‐face interviews, or telephone interviews if required. Interviews were conducted in English or in the participant's Aboriginal language (when assessments were conducted in a particular language, staff members present were competent in that language). Participants first answered the aPHQ‐9, Aboriginal culturally specific depression items,10 demographic questions, and questions about their experience of completing the questionnaires. Within a week of completing the aPHQ‐9, participants were interviewed using the major depressive episode/disorder, generalised anxiety disorder and post‐traumatic stress disorder modules of the Mini International Neuropsychiatric Interview (MINI) 6.0.0.13 The MINI interview was conducted by a second, local, trained, culturally competent member of staff who was unaware of participants’ aPHQ‐9 responses. The MINI is the most widely used structured psychiatric diagnostic interview instrument globally, and is validated in more than 100 countries with many different cultures. For reasons including insufficient trained personnel, it was not feasible to use the ultimate reference standard method of diagnosis of depression by an experienced culturally safe mental health clinician. For this analysis, participants were classified as meeting the criteria for “current major depressive episode” or “no current major depressive episode”.

Statistical methods

Participants who completed the aPHQ‐9 plus Aboriginal culturally specific depression items and the MINI within seven days of each other were included in the analysis. For participants who did not answer one aPHQ‐9 question, the answered items were summed and multiplied by 9/8 to rescale the sum and obtain the final score. The aPHQ‐9 identified those “at risk of depression” based on a total score of ≥ 10. The Aboriginal culturally specific depression item responses were scored as 0 (no), 1 (a little bit, a little more, or sometimes) or 2 (a lot, a lot more, or most of the time), creating a total score of between 0 and 14. To determine the relationships between Aboriginal culturally specific depression items and a MINI diagnosis of a major depressive episode, χ² tests were used. Items that showed statistically significant differences between depressed and non‐depressed participants (P < 0.001 on the χ² test) were entered into a stepwise discriminant function analysis (using the Wilks lambda method; probability of F as criteria for entry and removal were 0.05 and 0.10, respectively) to determine their combined ability to identify a major depressive episode. To explore the possibility of non‐linear associations, decision tree analysis was undertaken to determine whether using a Boolean logic‐based model would enhance the sensitivity and specificity of the aPHQ‐9 or the Aboriginal culturally specific depression items alone in identifying participants who have a major depressive episode (Supporting Information, appendix 2). The sensitivity and specificity of the scale developed from this analysis was compared with that for the aPHQ‐9 using generalised estimating equations, taking into account clustering, disaggregated by sex. Diagnostic odds ratios were calculated using the sensitivity and specificity as calculated using generalised estimating equation models. Analyses were undertaken using SPSS version 28.0 (IBM).

Results

A total of 500 participants completed all study assessments (including three participants who each did not answer one aPHQ‐9 question), and their characteristics are shown in the Supporting Information, table 2. Participants’ mean age was 43 years (standard deviation [SD], 15), 267 (53%) were female and 300 (60%) were the main income earner in their household. Full details of the sample are reported elsewhere.11 Just under half (225 [45%]) reported a previous diagnosis of depression and just over two‐thirds (345 [69%]) had been diagnosed with at least one chronic illness (eg, diabetes). The prevalence of a MINI‐derived major depressive episode was 22% (95% CI, 18–25% [108 participants]). Language during the interview was English only for 442 participants (89%), English and an Aboriginal language for 19 participants (4%), and an Aboriginal language only for 33 (7%) participants.

The distribution of answers to the seven Aboriginal culturally specific depression items, according to whether participants were diagnosed with a major depressive episode or not, is shown in Box 1. More than half of the participants endorsed (scored 1 or 2) all items except the items relating to using more substances than usual (alcohol and marijuana) and feeling homesick or lonely. More than a quarter of all participants (and 65/108 [60%] of those with a current major depressive episode) indicated that they think they worry too much or a lot of the time. All items showed significant differences (P < 0.001) between depressed and non‐depressed participants. The seven Aboriginal culturally specific depression items showed good internal consistency (Cronbach α = 0.83) with only the two worry items showing high inter‐item correlations (ρ = 0.71) and exploratory factor analysis indicating that only one factor was present. The summed score for the Aboriginal culturally specific depression items was positively correlated with the aPHQ‐9 total score (ρ = 0.76; Box 2) and was significantly higher for depressed respondents (mean, 6.5 [SD, 4.6] v mean, 14.8 [SD, 5.1]; P < 0.001).

Using the stepwise entry method, four of the seven Aboriginal culturally specific depression items entered the discriminant function: feeling spirit was weak (item 1), drinking more grog or smoking more marijuana (item 2), feeling anger build up (item 4) and having too much worry (item 5). The four‐item scale was significantly predictive of a major depressive episode (P < 0.001) and correctly classified 425 participants (85%) as depressed or not depressed. Using decision tree analysis, three items discriminated between depressed and non‐depressed participants: feeling spirit was weak (item 1), feeling anger build up (item 4) and having too much worry (item 5). These items also correctly classified 425 participants (85%). Examination of the derived decision tree indicated that the items were acting as a simple summative score. The three items used to identify depression in both analyses were used to derive a screening score. Two cut‐points (≥ 3 and ≥ 4) were compared with each other, the aPHQ‐9 (≥ 10 cut‐point) and the MINI major depressive episode diagnosis. The sensitivity, specificity, positive likelihood ratio, negative likelihood ratio and diagnostic ratio for predicting a major depressive episode diagnosis by the MINI were calculated (Box 3). This showed that there was little difference between the discriminatory capability of the three‐item Aboriginal culturally specific depression scale and that of the aPHQ‐9. It also showed that the score generated by the spirit, worry and anger items, using a cut point of ≥ 3, had the best diagnostic properties.

Discussion

Our data indicate that the seven depression items we studied, which were developed by Aboriginal and Torres Strait Islander people, provide a reliable and valid assessment of depressive symptoms. In addition, three of the seven items can be summed to provide a sensitive and specific screening scale for a major depressive episode, equivalent to using the full version of the aPHQ‐9. Despite these Aboriginal developed items being developed in partnership with Aboriginal men, they appear to be applicable to the women who participated in the study. The sensitivity and specificity of this three‐item scale is comparable to shorter versions of the PHQ‐9 in primary care. For example, in a sample of more than 2600 individuals in New Zealand primary care, a score of 2 or higher on the PHQ‐2 had sensitivity and specificity of 61% and 92%, respectively.14 The items relating to worry in Western psychology contexts are more typically associated with anxiety, being a cardinal symptom of generalised anxiety disorder in diagnostic manuals;2 they are not listed as a diagnostic symptom of depression and not included in the list of 52 symptoms used in the most commonly used depression instruments.4 In this context, it may be that too much worry is a result of the experience of multiple daily stressors and inequalities in social determinants (like poverty, housing, humbug, grief and loss, and racism) that are disproportionately experienced by Aboriginal and Torres Strait Islander people.15

The item enquiring about whether the participant feels their spirit was weak is not present in widely used measures of depression4,5 or other measures previously adapted or developed by and for Aboriginal and Torres Strait Islander peoples.16,17,18 In the original studies with Aboriginal men, spirit or “Kurunpa goes beyond metaphor; it is not only a feeling, or a means of expressing distress: it is the vessel of life force itself”.9,11 No definition of spirit was provided to participants in this study, so there may be questions about the universality of interpretation of this question. However, this item may reflect a core or cardinal experience of depression for Aboriginal and Torres Strait Islander people, and possibly for other First Nations peoples (or other non‐Western cultures); this is a key question that would be valuable to explore. Even when all the aPHQ‐9 items and Aboriginal culturally specific depression items are entered simultaneously, feeling that your spirit is weak “a little bit” or “a lot” is the strongest single discriminator of depression with item specificity of 94% and sensitivity of 49%. The importance of a weak spirit as a cardinal symptom of depression is also indicated in item 2 on the aPHQ‐9 — “spirit was sad”. It is also important to consider how these core symptoms would be engaged with through cultural and/or therapeutic modalities offered to Aboriginal and Torres Strait Islander people diagnosed with depression. Few therapeutic modalities for depression engage with issues of spirit, instead focusing on behaviour and cognition as the key drivers that shape affect.

While our study was adequately powered for establishing diagnostic test accuracy, the sample is not representative of the diversity of cultures of Aboriginal and Torres Strait Islander peoples and the sample may reflect a greater representation of older people than would be expected in a representative sample. Participants were from ten sites across six states and territories, which represents only a very small proportion of the more than 250 Aboriginal language groups in Australia. We need to confirm how these developed depressive items are understood and relate to the experience of depression in other Aboriginal and Torres Strait Islander language and cultural groups. Also, we only have a one‐off assessment with the seven Aboriginal and Torres Strait Islander items. Thus, we do not have data on test–retest reliability, sensitivity to change, or responsiveness to change. Additional study with the three and seven Aboriginal culturally specific depression items is needed.

Further, we used the MINI13 as the reference standard for the diagnosis of a major depressive episode in Aboriginal and Torres Strait Islander people. Arguably, the gold standard reference should be a semi‐structured clinical interview undertaken by an experienced, culturally safe consultant psychiatrist, or highly trained mental health clinician. In our study, the MINI was delivered by local clinicians sensitive to local language and expressions and context but the resources (personnel, time, funds) for more senior clinicians were unavailable to the study team. Also, even if these resources were available, they would still not address the underlying issue of whether there is a qualitatively different experience of life for those diagnosed with major depression compared with those who report depressive symptoms below the diagnostic threshold, or whether diagnosis represents an arbitrary cut‐point on a linear scale. We acknowledge all these limitations, and that some of these issues may not be resolvable with respect to the current progress in this field. A diagnosis of depression is required for access to pharmaceutical, interpersonal and other management strategies for depression and associated social and emotional wellbeing difficulties. Thus, using a brief diagnostic interview represents a pragmatic standard against which we could determine the utility of the Aboriginal and Torres Strait Islander‐developed depressive symptom screening scale.

Despite the limitations of our study, the data indicate that clinicians should consider the Aboriginal culturally specific depression items during discussions about depression. Furthermore, when Aboriginal and Torres Strait Islander people report feeling a weak spirit, anger and excessive worry, this should warrant further formal assessment that incorporates clinical and cultural perspectives.

Data sharing statement

In line with Indigenous data sovereignty and Aboriginal and Torres Strait Islander ethical approvals, no data sharing is available from this study.

Box 2 – Scatter plot of the 9‐item adapted Patient Health Questionnaire (aPHQ‐9) total scores versus the sums of the seven Aboriginal culturally specific depression items

---