Journal of Traumatic Stress Disorders & Treatment ISSN: 2324-8947

All submissions of the EM system will be redirected to Online Manuscript Submission System. Authors are requested to submit articles directly to Online Manuscript Submission System of respective journal.

Research Article, J Trauma Stress Disor Treat Vol: 5 Issue: 1

Diagnostic Accuracy and Operating Characteristics of the Posttraumatic Stress Disorder (PTSD) Checklist in the Postearthquake Population in Armenia

Ani Movsisyan1*, Anahit Demirchyan1, Vahe Khachadourian2, Haroutune K Armenian2, Marie Diener-West3 and Armen Goenjian4
1School of Public Health, American University of Armenia (AUA), Armenia
2Department of Epidemiology, Fielding School of Public Health, University of California, Los Angeles (UCLA), USA
3Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, USA
4Department of Psychiatry and Biobehavioral Sciences, UCLA/Duke University National Centre for Child Traumatic Stress, USA
Corresponding author : Ani Movsisyan
School of Public Health, American University of Armenia (AUA), 40 Marshal Baghramian Avenue, Yerevan 0019, Armenia
Tel: (+374 10) 32 40 40; Fax: (+374 60) 61 25 12
E- mail: [email protected]
Received: December 16, 2015 Accepted: February 16, 2016 Published: February 29, 2016
Citation: Movsisyan A, Demirchyan A, Khachadourian V, Armenian HK, Diener-West M, et al. (2016) Diagnostic Accuracy and Operating Characteristics of the Posttraumatic Stress Disorder (PTSD) Checklist in the Post-earthquake Population in Armenia. J Trauma Stress Disor Treat 5:1. doi:10.4172/2324-8947.1000154

Abstract

Diagnostic Accuracy and Operating Characteristics of the Posttraumatic Stress Disorder (PTSD) Checklist in the Postearthquake Population in Armenia

The objective of this study was to determine the diagnostic accuracy and the operating characteristics of the Posttraumatic Stress Disorder Checklist Civilian (PCL-C) in the post-earthquake population of Armenia. A community sample of 126 participants witnessing the 1988 Spitak earthquake were administered the Armenian-language version PCL-C and the Structured Clinical Interview for DSM-IV disorders (SCID). Receiver operating characteristics curve (ROC) analysis was used to estimate the diagnostic accuracy of the instrument. PCL-C exhibited overall high diagnostic accuracy in the study population (area under the ROC curve was calculated as 0.903, SE=0.026). The cut score of 50 was identified as the most optimal and the most efficient based on the Youden index with equal values placed on sensitivity (0.94; 95% CI=0.80–0.99) and specificity (0.81; 95% CI=0.71–0.88), and a base rate of 26.2%. However, the higher cut score of 54 yielded the prevalence estimate equivalent to the base rate. The study results render the Armenian-language version PCL-C as an accurate and valid instrument to screen for PTSD symptoms in the post-earthquake population. This paper highlights the need for future investigators to study the PCL-C operating characteristics in broader population subgroups in Armenia.

Keywords: Posttraumatic stress disorder (PTSD); PTSD Checklist Civilian version (PCL-C); Earthquake, DSM-IV disorders; SCID; Diagnostic accuracy

Keywords

Posttraumatic stress disorder (PTSD); PTSD Checklist Civilian version (PCL-C); Earthquake, DSM-IV disorders; SCID; Diagnostic accuracy

Introduction

Posttraumatic Stress Disorder (PTSD) is one of the most commonly studied and central psychopathologies in the aftermath of disasters [1-3]. It has been estimated that approximately 162 million people were affected by disasters in 2005 worldwide. In 2010, those estimates were reported to have reached more than 330 million people [4]. Post-disaster PTSD prevalence is estimated to range between 30–40% among the direct victims, 10–20% among rescue workers, and 5–10% in the general population [2,4,5]. Although PTSD symptoms have the potential to subside over time [6], there is substantial evidence on the persistence of posttraumatic symptoms and high rates of related co-morbidity decades (e.g. 30 to 40 years) after exposure to a traumatic event [7], especially when left untreated [8-10].
Considering the vast short-term and long-term consequences of PTSD [11,12], the routine screening of PTSD symptoms becomes important. Currently, one of the most frequently used self-report screening tests of PTSD symptoms is the PTSD Checklist (PCL) [13]. The PCL checklist is widely used for clinical screening of PTSD symptoms [14], and to determine PTSD prevalence rates in large epidemiologic investigations [15]. Since its introduction, PCL has been tested for diagnostic accuracy on different population groups and subgroups exhibiting considerable variation in terms of optimal cut scores [16]. The most recent critical review of diagnostic accuracy studies of PCL provides a rigorous examination of various factors accounting for the variation in the PCL operating characteristics and emphasizes the need to further expand the diagnostic accuracy studies of this screening test [16]. This review highlights that the operating characteristics of PCL, including sensitivity and specificity values, are not fixed across settings and might be amenable to various spectrum effects (i.e. demographics, disease severity, comorbidity and other sample characteristics).
Diagnostic accuracy is an indicator of the performance of a screening test and is measured by its agreement with a reference or “gold” standard [17]. The abovementioned critical review by McDonald & Calhoun identified 18 studies of diagnostic accuracy of PCL [16]. Our complementary literature search using the search terms for PTSD Checklist (PCL) and diagnostic accuracy (criterion validity) in PsycINFO, MEDLINE, EMBASE, and CINAHL retrieved additional 9 recently published studies of the PCL diagnostic accuracy that used a structured interview as the reference standard. The characteristics of these studies, including the information on the investigated subgroups are summarized in Table 1.
Table 1: Studies of the PCL diagnostic accuracy: Operating Characteristics.
The first investigation of the PCL diagnostic accuracy originally suggested a cut score of 50, which yielded sensitivity and specificity of 0.82 and 0.83, respectively [13]. However, further diagnostic accuracy studies of PCL recommended cut scores ranging from 29 [18] to as high as 66 in various populations [19]. Large disparity in the suggested cut scores is observed even across similar populations, such as primary care veteran patients [20-27], active military personnel and general community samples [28-30]. One of the recent studies suggests that the optimal cut scores of PCL might vary even between the genders in the same population [30]. This supports the argument that reliance on “conventional” cut scores of the PCL checklist might result in its misapplication [16].
No evidence is currently available on the PCL operating characteristics for the Armenian population. As shown in Table 1, most of the diagnostic accuracy studies of PCL have been conducted on various population groups in high-income settings with very limited information regarding its use in low and middle-income countries (LMIC). This is in line with the findings showing overrepresentation of high-income countries (HIC) in traumatic stress research [31,32]. Meanwhile, the pooled estimate of the lifetime prevalence for common mental disorders in LMIC is reported as high as 22.7% (95% CI=17.4–29.0%) [33] with a significant burden of unmet mental health needs, including trauma-related disorders [34]. It has also been estimated that the risks of experiencing a potentially traumatic event and developing mental health disorder are higher in LMIC as a result of higher rates of poverty, social exclusion and experiences of loss [35].
Since the 1988 Spitak Earthquake that struck the northern part of Armenia, with a magnitude of 6.9 on the Richter scale, causing over 25,000 deaths and leaving 500,000 people homeless, a number of psychopathological investigations have been conducted in the region with multiple follow-ups that found persistently high rates of PTSD symptoms among the survivors [36-40]. The Armenian-language version of PCL Civilian (PCL-C) was employed in 2012 follow-up investigation of 725 survivors [40,41]. Even though the Armenian-language version of PCL-C was found to have excellent psychometric properties [42], because of the lack of evidence on the operating characteristics of this measure for the Armenian population, the authors had to rely on the “conventional” cut scores (≥ 50) to estimate the prevalence of PTSD symptoms in this population. This study aims to fill in this gap and investigate the diagnostic accuracy and operating characteristics of PCL-C in the Armenian population, specifically in the post-earthquake population. Additionally, it will explore whether PCL-C differs in its accuracy across population subgroups defined by age and gender.

Materials and Methods

Participants
To ensure the comparability of the derived findings with previous investigations of the post-earthquake cohort [36-38,41,43,44], this study convened a new community sample of participants aged 39 years and older who were living in the city of Gyumri during 1988 Spitak earthquake. Gyumri was the largest urban area destroyed by the 1988 Spitak earthquake, and the vast majority of participants in previous post-earthquake investigations were recruited from this city. Sample size calculation was based on the formula for the calculation of standard error (SE) of the area under the receiver operating characteristic (ROC) curve suggested by Hanley & McNeil as a single global metric for assessing the diagnostic accuracy of a test [45]. With anticipated 25% estimated prevalence rate of PTSD symptoms based on previous investigation of this population and considering the evidence on the persistence of untreated PTSD over decades [10,36,46], an estimate of 0.88 for the area under the ROC curve (AUC), and after adjusting for the estimated homogeneity effect (1.2) due to the cluster sampling design, the sample size of 126 was calculated to yield an SE value not greater than 0.046.
Procedure
The study followed a cross-sectional design. For the recruitment of the study participants, a two-stage cluster sampling strategy was applied. During the first stage, clusters (electoral districts) were randomly chosen from the total of 81 geographically subdivided electoral districts of Gyumri city applying probability proportional to size systematic random sampling. Subsequently, starting addresses for the household visits in each selected district were randomly chosen using election lists of the Republic of Armenia. These lists included all the households in the city of Gyumri grouped in electoral districts. With the calculated sample size of 126 and the chosen cluster size of 9, 14 clusters were included in the study.
Two interviewers organized the data collection through household visits. In each cluster, the first attempt to recruit a study participant was conducted in the starting address. If the attempt was successful, the next recruitment attempt was initiated in the fifth household to the right/up from the prior one, while in the case of unsuccessful attempt the next household to the right/up was visited, until nine interviews per cluster were completed. Only one individual per household was included in this study (selected randomly among eligible household members if more than one). These measures assured sufficient diversity of the surveyed population within a cluster. For those consenting to participate, the assessment took place on the same day and was organized in two sessions with 10-15 min. breaks in between: first, one of the interviewers gave instructions and kept track of the self-administration of PCL-C to assure the completeness of the items. Next, the second interviewer (a graduate in psychology), who remained masked to the PCL-C completions results, rated the participant using SCID (Structured Clinical Interview for DSM Disorders) diagnostic interview [47]. Data collection took place in March 2013. The Institutional Review Board of the American University of Armenia reviewed and approved the study protocol.
Measures
The PTSD-Checklist Civilian (PCL-C) is a 17-item scale corresponding to the symptoms of PTSD defined by the DSMIV (Diagnostic and Statistical Manual of Mental Disorders) [6]. Respondents identify the extent to which they have experienced each symptom in the past 30 days using a five-point ordinal scale (ranging from “Not at all” to “Extremely”). Different scoring approaches can be applied to yield either a continuous measure of PTSD symptom severity (17–85) or a dichotomous indicator of diagnostic status [48]. The latter can include either an overall cut score or a symptom cluster approach coinciding with the DSM-IV criteria B–at least one re-experiencing symptom of the five, C–at least three avoidance/numbing symptoms of the seven, and D–at least two arousal symptoms of the five. The checklist does not assess the traumatic event in detail or the feelings of fear/helplessness or horror, which correspond to the Criteria A1 and A2 of PTSD diagnosis. Considering the time interval of 25 years between the index event of the 1988 earthquake and this assessment, and the potential of PTSD symptoms to subside over time [49], this study used SCID and PCL-C to capture the impact of any lifetime traumatic event, not necessarily the earthquake. The Armenian-language version of PCL-C has been finalized after several rounds of forward and backward translations and subsequently assessed for psychometric properties in the recent follow-up investigation of the post-earthquake cohort [42].
This study used the Structured Clinical Interview for DSMIV Disorders (SCID) Research Version Non-Patient edition as the “reference” standard for PTSD assessment. SCID is a semi-structured interview, a generally accepted instrument for making DSM-IV diagnoses [47]. It has been widely used to assess the operating characteristics of PCL (Table 1). Two members of the research team (AM and VK) translated the interview questions for the PTSD module of SCID into Armenian with several rounds of forward and backward translations until full correspondence between the translation and the original version was achieved. This was then pretested on a convenience sample of 10 individuals from the general population. SCID was administered by a graduate in psychology (AM), who underwent specific didactic and interview training series in SCID administration offered by the Biometrics Research Department of the Columbia University Medical Center [50].
Statistical analyses
Data were analyzed using Stata version 12 and MedCalc statistical softwares. Operating characteristics of the instruments including sensitivity (proportion of those with the disorder correctly identified by the test), specificity (proportion of those without the disorder correctly identified by the test), positive predictive value (proportion of those having the disorder from those testing positive), negative predictive value (proportion of those not having the disease from those testing negative), likelihood ratios (how much a test result changes the odds of having the disease), diagnostic efficiency (proportion of those correctly identified by the test), the base rate (true prevalence rate measured via the reference standard), and the Youden index J (estimated as [Sensitivity +Specificity-1] at any observed score) were calculated for PCL-C in comparison with SCID. The optimal cut score was chosen based on the point corresponding to the maximum value of the Youden index. Bias-corrected and accelerated bootstrapping (BCa, with 10000 iterations) was used to calculate 95% confidence intervals for Youden indices [51]. Weighted kappa coefficients (Κ (0.5)) were calculated to measure the level of agreement of PCL-C with SCID providing equal values to sensitivity and specificity (Κ (0.5)). Values for kappa were summarized according to the Landis & Koch magnitude categorization [52]. Internal consistency of the scales was calculated based on Cronbach’s α statistic. A nonparametric receiver operating characteristic (ROC) curve analysis was conducted to determine the AUC estimates. The nonparametric choice was justified by skewed test values [53] for PCL-C in the diseased group (skewness=0.58, kurtosis=2.45). Standard errors for the AUCs were calculated in accordance with Hanley & McNeil’s approach. In addition, separate ROC curves were generated for gender and age groups. Age was dichotomized at the mean of the sample (<56 years old and ≥ 56 years old). Differences in test performance (based on AUC comparison) between the demographic groups were statistically compared using the method suggested by Delong and colleagues [54]. Cases with three or less missing values on PCL-C were maintained in the analysis with the missing values replaced by the average value of all answered items (overall, there were 7 cases with three or less missing values). None of the cases had more than three missing values.

Results

Descriptive Analysis of the Sample
A total of 570 household visits were made of which 277 resulted in either no one at home or an unoccupied house. Out of the remaining 293 households contacted, 75 refused to participate without providing information about the eligible participants in the household, and 60 households did not have eligible participants. Overall, 244 eligible participants were found from 158 households. Of those, 126 completed the assessments (52%), and 118 (48%) selected participants refused or were unable to participate due to various reasons (i.e. not at home, busy at home, occupied with children, not in the mood, and poor health conditions).
Table 2 shows the demographic characteristics, the SCID-based prevalence rates of PTSD, and the PCL-C mean scores by gender. All the participants were Armenians. The majority of the sample was female (75.4%), married (66.7%), with a university (34.1%) or professional/technical (30.2%) education, and self-reported average (34.9%) or substantially below average (38.1%) standard of living. The mean age of the participants was 55.7 years. Significantly higher proportion of females was unemployed and retired compared to males. There was no significant between-gender difference in age, education level, marital status or family’s general standard of living. The mean PCL-C score was significantly higher for females. The results of SCID revealed that 33 out of the 126 screened (26.2%) met the criteria for current PTSD diagnosis (Criteria A, B, C, D, E). A significantly higher proportion of females had a PTSD diagnosis compared to males (32.6% vs. 6.5%, respectively). Of those diagnosed with current PTSD, the most commonly reported trauma that participants perceived as causing their PTSD symptoms was 1988 earthquake-related trauma (e.g. loss of children, loss of shelter, etc.; n=21; 63.6%), followed by a sudden loss of a close person that was not related to the earthquake (n=9; 27.3%).
Table 2: Descriptive analysis of the study sample by gender.
The mean score on PCL-C for the entire sample was 44.4 (SD=12.8). Participants with SCID-based current PTSD had a mean score on PCL-C of 57.1 (SD=6.6), which was significantly (t=10.4; p<0.005) higher than the mean score of 39.9 (SD=11.4) of participants with no PTSD diagnosis. Only 5.6% (7/126) of the entire sample reported ever seeing a mental health specialist. Of those diagnosed meeting the criteria for current PTSD, only 3 (9.1%) participants reported ever seeing a mental health specialist.
Operating Characteristics of the Armenian-Language PCL-C
Internal consistency of the Armenian-language version PCL-C total scale was good with Cronbach’s α 0.861 (95% CI=0.825–0.894). Internal consistencies for PCL-C clusters B, C, D were acceptable (α=0.807, 95% CI=0.749–0.850; α=0.724, 95% CI=0.644–0.790; α=0.737, 95% CI=0.658–0.803, respectively). Figure 1 depicts the ROC curve for PCL-C compared with the diagnosis of current PTSD based on SCID (Criteria A, B, C, D, E). PCL-C performed well in the study population, resulting in an AUC of 0.903 (SE=0.026; asymptotic 95% CI=0.852–0.953, z=15.58, p<0.005). Table 3 presents the operating characteristics for PCL-C. The findings indicated that PCL-C cut score of 50 resulted in the highest diagnostic efficiency (0.84) with a sensitivity of 0.94 (95% CI=0.80–0.99) and a specificity of 0.81 (95% CI=0.71–0.88) at the SCID base rate of 26.2%. The cut score of 50 corresponded to the highest point of Youden index (J=0.75, 95% CI=0.63–0.83) and the highest agreement after adjusting for chance agreement (Κ (0.5)=0.65, 95% CI=0.51–0.78). Using the cut score of 50, the estimated PTSD prevalence was 38.9%. The latter overestimated the base rate by 12.7%. Moving the cut score to 54 resulted in a prevalence estimate equivalent to the base rate (i.e. 26.2%). When assessing PTSD status using the symptom clusterbased approach for PCL-C, the agreement with SCID was calculated to be moderate for the avoidance/numbing cluster and slight for the re-experiencing and the hyperarousal clusters (Table 4). The AUC estimate for people aged below 56 was calculated as 0.883 (95% CI=0.803–0.963) and 0.918 (95% CI=0.852–0.984) for those aged 56 and above. The AUC estimates were 0.914 (95% CI=0.816–1.000) and 0.911 (95% CI=0.855–0.966) for males and females, respectively. Hence, no significant differences in the performance of PCL-C were found as a function of age (χ2=0.45, p=0.50) or gender (χ2=0.00, p=0.94).
Figure 1: Receiver operating characteristics (ROC) curve for the Armenian-language version PCL-C calculated against SCID.
Table 3: Operating characteristics of the Armenian-language version PCL-C by cut score.
Table 4: Operating characteristics of the Armenian-language version PCL-C relative to SCID for each of the three symptom clusters.

Discussion

This paper described the diagnostic accuracy and the operating characteristics of the Armenian-language PCL-C. The findings support the use of this instrument as an accurate self-report measure for PTSD symptom identification among the 1988 Spitak earthquake survivors in Armenia. Based on the present data, the recommended cut score of 50 [13] was shown to be optimal and the most efficient with a sensitivity of 0.94, a specificity of 0.81 and a base rate of 26.2%. The extant literature on the operating characteristics of PCL-C reveals that the cut score of 50 was shown to be optimal in various other populations (Table 1).
However, in the present study, this cut score overestimated the SCID-based prevalence rate of PTSD by 12.7%. This finding is consistent with the notion discussed by Terhakopian et al. [55], according to which the ability of PCL to accurately estimate the prevalence of a condition in a given population is strongly affected by the true prevalence (base rate) of the condition in that population.
In the present study, shifting the cut score to 54 yielded the closest estimate to the SCID-based prevalence of 26.2%. Consequently, the cut score that a researcher chooses to apply will depend upon the study aims and objectives. Since the base rate in the population highly affects the estimated prevalence and the diagnostic efficiency of PCL [16], researchers might want to choose cut scores other than 50 in order to obtain closer estimates to the true prevalence rate, if prevalence estimation is the major priority in the study.
Consistent with a number of international studies [16,28,56,57], the agreement between the self-administered PCL-C and SCID was substantial when using a dichotomized sum score of 50. However, when using a symptom cluster-based approach, the Armenianlanguage version PCL-C demonstrated lower agreement with SCID. This is consistent with recent investigations of PCL suggesting the superiority of the score-based approach over the symptom clusterbased approach in achieving higher agreement with a reference standard [28,58].
Although this study aimed to detect the impact of any lifetime traumatic event as a cause of PTSD, the majority of those meeting the criteria for current PTSD reported the 1988 earthquake-related trauma. The revealed prevalence rate of current PTSD (26.2%) in this sample is broadly in line with the previous investigations of the post-earthquake population in Armenia. Several studies conducted immediately after the earthquake revealed very high prevalence rates of PTSD that ranged from 65% to 95% in various subgroups [38,39,59], while the investigation conducted two years after the earthquake found that 49% of the population met the criteria for PTSD [36]. It is, however, worth noting that these investigations used different instruments to measure PTSD, which might undermine the comparability of these estimates. The 2012 follow-up investigation of the same cohort that used PCL-C reported a mean scale score of 36.3 [42], below the mean score of the present sample (44.4). The persistence of PTSD symptoms in this population was previously attributed to the severity of the disaster-related losses and the ongoing social and economic hardships causing high levels of stress and adversely impacting the quality of life of this population [38,41]. The persistence of the PTSD symptoms can also be explained by the poor access to adequate mental health services; as this study found, only 5.6% (7/126) of the entire sample reported ever seeing a mental health specialist. These findings are broadly in line with the investigations of long-term consequences of untreated PTSD reporting 30% to 83.7% prevalence rates of current PTSD in various population groups over two to four decades following the traumatic event [8-10].
This study has several limitations. Even though it aimed to recruit a representative sample of the population that had witnessed the 1988-earthquake, the majority of the included participants were married and unemployed women, who were more likely to be at home during the household visits. A vast number of selected eligible men were unavailable because they were either at work or were migrant workers out of country. The under-representativeness of the employed men and women in the sample might thus compromise the generalizability of the study findings. Moreover, given this study was primarily designed to assess the operating characteristics of PCL-C in the post-earthquake population of Armenia, the findings might not be generalizable to other subgroups of the Armenian population (i.e. other age groups or trauma types). Another limitation of this study relates to the use of the Armenian-translated version of SCID. Given SCID was administered by one interviewer in this study, it was not possible to assess the measure for inter-rater reliability, which might somewhat undermine the value of SCID as a reference standard. Nevertheless, all the semi-structured questions of SCID were subjected to many rounds of backward and forward translations, and the SCID administrator was comprehensively trained prior to applying the measure in the field. Finally, it should be noted that while the sample size was adequate for the AUC estimation for the entire sample, it was not specifically powered to detect AUC differences between the genders and the age categories. The lack of significant differences in the AUC estimates between the genders and the age categories might be attributed to the underpowered sample size and therefore, should be considered only an exploratory finding.
In conclusion, despite the foregoing limitations, this study was designed and conducted in accordance with the Standards for Reporting of Diagnostic Accuracy (STARD) requirements addressing all the items highlighted in the reporting statement [17]. The findings from this study support the use of the Armenian-language PCL-C as an accurate screening measure for PTSD symptoms in the postearthquake population. Authors recommend that future investigations conduct larger-scale studies to explore the operating characteristics of PCL-C in other subgroups of the Armenian population.

Acknowledgments

We would like to express our thanks and appreciation to all the study participants for their time and contribution.

References







































































Track Your Manuscript