- Split View
-
Views
-
Cite
Cite
Kelly C. Byars, Stacey L. Simon, James Peugh, Dean W. Beebe, Validation of a Brief Insomnia Severity Measure in Youth Clinically Referred for Sleep Evaluation, Journal of Pediatric Psychology, Volume 42, Issue 4, May 2017, Pages 466–475, https://doi.org/10.1093/jpepsy/jsw077
- Share Icon Share
Abstract
ObjectivesEvaluate psychometric properties of the Pediatric Insomnia Severity Index (PISI), a brief measure of insomnia severity. Methods Clinically referred youth (n = 462; 283 males, 179 females, mean age = 7.28 ± 2.05 years) and their caregiver(s) completed sleep evaluation including the PISI, Children's Sleep Habits Questionnaire, and sleep disorders inventory for students. Tests of reliability and validity and confirmatory factor analysis (CFA) were conducted to assess PISI psychometric properties. Exploratory analyses were conducted to examine insomnia severity by insomnia diagnosis. Results Measures of internal consistency for the PISI factor scores varied. CFA indicated that a two-factor model had optimal fit relative to a single-factor solution. Overall, convergent and discriminant validity of PISI factors were supported. Insomnia severity varied by diagnosis. Conclusions Findings provide preliminary support for the reliability and validity of the PISI within a large pediatric sample and for its clinical utility as a brief measure of insomnia severity.
Introduction
Pediatric insomnia is common, tends to persist without treatment, and has considerable negative consequences (Byars & Simon, 2014; Byars, Yolton, Rausch, Lanphear, & Beebe, 2012; Pollock, 1994; Sadeh, Gruber, & Raviv, 2002). Pediatric sleep difficulties, including insomnia, are associated with academic and cognitive difficulties, internalizing and externalizing behavior problems, and risk for obesity (Chen, Beydoun, & Wang, 2008; Reid, Hong, & Wade, 2009; Touchette et al., 2007). The myriad of negative consequences associated with untreated pediatric sleep disorders has spurred efforts to improve assessment and treatment (Burnham, Goodlin-Jones, Gaylor, & Anders, 2002; Lozoff, Wolf, & Davis, 1985; Meltzer, Johnson, Crosette, Ramos, & Mindell, 2010). Empirically supported treatments for pediatric insomnia are available and efficacious (Byars & Simon, 2014; Meltzer & Mindell, 2014).
The hallmark symptoms of pediatric insomnia are difficulty falling asleep (i.e., sleep-onset problems) and/or difficulty staying asleep (i.e., sleep-maintenance problems) (Meltzer, 2010). Diagnostic nosology has specified clinical subtypes for primary insomnia that include etiologies more common in young children (e.g., behavioral insomnia of childhood) as well as older children and adolescents (e.g., psychophysiological insomnia) (American Academy of Sleep Medicine, 2005a). Behavioral insomnia of childhood is characterized by sleep-onset or -maintenance problems resulting from learned sleep associations and/or bedtime resistance/limit-setting problems (American Academy of Sleep Medicine, 2005a American Academy of Sleep Medicine). Psychophysiological insomnia is characterized by sleep disruption associated with heightened arousal and problematic learned sleep-preventing associations (American Academy of Sleep Medicine, 2005a).
Recent revision to the International Classification of Sleep Disorders diagnostic criteria for insomnia included consolidation of all insomnia diagnoses under a single chronic insomnia disorder, but sleep-onset and sleep-maintenance problems included in the general diagnostic criteria for insomnia have remained essentially unchanged since first publication of the International Classification of Sleep Disorders (ICSD; American Academy of Sleep Medicine, 2014; American Sleep Disorders Association Diagnostic Classification Steering Committee, 1990; Sateia, 2014). To date, the bulk of pediatric insomnia outcome research has demonstrated effectiveness of behavioral intervention for treating both sleep-onset and sleep-maintenance problems (Meltzer & Mindell, 2014; Morgenthaler et al., 2006). Thus, convergence of empirical evidence and clinical experience suggests that evidence-based assessment of pediatric insomnia should focus on sleep-onset and sleep-maintenance difficulties.
Despite advances in pediatric behavioral sleep medicine, sleep problems in youth often are not identified in primary care settings (Blunden et al., 2004; Meltzer et al., 2010). Lack of routine assessment of pediatric sleep disturbances may in part be because of unavailability of brief, easily administered/interpreted, and clinically valid tools for measuring insomnia severity in youth. Polysomnography is the “gold standard” for assessment of medically based sleep disorders (e.g., obstructive sleep apnea [OSA]), but does not directly measure insomnia and is not recommended as an assessment tool for behavioral sleep problems (Reite, Buysse, Reynolds, & Mendelson, 1995; Standards of Practice Committee of the American Sleep Disorders Association, 1995). Actigraphy can be used to estimate habitual sleep/wake patterns at home, but can be expensive, time intensive to score, and historically has had limited clinical use because of poor third-party reimbursement (McCrae, Taylor, Smith, & Perlis, 2010; Morgenthaler et al., 2007).
There are a number of evidence-based psychometrically sound parent- and child-report pediatric sleep measures (see Lewandowski, Toliver-Sokol, and Palermo, 2011, for comprehensive review). For example, the Children's Sleep Habits Questionnaire (CSHQ) is a well-established multidimensional parent-report measure of sleep in school-age children with strong psychometric support for assessing behavioral and medical sleep symptoms in children (e.g., bedtime resistance, sleep-disordered breathing, daytime sleepiness; Lewandowski et al., 2011; Owens, Spirito, & McGuinn, 2000). However, the time required to administer, score, and interpret this measure may be impractical for screening purposes in primary care settings, nor even as a measure of insomnia severity during routine follow-up of youth undergoing treatment for insomnia. Furthermore, the CSHQ was not specifically designed to measure insomnia symptoms. In fact, there are no insomnia-specific measures that can be used with children, and adult insomnia-specific measures (e.g., the Insomnia Symptoms Questionnaire [Okun et al., 2009]; Brief Insomnia Questionnaire [Kessler et al., 2010]; Insomnia Severity Index [Bastien, Vallieres, & Morin, 2001] cannot simply be generalized for use with youth [Lewandowski et al., 2011]).
Given the lack of insomnia-specific pediatric measures and the increasing need to document health care outcomes (Forrest & Silber, 2014), we developed the Pediatric Insomnia Severity Index (PISI). The PISI is a brief parent-proxy measure designed to efficiently monitor primary clinical symptoms of pediatric insomnia that are common treatment targets in the context of clinical care for youth (4–10 years) with sleep disturbance. The PISI consists of six items, which take <5 min to complete and score. This article examines, within the applied context of a comprehensive pediatric sleep disorders center, the psychometric properties of the PISI, including reliability and content, convergent, and discriminant validity.
Methods
Development of thePISI
The PISI was developed by a team of sleep specialists including two psychologists, four pulmonologists, a neurologist, and a pediatric nurse practitioner. The scale was developed to assess the hallmark features of pediatric insomnia while considering core measures of sleep continuity typically targeted during insomnia treatment. Group consensus was reached regarding the following ICSD-II general insomnia criteria that would guide item content: (1) difficulty falling asleep (PISI Items 1 and 2), (2) difficulty maintaining sleep (PISI Items 3 and 4), and (3) daytime impairment (PISI Item 5). In addition, prior insomnia research informed group consensus regarding selection of the following sleep continuity measurement domains that were included in the scale: (1) sleep-onset latency (PISI Item 1), (2) frequency of night wakings (PISI Item 3), and (3) sleep duration (PISI Item 6) (Perlis, Jungquist, Smith, & Posner, 2005). Subsequent to identifying foundational content domains, the team reviewed previously validated pediatric sleep measures to identify insomnia-specific items to guide item development for the brief insomnia severity measure. Group consensus was reached regarding six insomnia-specific items. PISI items cover some of the same ground as the CSHQ, but no items are identical. After the final PISI form was developed and approved by the clinical team, use in the clinic setting was started in early 2009. None of the six items originally included in the PISI was eliminated from the scale before initial clinical implementation.
Participants
All study procedures were approved and overseen by the hospital’s institutional review board. The sample included 462 youth and their caregivers, seen for behavioral sleep medicine evaluation within an accredited sleep disorders center located in a tertiary care pediatric hospital. Eligible families received print and verbal information explaining the purpose of the study. Before formal enrollment, primary caregivers provided written informed consent. Inclusion criteria were child aged 4–10 years and a clinical diagnosis of insomnia confirmed during a diagnostic sleep evaluation conducted by a licensed psychologist certified in behavioral sleep medicine or a clinical resident/fellow under the direct supervision of the sleep psychologist. During the enrollment period from June 2009 to March 2015, 482 patients were eligible for participation. In total, 11 parents did not provide consent, and nine families had incomplete data, resulting in a final sample of 462 subjects (96% of eligible subjects; 283 males, 179 females, mean = 7.28 years old, SD = 2.05 years). The majority self-identified as non-Hispanic White (78%), with remaining participants Black (11%), Hispanic (2%), Asian (1%), or Multiracial (8%). In total, 62% of parents were married, 21% unmarried, 10% divorced, 4% separated, 2% remarried, and 1% widowed. In total, 25% reported annual income <$20,000, 27% reported $20,000–49,000/year, 26% reported $50,000–99,000/year, and 22% >$100,000/year.
Procedures
Participants were referred to an accredited sleep disorders center and were triaged for evaluation based on referral question and parent-reported history gathered during a telephone intake interview. Patients had a chief complaint of insomnia and underwent a comprehensive sleep evaluation with a behavioral sleep medicine clinician. The diagnostic work-up included a clinical interview and battery of sleep screening instruments (see measures). Patients with symptoms suggestive of an organic sleep disorder (e.g., OSA) were also triaged for evaluation with a board-certified sleep physician.
Measures
Demographic Information
On an intake questionnaire, parents reported demographic information in a multiple choice format, and relevant health conditions in a yes/no format.
PISI
CSHQ
The CSHQ (Owens et al., 2000) is a 33-item parent-report measure of sleep behavior and sleep disorder symptoms developed for use with children 4–10 years old. Caregivers use a 3-point Likert scale to rate the frequency of specific sleep symptoms/behaviors during the previous week (rarely to usually). The measure yields a total score and eight subscale scores for which higher scores represent more problematic sleep. The CSHQ has demonstrated validity for use with preschool and school-aged children and has been shown to differentiate clinical from control groups (Goodlin-Jones, Sitnick, Tang, Liu, & Anders, 2008; Owens et al., 2000). Published psychometrics demonstrate adequate internal consistency (0.68–0.78) and test–retest reliability (0.62–0.79) for clinical and community comparison groups (Owens et al., 2000). A total score >41 appears to be the most sensitive clinical cutoff for identifying sleep problems in children (Owens et al., 2000). Primary caregivers completed the CSHQ.
Sleep Disorders Inventory for Students
The sleep disorders inventory for students (SDIS; Luginbuehl, Bradley-Klug, Ferron, Anderson, & Benbadis, 2008) is a paper-pencil parent-report measure screening for sleep disorders that have an organic basis, including OSA, narcolepsy, periodic limb movement disorder (PLMD), and delayed sleep phase syndrome (DSPS). Items are rated using a Likert scale ranging from 1 (the child never exhibits this behavior) to 7 (child exhibits behavior multiple times per hour daily or nightly). Factor scores for each sleep disorder domain and a total sleep disorder score are computed. T-scores of 65 and higher are considered to be suggestive of a high risk for a sleep disorder. The SDIS-Child version is 41 items and is used with children 2–10 years old. Comprehensive psychometrics on the instrument demonstrate adequate internal consistency, test–retest reliability, and validity (Luginbuehl et al., 2008). Primary caregivers completed the SDIS.
Sleep Diagnosis
ICSD-II (American Academy of Sleep Medicine, 2005a) sleep diagnoses were given by a behavioral sleep medicine clinician. Every subject included in the study met ICSD-II criteria for a primary insomnia diagnoses as follows: 107 participants (23%) were diagnosed with behavioral insomnia of childhood (BIC) sleep-onset association type, 111 (24%) with BIC limit-setting type, 141 (31%) with BIC combined type, and 103 (22%) with psychophysiological insomnia. In total, 29% of participants also met criteria for a secondary sleep disorder diagnosis, and 13% met criteria for three sleep disorder diagnoses.
Analysis Plan
This study examined the psychometric properties of a two-factor conceptual model reflecting primary clinical symptoms of pediatric insomnia (sleep-onset and sleep-maintenance problems) and relevant sleep continuity variables (sleep-onset latency, frequency of night wakings, and total sleep time). The factor domains were conceptualized as follows: (1) sleep-onset problems (aggregate of PISI Items 1, 2, and 6) and (2) sleep-maintenance problems (aggregate of PISI Items 3, 4, and 6). PISI factor scores were computed by summing item-level responses loading on each factor. One item (Item 5) was excluded from model testing because of poor inter-item correlation. In retrospect, Item 5 (sleepiness) was constructed to measure daytime consequences of pediatric insomnia, but was likely too narrowly defined in light of the broad range of secondary effects of pediatric insomnia (e.g., sleepiness, fatigue, attention, concentration or memory difficulty, impaired psychosocial, academic, or vocational functioning, mood and/or behavioral disturbance, sleepiness).
SPSS Statistics software version 23 and Mplus version 7.4 were used for data analysis. Missing data for the CSHQ and SDIS were typically the result of the omission of one or two items on a multi-item subscale. Capitalizing on the strong internal consistency of these measures, in cases where <50% of subscales items were missing, composites were prorated from the remaining items for the subscale (Beebe et al., 2007). If >50% of the item-level responses making up a subscale were missing, no data for the measure were included in the final data analysis. After proration, <2% of any given validation measure had missing data in the final sample of 462, all of whom had fully completed the PISI (to allow for a complete assessment of psychometrics for the PISI, cases were dropped in which the PISI was incomplete). Refer to Table 2 where n values are reported for completed study measures.
All statistical tests were evaluated at the p < .05 significance level and constituted two-tailed tests. Because the large sample raised the potential for statistically significant effects of dubious clinical relevance, we also examined effect sizes. Following convention, a correlation between .10 and .29 was considered a small effect, .30 and .49 a medium effect, and ≥.50 a large effect (Cohen, Cohen, West, & Aiken, 2003).
Reliability and Construct Validity of PISI
Internal consistency reliability of the PISI factor scores was examined through Cronbach’s α.
Confirmatory factor analysis (CFA) procedures were used to confirm superior model fit of a clinically relevant and conceptually derived two-factor model over a single-factor solution and to test for invariance across sex. Although we report χ2 for model fit, that statistic is considered overly sensitive in large samples, so we set a priori criteria for an acceptable fit as a root mean square error of approximation (RMSEA) ≤0.08 and a Comparative Fit Index (CFI) > 0.95 (Hu & Bentler, 1999). PISI item-level response data were treated as ordinal (ordered categorical), and CFA parameter estimates were obtained via a categorical response variable parameter estimation algorithm (i.e., WLSMV; see Muthen and Muthen [1998-2015]; p. 608). The DIFFTEST procedure in Mplus version 7.4 (Muthen and Muthen [1998-2015]; p. 450) was tested for the invariance of the proposed CFA model across gender. To avoid parameter estimation nonconvergence issues, thresholds for the five sleep questionnaire items were constrained to equality and the scale factors for the five sleep questionnaire items were fixed at unity between male and female participants (see Muthen and Muthen [1998-2015]; p. 685–686).
Convergent and Discriminant Validity of the PISI
Correlations were used to assess convergent validity (correlations with measures of conceptually linked constructs) and discriminant validity (lack of correlation with measures of conceptually distinct constructs). For parsimony, we focused on subscales with the clearest conceptual relationships. See Table I for a list of the relationships between the PISI and validation measures that were determined a priori to support convergent and discriminant validity. Finally, we assessed whether PISI factor scores varied by clinical diagnosis via an exploratory analysis of variance (ANOVA).
PISI . | ||
---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . |
Child Sleep Habits Questionnaire | ||
Bedtime resistance | Convergent | Discriminant |
Sleep-onset delay | Convergent | Discriminant |
Night waking | Discriminant | Convergent |
Parasomnias | Discriminant | Convergent |
SDB | Discriminant | Convergent |
Sleep disorders inventory for students–child | ||
OSA | Discriminant | Convergent |
PLMD | Discriminant | Convergent |
DSPS | Convergent | Discriminant |
PISI . | ||
---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . |
Child Sleep Habits Questionnaire | ||
Bedtime resistance | Convergent | Discriminant |
Sleep-onset delay | Convergent | Discriminant |
Night waking | Discriminant | Convergent |
Parasomnias | Discriminant | Convergent |
SDB | Discriminant | Convergent |
Sleep disorders inventory for students–child | ||
OSA | Discriminant | Convergent |
PLMD | Discriminant | Convergent |
DSPS | Convergent | Discriminant |
Note. Convergent validity would be supported if the PISI factor score displayed significant positive correlations with the sleep measure subscales hypothesized to be most conceptually consistent with that factor domain. This is indicated in the table with “Convergent.” Discriminant validity would be supported by minimal correlations between the PISI factor score and the sleep measure subscales hypothesized to be most conceptually distinct from that factor domain, indicated in the table with “Discriminant.”
DSPS = delayed sleep phase syndrome; OSA = obstructive sleep apnea; PISI = Pediatric Insomnia Severity Index; PLMD = periodic limb movement disorder; SDB = sleep disordered breathing.
PISI . | ||
---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . |
Child Sleep Habits Questionnaire | ||
Bedtime resistance | Convergent | Discriminant |
Sleep-onset delay | Convergent | Discriminant |
Night waking | Discriminant | Convergent |
Parasomnias | Discriminant | Convergent |
SDB | Discriminant | Convergent |
Sleep disorders inventory for students–child | ||
OSA | Discriminant | Convergent |
PLMD | Discriminant | Convergent |
DSPS | Convergent | Discriminant |
PISI . | ||
---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . |
Child Sleep Habits Questionnaire | ||
Bedtime resistance | Convergent | Discriminant |
Sleep-onset delay | Convergent | Discriminant |
Night waking | Discriminant | Convergent |
Parasomnias | Discriminant | Convergent |
SDB | Discriminant | Convergent |
Sleep disorders inventory for students–child | ||
OSA | Discriminant | Convergent |
PLMD | Discriminant | Convergent |
DSPS | Convergent | Discriminant |
Note. Convergent validity would be supported if the PISI factor score displayed significant positive correlations with the sleep measure subscales hypothesized to be most conceptually consistent with that factor domain. This is indicated in the table with “Convergent.” Discriminant validity would be supported by minimal correlations between the PISI factor score and the sleep measure subscales hypothesized to be most conceptually distinct from that factor domain, indicated in the table with “Discriminant.”
DSPS = delayed sleep phase syndrome; OSA = obstructive sleep apnea; PISI = Pediatric Insomnia Severity Index; PLMD = periodic limb movement disorder; SDB = sleep disordered breathing.
Pediatric Insomnia Severity Scale, raw scores (n = 462) | |
Factor scores | |
Sleep-onset problems | 9.41 ± 3.93 |
Sleep-maintenance problems | 8.30 ± 3.32 |
Child Sleep Habits Questionnaire, raw scores (n = 459) | |
Subscales | |
Bedtime resistance | 10.83 ± 3.22 |
Sleep-onset delay | 2.25 ± 0.83 |
Night waking | 6.52 ± 1.63 |
Parasomnias | 11.46 ± 2.61 |
Sleep-disordered breathing | 3.83 ± 1.28 |
Sleep disorders inventory for students–child version, T-score (n = 460) | |
Subscales | |
Obstructive sleep apnea | 57.53 ± 9.80 |
Periodic limb movement disorder | 60.49 ± 8.94 |
Delayed sleep phase syndrome | 63.50 ± 14.36 |
Pediatric Insomnia Severity Scale, raw scores (n = 462) | |
Factor scores | |
Sleep-onset problems | 9.41 ± 3.93 |
Sleep-maintenance problems | 8.30 ± 3.32 |
Child Sleep Habits Questionnaire, raw scores (n = 459) | |
Subscales | |
Bedtime resistance | 10.83 ± 3.22 |
Sleep-onset delay | 2.25 ± 0.83 |
Night waking | 6.52 ± 1.63 |
Parasomnias | 11.46 ± 2.61 |
Sleep-disordered breathing | 3.83 ± 1.28 |
Sleep disorders inventory for students–child version, T-score (n = 460) | |
Subscales | |
Obstructive sleep apnea | 57.53 ± 9.80 |
Periodic limb movement disorder | 60.49 ± 8.94 |
Delayed sleep phase syndrome | 63.50 ± 14.36 |
Pediatric Insomnia Severity Scale, raw scores (n = 462) | |
Factor scores | |
Sleep-onset problems | 9.41 ± 3.93 |
Sleep-maintenance problems | 8.30 ± 3.32 |
Child Sleep Habits Questionnaire, raw scores (n = 459) | |
Subscales | |
Bedtime resistance | 10.83 ± 3.22 |
Sleep-onset delay | 2.25 ± 0.83 |
Night waking | 6.52 ± 1.63 |
Parasomnias | 11.46 ± 2.61 |
Sleep-disordered breathing | 3.83 ± 1.28 |
Sleep disorders inventory for students–child version, T-score (n = 460) | |
Subscales | |
Obstructive sleep apnea | 57.53 ± 9.80 |
Periodic limb movement disorder | 60.49 ± 8.94 |
Delayed sleep phase syndrome | 63.50 ± 14.36 |
Pediatric Insomnia Severity Scale, raw scores (n = 462) | |
Factor scores | |
Sleep-onset problems | 9.41 ± 3.93 |
Sleep-maintenance problems | 8.30 ± 3.32 |
Child Sleep Habits Questionnaire, raw scores (n = 459) | |
Subscales | |
Bedtime resistance | 10.83 ± 3.22 |
Sleep-onset delay | 2.25 ± 0.83 |
Night waking | 6.52 ± 1.63 |
Parasomnias | 11.46 ± 2.61 |
Sleep-disordered breathing | 3.83 ± 1.28 |
Sleep disorders inventory for students–child version, T-score (n = 460) | |
Subscales | |
Obstructive sleep apnea | 57.53 ± 9.80 |
Periodic limb movement disorder | 60.49 ± 8.94 |
Delayed sleep phase syndrome | 63.50 ± 14.36 |
Results
Reliability and Construct Validity of PISI
Internal consistency analyses for the PISI factor scores (Cronbach’s α) yielded the following reliability indices: sleep-onset problems α = .81 and sleep-maintenance problems α = .62.
Mean scores and standard deviations for study measures are presented in Table II.
Convergent and Discriminant Validity of the PISI
Results from analyses examining the convergent and discriminant validity of the PISI are presented in Table III. All hypotheses (eight of eight) regarding PISI factor scores/sleep screening subscales for convergent validity were supported by statistically significant positive correlations, and half represented medium to large effects. Although five of eight PISI discriminant validity associations were also statistically significant, all effect sizes were small and of dubious clinical significance.
PISI . | ||||
---|---|---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . | ||
Expected . | Actual . | Expected . | Actual . | |
Child Sleep Habits Questionnaire Subscales | ||||
Bed resistance | + | 0.27*** | 0 | 0.02 |
Sleep-onset delay | + | 0.66*** | 0 | 0.17** |
Night waking | 0 | −0.12* | + | 0.37*** |
Parasomnias | 0 | 0.03 | + | 0.16** |
Sleep-disordered breathing | 0 | 0.00 | + | 0.09* |
Sleep disorders inventory for students–child subscales | ||||
Obstructive sleep apnea | 0 | 0.16** | + | 0.21*** |
Periodic limb movement disorder | 0 | 0.27*** | + | 0.30*** |
Delayed sleep phase syndrome | + | 0.70*** | 0 | 0.17** |
PISI . | ||||
---|---|---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . | ||
Expected . | Actual . | Expected . | Actual . | |
Child Sleep Habits Questionnaire Subscales | ||||
Bed resistance | + | 0.27*** | 0 | 0.02 |
Sleep-onset delay | + | 0.66*** | 0 | 0.17** |
Night waking | 0 | −0.12* | + | 0.37*** |
Parasomnias | 0 | 0.03 | + | 0.16** |
Sleep-disordered breathing | 0 | 0.00 | + | 0.09* |
Sleep disorders inventory for students–child subscales | ||||
Obstructive sleep apnea | 0 | 0.16** | + | 0.21*** |
Periodic limb movement disorder | 0 | 0.27*** | + | 0.30*** |
Delayed sleep phase syndrome | + | 0.70*** | 0 | 0.17** |
Note. *p < .05, **p < .01, ***p < .001. Expected associations reflect those detailed in Table 1 that would reflect convergent validity evidence because of significant positive (+) associations versus evidence of discriminant validity via minimal associations (0). To aid interpretation, correlations between .30 and .49 (medium effects) are in bold, while those of .50 or greater (large effects) are in bold and underlined.
PISI . | ||||
---|---|---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . | ||
Expected . | Actual . | Expected . | Actual . | |
Child Sleep Habits Questionnaire Subscales | ||||
Bed resistance | + | 0.27*** | 0 | 0.02 |
Sleep-onset delay | + | 0.66*** | 0 | 0.17** |
Night waking | 0 | −0.12* | + | 0.37*** |
Parasomnias | 0 | 0.03 | + | 0.16** |
Sleep-disordered breathing | 0 | 0.00 | + | 0.09* |
Sleep disorders inventory for students–child subscales | ||||
Obstructive sleep apnea | 0 | 0.16** | + | 0.21*** |
Periodic limb movement disorder | 0 | 0.27*** | + | 0.30*** |
Delayed sleep phase syndrome | + | 0.70*** | 0 | 0.17** |
PISI . | ||||
---|---|---|---|---|
Sleep measure . | Sleep-onset problems . | Sleep-maintenance problems . | ||
Expected . | Actual . | Expected . | Actual . | |
Child Sleep Habits Questionnaire Subscales | ||||
Bed resistance | + | 0.27*** | 0 | 0.02 |
Sleep-onset delay | + | 0.66*** | 0 | 0.17** |
Night waking | 0 | −0.12* | + | 0.37*** |
Parasomnias | 0 | 0.03 | + | 0.16** |
Sleep-disordered breathing | 0 | 0.00 | + | 0.09* |
Sleep disorders inventory for students–child subscales | ||||
Obstructive sleep apnea | 0 | 0.16** | + | 0.21*** |
Periodic limb movement disorder | 0 | 0.27*** | + | 0.30*** |
Delayed sleep phase syndrome | + | 0.70*** | 0 | 0.17** |
Note. *p < .05, **p < .01, ***p < .001. Expected associations reflect those detailed in Table 1 that would reflect convergent validity evidence because of significant positive (+) associations versus evidence of discriminant validity via minimal associations (0). To aid interpretation, correlations between .30 and .49 (medium effects) are in bold, while those of .50 or greater (large effects) are in bold and underlined.
PISI ScoresAcross Diagnostic Subgroups
PISI factor scores according to insomnia diagnosis are presented in Table IV. One-way ANOVAs indicated that insomnia severity scores varied by diagnosis for sleep-onset problems [F(3,458) = 9.34, p ≤ .001] and sleep-maintenance insomnia [F(3,458) = 4.07, p = .007]. Patients with BIC limit-setting type insomnia displayed the greatest insomnia severity scores for both sleep-onset problems and sleep-maintenance problems relative to all other diagnoses, followed by BIC combined type, psychophysiological insomnia, and BIC sleep-onset type in that order.
PISI factors . | Insomnia diagnosis . | |||
---|---|---|---|---|
BIC sleep-onset association . | BIC limit setting . | BIC combined . | Psychophysiological insomnia . | |
Sleep-onset problems | 7.88 ± 3.90 | 10.31 ± 3.63 | 10.08 ± 3.90 | 9.09 ± 4.09 |
Sleep-maintenance problems | 7.50 ± 3.05 | 9.05 ± 3.30 | 8.38 ± 3.45 | 8.21 ± 3.32 |
PISI factors . | Insomnia diagnosis . | |||
---|---|---|---|---|
BIC sleep-onset association . | BIC limit setting . | BIC combined . | Psychophysiological insomnia . | |
Sleep-onset problems | 7.88 ± 3.90 | 10.31 ± 3.63 | 10.08 ± 3.90 | 9.09 ± 4.09 |
Sleep-maintenance problems | 7.50 ± 3.05 | 9.05 ± 3.30 | 8.38 ± 3.45 | 8.21 ± 3.32 |
Note. BIC = behavioral insomnia of childhood; PISI = Pediatric Insomnia Severity Index.
PISI factors . | Insomnia diagnosis . | |||
---|---|---|---|---|
BIC sleep-onset association . | BIC limit setting . | BIC combined . | Psychophysiological insomnia . | |
Sleep-onset problems | 7.88 ± 3.90 | 10.31 ± 3.63 | 10.08 ± 3.90 | 9.09 ± 4.09 |
Sleep-maintenance problems | 7.50 ± 3.05 | 9.05 ± 3.30 | 8.38 ± 3.45 | 8.21 ± 3.32 |
PISI factors . | Insomnia diagnosis . | |||
---|---|---|---|---|
BIC sleep-onset association . | BIC limit setting . | BIC combined . | Psychophysiological insomnia . | |
Sleep-onset problems | 7.88 ± 3.90 | 10.31 ± 3.63 | 10.08 ± 3.90 | 9.09 ± 4.09 |
Sleep-maintenance problems | 7.50 ± 3.05 | 9.05 ± 3.30 | 8.38 ± 3.45 | 8.21 ± 3.32 |
Note. BIC = behavioral insomnia of childhood; PISI = Pediatric Insomnia Severity Index.
Discussion
This article reports the psychometric properties of the PISI based on testing with 462 youth with clinically diagnosed insomnia. Overall, the current study findings provide preliminary support for reliability and validity of the PISI. Although initial conceptualization, design, and clinical implementation of the PISI preceded publication of the Journal of Pediatric Psychology (JPP) guidelines for measure development (Holmbeck & Devine, 2009), the process for developing the PISI was fundamentally consistent with the most important JPP criteria including (1) clearly establishing a research and/or clinical need for the instrument and (2) using a rigorous, conceptually driven, and empirically based methodology for developing item/scale content. In addition, the current study analyses and resulting psychometric data presented herein, largely align with the JPP guidelines for evaluating (1) reliability, (2) factor structure, (3) convergent and discriminant validity, and (4) clinical utility of newly developed measures (Holmbeck & Devine).
Measures of internal consistency for the PISI factor scores varied with the sleep-onset factor (Cronbach’s α = .81) showing acceptable reliability and the sleep-maintenance factor (Cronbach’s α = .62) showing less optimal reliability (Nunnally & Bernstein, 1994). Reliability levels were related to the brevity of the scales and to interitem correlations that varied by factor. The sleep-onset factor was characterized by medium to large interitem correlations (.43–.86), while marginal reliability for the sleep-maintenance factor was a function of statistically significant, but small to medium interitem correlations (.22–.45).
An interpretive approach that focused on the hallmark symptoms of pediatric insomnia was partially supported by CFA. A single-factor solution demonstrated less optimal fit, both conceptually and statistically relative to the proposed two-factor model. The conceptually driven two-factor model (sleep-onset problems and sleep-maintenance problems) that is reflective of current insomnia nosology had an excellent CFI. However, the RMSEA suggested poorer fit.
Although reliability indices and fit statistics were not uniformly supportive of the proposed two-factor structure, on balance, given the clinical utility of differentiating sleep-onset difficulties from sleep-maintenance difficulties, we recommend the two-factor solution as the most suitable interpretive approach.
Findings regarding convergent validity of the PISI are promising and provide preliminary support for use of the PISI as a brief instrument to examine both sleep-onset and sleep-maintenance problems in pediatric insomnia patients. All (eight of eight) of the hypothesized correlations supporting convergent validity of the PISI were confirmed. Sleep-onset problems on the PISI were significantly linked to established validation measures of bedtime resistance, sleep-onset delay, and DSPS (reflective of sleep-onset insomnia symptoms). Sleep-maintenance problems on the PISI were significantly linked to established validation measures of night wakings, difficulties falling back asleep, and symptoms related to sleep fragmentation, including OSA, parasomnias, and restless sleep. Of note, those validation measures are nested within much longer questionnaires that are, because of their length and scoring complexity, difficult to integrate into routine screening in general pediatric settings or in serial assessments to track insomnia treatment effects.
It is noteworthy that significant effects observed for convergent validity analyses with PISI factors and screening subscales most aligned with pediatric insomnia symptoms (e.g., sleep-onset delay [r = .66], protracted sleep-onset secondary to delayed circadian rhythm [r = .70], and night waking [r = .37]) were largest and represented medium to large effect sizes. Although bedtime resistance is a common feature of pediatric insomnia, the small effect observed for bedtime resistance (r = .27) may be related to clinical presentations when bedtime resistance disrupts the prebed routine, but does not necessarily directly impact sleep-onset latency once the child is eventually in bed. Convergent validity associations with screening subscale measures of sleep fragmentation (i.e., parasomnias, OSA, and PLMD) were in the expected direction, but most (75%) represented small to negligible effects (range 0.09–0.21). Although OSA, parasomnias, and PLMD may share sleep fragmentation as a common problem domain along with insomnia, they are distinct disorders with different cardinal symptoms from insomnia and thus may not be optimal convergent validity measures.
Discriminant validity testing with the PISI was less supportive of a priori expectations. However, when there is shared method variance (in this case, the same response format and same reporter for the PISI and its validation measures), it is not unusual to see some degree of correlation between measures; what is more important is that convergent validity analyses show notably stronger correlations than discriminant validity analyses (Campbell & Fiske, 1959), which was overwhelmingly the case here. As noted, convergent validity associations with direct measures of insomnia symptoms represented medium to large effect sizes, whereas discriminant validity associations tended to be small to negligible, with statistical significance reached primarily because of the large sample.
Although the PISI was not intended to be used as a diagnostic tool nor a scale for clarifying insomnia etiology, we explored how scores varied across clinically confirmed insomnia diagnoses. Both PISI factor scores varied significantly by diagnosis, with patients with BIC limit-setting type insomnia generally faring worst, and BIC sleep-onset type generally faring best. However, there was considerable within-diagnosis variability, and PISI scores will likely vary depending on the unique circumstances of the child. Use of the PISI factor scores for assessing insomnia severity may help to prioritize insomnia treatment targets.
Clinical Use of the PISI
To be clear, the PISI is not a replacement for thorough diagnostic evaluation. Sleep medicine practice guidelines make clear the critical importance of understanding the primary etiologic mechanisms for insomnia to guide empirically supported therapies. Formal sleep disorders evaluation requires a comprehensive sleep assessment that considers relevant developmental, psychological, psychiatric, and medical factors that may impact sleep along with sleep-specific variables including the sleep environment, routines, schedule, and sleep hygiene. Insomnia evaluation should always include a thorough sleep history and screening for sleep disorders in the context of a clinical interview (Chesson et al., 2000). Other sleep measures obtained may include actigraphy, sleep diaries, parent-proxy or self-report questionnaires, symptom checklist, or psychological screening measures (Chesson et al.).
However, such intensive diagnostic procedures are not well suited for brief screening of insomnia symptoms or ongoing assessment during the course of clinical care for pediatric patients. The PISI was developed to fill that void. Overall, the present data support its reliability and validity, which are impressive considering its striking brevity and ease of interpretation. These latter features, though more difficult to quantify than psychometrics, are essential in busy clinical settings. The ease in which the PISI is integrated clinically is attested to by its continuous use in a busy behavioral sleep medicine clinic for the better part of a decade, where it is administered both at baseline (as described here) and across follow-up visits to help guide treatment decisions. Parents can readily complete the PISI at the time of check-in, and clinicians can quickly compute the sleep-onset (sum of Items 1, 2, and 6) and sleep-maintenance (sum of Items 3, 4, and 6) factor scores, allowing an immediate point of reference for clinical conversations and symptom tracking over time. Preliminary data indicate the PISI is sensitive to treatment changes and thus may have promise as a tool that can aide in decisions about treatment termination (Byars & Simon, 2014).
Study Limitations and Future Directions
It is important to consider the study findings within the context of specific methodological limitations. Findings are potentially biased by shared method variance. All validity testing for the PISI was conducted with sleep-specific paper-pencil measures completed by primary caregivers. Although tests of convergent validity were significant and represented medium to large effect sizes for insomnia-specific items, potential for reporter bias on measures with overlapping/similar content must be considered. In addition, while discriminant validity analyses demonstrated lack of correlation or weak associations between insomnia-specific symptoms on the PISI and validated measures of symptoms from conditions distinct from insomnia (e.g., OSA, PLMD), findings from tests of discriminant validity that examined dissimilar core insomnia symptoms on the PISI and CSHQ were not measuring distinct constructs and thus should be interpreted with caution. In addition, as previously discussed, CFA findings were mixed, though on balance the two-factor model seemed most suitable for interpretation.
The current study involved a clinical sample within a single sleep disorders center. Further examination in multisite clinical settings as well as community and primary care settings will help determine whether the PISI can discriminate between those children with and without sleep problems, as well as the best cutoff scores as a screener. In addition, it remains to be seen if the PISI could be used as a measure to differentiate insomnia patients from patients with other sleep disorders commonly seen for evaluation in comprehensive sleep disorders centers (e.g., OSA, parasomnia, restless legs syndrome/PLMD). Also, it would be helpful to examine validity with other sleep measures, such as sleep diaries or actigraphy. Further validation with a more racially diverse sample, and using Spanish and other language translations of the measure is recommended to ensure that the PISI is valid across cultures and racial/ethnic groups.
Even so, the PISI is the only empirically supported brief insomnia-specific measure developed for use with pediatric patients. The current study findings provide preliminary support for the reliability and validity of the PISI as a measure of insomnia severity in a large clinical sample of children and adolescents referred for sleep evaluation. These findings, combined with previous evidence demonstrating sensitivity of the PISI to treatment changes, suggest that it has promise for use in pediatric sleep medicine clinics and as a screener in broader clinical care.
Disclosures
The authors have no financial or other relationship that might lead to a conflict of interest. The study did not involve the use of off-label or investigational drugs.
Funding
This research study was completed at Cincinnati Children’s Hospital Medical Center without external financial support. Internal divisional funds (Behavioral Medicine and Clinical Psychology) were used to support completion of the study. The study did not involve the use of off-label or investigational drugs. None of the authors have any conflicts of interest.
Conflicts of interest: None declared.
Acknowledgments
The authors thank the staff in Cincinnati Children’s Hospital Medical Center (CCHMC), Division of Behavioral Medicine and Clinical Psychology—Data Core for their assistance with data management and storage and the CCHMC Sleep Disorders Center Faculty and Staff for their collaboration in providing clinical care for children with sleep disorders. We also thank the many families that participated in the study.