Comparison of intervention fidelity between COPE TEEN and an attention-control program in a randomized controlled trial

Abstract

Fidelity in implementing an intervention is critical to accurately determine and interpret the effects of an intervention. It is important to monitor the manner in which the behavioral intervention is implemented (e.g. adaptations, delivery as intended and dose). Few interventions are implemented with 100% fidelity. In this study, high school health teachers implemented the intervention. To attribute study findings to the intervention, it was vital to know to what degree the intervention was implemented. Therefore, the purposes of this study were to evaluate intervention fidelity and to compare implementation fidelity between the creating opportunities for personal empowerment (COPE) Healthy Lifestyles TEEN (thinking, emotions, exercise, and nutrition) program, the experimental intervention and Healthy Teens, an attention-control intervention, in a randomized controlled trial with 779 adolescents from 11 high schools in the southwest region of the United States. Thirty teachers participated in this study. Findings indicated that the attention-control teachers implemented their intervention with greater fidelity than COPE TEEN teachers. It is possible due to the novel intervention and the teachers’ unfamiliarity with cognitive-behavioral skills building, COPE TEEN teachers had less fidelity. It is important to assess novel skill development prior to the commencement of experimental interventions and to provide corrective feedback during the course of implementation.

Introduction

Fidelity of behavioral interventions, including the methodological strategies to monitor and strengthen the interventions [1], has received considerable attention in the past 3 decades. Fidelity is necessary for accurate assessment and interpretations of treatment effects [2]. Intervention outcomes are a result of what components the intervention contains and how the intervention was delivered rather than just the number of intervention components delivered [3].

Researchers in the early 1980s began to focus on improving characteristics of delivered interventions, including their strength, integrity and effectiveness [4]. Waltz et al. [5] proposed additional methodological improvements to strengthen an intervention, including the assessment of adherence and competence. They advocated for the benefits of using treatment manuals and the need for conducting manipulation checks. The dimensions of treatment receipt and enactment were introduced a short time later [6]. A landmark manuscript focusing on fidelity was published in 2004 by the Treatment Fidelity Workgroup of the NIH Behavior Change Consortium [7]. They recommended evaluating five components of fidelity: study design, provider training, treatment delivery, treatment receipt and enactment of treatment skills. Gearing et al. [8] published a review of aspects of fidelity that were identified in published manuscripts between 1980 and 2009. They developed a guide for evaluating fidelity in four of the five targeted components recommended by the Treatment Fidelity Workgroup. The authors provide a table outlining the assessment of major fidelity components and recommend assessing intervention design, intervention training, monitoring the intervention and monitoring the intervention receipt. Within each category, specific points are categorized for the protocol, execution, maintenance, feedback, threats and measurement. Gearing et al. [8] did not include enactment because an intervention may be delivered with fidelity, but the participant(s) may not be willing or able to apply it.

Despite a researcher’s best intention to design and implement an intervention with fidelity, it is common for one or more aspects of fidelity to not be completed and/or documented [8, 9]. Therefore, it is helpful that ongoing work in behavioral research focus ‘strongly on quantification of treatment fidelity rather than assuming that fidelity was achieved because of rigorous design plans’ [10] (p. 53). Documenting the implementation of each component and subsequent quantification can aid researchers in better evaluating intervention fidelity and also may lend understanding of treatment effects or lack thereof.

Beyond a rigorous design and plan for monitoring fidelity of an intervention, researchers often are challenged with lack of adherence to program protocol resulting in the inevitable adaptation of their intervention when it is implemented [11]. ‘It is also necessary to know how a specific intervention should be implemented and under which circumstances it can be successful’ [12] (p. 1). Carvalho et al. [11] identified five types of adaptations that took place in 12 sites that implemented evidence-based interventions, which include (i) changes to educational materials, (ii) changes to intended audience, (iii) changes to program delivery, (iv) adding new activities and (v) deleting core elements. It is important to be aware of the types of adaptations that occur frequently in intervention research and monitor which occur during implementation.

The creating opportunities for personal empowerment (COPE) Healthy Lifestyles TEEN (Thinking, Emotions, Exercise and Nutrition) (COPE TEEN) program is a one-semester cognitive-behavioral skills building (CBSB) intervention to improve a teen’s physical and mental health. This program includes cognitive reframing, problems solving, stress management, goal setting, physical activity and nutritional information (Table I). High school health teachers implemented the intervention 1 day per week during their regular scheduled health class (∼50 min in length). Most teachers are unfamiliar with CBSB strategies and many learned these techniques for the first time during training for this intervention. The Healthy Teens control program also was taught over the course of one semester by health teachers in their regular scheduled health class. Healthy Teens was based on increasing a teen’s health literacy and included topics familiar to health teachers such as first aid, sun safety and transportation safety (Table I). We were interested to learn if there would be differences in fidelity between each group due to the novel content learned by the COPE TEEN teachers. Therefore, the purposes of this study are to (i) discuss intervention fidelity in the COPE Healthy Lifestyles TEEN program, (ii) describe the fidelity of intervention design, training, delivery and receipt in the COPE TEEN group and (iii) compare fidelity to the intervention between the COPE TEEN and Healthy Teen intervention groups in a prospective blinded (to teachers and participants) randomized controlled trial (RCT) to promote mental and physical health in high school teens.

Methods

Sample/participants

Health teachers (N = 30) were requested by their school and/or district leadership to participate in the RCT by delivering the intervention in their health courses during one semester. All health teachers referred by their district were eligible to participate.

Students were invited to participate if they were aged 14–16 years and enrolled in health education courses at 11 high schools from two school districts in a large metropolitan city in the southwest United States. To participate in the study, the teen needed to provide assent, parental consent and be free from medical conditions that would not allow them to participate in the physical activity component of the program if they were randomized to the COPE TEEN group. Parents of the teens were invited to participate, but it was not mandatory. A more detailed description of the study methods has been published previously [13].

Data collection

Data were collected between December 2009 and December 2012. Teachers in both intervention groups were introduced to the study during a 1-day training prior to the start of a school semester. The training for the COPE TEEN teachers consisted of (i) a review of the literature on adolescent/childhood obesity and mental health problems, (ii) description of the research study aims, objectives and procedures, including prior feedback from teachers, (iii) review of the research protocol, (iv) completion of consent and background questionnaire including teaching experience and teaching satisfaction as well as demographic information, (v) orientation to the program manual, (vi) explanation and demonstration of CBSB, (vii) review of all session content along with emphasis of key elements, (viii) demonstration of and practice implementing available physical activities that could be done within the limited space of a classroom and (ix) use of pedometers. The training for the Healthy Teens teachers consisted of (i) review of the research protocol, (ii) completion of consent and background questionnaire, including teaching experience and teaching satisfaction as well as demographic information, (iii) orientation to the program manual and (iv) review of all session attention-control content along with emphasis of key elements and (v) use of pedometers.

After the teacher training, research team members introduced the study to all students in each health class participating during the first week of the semester and sent consent/assent packets home with all teens who expressed interest in study participation. Students with assent and parental consent were enrolled in the study. Enrollment in the study included completing questionnaires at baseline, post-intervention, 6-month follow-up and 12-month follow-up time points. Parent participation in the study included questionnaires at baseline and post-intervention. Additionally, students who were identified as overweight or obese from calculated body mass index (BMI) were asked to assent (parental consent) to provide a fasting finger stick blood sample for analysis of blood lipids.

Whether students were enrolled in the study or not, all adolescents in the participating health education courses received either (i) a 15-week, 15-session multi-component educational and CBSB intervention with physical activity (COPE TEEN) or (ii) a 15-week, 15-session attention-control program (Healthy Teens) focusing on common adolescent health topics.

Teachers at the 1-day training received a $25 gift card, a t-shirt and a universal serial bus storage device with study materials. Each teacher received $100 for completing questionnaires at the end of the semester. All student and parent participants in the study received gift card incentives for participation; a total of $55 for the students (an additional $30 for blood sampling) and a total of $40 for the parent.

Observations and measure

Teachers were observed during intervention implementation by research team members. The goal was to arrive to observe the session unannounced at four sessions (25%). Four of the research team members were trained to use a standardized fidelity observation form by the principal investigator and the project manager. Inter-rater reliability was 90%. Observers were not blinded to the group assignment.

The fidelity observation form was created by reviewing literature for relevant information that was used to assess fidelity in prior intervention studies through observational methods [14, 15]. The goal was to create an easily completed form to evaluate intervention delivery including observations focusing on (i) teacher preparation, (ii) presentation of learning objectives, (iii) teacher delivery of intervention, (iv) adherence to lesson plan including critical intervention input subcategories of role play, homework and physical activity (COPE TEEN), (v) participation level of students and (vi) cognitive behavior skills building practice by students (COPE TEEN). Participation of students in both intervention groups and cognitive behavior skills building practiced in the COPE TEEN group were included because each was emphasized in the classroom as part of the intervention. Each of the categories was evaluated with a 5-point Likert scale. There were additional dichotomous questions (e.g. yes/no) under each category (e.g. ‘PowerPoint® displayed’, ‘Teacher addressed all learning objectives’ and ‘Utilized examples in teacher’s manual’). These additional questions were intended to assist the observer in rating the item on the Likert scale (Table II). The responses from the observation form were analyzed individually rather than with a summation score. The form was evaluated for content validity by six research team members. A score of 4 out of 5 or greater on the Likert scale and >80% of affirmative responses on the dichotomous items were chosen to be considered a desirable level of fidelity [16]. Training consisted of reviewing the protocol for conducting an observation and items on the observation form, discussing selection of responses and selecting the day for an observation. Background education of observers included (i) educational training of the intervention content in both the COPE TEEN and Healthy Teen groups, (ii) use of the observation form and (iii) practice observational sessions to improve inter-rater reliability. Prior to independently observing an intervention session, two or three observers monitored the same session and compared results.

Ethical considerations

The study was approved by the university Institutional Review Board and each participating school district.

Data analysis

Data analysis included descriptive statistics and logistic regression for repeated measures using generalized estimating equations (GEE) models for the binary responses and the Likert-scale responses on the observation form comparing teacher’s implementation between the two groups. All GEE models controlled for years of teaching experience as it was significantly higher in the Healthy Teen teachers at baseline [Healthy Teens = 17.09 years (SD = 8.27), COPE TEEN = 11.43 years (SD = 6.24); P = 0.04]. Power was sufficient to identify a one standard deviation difference.

Results

All health teachers approached about involvement in the study consented to participate. Thirty teachers were observed implementing the interventions, including 16 Healthy Teens teachers and 14 COPE TEEN teachers. Teachers in both groups were similar in age and years teaching health courses and education (Table III). Healthy Teen teachers had significantly more years teaching overall than COPE TEEN teachers. A total of 117 observations were completed [Healthy Teens = 61 observations, (M = 3.81 per teacher); COPE TEEN = 56 observations, (M = 4.00 per teacher)].

Fidelity in both groups

The proportion of teachers completing the fidelity components varied (Fig. 1). There was only one component that was completed <50% of the time. Eleven percent of the learning objectives were referenced but not explained by the teacher. There were five components that were completed by between 50 and 70% of the observations. These items related to learning objectives, instruction and adherence to lesson plan. There were 14 components that were completed between 71 and 89% of the time. These items related to learning objectives, preparation, instruction, adherence to lesson plan and practicing skills. There were 10 components that were completed >90% of the time and included components regarding preparation, instruction, adherence to lesson plan, active participation and practicing skills.

Narrative review of fidelity in COPE teachers

We completed a detailed accounting of implementation fidelity in the COPE TEEN teachers for the four components of intervention fidelity based on recommendations by Gearing et al. [8] (Table IV). Overall, most components were addressed in this study, but there were several aspects of fidelity that were not fully measured or addressed. Two examples include (i) a mechanism was not in place to assess teacher skill development prior to intervention implementation and (ii) a rigorous protocol was not in place to provide corrective feedback when protocol deviations occurred during implementation.

Comparison of fidelity between groups

Adherence to intended core elements (e.g. Likert scale items) were documented with two significant differences (Table V). Healthy Teen teachers demonstrated increased clarity in describing the learning objectives compared with the COPE TEEN teachers [OR: 0.51; 95% confidence interval (CI): 0.35–0.75; P = 0.001], and the session content was delivered as detailed in the manual by teachers in the Healthy Teen group (OR: 0.68; 95% CI: 0.46–1.0; P = 0.05) when adjusted for teaching years. Adherence to prescribed interventionists’ behaviors and teens’ responses (e.g. dichotomous questions) also were analyzed (Table II). Several significant differences were present between the two groups. All differences favored the Healthy Teens teachers, including (i) more teachers displayed the PowerPoint® presentations during the intervention (OR: 0.05; 95% CI: 0.00–0.57; P = 0.02), (ii) all aspects regarding the learning objectives were presented (see Table V for specifics), (iii) summarization of important points and discussion was completed (OR: 0.15; 95% CI: 0.04–0.49; P = 0.002), (iv) discussion of completed homework and answering questions was completed (OR: 0.39; 95% CI: 0.15–1.01; P = 0.05) and (v) more active participation of students, demonstrated by students asking questions occurred (OR: 0.23; 95% CI: 0.07–0.71; P = 0.01).

Discussion

This study allowed for careful assessment of the fidelity of intervention implementation. The fidelity observation form created for this intervention was comprehensive and addressed important fidelity components. Overall, 17 of the 31 (55%) dichotomous items were completed by the teachers 80% or more of the time. Lower fidelity (<70%) was observed for components related to learning objectives, homework, teacher’s manual being present and discussing all PowerPoint® slides. The greatest fidelity (>90%) was observed in teachers addressing questions, students’ active participation and several aspects of implementing the intervention.

It was enlightening to compare implementation between the COPE TEEN and the Healthy Teens experimental groups implementing different curricula. This is the first study that we are aware of to document the comparison of fidelity of implementation between an experimental intervention group and an attention-control group. In this study, significant differences were identified in fidelity for implementation between COPE TEEN and Healthy Teens. All significant differences in implementation favored the Healthy Teen teachers and occurred in all aspects of presenting the learning objectives, how appropriately the material was presented, if the PowerPoint® presentation was displayed during implementation, summarization of important points by the teacher, discussion of completed homework and student participation by asking questions. Fidelity may have been greater in the Healthy Teen teachers because they had significantly more years teaching experience compared with the COPE TEEN teachers. It is challenging to determine how much of an effect the limitations in fidelity had on the study’s outcomes. Despite less fidelity in the COPE TEEN teachers, important outcome differences were achieved between the two groups of students. The COPE TEEN group had significantly (i) greater number of steps per day (P = 0.03), (ii) less BMI (P = 0.01), (iii) less depression in the students who started the study with severe depressive symptoms (P = 0.02), (iv) lower alcohol use (P = 0.04) and (v) higher average scores on all Social Skills Rating Scale (P < 0.05) and higher health course grades (P = 0.01) [17]. Although intervention fidelity was less in the COPE TEEN teachers, positive outcomes were achieved, which may indicate that teens who receive the COPE TEEN intervention may receive even more benefit in the future if the intervention achieves greater fidelity.

Some possible effects of the fidelity differences between groups to consider include (i) the likelihood that the scripted sessions were not taught per the manual when the PowerPoint® presentations were not displayed, (ii) the students’ lack of clear understanding of the content for the session when the learning objectives were not discussed per the manual, (iii) teacher reinforcement of session content may not have been sufficient without the discussion of each session’s homework assignment, (iv) fewer questions posed by the students may have indicated that the students were less ‘engaged’ in the session and (v) cognitive behavior skills building practice may not have been appropriately presented and practiced when the materials were not appropriately presented.

None of the teachers in either group implemented the intervention with 100% fidelity. Durlak and DuPre [16] reviewed literature on the influence of implementation on program outcomes. They noted that it is unrealistic to expect a perfect or near-perfect implementation. Their review of interventions identified positive program results with levels around 60% with most reviewed studies achieving 80% fidelity.

Numerous factors are interrelated when evaluating intervention outcomes, including adherence and competence [18]. Adherence and competence have been shown to independently predict outcomes [19]. Adherence to intended intervention is an important aspect of fidelity. In this study, the Healthy Teens teachers adhered more closely to the proscribed intervention. Importantly, the content of the Healthy Teens program was aligned more closely with what would be expected as core knowledge of health teachers and district curriculum requirements (Table I). There were minimal new knowledge or skills that needed to be learned to implement the Healthy Teen program as it included content across a variety of common teen health topics. The Healthy Teen program did not include any CBSB activities and may have been easier and more familiar to teach. Conversely, the COPE TEEN curriculum contained new content about cognitive-behavior skills building that the teachers had to first learn themselves and then teach their students. New content centered primarily on understanding and implementing CBSB activities (Table I) [13]. CBSB activities embedded in COPE TEEN included setting goals, increasing communication, recognition of unhealthy habits, awareness of stress responses and interconnection of thoughts and actions. Skills developed during this intervention help the teen to recognize and think about their unhealthy behaviors and have been shown to improve behavior change [17]. We theorize that the decreased adherence to the COPE TEEN intervention may be related to the teachers’ limited proficiency in teaching the new content. We anticipate with added exposure and experience in teaching the curriculum, the intervention teachers’ adherence and competence would improve.

Despite the limitations in fidelity for implementing the intervention in the COPE TEEN teachers, significant favorable results were found in the study [20]. One theoretical model of program implementation proposes that the effects of fidelity on program outcomes are moderated by participant responsiveness [21]. The active participation of students and teacher responsiveness to questions may have facilitated achievement of some anticipated program outcomes. Wenz-Gross [22] introduced a year-long curriculum to preschool teachers to improve problematic behaviors. They provided direct support for 2 years and then monitored implementation the third year. It was noted that, by year 3, the teachers were able to independently implement the intervention with high fidelity. This finding may indicate that individuals that are trained but new in delivering interventions will need more (i) frequent educational sessions and (ii) on-going supervision and consultation throughout the intervention program to improve their confidence and ability to deliver the intervention as planned as well as to emphasize key components of the intervention.

Another factor to consider when evaluating outcomes in an intervention program is the degree to which the program was adapted. Carvalho et al. [11] suggested, ‘The tension between fidelity and adaptation might well be reframed as a natural process of program evolution’. Durlak and DuPre [16] identified factors affecting implementation in five categories including characteristics of innovations, individuals and communities and features associated with the prevention delivery and support systems. In our intervention, teacher adaptations to the interventions occurred frequently when changes were made to the PowerPoint® presentations. Other teachers omitted curriculum activities or added new materials not in the manualized intervention. These changes were unexpected because these interventions were created for the intended age of the students, were culturally relevant and designed to be implemented in a health course. Future healthy lifestyle interventions in the school setting will benefit from additional teacher input and collaboration during the development phase when critical intervention content and planning for the delivery of that content is created. In addition, a protocol must be in place prior to implementation of the intervention to address deviations that are observed during fidelity checks.

Lessons learned and implications for future intervention research

Durlak and DuPre [16] suggest eight steps that can be taken to improve implementation: ‘(i) specify the essential ingredients of an intervention; (ii) collaborate with change agents in field settings to tailor the program to the target setting; (iii) obtain a clear commitment to administer the agreed-upon intervention; (iv) train change agents to conduct the program effectively; (v) provide on-going supervision and consultation once the program has begun; (vi) be ready for unexpected problems; (vii) do pilot work and (viii) designate staff with responsibilities for implementation’ (p. 14). LaChausse et al. [23] also identified a need for an enhanced comprehensive teacher training rather than a one shot curriculum training to improve a teacher’s implementation fidelity.

In our study, we recognized the need to have teachers involved early in the process. School administration and school boards must demonstrate interest and buy-in initially, but interest in the intervention program and research study must quickly filter down to the teachers to improve direct interaction with the teachers regarding their participation. Another important factor is to obtain teacher input regarding the intervention at several planning stages. Information elicited from the teachers prior to implementation can help to adapt the intervention as needed prior to implementation and will improve the integrity of the intervention. A debriefing after completion of each session will allow teachers to share their experience implementing the intervention and will provide more feedback for future revisions and improvements to the interventions. In our post-implementation evaluation survey of the program, 79% of COPE teachers indicated they would recommend one or more things changed in the intervention. Less than half (43%) of the COPE TEEN teachers indicated they had enough time to present all of the PowerPoint® slides, fully discuss the content and initiate an in-class physical activity. Fifty seven percent of the COPE TEEN teachers indicated the program provided them with new skills and/or useful knowledge. We recommend assigning a study team member as a mentor to assist the teacher during the first few intervention sessions to support the teacher as he/she implements new information. Several teachers also indicated that they would have liked a more flexible curriculum so that they could align the COPE TEEN session content with required school district core health curriculum as much of the same content (i.e. coping, stress, healthy nutrition and physical activity) is included in both the COPE TEEN and the school district health curriculum. Ultimately, we believe we would have had more buy-in for delivery of the intervention and, therefore, improve fidelity, if the teachers were able to have early more input and direct support in the delivery of the intervention.

Limitations

This study had several limitations including non-blinded observers, a small sample size and, occasionally, due to scheduling issues, the class observations were announced to the teacher prior to the planned session. Every attempt by the observers was made to arrive at the classroom unannounced for the observation of the intervention. Teachers were able to choose which day of the week that they wanted to present the COPE TEEN or Healthy Teens intervention. Often, the predetermined day for delivery of the interventions changed due to school scheduling or other classroom demands and the observer arrived when the teacher was not delivering the intervention. To avoid numerous missed observation opportunities, some observations were scheduled with the teacher on a specific day. Additionally, the observations are only a random sample of the program sessions, and the results are assumed to generalize to all program sessions. Our intervention lacked a rigorous protocol for corrective feedback and ongoing support/coaching for program facilitators, which we now recognize as vitally important to include in future intervention work. We also did not systematically document teacher adaptation of the interventions.

Although coaching has been documented as a beneficial component to increase implementation fidelity (particularly when program content and skills are novel), there are challenges inherent in translating this model into a school or public health setting [24]. Some appropriate coaching strategies in the school setting to consider may include peer supervision, coaching via telephone, creation of an informational intervention blog or delivery of session specific tips via email prior to each lesson.

Conclusions

Fidelity to the intervention is essential to measure in intervention research. Each aspect of fidelity needs to be carefully addressed early in the planning process of the study’s implementation. To sustain intervention programs in the school setting, collaboration with those individuals who will be responsible for the delivery of the intervention long after the research team leaves the setting, is paramount. Early and sustained input from teachers during the development of the intervention content, protocol planning and implementation of the intervention is vital. Adaptation, without loss of key programmatic ingredients, may be necessary to accommodate implementation of healthy lifestyle interventions such as COPE TEEN in schools. Frequent monitoring by the research team with planned corrective follow-up and support is necessary to improve the delivery fidelity of the intervention.

Acknowledgements

Special thanks to Kim Weberg, Kristine Hartmann, Alan Moreno, Meghan Paradise and Jonathon Rose for all of their efforts on and dedication to this study.

Funding

This work was supported by the National Institutes of Health/National Institute of Nursing Research (grant number 1R01NR012171) (PI: Bernadette Melnyk).

Conflict of interest statement

None declared.

References