INTEGRATE-HTA: adopting and implementing an integrated perspective on complex interventions

Most public health interventions can be considered complex. According to the UK Medical Research Council (MRC), complex interventions are characterized by several interacting components, the number and difficulty of behaviours required by those delivering or receiving the intervention, multiple groups or organizational levels targeted, many and variable outcomes, and explicitly permitted flexibility or tailoring of the intervention.¹

In order to be useful for decision-making, systematic reviews and health technology assessments (HTAs) cannot ignore this complexity. They should explicitly consider the factors that make the intervention (or its context) complex and derive conclusions in a way that integrates across various outcomes. The EU-funded project ‘Integrated health technology assessment for the evaluation of complex technologies’ (INTEGRATE-HTA) set out to develop and apply methodological guidance on these issues. While the focus of the project was on HTA, most of the methodological guidance applies equally to systematic reviews.

The overarching product was the INTEGRATE-HTA model, a step-wise approach to conduct an integrated HTA.² In a first step, stakeholder advisory panels are employed to scope the HTA.³ In a second step, the intervention and its implementation in context are conceptualized and structured based on a logic model and comprehensive framework (see below); patient preferences and possible modifiers of intervention effects are identified.⁴ In a third step, using a variety of methods, evidence is pulled together on economic, ethical, social and other aspects⁵; an example on effectiveness is provided below. In a fourth step, the evidence is inserted back into the logic model to visualize inter-relationships across all aspects and outcomes of the HTA. In a fifth step, the results are presented to facilitate decision-making. All INTEGRATE-HTA guidance is listed in Table 1 and can be downloaded from http://www.integrate-hta.eu/downloads/.

In the following, we highlight three methodological advances of particular relevance to Cochrane Public Health reviews as well as public health systematic reviews conducted outside Cochrane.

Structuring complexity of public health interventions through logic models

Systematic reviews often ignore the system, within which a public health intervention is implemented, and the potential interplay and co-evolution of the intervention and the system. Conversely, a formal consideration of the different dimensions of complexity should be the starting point of every systematic review of a public health intervention, as complex problems must be structured before they can be solved.

Logic models, described as ‘… a graphic description of a system … designed to identify important elements and relationships within that system’,⁶ are one important means of conceptualizing and handling complexity. When evaluating public health interventions, logic models can serve an instrumental purpose at every stage of the systematic review process, from formulating the question and defining the intervention to interpreting results and making them relevant for decision-makers.⁶

We developed guidance on how to apply logic models through a combination of (i) systematic searches for published examples of logic models, (ii) searches for existing guidance on the use of logic models in primary research, systematic reviews and HTAs, (iii) development of logic model templates and (iv) application of these templates in several systematic reviews and one HTA of distinct interventions.⁷ We distinguish between three types of logic modelling, and offer direction on how to choose between these. Two templates—one for ‘system-based logic models’ that seek to represent structure and one for ‘process-orientated logic models’ that focus on processes or activities⁸—provide a useful starting point for developing a systematic review-specific logic model.

The use of logic models in systematic reviews has been recommended by many different groups including Cochrane Public Health but, to date, there has been little guidance on how to implement this recommendation. This is the gap that the new methodological guidance has attempted to close. The ongoing Cochrane Public Health review on environmental interventions to reduce the consumption of sugar-sweetened beverages and their effects on health⁹ represents one of several examples of how the system-based logic model template has been applied in practice.

Capturing context and implementation for public health interventions

The effectiveness of public health interventions, as well as their success in reaching all relevant target populations, is critically influenced by their implementation in a given context; indeed, effectiveness, implementation and context are inextricably linked.¹⁰ To date, systematic reviews have often failed to capture context and implementation in appropriate ways, which constitutes a major barrier for appraising transferability and applicability of findings.

The Context and Implementation of Complex Interventions (CICI) framework provides a framework to facilitate the systematic and comprehensive assessment, documentation and graphical presentation of setting, context and implementation of public health interventions. It was developed in an iterative way: an initial framework based on a scoping review was tested in rapid assessments, revealing conceptual immaturity. Thus, pragmatic utility concept analysis was undertaken to advance the concepts of context, implementation and setting.¹¹ Based on the findings, the framework was revised and applied in several quantitative systematic reviews as well as one qualitative systematic review, one HTA and one applicability assessment of distinct interventions (Polus et al., submitted manuscript). Lessons learnt from these applications were incorporated.

Within the CICI framework, the context dimension comprises seven domains (i.e. geographical, epidemiological, socio-cultural, socio-economic, ethical, legal and political), the implementation domain consists of five domains (i.e. implementation theory, process, strategies, agents and outcomes). Interactions between a public health intervention and the way it is implemented in a given setting and context can occur on a micro, meso and macro level (Pfadenhauer et al., submitted manuscript). Definitions and characteristics of all dimensions and domains are provided.

The importance of context and implementation for public health interventions is uncontested. The new methodological guidance presents an overarching framework to tackle these aspects and comprises a set of ready-to-use tools, including data extraction tools for qualitative and quantitative systematic reviews. A review of the effectiveness of interventions to reduce exposure to lead through consumer products and drinking water,¹² conducted as part of the ongoing development of a WHO guideline, shows that the quantitative data extraction tool can easily be put into practice but also highlights that reporting on context and implementation in primary research is scarce.

Synthesizing effectiveness of heterogeneous public health interventions

Systematic reviews of public health interventions are often shaped by a broad research question and multidisciplinary searches, leading to an evidence base characterized by much heterogeneity in populations in different settings and contexts, interventions and their implementation, comparisons and outcomes, as well as in methodological characteristics. Moreover, randomized controlled trials, the mainstay of Cochrane reviews, may not be the most feasible or appropriate study design. It is now increasingly accepted that for many effectiveness reviews much evidence resides within other study designs,¹³ raising questions with regard to the types of study designs to include, how to assess their risk of bias and how to synthesize and interpret findings.

Our thinking was strongly influenced by two recent series of articles published in Research Synthesis Methods¹⁴ and the Journal of Clinical Epidemiology.¹⁵ Drawing on these and work within the INTEGRATE-HTA project, we defined relevant aspects to guide the reviewer in how to (i) assess heterogeneity, (ii) make choices regarding the inclusion of study designs and (iii) select an appropriate method of evidence synthesis. In short, there is no one-size-fits-all solution on how to make decisions about and synthesize heterogeneous evidence for public health interventions; relevant solutions include standard meta-analytical and more sophisticated quantitative meta-analytical approaches as well as graphical and narrative approaches.

The new methodological guidance presents a useful starting point for deciding on appropriate methods of evidence synthesis in Cochrane and non-Cochrane systematic reviews of public health interventions. A systematic review of the effectiveness of home-based and reinforced home-based palliative care, conducted as part of the INTEGRATE-HTA project (Burns et al., submitted manuscript), shows how harvest plots, a graphical means of evidence synthesis, can be combined with gap analysis and expert consultations to deliver more in-depth insights into the findings obtained through an effectiveness review.

Conclusions

The INTEGRATE-HTA project has produced a suite of methods for making sense of different aspects of ‘complex interventions’ or, more often, interventions implemented within and interacting with complex systems. In our view, a significant strength of the project was that methods development took place in a highly interdisciplinary and interactive way, and in parallel with empirical testing. Most of these methods can be directly applied in Cochrane and non-Cochrane reviews of effectiveness of public health interventions, as well as in qualitative systematic reviews. Importantly, adopting the INTEGRATE-HTA perspective and a combination of the methods to make this perspective a reality will lead to a more holistic understanding of a public health intervention. We believe that this suite of methods is potentially useful to many of those engaged with commissioning, conducting or using systematic reviews and HTAs of public health interventions.

Acknowledgements

The research leading to this publication is part of the project INTEGRATE-HTA and has received funding from the European Union Seventh Framework Programme under grant agreement no 306141.

References