- Split View
-
Views
-
Cite
Cite
Anthony G. Threlfall, Soraya Meah, Alastair J. Fischer, Richard Cookson, Harry Rutter, Michael P. Kelly, The appraisal of public health interventions: the use of theory, Journal of Public Health, Volume 37, Issue 1, March 2015, Pages 166–171, https://doi.org/10.1093/pubmed/fdu044
- Share Icon Share
Abstract
Public health decision-making is hampered by inappropriate adherence to underpowered randomized controlled trials (RCTs) which give inconclusive results and lead to decision-makers being loath to recommend interventions with strong theoretical and observational support.
We outline situations in which robust decisions about health interventions can be made without trial evidence. We present a new approach in which theory, causal models and past observations are given proper regard in the decision-making process.
Using our approach, we provide examples where the use of causal theories and observations in areas, such as salt reduction, smoking cessation and gardening to improve mental health, is sufficient for deciding that such interventions are effective for improving health without needing the support of underpowered RCTs. Particularly where RCT evidence is inconclusive, our approach may provide similar aggregate health outcomes for society for vastly lower cost.
When knowledge and theoretical understanding are unable sufficiently to reduce doubt about the direction of effect from an intervention, decisions should be made using evidence-based medicine approaches. There are, however, many cases where the combination of robust theory, causal understanding and observation are able to provide sufficient evidence of the direction of effect from an intervention that current practice should be altered.
Introduction
Commissioning services and interventions to meet the health needs of individuals and communities poses many challenges. To make the best decisions about health interventions there needs to be a transparent consideration of all the relevant evidence. We propose that a decision-theory approach allows a range of relevant evidence forms, and thus provides a more flexible and robust mechanism to support commissioning decisions rather than a hypothesis-testing approach which relies predominantly on randomized controlled trials (RCTs).1
A decision-theory approach utilizes relevant knowledge, theory and data both from observational and experimental studies to evaluate the likely efficacy of an intervention. If from this process it can be demonstrated that an intervention is sufficiently unlikely to cause net harm, then we can move to estimate cost-effectiveness. That is, we assess if the benefit relative to its cost is sufficient for the intervention to be recommended for application to population groups under consideration. This contrasts with the hypothesis-testing approach in which decisions about the efficacy of an intervention are made solely by using the findings of scientific studies that use statistical testing to evaluate their efficacy. The hypothesis-testing approach is central to evidence-based medicine but in practice groups charged with reaching decisions about health interventions for populations also use additional evidence alongside scientific, methodological and philosophical judgements.2 Rarely, however, is the additional evidence and reasoning underpinning decisions transparently described in a framework. The use of a decision-theory approach is in keeping with the original articulation of evidence-based medicine which required clinicians to use clinical judgement, patients' views and available scientific evidence in making decisions.3 The proposed approach increases transparency in the decision-making process so that the reasoning underpinning a treatment or commissioning decision can be seen, which may enhance the ability of patients and public to engage with the process.
This paper argues that consideration of theory, which is usually based on robust evidence, as a form of evidence is often important in reaching the best health decisions for populations. It is acknowledged that when RCT evidence is available, it is usually of value, and often of great value, but a robust decision about the efficacy of a health intervention based on scientific standards can in some situations be made in the absence of RCT evidence. The paper sets out a description of situations in which robust decisions can be made without RCT evidence and describes a context for decision-making where RCT evidence is not necessary.
Role of theory in evaluating interventions: establishing the principle
In a celebrated paper by Smith and Pell4 on the role of parachutes in preventing death when exiting a plane at high altitudes, no RCT evidence was found to support their use. Indeed, people find risible the notion that an RCT is required to prove a parachute's worth.
Parachutes have the backing of strong theoretical evidence where the strength of evidence is not from controlled trials with humans but from the physical sciences. These disciplines have laws, rules and theories based on observational experiments which can be used to understand and predict outcomes. The rules of geometry used in architecture to ensure that buildings do not fall down are a prime example. There is a whole world of human knowledge and understanding of this type and other forms of reasoning which do not require an RCT to demonstrate whether something works.2,5 The important question is not whether it is ‘scientific’ to make decisions of efficacy without particular forms of evidence, but how far, and in what circumstances, one is willing to go without statistically significant RCTs or similar evidence in matters of health. Black6 for example has argued for other forms of evidence, notably observational, as a useful means of assessing the value of an intervention. This can be extended within a decision-theory approach to include robust theory, as an acceptable form of evidence on which to base a health decision.
Role of intervention trials in evaluation of health interventions
New trials of interventions are needed when, after considering theory, past observation and previous studies, people do not know with sufficient certainty if an intervention causes net harm or benefit. The primary role of an intervention trial is to reduce doubt about whether the intervention produces on average more benefit than harm, i.e. the direction of effect of the intervention in a population group. Doubt about the direction of effect provides the ethical basis underpinning assignment of individuals within intervention trials. It is considered ethical to assign individuals to different interventions if there is sufficient doubt among investigators about outcome and unethical to assign individuals if it is believed with enough certainty that one group is knowingly advantaged. Once a carefully-conducted trial is successfully completed there exists evidence about the direction of effect of the intervention in a group. If the difference between groups is statistically significant it is accepted that statistical uncertainty about the outcome is sufficiently reduced to allow it to be used as robust evidence for decision-making. However, as most public health trials aim to demonstrate average benefit at the group level and often show a range of differing benefits among individual participants, there will often be some doubt about the outcome of the intervention for a particular individual.
Role of observational evidence, theory and causal models in evaluation of health interventions
‘Beyond reasonable doubt’ is a test used in many legal systems to allow a criminal conviction. The evidence considered in these systems to reach a judgement about guilt rarely if ever comes from RCTs but nonetheless can be sufficient to convict an individual. Similarly, there are many health interventions for which the outcome is beyond reasonable doubt based upon evidence other than that from intervention trials. An outcome can be predicted using a causal model, and robust theory and past observation can show that the intervention repeatedly produces the same outcome. Some interventions in health care produce an effect so large that background factors cannot mask it. Before the invention of the RCT there was a good deal of medical progress. It did not need an RCT to tell us that penicillin worked, that insulin could prevent death from diabetes, that aspirin was an effective analgesia or that appendicectomy prevented death from acute peritonitis. For these treatments, it was clear from observation that many of those who would have died without the treatment did not do so with treatment or that improvement was tangible. The effect size for such treatments is so large that the possibility that it has been caused by bias is remote. This, in the literature, has been identified as having a high signal-to-noise ratio (SNR).7 Observational studies for treatments with a high SNR have been quite sufficient to satisfy the scientific community of causation in a particular direction, and that the result is not simply a coincidental association.
Doubt about the outcome from a health intervention is much reduced if there is a credible biological explanation of causation and the intervention fits with this explanation. Consider again the case of acute appendicitis. Observation alone shows that untreated by surgery, it may well lead to death; so treated, it will lead to almost certain survival, implying a very high SNR. The likelihood that surgery will improve survival is bolstered by a plausible biological explanation, based on both germ theory and knowledge of anatomy and physiology (that people can live without an appendix, without any apparent loss of functioning). When a robust theory explaining how an intervention works is not present there is increased doubt about the outcome and very strong observational evidence is needed to demonstrate efficacy. For example, when the UK Parliament Science and Technology Committee (2009–10)8 considered the merits of homeopathy, it noted the lack of a plausible scientific explanation about how it works and therefore insisted that conclusions about the evidence on the efficacy of homeopathy should be derived from well-designed and rigorous RCTs.
Theory and causal models can be used to establish beyond reasonable doubt how a public health intervention causes an outcome, to predict the population health outcome from the use of an intervention and to derive beneficial interventions. For example, it is our theoretical understanding that allows us not to doubt that health outcomes will be improved following a natural disaster if emergency aids, such as shelter and clean water, are provided. In this example the strength of evidence from non-trial sources is so strong that trials to show shelter and clean water are effective are not needed.
The role of interventional trials when theory and observation can establish beyond reasonable doubt the direction of effect from an intervention
Health interventions for which there is evidence from theory and observation that establish beyond reasonable doubt the direction of effect in a population include, but are not limited to, quarantine against infectious diseases, the chlorination of drinking water, and filtration and extraction systems to reduce particulate matter in the air (whether from tobacco smoking, or transport or industrial pollution, or the heating of houses). For these types of interventions the role of new study evidence is not to demonstrate the direction of effect, because it is already established. The role is to estimate the size of effect from different approaches which can then be used to determine the most cost-effective approach for different circumstances.
This change of emphasis is crucial for reaching decisions as it alters the interpretation of intervention trial data and allows its proper use within a decision-theory approach. In particular, it changes the relevance given to the confidence interval around the point estimate crossing a relative risk (RR) of unity. The reason for 95% confidence intervals in respect of intervention trials is to ensure that the probability that the intervention on average does more harm than good is acceptably low. That is, it is to determine the direction of change. When testing the null hypothesis that an intervention produces net benefit, a confidence interval for RR that crosses unity indicates that harm cannot be ruled out with sufficient statistical confidence. When the decision-maker knows the direction of effect and is interested only in the size of effect, the confidence interval is relevant only as an indicator of the confidence that the point estimate from the study is the true value of effect that would be observed if the intervention were repeated in a similar population.
Consider using either a hypothesis-testing or a decision-theory approach to make recommendations about providing clean water and shelter following a natural disaster. A problem emerges for the hypothesis-testing approach to recommending interventions if searches only find trials, which due to their small sample size or particular circumstances, are unable to demonstrate the benefits of clean water and shelter. Using this approach, the entire weight is given to the study findings. A systematic review of the study evidence would report no conclusive evidence to recommend any intervention. Using a decision-theory approach, a decision-maker is able to use all relevant evidence, including theory and understanding of cause and effects to interpret the results of any available study evidence. If a decision-maker had robust theoretical evidence to indicate that a type of intervention produced net benefit they would weigh this evidence against the quality of the study evidence and would not overturn existing theory based on the results of small studies which due to their size were unlikely to accurately estimate the true effect.
By using a transparent decision-theory approach, decision-makers can use study evidence to challenge existing theory but can avoid drawing the wrong conclusion from weak study evidence. Using observational evidence, theory and causal models, a judgement can be reached about the doubt around the direction of effect from an intervention which can be used as evidence and weighed against any existing study evidence.
Consideration of cost-effectiveness
When making recommendations for public health interventions, an important consideration is cost-effectiveness, for which an estimate of the size of effect of an intervention is needed. The standard approach used for health technology assessments of new drugs and new clinical interventions is to establish efficacy using RCT evidence or similar and then to calculate cost-effectiveness.9,10 When theory, causal models and past observation have established beyond reasonable doubt the direction of effect from an intervention, it is still essential to consider cost-effectiveness, because the intervention's benefits might not be worth the cost. If good quality intervention studies are available and applicable to the situation, then the estimates of effect from those studies can be used in estimating cost-effectiveness. If study evidence is of poor quality, unlikely to be transferable to the situation, or does not exist, then cost-effectiveness can be considered by asking what size of benefit would be required for the intervention to be judged cost-effective.
Here the advantage of a decision-theory approach is that investigations of cost-effectiveness are no longer limited only to those public health interventions which have been shown to be effective in intervention trials. This allows formal consideration of many public health interventions designed to deliver small but important effects at relatively low cost, which are infrequently the subject of intervention trials large enough to detect the effect sizes at the standard level of statistical confidence.
Examples
Action to reduce salt intake to lower cardiovascular events in populations is an example of a public health intervention which is strongly supported by a wide body of evidence but for which evidence from RCTs alone is less clear.1 The following examples illustrate how a transparent consideration of additional theory, knowledge and reasoning might lead to the adoption of interventions that are rejected using current approaches.
Preventing smoking relapse in pregnant smokers who quit during pregnancy
Smoking during pregnancy can be associated with complications during labour, increased risk of miscarriage, premature birth and still birth. However, although many women give up smoking when pregnant, relapse rates are high; up to 70% of women who quit return to smoking within the first 6 months of the birth.11
Studies of behavioural techniques such as advice, information and encouragement have been undertaken to try to reduce the number of pregnant women who have quit smoking relapsing back to being smokers. A meta-analysis of 11 trials found that for a pregnant woman who has quit smoking and receives a behavioural intervention the RR of remaining smoke free at 12 months of follow-up is 1.07, with 95% CI (0.98 to 1.18).12 This evidence indicates that behavioural interventions are possibly effective, but as the confidence interval from the pooled studies includes unity, the authors concluded that there is insufficient evidence to support the use of any specific behavioural intervention for helping smokers who have successfully quit for a short time to avoid relapse. The National Institute for Health and Care Excellence) (NICE) looked at this before the theory of this paper was developed. It took the view that it could not recommend the intervention because the effect had not been sufficiently proven.13 This in practice means that in the UK there has not been any widely supported effort made to help women who have stopped smoking during pregnancy to remain smoke free.
The proposed decision-theory approach would consider this study evidence but would also transparently consider theory and the mechanism by which the behavioural technique interventions cause benefit. Here, harm is caused if the behavioural intervention such as a motivational telephone call causes more women to start smoking again than the absence of such a call. If after consideration of theory and causal mechanisms there is little doubt that the net effect from the behavioural intervention is beneficial, then it is very likely that the lower end of the confidence interval below 1 is an artefact of too small a study size and that the true value of the RR is above 1. Based on the view that the true effect size is very likely to be above unity, the decision-maker should then undertake a cost-effectiveness analysis. This would estimate how large a reduction in the percentage of quitters relapsing would have to occur in a population for the intervention to be cost-effective and use this information to reach a decision.
When cost-effectiveness is considered, it is clear that an inexpensive intervention which prevented even a small proportion of women from starting smoking would be a good use of resources. This is because the health gain from quitting smoking is large. The gain from quitting smoking has been estimated, after discounting, to be about two quality adjusted life years (QALYs).14 However, many of those who relapse following this intervention will stop smoking anyway, well before their last years of life. This reduces the QALY gain attributed to the intervention, and a conservative estimate might propose that the overall gain accruing to a pregnant woman remaining smoke free would be halved, leaving a gain of one QALY. If each QALY is valued at £20 000 (the threshold often used to determine if a treatment should be available to NHS patients) then the value (to the NHS) for each permanent quitter gained by an intervention is £20 000. If an intervention such as personalized advice by telephone (1 h per woman) cost £50 per hour it would only have to persuade 1 in 400 of the women who are currently non-smokers to permanently remain smoke free to be cost-effective.
An intervention that had a reduction in relapse at 12 months equivalent to an RR of 1.07 (the RR found in Hayek et al.'s12 systematic review) would produce 7 additional women smoke free at 12 months for every 100 participants. Although some women may return to smoking later in life, such an intervention would be very likely to result in sufficient permanent quits to be highly cost-effective and would be very unlikely to cause net harm. If applied across the nation, an intervention which would prevent in the order of 5000 women each year from re-starting smoking would yield health gains of some £100 million each year. If aggregated over time and discounted at 3.5% per year, its present value would be of the order of £2 billion (the amount that the NHS would have to put aside to gain the same health gains by a different method). These potentially large benefits could be gained if we correctly consider all the evidence in a decision-theory approach and do not automatically reject all behavioural interventions based on studies with small sample sizes.
Mental well-being and the elderly
An evidence review commissioned by NICE to look at interventions to improve mental health in elderly people found three studies that examined the role of gardening.15 The authors found critical flaws in each study and concluded that there is no robust evidence on the effectiveness of gardening interventions in improving mental well-being. Similarly in the same review, based upon the existence of three poor quality studies that investigated the effect of volunteering, they state that there is no robust evidence on the effectiveness of volunteering in improving the mental well-being of older volunteers or older clients. These conclusions, based upon a very limited subset of data, were used as evidence to help produce NICE guidance. NICE guidance on mental well-being for elderly people16 has no recommendation about the benefits of gardening or volunteering on mental well-being. Using a decision-theory approach, this limited intervention study evidence on the effectiveness of gardening and volunteering would be considered in the light of a theoretical understanding of well-being and observational evidence. Such a consideration would be likely to conclude that the direction of effect from interventions that increase participation in gardening and volunteering is beyond reasonable doubt and allow consideration of the cost-effectiveness of interventions which show promise. Given that the cost of providing some gardening and volunteering interventions could be very small (particularly if administered by volunteers!) these interventions would stand a good chance of being cost-effective.
Discussion
This paper argues for a decision-theory approach to public health decision-making in which theory, causal models and past observations are all given weight and can be used as evidence to reach a decision about an intervention. Often theory will be used in a process of considering how an intervention causes an outcome. This causation process can then be described transparently, assisting the interpretation of existing intervention trial data. When knowledge and theoretical understanding are unable sufficiently to reduce doubt about the direction of effect from an intervention, decisions should be made using standard evidence-based practice and based on the best available evidence which often comes from RCTs. However, when there exists robust theory, a causal understanding and observations that taken together sufficiently reduce doubt about the direction of effect from a public health intervention, current practice should be altered so that this theoretical evidence is given proper regard in the decision-making process.
A lack of trial evidence and observational studies of poor quality are particular issues in public health, which hold back efforts to tackle population health issues.1 In such cases, application of standard evidence-based practice can lead to the rejection of interventions which are both safe and cost-effective but have not been shown to be effective in intervention trials. A decision-theory approach would allow proper consideration of the cost-effectiveness of preventive interventions.
Accepting that a robust theory can predict the direction of change caused by a public health intervention must not lead to the uncritical acceptance of interventions. For different population groups care and transparency is needed in deciding the direction of change of an intervention and decisions should be open to challenge from new evidence.
The proposed decision-theory approach can give sufficient weight to strong RCT evidence but also allow transparent scrutiny of the context and limitations of RCT evidence. The outcome of a trial of a public health intervention that aims to achieve a behavioural change, such as reducing alcohol consumption, is influenced by the characteristics of the subject population. The proposed approach allows consideration of population differences and the likelihood of study results being achieved in different population groups observed in a trial depended on the population being considered.
Considering all relevant evidence in a process that aims to reduce doubt about the direction of effect from an intervention is likely to be beneficial to making good decisions. A randomized trial is a powerful tool for reducing uncertainty about the outcome from an intervention but it does not eliminate doubt. The issue then for decisions about health interventions for populations and individuals is not eliminating all doubt that an intervention will be beneficial but identifying, understanding and transparently describing sources of doubt and taking steps to manage the risk posed by intervening or not intervening. This we believe is best achieved using an approach that allows all relevant evidence to be considered.
If after transparently considering all relevant evidence decision-makers have doubt about the direction of effect of a public health intervention, they should be cautious and not proceed without strong evidence from trials. If they decide that net harm from an intervention is highly unlikely but want additional evidence to quantify the benefits in particular population groups there are approaches that can be used. For example, before the full-scale implementation of an intervention, additional information can be gained by instituting a series of pilot studies with adequate evaluation, and after implementation, by the appropriate use of audit and monitoring. The proposed approach aims to appropriately use all available evidence and if, after making recommendations, evidence emerges about the efficacy of an intervention in practice, this evidence should be used when reviewing past decisions.
Conclusions
The assessment of public health interventions should not invariably follow the same paradigm as that of the appraisal of pharmaceutical drugs. Theory can and should be used more widely and more explicitly. Often theoretical understanding can help interpret trial results and help decision-making by identifying and understanding doubt about the likelihood of a positive outcome from an intervention. Theory, causal models and observational evidence can be used and given weight within a transparent decision-theory approach to improve health decisions. This paper does not endorse either a carte blanche attitude towards the evaluation of public health interventions or lower standards of evidence for them. Rather, it proposes the explicit incorporation of relevant additional evidence from a range of disciplines into the decision-making process to make more rational decisions about interventions which in some instances are failed by a hypothesis-testing approach.