Proposal for Impact Calculators
Background: The following is an excerpt from this manuscript:
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM.Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006 Oct;21(5):688-94. Epub 2006 Aug 31.
Rationale: While the individual RE-AIM dimensions are useful in their own right, there may be times (e.g. when having to choose among different programs) that it would be helpful to have an overall summary RE-AIM score. This exercise is of course dependent upon many assumptions, but worth considering (for an alternative or complementary graphical display approach see Visual Displays). Our proposal below a) considers some of the multi-level and setting impact issues involved in potential translation; and b) addresses some of the public health perspective issues by combining different outcomes (e.g. often reach and efficacy seem to be inversely related- ignoring reach may seriously overestimate the population estimates of impact).
Proposed measures: We propose one standard measure of “Individual Impact,” and one standard measure of “Organization/Setting level Impact,” to be calculated as specified below. Following this basic description, we discuss other approaches to calculating impact and our justification for including adoption and implementation.
PROPOSAL 1: Individual Level Impact
Calculation: Individual Impact = Sum across target behaviors of (Reach X Average of individual change at Long-Term Follow-up)
Reach = Percent of eligible participants approached who participate divided by number of eligible participants (see Reach Calculator for detailed assistance with calculation of reach estimator. Data source links to obtain local estimates of your target population can be found on our Links Page). Note- this measure should be adjusted for attrition and missing data. Our current thinking is that an intent to treat or imputation procedure be used.
Average of individual change at 12-month follow-up. There are at least two possible ways to do this:
- calculate for each target behavior the effect size (M change from baseline to 12 months in Tx condition- Mean change in control condition) / common S.D. then average across behaviors
- calculate for each target behavior the percent change from baseline, or possibly the percent of persons who achieve some criterion for “clinical or public health significance”- and use this measure to calculate a “risk or goal achievement difference score” by comparing treatment to control conditions
PROPOSAL 2: Organizational/Setting Level Impact
For each study conducted in a community or organizational setting (e.g. school, worksite, religious organization, medical facility) calculate Organizational/Setting level Impact
Calculation = Adoption X Implementation
Adoption = Percent of organizations eligible and approached for participation who ended up participating. Note- should be adjusted for attrition of settings/organizations similar to how attrition handled at individual level above using some type of intent to treat analysis.
Implementation = Average percent (across intervention agents and across different intervention components) extent to which protocol was implemented as intended. Note- Ideally, one would also multiply adoption and implementation by “Institutionalization” or the extent to which sponsoring organizations continue to offer the intervention on a long-term basis. Unfortunately, such data are hardly ever available.
Background: (Following is an except from a recent manuscript submitted for publication)
Realize that public health impact involves more than just efficacy. Our research training and current review criteria all emphasize producing large effect sizes under tightly controlled conditions. However, to make a real-world impact, several other criteria are also necessary.
- At the individual level, several research groups have proposed that Impact (I) = Reach (R) x Efficacy (E). Various authors, including Abrams, et al., have reminded researchers that it is not enough to produce a highly efficacious intervention. To have broad public health impact, an intervention must also have high reach. To the Impact = R x E formula, we would add a third component of Implementation. As discussed by Basch, et al., a program cannot be effective if it is not implemented. Thus, we propose that individual level Impact = R x E x I.
- This individual level focus is, however, not sufficient. To have true impact, an intervention also has to be acceptable to and adopted by a variety of intervention settings, and to be implemented relatively consistently by different intervention agents. In other words, the setting or organizational level impact formula should be Adoption (A) X Implementation = Organizational Impact (OI). Although many authors have discussed issues of nesting and setting factors and how to adjust individual level effects for issues of non-independence, to our knowledge the A x I = OI formula for estimating the impact of an intervention across settings has not been discussed with the exception of an early proposal by Kolbe that Impact = Effectiveness x Dissemination x Maintenance. It is important to emphasize that in terms of overall public health effect, Adoption and Implementation are equally important as Reach and Efficacy, and that we need a much greater emphasis on studies of organizational and system level factors