ASSESSMENT using RE-AIM and PRISM

This section provides guidance on data collection using RE-AIM and PRISM. We posit that RE-AIM and PRISM can be used temporally (before, during, and after implementation), in different settings (clinic, community, corporate), with multiple levels of stakeholders, and for a variety of targeted audiences and outcomes. Depending on the phase of implementation, sources of data, and questions you are answering, either qualitative approaches, quantitative approaches, or mixed methods may be most appropriate. We strongly advocate mixed methods approaches in collecting and analyzing data about the RE-AIM dimensions and contextual factors in PRISM.

Navigation Links:

Key RE-AIM Papers

Questions to ask about RE-AIM Dimensions

Considerations for applying RE-AIM Dimensions

Qualitative Data

Quantitative Data

RE-AIM Consort

KEY PEER-REVIEWED PAPERS HIGHLIGHTING APPROACHES TO ASSESSMENT USING RE-AIM AND PRISM:

Holtrop JS, Rabin BA, Glasgow RE. Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res. 2018;18(1):177. doi: 10.1186/s12913-018-2938-8.

King DK, Shoup JA, Raebel MA, et al. Planning for Implementation Success Using RE-AIM and CFIR Frameworks: A Qualitative Study. Front Public Health. 2020;8:59. Published 2020 Mar 3. doi:10.3389/fpubh.2020.00059

Harden SM, Gaglio B, Shoup JA, et al. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev. 2015;4:155. Published 2015 Nov 8. doi:10.1186/s13643-015-0141-0

Glasgow RE, Estabrooks PE. Pragmatic Applications of RE-AIM for Health Care Initiatives in Community and Clinical Settings. Prev Chronic Dis. 2018 Jan 4;15:E02. doi: 10.5888/pcd15.170271. PMID: 29300695; PMCID: PMC5757385.

Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008 Apr;34(4):228-43. doi: 10.1016/s1553-7250(08)34030-6. PMID: 18468362. https://pubmed.ncbi.nlm.nih.gov/18468362/

McCreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, Frank JW, Hess PL, Burke RE, Battaglia CT. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. 2019 Nov 25;9(6):1002-1011. doi: 10.1093/tbm/ibz085. PMID: 31170296.

QUESTIONS TO ASK ABOUT RE-AIM DIMENSIONS WHEN EVALUATING PROGRAMS AND POLICIES

Return to Top

CONSIDERATIONS FOR APPLYING RE-AIM DIMENSIONS ACROSS SETTINGS AND PHASES OF PROJECTS

Table 1–Examples PDF

 

Citation: Harden SM, Smith ML, Ory MG, Smith-Ray RL, Estabrooks PA, Glasgow RE. (2018). RE-AIM in clinical, community, and corporate settings: Perspectives, strategies, and recommendations to enhance public health impact. Frontiers in Public Health – Public Health Education and Promotion. https://doi.org/10.3389/fpubh.2018.00071

QUALITATIVE DATA COLLECTION ASSESSING RE-AIM DIMENSIONS

Several focus group guides and individual interview guides have been graciously shared as examples of qualitative data collection tools that may be adapted for your use for research and practice, based on stakeholder priorities.

RE-AIM Workgroup Templates

The National Working Group on RE-AIM Planning and Evaluation Framework would like to share templates of focus group and one-on-one interview guides. These are by no means comprehensive of all styles and types of qualitative data collection, but rather a template for you to adapt based on your research and stakeholder priorities.

 

Example Qualitative Data Collection Tools

Some of our colleagues have graciously shared their qualitative data collection tools. We are in the process of compiling this information for upload. These are the intellectual property of the research team but available for your perusal. Contact information was up to date upon posting. Please email Samantha Harden if there are issues.

Topic
Area

Resources
Available

Scope
of Project (timeframe, funds)

Contact
Information

Implementing alcohol
screening, brief intervention, and referral

1. Key Informant Interview Guide
for Nurse Managers,

2. CFIR Interview Guide
(With RE-AIM domains indicated) for alcohol screening (behavioral
intervention) and implementation strategy

3. Breathewell Study Planning Team
interview guide: Items include proposal, planning, design, implementation,
design and adoption, and maintenance phases

 

Diane
K. King, PhD Director Center for Behavioral Health Research and
Services Institute of Social and Economic Research University of Alaska
Anchorage
907-786-1638

The “I Decide” team
developed an interview guide based on the Diffusion of Innovation Theory and
has many RE-AIM constructs.

Patient and staff
pre-intervention (“baseline”), during
intervention (“follow-up”), and
maintenance (“post-intervention”)
interview guides for DECIDE-LVAD.

 

Daniel
Matlock, PhD [email protected] and Jocelyn
Thompson, MA I DECIDE:LVAD
Adult and Child Consortium for Health Outcomes Research and Delivery Science
(ACCORDS) University of Colorado School of Medicine Mail Stop F443 13199 E.
Montview Blvd, Suite 210  Aurora, CO 80045 (p) 530-906-1081
(e)[email protected] https://patientdecisionaid.org/lvad/

Weight Management
Intervention for Endometrial Cancer Survivors

Focus group pre-mortem script-
Zoom Version with participants  

Medical student project,
$2000 max funding for participant compensation

Shannon
Armbruster, MD, MPH  Carilion Clinic
Gynecologic Oncology (p) (540) 581-0275 (e) [email protected]   Samantha
Harden, PhD Dept. Obstetrics/Gynecology Virginia Tech
Carilion School of Medicine (p) (540) 231-9960 (e) [email protected]

Older Ghanaian Adults’
Perceptions of Physical Activity Focus Group Script

Pre-implementation focus group
script with participants

Balis L, Sowatey G,
Ansong-Gyimah K, Ofori E, Harden SM. Older Ghanaian adults’ perceptions of
physical activity: an exploratory, qualitative study. University of Wyoming
Center for Global Studies Faculty International Research Grant, 2017-2018.

Laura
Balis, PhD Assistant Professor and Health Specialist University of
Arkansas System Division of Agriculture Cooperative Extension Service 2301
South University Avenue Little Rock, AR 72204 (p) 501-671-2099 (e)
[email protected]

 

QUANTITATIVE DATA COLLECTION ASSESSING RE-AIM DIMENSIONS

An advantage of RE-AIM is that many of the quantitative outcomes in the RE-AIM dimensions are objective and countable (e.g., number of individuals who received the program out of those who could benefit from it). Others may require quantitative measures (e.g., assessment of fidelity of implementation of a program or intervention). Any quantitative data collection used should be as pragmatic as possible to reduce burden on program participants, interventionists, administrators, and other stakeholders. Examples of important data that can be tracked quantitatively within each RE-AIM domain are:

RE-AIM Dimension

Definition

Examples of Quantitative Assessment

Important Considerations

Reach

The number of people and percent of the target population who are impacted, and the extent to which those reached are representative and include those at most risk.

  • Size of the target population

  • Number of individuals receiving the program/intervention

  • Quantitative comparison of characteristics of the individuals receiving the program to the target population (e.g., risk factors, sociodemographic factors, etc.)

  • Accuracy of estimated or observed size of target population

  • Accuracy of tracking of individuals receiving the program/intervention

  • Identification of most important indicators of representativeness

Effectiveness

A measure of the impact on health, mental health, or other individual-level primary outcome, including positive, negative, and unintended consequences.

  • Objective measure of health outcome (e.g., biomarkers)

  • Objective measure of health behavior (e.g., observational rating)

  • Self-report rating of health outcome (e.g., quality of life)

  • Self-report rating of health behavior (e.g., food frequency questionnaire)

  • Quantitative comparison of outcomes across subgroups (e.g., by risk factors, sociodemographic factors, etc.)

  • Reliability and validity of objective measures

  • Reliability and validity of self-report measures

  • Existence of high quality measures

  • Feasibility and burden of effectiveness measures on individuals and settings involved

  • Comparisons across most important indicators of representativeness

  • Relevance of effectiveness outcomes to key stakeholders

Adoption

The number and percent of settings/providers/interventionists who participate, and the extent to which these are representative of those who the target population will use or visit.

  • Number of settings/providers/interventionists invited

  • Number participating

  • Quantitative comparisons of characteristics of the settings/providers/interventionists participating to those who the target population will use or visit

  • Clarity regarding who the adopters are

  • Changes in adopters from planning through evaluation phases

  • Accuracy in tracking settings/providers/interventionists invited and participating

  • Identification of most important indicators of representativeness

Implementation

Level of adherence to program delivery as intended, including extent to which elements are implemented and/or adapted. Cost of delivering the program.

  • Fidelity or adherence checklists or observation

  • Proportion of intended program/intervention elements delivered as intended

  • Quantity of adaptations made

  • Ratings of extent of adaptations made

  • Cost of program delivery (e.g., time-based activity costing)

  • Reliability and validity of fidelity or adherence assessment

  • Complexity of tracking adaptations

  • Accuracy in rating extent of adaptations

  • Feasibility and burden of tracking adaptations

  • Feasibility and burden of tracking costs

Maintenance

The degree to which the program is sustained (at the setting level) and to which the effects of the program are maintained (at the individual level).

  • Level of intent to continue program delivery (setting)

  • Actual continuation of program delivery at later time points (setting)

  • Maintenance or improvement of health outcomes or health behaviors at later time points (individual) 

  • Clarity regarding who decides to continue program delivery over time (setting)

  • Reliability and validity of intent assessment (setting)

  • Reliability and validity of health outcomes and behaviors (individual)

  • Feasibility and burden of measuring maintenance (both setting and individual)

RE-AIM AND CONSORT

Additionally, quantitative data in RE-AIM are relevant to the CONSORT diagram required by many journals to transparently report the screening, identification, recruitment, enrollment, and participation of people, organizations, and other entities in a project. The Extended CONSORT Diagram links quantitative data specified in RE-AIM to metrics used in reporting study flow:

Combining the CONSORT Statement and Aspects of the RE-AIM Framework

This document proposes seven additional items to the existing CONSORT (Consolidated Standards of Reporting Trials) criteria. For more information on the CONSORT click here. The items increase awareness of and reporting on external validity. Download checklist here and find additionally supplemental checklist resources here.

FOR ADDITIONAL EXAMPLES OF ASSESSMENT APPROACHES — USED FOR EACH RE-AIM DIMENSION AND FOR PRISM CONTEXTUAL FACTORS, PLEASE SEE:

ASSESSING REACH

ASSESSING EFFECTIVENESS

ASSESSING ADOPTION

ASSESSING IMPLEMENTATION

ASSESSING MAINTENANCE

ASSESSING CONTEXTUAL FACTORS

Return to Top