Table 2 Key characteristics of systems evaluation studies that inform the ENCOMPASS framework
Title | Evaluating system change: a planning guide – Hargreaves 2010 [19]. | Applying complexity theory: A review to inform evaluation design – Walton 2014 [12]. | Guidance on systems approaches to local public health evaluation – Part 2: What to consider when planning a systems evaluation – Egan et al. 2019 [17]. | Evaluation of public health interventions from a complex systems perspective: A research methods review – McGill et al. 2021 [5]. | An action-oriented framework for systems-based solutions aimed at childhood obesity prevention in US Latinx and Latin American populations – Garcia et al. 2021 [9]. | A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance – Skivington et al. 2021 [18]. |
|---|---|---|---|---|---|---|
Study aim | To provide guidance on how to plan the evaluation of a system change intervention | To identify themes to take into account when applying a complexity lens to evaluation | To provide guidance on evaluating public health interventions that take a systems approach | To classify the various types of methods employed in systems evaluations and examining the evidence produced by those methods | To provide a roadmap for designing, implementing, evaluating and sustaining whole-of-community systems changes interventions | To support researchers in identifying key intervention questions and in designing and conducting research from a systems perspective |
Identified theoretical elements | (1) Understanding the system (2) Considering emergence and other complexity concepts (3) Defining level and unit of analysis (4) Appropriately timing the evaluation (5) Considering participatory methods (6) Using multiple and mixed methods (7) Defining theories of change | (1) Defining evaluation questions and focus (2) Understanding the system (3) Identifying levers of change | (1) Considering context (2) Developing, refining, testing and retesting programme theory (3) Engaging stakeholders (4) Identifying key uncertainties (5) Refining intervention | |||
Identified evaluation stages | (1) Understanding the conditions and dynamics of the system: boundaries, relationships and multiple perspectives (2) Understanding the elements and dynamics of the intervention: theory of change, intended outcomes, and the monitoring of unintended outcomes (3) Determining the users, purposes and methods of the evaluation | (1) Theorising (2) Prediction (simulation) (3) Process evaluation (4) Impact evaluation (5) Further prediction (simulation) | (1) Fostering multisectoral team (2) Mapping the system, context and drivers (3) Envisaging system-wide changes (4) Effecting system-wide changes (5) Monitoring, learning and adapting (6) Scaling and sustaining | (1) Developing intervention or identifying intervention (2) Feasibility and pilot testing (3) Evaluation (4) Implementation | ||
Examples of methods | - Agent-based modelling - Appreciative inquiry, reflective practice - Case studies, interviews, focus groups - Document reviews - Observations - Outcome mapping, concept mapping - System pattern analyses - Time-trend analysis - Tracking activities | - Agent-based modelling - Interviews - Network analysis - Qualitative comparative analysis - Time-trend analysis | - Agent-based modelling - Concept mapping - Network analysis - Qualitative research with a systems lens - Systems dynamic modelling | - Network analysis - System framing - System mapping - System modelling | - Adaptive policy approaches - Causal loop diagram - Group model building - Interviews - Network analysis - Scenario modelling | - Group model building - Social network analysis - System modelling |