Table 2 Operational Definitions of Risk of Generalizability Biases
Risk of Generalizability Bias | Questions to Ask | Increased Presence with Small Sample | Hypothesized Influence of the Presence of Risk of Generalizability Bias | Example | ||
|---|---|---|---|---|---|---|
Pilot | Larger-Scale Efficacy/Effectiveness | Pilot | Larger-Scale Efficacy/Effectiveness | |||
What is the potential for difference(s) between… | ||||||
Intervention Intensity Bias | …the number and length of contacts in the current study and future evaluations of the intervention? | Yes | More frequent and longer contacts result in more effective intervention | Fewer and shorter contacts results in less effective intervention compared to pilot | 19 lessons delivered (Salmon 2008 [34])a | 6 lessons delivered (Salmon 2011 [37])a |
Implementation Support Bias | …the amount of support provided to implement the intervention in the current study and future evaluations of the intervention? | Yes | Greater amounts of support to implement the intervention results in more effective intervention | Reduced support to implement the intervention results in less effective intervention compared to pilot | “During the intervention, weekly, audio-taped debriefing meetings were held with the interventionists and project investigators to troubleshoot any problems with each session and to plan for the following sessions.” (Beech 2003 [74]) | |
Intervention Delivery Agent Bias | …the level of expertise of the individual(s) who deliver the intervention in the current study compared to who will deliver the intervention in future evaluations? | Yes | Higher levels of expertise delivering the intervention results in more effective intervention | Lower level of expertise to deliver the intervention results in less effective intervention compared to pilot | “…the programme was delivered by the researcher, a PE trained specialist, with extensive experience in the primary classroom.” (Riley 2015 [75]) | “Classroom teachers were responsible for the planning and the delivery of all movement-based lessons during the intervention.” (Riley 2016 [76]) |
Target Audience Bias | …the demographics of those that received the intervention in the current study to those who will receive the intervention in future evaluations? | No | Delivering intervention to more conducive, convenience sample or sample that is not representative of target population results in more effective intervention | Delivering intervention to sample of whom the intervention is intended results in less effective intervention compared to pilot | “Although our sample size was... predominately white, and well-educated…” (Sze 2015 [77]) | |
Intervention Duration Bias | …the length of the intervention provided in the current study to the length of the intervention in future evaluations? | No | Shorter duration results in more effective intervention | Longer duration less effective intervention compared to pilot | 4-week intervention (Wilson 2005 [78]) | 17-week intervention (Wilson 2011 [79]) |
Setting Bias | …the setting where the intervention is delivered in the current study and the intervention delivery setting in future evaluations? | No | Delivering intervention in a more conducive, convenience location that is not representative of the target setting results in more effective intervention | Delivering intervention in a location more representative of target setting results in a less effective intervention compared to pilot | Intervention delivered on university campus b | Intervention delivered in community setting b |
Measurement Bias | …the measures employed in the current study and the measures used in future evaluations of the intervention for primary/secondary outcomes? | Yes | Use of less reliable or valid measures of primary/secondary outcomes results in more effective intervention | Use of more reliable and valid measures results in less effective intervention compared to pilot | Pedometer used to measure physical activity (Lubans 2009 [80]) | Accelerometer used to measure physical activity (Lubans 2012 [81]) |
Directional Conclusions | Are the intervention effect(s) in the hypothesized direction? | No | Less effective intervention | Reduces intervention effectiveness | “The decline in physical activity among the participants was not anticipated…” (Cliff 2007 [82]) | |
Outcome Bias | Is the primary outcome for future evaluations of the intervention measured in the current study? | No | Absences of measuring primary outcome results in more effective intervention | Absence of primary outcome collected in pilot results in less effective intervention tested in well-powered trial | Nutrients sold per day and number of items sold per day in school cafeterias (Hartstein 2008 [83]) | Self-reported daily dietary intake of students (Siega-Riz 2011 [84]) |