Skip to main content

Table 2 Operational Definitions of Risk of Generalizability Biases

From: Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: a systematic review and meta-analysis

Risk of Generalizability Bias Questions to Ask Increased Presence with Small Sample Hypothesized Influence of the Presence of Risk of Generalizability Bias Example
Pilot Larger-Scale Efficacy/Effectiveness Pilot Larger-Scale Efficacy/Effectiveness
  What is the potential for difference(s) between…      
Intervention Intensity Bias …the number and length of contacts in the current study and future evaluations of the intervention? Yes More frequent and longer contacts result in more effective intervention Fewer and shorter contacts results in less effective intervention compared to pilot 19 lessons delivered (Salmon 2008 [34])a 6 lessons delivered (Salmon 2011 [37])a
Implementation Support Bias …the amount of support provided to implement the intervention in the current study and future evaluations of the intervention? Yes Greater amounts of support to implement the intervention results in more effective intervention Reduced support to implement the intervention results in less effective intervention compared to pilot “During the intervention, weekly, audio-taped debriefing meetings were held with the interventionists and project investigators to troubleshoot any problems with each session and to plan for the following sessions.” (Beech 2003 [74])  
Intervention Delivery Agent Bias …the level of expertise of the individual(s) who deliver the intervention in the current study compared to who will deliver the intervention in future evaluations? Yes Higher levels of expertise delivering the intervention results in more effective intervention Lower level of expertise to deliver the intervention results in less effective intervention compared to pilot “…the programme was delivered by the researcher, a PE trained specialist, with extensive experience in the primary classroom.” (Riley 2015 [75]) “Classroom teachers were responsible for the planning and the delivery of all movement-based lessons during the intervention.” (Riley 2016 [76])
Target Audience Bias …the demographics of those that received the intervention in the current study to those who will receive the intervention in future evaluations? No Delivering intervention to more conducive, convenience sample or sample that is not representative of target population results in more effective intervention Delivering intervention to sample of whom the intervention is intended results in less effective intervention compared to pilot “Although our sample size was... predominately white, and well-educated…” (Sze 2015 [77])  
Intervention Duration Bias …the length of the intervention provided in the current study to the length of the intervention in future evaluations? No Shorter duration results in more effective intervention Longer duration less effective intervention compared to pilot 4-week intervention (Wilson 2005 [78]) 17-week intervention (Wilson 2011 [79])
Setting Bias …the setting where the intervention is delivered in the current study and the intervention delivery setting in future evaluations? No Delivering intervention in a more conducive, convenience location that is not representative of the target setting results in more effective intervention Delivering intervention in a location more representative of target setting results in a less effective intervention compared to pilot Intervention delivered on university campus b Intervention delivered in community setting b
Measurement Bias …the measures employed in the current study and the measures used in future evaluations of the intervention for primary/secondary outcomes? Yes Use of less reliable or valid measures of primary/secondary outcomes results in more effective intervention Use of more reliable and valid measures results in less effective intervention compared to pilot Pedometer used to measure physical activity (Lubans 2009 [80]) Accelerometer used to measure physical activity (Lubans 2012 [81])
Directional Conclusions Are the intervention effect(s) in the hypothesized direction? No Less effective intervention Reduces intervention effectiveness “The decline in physical activity among the participants was not anticipated…” (Cliff 2007 [82])  
Outcome Bias Is the primary outcome for future evaluations of the intervention measured in the current study? No Absences of measuring primary outcome results in more effective intervention Absence of primary outcome collected in pilot results in less effective intervention tested in well-powered trial Nutrients sold per day and number of items sold per day in school cafeterias (Hartstein 2008 [83]) Self-reported daily dietary intake of students (Siega-Riz 2011 [84])
  1. aAlthough not labeled as a pilot study, the example illustrates the presence of the risk of generalizability bias in one study and altered in the subsequent trial
  2. bHypothetical example of the risk of generalizability bias as it could operate in a pilot to larger-scale efficacy/effectiveness trial