Skip to main content

Table 5 Descriptive statistics of signalling questions that contributed most to the high risk of bias

From: The effect of changing the built environment on physical activity: a quantitative review of the risk of bias in natural experiments

Bias domain Signalling question Judgement Frequency of eligible outcomes Percentage of eligible outcomesa
Bias due to confounding Did the authors use an appropriate analysis method that adjusted for all the critically important confounding domains? Yes 0 0 %
No 15 100 %
Critically important confounding domains not controlled for    
Differences in baseline outcome measurements - 10 66.6 %
Differences in baseline demographic characteristics - 9 60 %
Any unusual events - 4 27 %
Socioeconomic or political influences - 2 13 %
What variables were used to match intervention and control sites?    
Demographic variables (e.g., ethnicity, income, education) - 5 62.5 %
Features, facilities or amenities - 5 62.5 %
Size - 2 25 %
12.5 %
Land use - 1
Same neighbourhood - 1 12.5 %
Is the control site well matched to the intervention site? Yes 4 26.7 %
No
No information
9
2
60 %
13.3 %
Were there multiple control sites? Yes 9 60 %
No 6 40 %
Bias in selection of participants into the study Is there a fully justified sample size calculation? Yes 0 0 %
No 15 100 %
Is there a clear and sufficient description of the sample? Yes 5 33 %
No 7 47 %
Not applicableb 3 20 %
Bias in measurement of interventions Did the authors describe…    
… what was modified in the intervention? Yes 15 100 %
No 0 0 %
… where the intervention was implemented? Yes 9 60 %
No 6 40 %
… how long it took to construct the intervention? Yes 4 26.7 %
No 8 53.3 %
No (and potential overlap with intervention construction) 3 20 %
Bias due to departures from intended interventions Was individual-level intervention exposure measured? Yes 4 67 %
No 2 33 %
Was individual-level intervention exposure measured objectively? Yes 0 0 %
No 4 100 %
Bias in measurement of outcomes Was the outcome measure valid and reliable? Yes 7 47 %
No 8 53 %
Were the outcomes measured over a period of more than one week at each time point? Yes 3 37.5 %
No 5 62.5 %
Were there multiple follow-up time points? Yes 4 27 %
No 11 73 %
Bias in selection of the reported result Was a study protocol published? Yes 0 0 %
No 15 100 %
Did the authors provide a clear and compelling justification for not publishing a study protocol? Yes 0 0 %
No 15 100 %
  1. aThis percentage is based on the total number of outcomes eligible for that particular signalling question, rather than the total number of outcomes included in this review
  2. bThese studies performed an appropriate analysis to control for differences between intervention and control groups at baseline