Table 1 Summary of the seven bias domains and types of signalling questions added to the ACROBAT-NRSI

1) Bias due to confounding

Confounding occurs when one or more variables also explain the observed relationship between exposure and outcome.

The following four critically important confounding domains were identified: (1) baseline outcome measurements; (2) baseline demographic characteristics (including age and gender as a minimum standard); (3) any unusual events; and (4) socioeconomic or political influences. Following this, a number of signalling questions were also added to this bias domain concerning the control site; including how well the control and intervention site were matched in terms of built environment features and population demographics, whether there were multiple control sites, and whether any significant changes occurred to the control site during the study period.

2) Bias in selection of participants into the study

This bias domain refers to the exclusion of eligible participants that biases the outcome.

Signalling questions were added to determine whether a fully justified sample size calculation was carried out, and whether both the sampling criteria and the sample were clearly described.

3) Bias in measurement of interventions

Bias in this domain occurs when intervention status is misclassified; that is, when errors in measuring participants exposure to the intervention biases the estimated effect of the intervention.

Signalling questions were added concerning whether the selection of the sampling site was appropriate and justified, and also whether the intervention was clearly reported in terms of what was modified, where the intervention was implemented, and how long it took to construct the intervention.

4) Bias due to departures from intended interventions

This bias domain refers to systematic differences between intervention and control groups due to departures from the intended intervention.

Signalling questions were added to consider whether any delays or changes in intervention construction impacted upon the study, and whether individual-level intervention exposure was measured.

5) Bias due to missing data

Studies that have missing data increase the risk of selection bias, thus resulting in a misrepresented sample.

Signalling questions were added for the response rates at baseline, follow-up, and the overall response rate.

6) Bias in measurement of outcomes

Bias can occur when there are errors in measuring outcomes of the intervention.

Additional signalling questions related to whether outcome measures were clearly described, valid and reliable, timing of measurements, whether there were multiple follow-up time points, and potential performance biases due to participants’ awareness of the study.

7) Bias in selection of the reported result

This domain refers to the selective reporting of fully reported results.

Signalling questions added to this section focused on whether a pre-registered study protocol was published specifying the objectives and methods of the study.