Systematic comparative validation of self-report measures of sedentary time against an objective measure of postural sitting (activPAL)

Study design and sample

The study is a validation study of self-report measures of sedentary time based on the TAxonomy of Self-report Sedentary behaviour Tools (TASST) framework [8] (Fig. 1) to provide a systematic comparison process. Currently there are at least 37 different self-report tools for measuring sedentary time in adults, representing at least 140 individual questions [8]. Attempting to provide comparative validation for all of them would be pragmatically very difficult because of burden on respondent and cost, and limit the results to information about existing tools only. Instead, a systematic approach was used which allowed testing and validation of specific characteristics of self-report tools and the assessment of how the combinations of these characteristics influenced measurement characteristics. It was considered more useful to derive information about the taxonomic characteristics that could then be generalised. Eighteen self-report measures of sedentary time, representing combinations of the two main domains of the TASST framework, were validated against an objective measure of sedentary time. Eighteen self-report measures of sedentary time representing all relevant branches of TASST were validated against an objective measure of sedentary time. Cross-sectional data were gathered in three Scottish cohorts of older adults; the Lothian Birth Cohort 1936 (LBC1936) [9], and the West of Scotland Twenty-07 study (T07) 1930s and 1950s birth cohorts [10]. The current study is part of the MRC funded Seniors USP (Understanding Sedentary Pattern) project, which aims to understand determinants of sedentary behaviour. For the Seniors USP project a total of 1757 cohort members (n = 524 LBC1936; n = 1233 T07) were invited to participate in the study. This study was embedded within Wave 4 of the LBC1936 cohort study but ran as a separate data collection wave for T07.

Subjective measurement of sedentary behaviour

Eighteen self-report measures of sedentary time were constructed, based on the TASTT framework, to provide a representative sample of characteristics in current and future self-report tools of sedentary time for use in an adult population. The selection focused on the first two domains of TASTT, namely the type of assessment and the recall period. These two domains cover the majority of the variations found in self-report measures and are most likely to influence the measurement characteristics [8] .

The systematic selection covered five different types of assessment (Taxon 1):

• two single item direct measures of sedentary time (Taxon 1.1.1), where the term direct refers to the TASST nomenclature for measures that ask individuals to recall total sedentary time directly (as opposed to using a proxy measure). This should not be confused with the term direct often used to refer to objective measure of sedentary time.

• a direct question about the total time spent sitting;

• a visual analogue scale of the proportion of the day spent sitting (Fig. 2);

• a single item proxy measure (TV time) (Taxon 1.1.2)

• a composite measure based on pattern (Taxon 1.2.1) asking respondents to report the number of bouts of sitting and their average duration (multiplied together to get total duration);

• a composite measure based on the sum of behaviours (Taxon 1.2.2.1) asking respondents to report the time spent in 13 specific SBs (extracted from [3]): watching TV, work, using computer/screen for leisure, reading, listening or playing music, engaging in seated hobbies, talking, eating, self-care, performing activities of daily living, napping, sitting in transport and sitting during leisure activities outside the home (e.g. watching a play at the theatre)); and

• a composite measure based on the sum of domains (Taxon 1.2.2.2) asking respondents to report the time spent in four different domains (extracted from [3]): work, home, transport and leisure).

Each of these types of measure was applied across three different recall periods (Taxon 2); previous day recall, previous week recall (sometimes called 7-day recall), and unanchored recall (respondents are asked about their usual behaviour without reference to the present, for example a usual week).

All self-report measures also used a temporal unit (Taxon 3) of a day, asking for time spent sitting on a day within the recall period. The assessment period (Taxon 4) was not defined, meaning that all days within the recall period were considered. The actual questions used are available in the Additional file 1.

Objective reference measure

Self-report tools were validated against the same reference measure; average daily sedentary time derived from seven full days of objective measurement. Sedentary time was measured objectively using an activPAL activity monitor (activPAL3c, PAL Technologies Ltd., UK). This small (53 × 35 × 7 mm) and light (15 g) tri-axial activity monitor is worn on the anterior thigh. The monitor samples acceleration at 20 Hz, which is then categorised into time spent in sedentary or upright posture based on thigh inclination. Monitors were waterproofed so participants could wear the monitor for the full measurement period without having to remove and reattach it. They were heat-sealed inside plastic tubing (layflat tubing, Packaging Aids Ltd., UK), and attached by a researcher to the dominant leg using a hypoallergenic double sided adhesive pad (PAL stickies, PAL Technologies Ltd., UK) and covered by a waterproof dressing (Opsite Flexifix, Smith & Nephew, UK). Besides measuring the number of steps and number of sit-to-stand transitions per day, the activPAL classifies activity behaviour into time spent sitting/lying, standing, and walking. The activPAL is valid for step count, time spent sedentary (sitting and lying), standing and walking, and is currently regarded as a gold standard for the objective measurement of sedentary/sitting time and pattern of SB [11–14].

As the activPAL monitor is unable to distinguish between sitting/lying and sleeping, participants were asked to keep a diary reporting the time they fell asleep the previous night and the time they woke up for each day of monitoring.

Procedure

Participants received the first questionnaire at home via post, containing questions about sedentary time for the unanchored recall period. Participants were then met on two occasions. The first visit took place at a clinical research facility for the LBC1936 cohort and at home for the T07 cohort. During this visit the unanchored SB questionnaire was retrieved and a researcher attached the activPAL activity monitor. Participants were asked to wear the activPAL for at least 7 full days continuously including overnight and during bathing/swimming, while continuing their normal daily activities. Day one was the first day of continuous activity monitoring and corresponded to the day after the first visit. During the first visit, the participants were given the two remaining questionnaires. The first questionnaire, asking about sedentary time on the previous day, was completed on day 3 (about SB on day 2) of the monitoring period. The second questionnaire, asking about sedentary time during the previous week, was completed on day 8 (covering the period of day 1 to 7). After day 8, the participants came back to the research facility, or the researcher visited the participants at home, where the researcher removed the activPAL activity monitor and collected and checked the questionnaires.

Cohort data

Demographic information about the participants was obtained either through primary data collection as part of the Seniors USP project, or from the most recent wave of cohort data at which it was collected. Age was reported as an average for each cohort (one LBC1936 cohort and two age cohorts from the T07 study). Gender, socio-economic status (lifetime social class based on highest achieved household occupation, stratified as high (professional, managerial) or low (skilled non-manual, skilled manual, semi-skilled and unskilled), and highest level of education were obtained from previously-collected cohort data. Self-reported health limitations were measured concurrently (categorised as no long standing illness, a long standing illness that doesn’t affect activities, and a long-standing illness that does affect activities). Body mass index (BMI) was calculated from weight measured concurrently, and height measured at last cohort wave.

Data processing

Data were downloaded from the monitor using the activPAL proprietary software (version 7.2.32, PAL Technologies Ltd., UK). Participants with less than seven full 24 h days of activPAL data, or with incomplete sleep diaries, were excluded, in order to avoid making any assumptions about wear time. The statistical programming environment and language R (6) was used to combine the activPAL and sleep diary data to compute waking day sedentary time and number of sedentary bouts for each 24 h period (from midnight) and to perform all the analyses. Objective total sedentary time and number of sedentary bouts were calculated as the mean daily recorded sedentary time and the mean daily number of sedentary bouts over the seven days. Similarly, the proportion of the day spent sedentary was computed as the average over seven days of the daily proportion of sedentary time during waking hours.

Outcomes from all self-report measures had a temporal unit of a day, thus the value reported by the participant was used directly as the value of sedentary time per day in each category. Total sedentary time for the composite self-report measures was calculated as a sum of domains (4 items), or a sum of behaviours (13 items). Finally, for the composite pattern measure, the number of bouts was multiplied by the average sedentary time per bout.

Analysis

The amount of data lost because participants did not answer the self-reported sedentary time questions or provided unreadable or obviously incorrect answers (e.g. more than 24 h of total sedentary time per day) was used to provide an indication of ease of use. Self-report SB was compared with objective SB using Bland-Altman plots and statistics to assess agreement. ActivPAL data were treated as “gold standard” and the Bland and Altman plots computed using Krouwer’s method [15]. Pearson correlation was used to assess criterion validity. The mean difference between objective data and self-report measures was used to gauge the accuracy of the self-report measure (systematic error). This value can also be used as a correction factor to be added to values obtained with a specific self-report tool in the right taxon in order to gain more accurate estimates sedentary time. The standard deviation of the difference between objective data and self-report measures was used to gauge the precision of the self-report measure (random error).

Response and sample descriptive statistics

Data were pooled from all three cohorts. Seven hundred and seventy three participants agreed to wear the activPAL; of those, 700 provided seven full days of data and were included in this analysis. Cohort ages were 64 (T07 1950s cohort, n = 310), 79 (LBC1936, n = 271), and 83 (T07 1930s cohort, n = 119). There were slightly more women (n = 361, 52%) than men. Participants had a mean BMI of 27.6 ± 4.5 kgm^− 2 (range 17.0 to 50.5), with the majority from higher socioeconomic classes (n = 422, 60%), and 206 (29%) achieved degree level of education. About half the participants (n = 327, 47%) reported they had no long-standing health conditions, and only 160 (23%) reported their long-standing health condition limited their life.

Validation

For each of the 18 self-report measures, results from Bland-Altman statistics including the bias in mean and limits of agreement (LOA), standard deviation of the difference, percentage of data loss, and Pearson correlation coefficient and p-values are reported in Table 2. Bland and Altman plots are presented in Fig. 3 for the direct measure of total sedentary time, direct measure using a visual analogue scale, proxy measure (TV time) and composite measure based on sum of behaviours for a previous day recall. All other plots are in the Additional file 2.

Self reported measures	TASST Taxon (Fig. 1)	% data loss	B-A bias Correction factor (min) [%]	B-A Lower LOA (min) [%]	B-A Upper LOA (min) [%]	St. dev of difference (min) [%]	correlation	p-value
Previous day recall
Single item total sedentary time (min)	1.1.1/2.1	1.0	207.0	− 146.8	560.8	180.5	0.20	< 0.001
Single item Proportion of the day (%) [min]	1.1.1/2.1	0.7	12.9 [123.8]	−22.1 [−212.2]	47.8 [458.9]	17.8 [170.9]	0.28	< 0.001
Single item proxy TV time (min)	1.1.2/2.1	2.4	439.0	144.8	733.2	150.1	0.24	< 0.001
Composite Sum of domains (min)	1.2.2.2/2.1	0.7	130.3	− 272.5	533.1	205.5	0.23	< 0.001
Composite Sum of behaviours (min)	1.2.2.1/2.1	4.3	− 145.2	−651.2	366.7	258.2	0.23	< 0.001
Composite Pattern (min)	1.2.1/2.1	9.7	138.0	− 471.6	747.5	311.0	0.17	< 0.001
Previous week recall
Single item total sedentary time (min)	1.1.1/2.2	4.9	221.3	− 149.9	563.9	182.1	0.23	< 0.001
Single item Proportion of the day (%) [min]	1.1.1/2.2	0.9	11.8 [113.2]	−19.3 [− 185.3]	42.9 [411.8]	15.9 [152.6]	0.36	< 0.001
Single item proxy TV time (min)	1.1.2/2.2	2.6	406.2	80.6	731.9	166.1	0.23	< 0.001
Composite Sum of domains (min)	1.2.2.2/2.2	3.9	34.5	− 412.9	482.1	228.3	0.30	< 0.001
Composite Sum of behaviours (min)	1.2.2.1/2.2	16.9	− 244.9	− 755.2	265.3	260.3	0.32	< 0.001
Composite Pattern (min)	1.2.1/2.2	14.6	99.1	− 528.8	726.9	320.3	0.23	< 0.001
Unanchored (usual day)
Single item total sedentary time (min)	1.1.1/2.4	2.4	250.6	− 135.1	549.2	174.6	0.20	< 0.001
Single item Proportion of the day (%) [min]	1.1.1/2.4	2.3	16.1 [154.6]	−18.9 [− 181.4]	51.1 [490.6]	17.9 [171.8]	0.32	< 0.001
Single item proxy TV time (min)	1.1.2/2.4	1.9	415.0	124.7	705.4	148.1	0.26	< 0.001
Composite Sum of domains (min)	1.2.2.2/2.4	1.9	77.9	− 372.8	528.8	230.0	0.16	< 0.001
Composite Sum of behaviours (min)	1.2.2.1/2.4	9.1	− 219.8	− 725.3	285.6	257.9	0.33	< 0.001
Composite Pattern (min)	1.2.1/2.4	7.1	472.9	−33.8	979.6	258.5	0.02	0.624

Single item proportion of the day was converted into sitting time per day in minutes assuming a standard 16 h waking day

Accuracy

The Bland and Altman statistics show that the differences in sample mean measured by the self-report variable from the objective measure were often large, ranging from a 473 min underestimation for a composite measure based on pattern and an unanchored recall period, to a 245 min overestimation for a composite measure based on sum of behaviours and a previous week recall period. With the exception of composite measures based on sum of behaviours, all of the self-report measures underestimated time spent sitting. In general, the type of assessment was more of an influence on size of bias than the recall period. Within each recall period, composite measures based on sum of domains consistently had the lowest bias.

Precision

The Bland and Altman plots (Fig. 3 and Additional file 2) show that the limits of agreement were very wide (Table 2) ranging from around 600 min to 1000 min. Proxy measures, apart from a few cases, consistently underestimated total sedentary time. For all other self-report measures the error could be in either direction (under or over reported). However, there were marked trends, with error decreasing proportionally to sedentary time in several self-report measures (Fig. 3 and Additional file 2). For evaluation of total sedentary time using a single question there was a tendency for individuals engaging in less sedentary time to over report and those engaging in more sedentary time to under report.

The biggest influence on the precision of self-report measures (standard deviation of the difference) was the type of assessment used, with recall period having less influence. Across all recall periods, the proxy measure (TV time) and the assessment with a visual analogue scale of the proportion of the day spent sedentary (single item direct measure of sitting) had the lowest standard deviation of the difference (around 160 min). Composite measures based on pattern and the sum of behaviours had by far the lowest precision of those assessed around 320 min and 260 min respectively.

Correlation

Correlations of the self-report measure with the objective measure were all low, with correlation coefficients ranging from 0.02 to 0.36. Many of the self-report measures clustered around similar values; out of 21 measures, 13 (62%) were in the range 0.2 to 0.3, while 8 (28%) were either 0.23 or 0.24. Within each recall period, a single item based on proportion of the day and a composite measure based on sum of behaviours tended to have higher correlation with objective measures. Recall period did not appear to have a large influence on correlation; however, for those types of assessment with higher correlation coefficients, previous day recall tended to be worse than either previous week or unanchored, but that difference was not statistically significant. The composite measures based on pattern had a particularly poor correlation with objective measures. Self-reported number of sedentary bouts showed a significant correlation (p = 0.017) with the objective measure for only a previous day recall period, but this was very low (correlation coefficient of 0.05). This suggests that self-report number of sedentary bouts is unlikely to be valid measure.

Data loss

The amount of data lost because participants did not answer the self-reported sedentary time questions or provided unreadable or obviously incorrect answers (e.g. sedentary time larger than 24 h) is reported as percentage in Table 2. Overall, the previous day recall period had a slightly lower percentage of data loss and the previous week recall period the highest. The lowest percentages of data loss were for a composite measure based on sum of domains with a previous day recall period and the single item assessment of the proportion of time sitting for both previous day and previous week recall periods. However, differences in data loss between most self-report measures were relatively small. The exceptions were the number of sedentary bouts, composite measures based pattern, and the sum of behaviours, which had consistently large proportion of data loss. The source of data loss was different between measures; most data loss for composite measures based on the sum of behaviours was due to obviously incorrect answers, whereas the pattern metric was more evenly balanced between data not reported and data that were obviously incorrect.

Comparison between self-report measures

Overall accuracy is low and criterion validity is relatively similar for most tools. However precision and data loss show more salient differences. Figure 4 shows a comparative plot of the precision and percentage of data loss for the 18 self-report measures of sedentary time. This plot can be used to guide the selection of self-report measures and to identify characteristics of these measures influencing measurement. Eight of the 18 measures cluster with less than 4% of data loss and 180 min (3 h) of error. Amongst these, the most advantageous appear to be proxy measures (TV time) and direct measure of the proportion of daily sedentary time using a visual analogue scale, regardless of the recall period. Composite measures appear to be subject to more error. The amount of data loss and error grow with the complexity of the composite measure, either because of the number of items included (sum of behaviours) or difficulty in estimating the components (pattern).

Recommendations

This comparative validation study clearly shows that no self-reported tool of sedentary time provides a measurement of sedentary time with the same accuracy, precision and validity of objective SB measures. Therefore, when possible, objective measures should be used instead of self-report tools. Using the results of this study, in conjunction with the TASTT framework, some recommendations can be made about choosing the best possible self-report tool to measure SB if an objective measure is not possible. These are summarised as a flow chart in Fig. 5 and will depend primarily on whether a survey or study already uses a pre-existing measure, and on the main aim of the study [21]. Recommendations are expressed here in terms of taxa within the TASST framework, but this can be translated to specific self report tools using Additional file 3 which provides a table mapping the existing tools identified in a previous review [8] against the TASST domains assessed in the current validation study. For an existing survey that already includes assessment of sedentary time, it is probably only worth changing this assessment if the tool used does not fall within the type of assessment covered by a single item (direct taxon 1.1.1 or proxy taxon 1.1.2) with any recall period, as any gain in precision or reduced data loss are likely to be small (Fig. 4). In this case, the continuity of data collected with previous samples/studies is probably more important than moderate improvements in future data collected, especially in terms of population surveillance. In the case of a new survey or study, or the first introduction of a sedentary time assessment, the choice should be guided by the aim of the survey or tool.

Surveillance

If the primary aim is surveillance of total sedentary time, then using a visual analogue scale and either a previous day or previous week recall period would give the most accurate and precise results. However, if the aim of the surveillance is to look into more details at a specific SB or a specific domain where SB occurs, but an estimate of total sedentary time is still required, then composite measures should be adopted. A composite measure based on sum of domains would be preferable to one based on sum of behaviours for surveillance of total sedentary time, and should be used unless the aim of the survey or study is to monitor distribution of time between SBs. For this type of assessment, however, gains can be made in terms of reduced data loss by adopting short (previous day) or unanchored recall periods.

Epidemiology

If the primary aim is epidemiological research, then strong correlation and low random error are the most important measurement characteristics to consider. Using a proxy measure with an unanchored recall period or adopting a visual analogue scale assessment of the proportion of the day spent sitting provide the most valid measures with the lowest data loss. These should be preferentially used over other types of assessment and recall periods. However, a recent consensus recent consensus highlight that understanding the context of sedentary time is a research priority [22]. In this case composite measure would be more appropriate. Choosing the appropriate recall period might avoid unwanted data loss (Fig. 4).

Strengths and limitations

The main limitation to this study is the lack of objective detection of waking time. The analysis relies on waking day data and this is ascertained using a sleep diary. While these are generally considered reliable and valid [23], they are not free of bias and error. Consequently, they may have degraded the quality and accuracy of the reference measure. Although automated methods to detect sleep show promise [24, 25], they do not currently offer a sufficient advantage over a sleep diary.

Finally, the results should be interpreted with care because they are based on a sample of older adults from three ongoing cohorts, only some of whom were employed, and might not be directly generalisable to self-report assessment in children or adult populations, for different cultural contexts, or in those perhaps less interested in their health than those who volunteer for repeat data collections within an ongoing research cohort. While the systematic approach taken in this comparative validation process should provide generalisable information for all self-report tool, replication of this process would provide definite proof of consistency of the findings.

Future

The results show that it is unlikely that great improvements in accuracy can be gained by developing new questionnaires or adapting existing ones. The heteroscedasticity (the variability of a variable is unequal across the range of values of a second variable that predicts it) present in several of the self-report measures suggest that part of the error is not entirely random and might have some deterministic sources. This suggests that individual answers could be corrected with some calibration equations using respondent characteristics. However, a recent study found that calibrating a single item direct measure of total sitting measured on 183 blue-collar workers based on a prediction model using standard demographic information only improved accuracy by 10 to 30% [26]. Future research should certainly consider exploring calibration of data, however this may lead to overfitting the data or increased burden. Another potential route for improvement might be using adaptive testing and presenting type of assessment and recall period tailored to the individual respondent.

There is an increasing interest in studying SBs in more detail and context is seen as key [11], so more questionnaires using composite assessment are appearing [27]. These composite measures are a trade-off. They provide information about time spent sedentary in specific behaviours or domains and still enable an estimate of total sedentary time to be made. However, the subscales within these composite measures are never validated, so the quality of the information on time spent in specific domains or behaviours is really unknown. Additionally, as seen within this study, the measurement characteristics of these sums to assess total sedentary time are inferior to other type of assessments. In the future, specific validation of sub-scales to ascertain their individual validity should be performed against an appropriate reference measure other than total sedentary time.