Importance: Identification of individual-level barriers associated with decreased activity in older age is essential to inform effective strategies for preventing the health outcomes associated with high sedentary behavior and lack of physical activity during aging. Objective: To assess cross-sectional and prospective associations of a large set of factors with objectively assessed sedentary time and physical activity at older age. Design, Setting, and Participants: This population-based cohort study was conducted among participants in the Whitehall II accelerometer substudy with accelerometer data assessed in 2012 to 2013. Among 4880 participants invited to the accelerometer substudy, 4006 individuals had valid accelerometer data. Among them, 3808 participants also had factors assessed in 1991 to 1993 (mean [SD] follow-up time, 20.3 [0.5] years), 3782 participants had factors assessed in 2002 to 2004 (mean [SD] follow-up time, 9.1 [0.3] years), and 3896 participants had factors assessed in 2012 to 2013 (mean follow up time, 0 years). Data were analyzed from May 2020 through July 2021. Exposures: Sociodemographic factors (ie, age, sex, race and ethnicity, occupational position, and marital status), behavioral factors (ie, smoking, alcohol intake, and fruit and vegetable intake), and health-related factors (ie, body mass index, 36-Item Short Form Health Survey (SF-36) physical and mental component summary scores [PCS and MCS], and number of chronic conditions) were assessed among 3808 individuals in 1991 to 1993; 3782 individuals in 2002 to 2004; and 3896 individuals in 2012 to 2013. High alcohol intake was defined as more than 14 units of alcohol per week, and high fruit and vegetable intake was defined as twice daily or more. Main Outcomes and Measures: Accelerometer-assessed time spent in sedentary behavior, light-intensity physical activity (LIPA), and moderate to vigorous physical activity (MVPA) in 2012 to 2013 were analyzed in 2021 using multivariate linear re