What is margin of error and why does it matter?
Does an online panel represent the general U.S. population?
How do you know your results reflect participation in the U.S. population?
How is this study different than others already out there?
How do you know your estimate of bicycling participation is accurate?
Who is Breakaway Research Group?
All estimates based on samples rather than entire populations are subject to some sampling error. The larger the sample, the better it estimates the population and the smaller the error. The margin of error reflects the maximum expected difference between the true population parameter and the sample estimate.
For example, a total of 14,476 adults ages 18 and over completed the PeopleForBikes participation study in 2018. Among adults, 26% reported that they rode a bicycle at least one day within the past 12 months. The margin of error based on a total population of 249,193,000 adults is +/- .8% at a 95% level of confidence.
All estimates based on samples rather than entire populations are subject to some sampling error (see margin of error for details). A multi-modal approach, like the one used by the U.S. Census yields the most representative sample. However, such an approach is both time-consuming and expensive.
Historically, phone-based surveys dominated as the method of choice among those looking for a representative sample of U.S. residents. However, the prevalence of cell-only households has changed that approach dramatically. In 2013, the CDC estimated that 39.4% of American households were cell only, a lifestyle particularly prevalent among the young and those in lower socioeconomic status groups.
Given the prevalence of cell-only homes, online panels specifically built to represent the U.S. population are now the standard approach. The Pew Research Center estimated that 86% of adults used in the Internet in 2013. Online panels do have biases, but these can be mitigated with an appropriate sampling and weighting strategy.
First, we fielded using one of the largest online panel providers in the U.S. (SSI). SSI takes great care to build and maintain an online sample that is representative of the U.S. population. Second, we weighted the final sample to reflect the U.S. population for age, gender, region, income, and ethnicity. Weighting targets were generated based on the U.S. Census Bureau Current Population Survey.
There are two distinct lines of existing research:
- Research designed to measure participation in a variety of recreational activities including bicycling (e.g., NSGA, OIA/PAC)
- Research focused only on transportation bicycling, which is typically conducted and/or funded by government entities
The PeopleForBikes study was designed specifically to benchmark participation across all types of bicycling and all types of riders.
Although all estimates based on samples rather than entire populations are subject to error (see margin of error for details), we took the following steps to ensure that our estimate is as accurate as possible:
- Bicycling participation was defined very broadly. Respondents indicated how many days in the past 12 months they “rode a bicycle of any type outside for any reason.”
- Bicycling participation was measured in the context of ten activities.
- Research suggests that measuring bicycling participation in the context of a single activity artificially inflates reported participation whereas asking about bicycling participation in the context of more than 25 activities artificially reduces reported participation.
- The list of activities in which bicycling is measured includes leisure activities, transportation, sports, and chores so it doesn’t set a cognitive framework for either recreational or transportation bicycling.
- The ten activities were presented in random order across respondents to control for order effects.
- To control for positive response bias, respondents who reported having visited a fictional website within the past 12 months were terminated from the study. In 2018, respondents who had particularly
fast response times were disqualified as an additional quality control measure.
- Pre-testing indicated that 58% of those who reported having visited the fictional website reported having participated in all ten activities compared to only 5% of those who didn’t report having visited the fictional website.
- Participation estimates were externally validated. The Pew Research Center (2018) estimated that 74% of adults in the U.S. read at least one book in the past year. The results of the 2018 U.S. Bicycling Participation study suggest that 69% of adults read a book at least one day in the past year. Some of this difference may be explained by differences in how the two studies treat audiobook listening and future waves of this study will implement additional measures and respondent quality check to further improve the reliability of results. We also solicited feedback from an external advisory panel comprised of the following members:
- Ralph Buehler: Associate Professor, Urban Affairs & Planning, Virginia Tech
- Charles Chancellor: Associate Professor, Department of Parks, Recreation, and Tourism Management, Clemson University
- Jennifer Dill: Professor, Nohad A. Toulan School of Urban Studies & Planning, Portland State University
- Susan Handy: Professor, Department of Environmental Science and Policy, University of California at Davis
- Kevin Krizek: Professor, Programs in Environmental Design & Environmental Studies, University of Colorado
- Anne Lusk: Research Scientist, Harvard School of Public Health
- Nancy McGuckin: Travel Behavior Analyst
- Chris Monsere: Associate Professor, Civil and Environmental Engineering, Portland State University
- Elliot Rossen: Executive Vice President, Market Opportunities and Innovation, GfK Consumer Experiences North America
- Dean Runyan: President, Dean Runyan and Associates
- Elaine Zanutto: Vice President of Methods and Analytics, Naxion
Our founder named the company Corona because the word means “light.” It’s the knowledge that surrounds and illuminates an issue; exactly what we do. Our firm’s mission is to provide accurate and unbiased information and counsel to decision makers. We provide market research, evaluation, and strategic consulting for organizations both small and large. Learn more at CoronaInsights.com
Other questions? Contact us.