So you’re planning to do an online survey and you’re wondering, “how many completes do I need?” There is a short answer, but first, some context.
How confident is confident enough?
You want to have enough data so that you can feel confident in the results. The question is, how confident do you want to be? Are you looking for directional data or do you want something that you can calculate how well it actually represents the target population?
When you watch the news, you sometimes hear polling results. A common description is, “this data is based on a 95 percent confidence interval plus or minus 3 percent.” What does that mean? It simply means that we know that if we were to do this same poll over and over again that we’re confident that the data would be replicated repeatedly with a precision level of plus or minus 3 percent. That’s pretty darn representative, and I’d feel good about that data.
So how would you figure out what that is? Well, if you really do want reliable data, something that represents a broader target market; then you need to have an estimate of that target market’s variability. In fact, one of the biggest myths in market research is that the sample size required is based on the population size (for the population of interest). That’s not quite true. The formula used by statisticians to determine the required sample size is based on the variability of the population. Now, those of us who do a lot of consumer market research know that there are common levels of variability that we’re likely to encounter. Based on this, many market researchers will use rules of thumb. Given the variability we typically encounter in consumer markets, we know that a sample size of 400 to 600 completes will often give us very nice data. In some cases though, that can be overkill. For example, if you’re studying a market where you know (from past research) that most people have very similar opinions and similar behaviors, then you can get away with a smaller sample size and still have representative data.
The reality is that sometimes the sample size requirement is not based on objective needs, but on political ones. You may have internal colleagues who are simply stuck on certain numbers when it comes to survey samples—we see it happen all of the time. Some people believe that they must have 500 completes to feel good about the data, or for some the number is 1,000 or more. If budgets permit, it’s certainly nice to have all that data to play with.
If you plan to do any subgroup analysis, that becomes a sample size consideration. For example, if you know you’re going to want to analyze data by gender, you might want to make sure that you have at enough data so that you can compare the two groups. Or, if you have a question in your survey about purchase intent and you have 4 answer options, you may want enough data to use those 4 “buckets” for subgroup analysts. In reality, many market research sample sizes are based on subgroup analysis needs.
SAMPLE SIZE CALULATORS
There are sample size calculators also, like this one. And if you’re really interested in getting into the statistics of sample size calculations here is a great article.
THE SHORT ANSWER
For some consumer studies, directional data of 100 responses can be used for quick hits. However, if you want data you that can be used to extrapolate your findings to the broader population, then a sample size of 400-600 is preferred…but add in some political needs or sub-group analysis, and you may need to go higher.