We can learn a lot about questionnaire design by looking at examples of what not to do. Below are three problematic questions. Take a look: can you spot the errors?
Example 1: Political Activity
This is from a survey that in previous questions had determined the participant had bought a pair of Brand X jeans within the past 30 days.
Did you spot the errors?
In Example 1, no time frame is specified. Are we asking people if they have really ever done any of these things? Will it be useful if they did these things 20 years ago, but not within the past 5 years? Probably not.
In Example 2, there are a few issues. First, it is unclear if the participant should check one or “all that apply.” The second problem is that some of the items are vague. Researching what? Organizing what? If a lot of people select those items, how useful will that data be? We wouldn’t really know how to interpret it. And the most egregious error in example 2: no options for “Other, please specify” or “None of these.”
While we can’t say from these static pictures for sure, a possible issue with both examples 1 and 2 is whether the lists were randomized or not. Did every survey taker see those lists in the same order? If so, there is a risk that the responses would be subject to “first order bias.” Alas, we know that people tend to focus more on the top of a list than on the bottom. So randomizing lists is important. Of course, always anchor logical choices (like, "None of the above”) at the bottom of a list.
What about example 3? The problem here is that the question is too onerous. We can’t expect participants to hunt for receipts or to have accurate recall. And really, is it realistic for them to subtract out tax and, if applicable, shipping? This question is going to collect some very questionable data.
Questionnaire Design Solutions
These three examples point to some great checklist items for making sure your questions are precise enough to be meaningful and easy enough for the respondent to give accurate information.
1. Specify a time frame. If you are asking participants about past behaviors, or future ones, specify a time frame.
2. Make sure the answer options in a list are complete. When asking people to select attitude and behaviors from a list, consider an “Other, please specify” option. You might get some write-in answers that are unexpected but useful. Also, always offer a “None” or “None of these options.” You don’t want people simply picking an item so they can progress to the next question in the survey.
3. When applicable, randomize.
4. Give clear instructions. For list style questions, always state “Select one” or “Check al that apply.” Even if the survey software enforces logic (so that a person can not select more than one item if it is programmed as a single choose question), you don’t want people to only select one item when really you do want “all that apply.”
5. Keep it simple. Don’t make participants work hard to give you accurate information. If you really do need to collect pricing or budget information, ask for it in ranges. In our jeans example, perhaps something like,” How much did you spend on your most recent purchase of Brand X jeans?” With ranges of:
a. $100 or more
e. Under $40.
Good Surveys, Good Data
Poor questionnaire design can lead to high drop out rates (participants literally dropping out of a survey because it is to long, boring or onerous) and weak data. By following some basic best practices, you will avoid many common questionnaire design problems.
Kathryn Korostoff is the founder of Research Rockstar, an online training company dedicated to teaching market research best practices. She can be reached at KKorostoff@ResearchRockstar.com. For training classes, please visit www.ResearchRockstar.com/classes