OPEN-ENDED QUESTIONS: Great In Small Doses

Most online surveys rely heavily on “closed-ended” questions. For example, we often have questions that are followed by five-point scaled responses (such as “very dissatisfied” up to “very satisfied”), or that allow the survey taker to “check all that apply” or “check one” from a list. Those are all common approaches in an online survey and are often very appropriate choices. However, one or two carefully constructed open-ended questions can give you a wealth of information.

An open-ended question is a question that is literally open, so a question is displayed and is followed by a blank text box, in which the respondent can type an answer. An example of this is, “Please describe in your own words your last shopping experience at our store?” or “Please describe what’s most important to you when you’re purchasing blue jeans?” Here’s another common example, brand awareness: “For your next purchase of an automobile, what brands are you most likely to consider?”

The Value

Open-ended questions are useful for three types of goals:
1. When you want to hear people describe their opinion or attitude in their own words: What words do they use to describe their satisfaction or experience with something?
2. When you want to measure awareness: Open-ended questions are a great way to measure awareness. What is their top of mind awareness for a brand, a product. For an awareness question, do they name your brand? Do they name brands that you wouldn’t even have considered in your competitive set? It happens, and it’s fascinating when it does…maybe not happy, but fascinating.
3. When you want to discover something: A great discovery question is, “What else can we do to improve your experience with our product?” or “What do you like best about X?” and “like least?” Simple questions like these can elicit very unexpected answers; answers that may reveal important opportunities.

The Catches

Use them judiciously: Unfortunately, people don’t like to type, so it’s a balancing act. We can have one or two open-ended questions and expect meaningful results, but we have to understand that we can’t have an entire survey constructed of open-ended questions. People get tired or bored, and they’ll drop out. A good general rule of thumb is you want no more than one open-ended question for every ten closed-ended questions.

Be realistic: Some people really don’t like to type and might only give you a one or two word response. You can’t really force them to write in complete sentences or in grammatically correct words. Not all of the respondents are going to answer an open-ended question at all, let alone answer as clearly and completely as you’d like. Typically, you’ll get 30 to 40 percent of the people who will write something that’s at least minimally helpful, and obviously, it will vary by topic.

Plan for analysis time: Somebody is actually going to have to read through these text responses, perhaps even code them, and that takes time. One inexpensive option is to use a word cloud tool to see what words occur the most frequently, and that will save you some time.

WORTHWHILE? YES

Market research professionals routinely use open-ended questions, but they use them prudently. We don’t want to wear out our respondents by asking for too much effort.