Ask A Good Product Manager

Question: What is the best way to design a questionnaire/survey to get customer feedback? How do we know that we have the right questions and are getting the right answers?

Answer from Saeed Khan of On Product Management: These are two very interesting questions. Let me answer each one independently:

1. What is the best way to design a questionnaire/survey to get customer feedback? Surveys, in general are very useful tools, but have limitations that need to be understood. The first step is to decide whether a survey is the right medium to get the answers you want.
   • What are your objectives?
   • What questions do you want answered when you are done? And these aren’t necessarily the specific survey questions you intend to ask, but the key insights you want to get out of the survey.
   • Are these insights mostly quantitative, qualitative, or both?
   • Do you need an online survey, a phone survey or a paper-based survey?
   • How many completed surveys will you need to make the survey data useful?
   • What response rate will you need to get the amount of data you’re looking for?
   • What is the incentive for the survey participants to fill out the survey?

   This question bears more discussion. The participants are going to spend some of their time filling it out, but to what end?

   Will it just be information that you collect and use but don’t share with them? Is there some outcome from the survey that makes it valuable to them to fill out? And I’m not suggesting you offer monetary or other inducements to get customers to fill out the survey. The assumption is that you are surveying customers to gain some insights that will be used to benefit both yourself as well as them. If that is not the case, i.e. the benefits are only for you, then you’re going to have a harder time getting people to spend their time.

   One inducement is to agree to share the survey results with the participants. A lot of people like this as it helps them understand how their opinions and answers compare to those of the larger group. An added benefit is that it can help build a stronger relationship with them and you may be able to have follow up conversations with a subset of the survey participants and derive deeper insights on the subject at hand. Keep a few things in mind:

   • A lot of vendors run customer surveys. You’re competing with those vendors for your customers’ time and mindshare.
   • Make sure the survey topic is relevant and to the point. If the topic resonates with people, they are more likely to consider participating.
   • Web surveys have a notoriously low response rate. In my experience it has been around 5%-7% at best with customer surveys.
   • Your results may vary, but plan for the low end. If you want mostly quantitative data, then a focused web survey may suffice.
   • Phone surveys, while more effort per respondent, will provide better qualitative information than web surveys. They’re also good for quantitative research, so if you have a live person on the phone, why not go for both. If what you need is smaller amount of deep data, phone surveys work well.
   • Paper-based survey may sound so 20th century, but can be very effective tools for gathering large amounts of quantitative and some additional qualitative data. This is especially true if the people filling out the survey can be held “captive” for a while, such as at a user group meeting or other similar event. i.e. they have both the interest AND the time to fill it out.

   I’ve had very good success with paper-based surveys at company user group meetings, with a response rate approaching about 70%! This came with a commitment to share back the results with the user groups at subsequent meetings.

2. How do we know that we have the right questions and are getting the right answers? To begin, I’ll go back to the first couple of bullets that I mentioned in question 1:
   • What are your objectives?
   • What questions do you want answered when you are done? And these aren’t necessarily the specific survey questions you intend to ask, but the key insights you want to get out of the survey.

   The questions you ask must be aligned with the objectives of the survey. What are you going to do with data once the surveys are compete and tabulated? Ask yourself if the questions you are planning on asking will best serve those objectives? If not, then rephrase the questions or ask different questions.

   If there is a lack of confidence on whether the questions you are planning on asking are the right ones, then test out the survey with a small group and see if their answers give you what you need. If so, then you’re set. If not, then use the responses to help shape better questions. Keep in mind that survey results always have margins of error and levels of confidence associated with them. Combined, these are a measure of the statistical validity of the results. In general, the larger the sample size and # of respondents, the high the confidence level and lower the margin of error. I won’t go into the details of this as you can find more info on this on various sites on the web, but you should try to keep this in mind, in case someone decides to challenge the data you collect or the conclusions you draw from the data.

   Overall, you should keep a few things in mind: Surveys are simply one of many data and information collection tools at your disposal. They are not a panacea for gaining true insight and understanding. Use them to get a high level understanding of topics of interest and then, in many cases, drill down a bit deeper with some of the survey respondents to collect better contextual information around the responses. For a Product Manager, it is this contextual information combined with support quantitative data that will help you truly make the case for whatever initiatives you are championing.