The definitive guide to creating effective Survey

Introduction 

Surveys are a simple and effective way to gather information about your users. They have been more and more widely used by not only marketers or strategists, but also by user experience researchers. They can provide valuable information about users’ attitudes, preferences, problems and experiences and advise on how to improve UX (user experience) in your product. You can use it to quickly gather information from a big sample of participants, validate your hypotheses, decide on the next features of the product or you can use it as a means to satisfy your stakeholders, such as when making higher risk decisions.

What users say they do and what they actually do often aren’t the same. That’s why surveys should be complementary to other methods that are better suited for capturing behaviour (such as usability testing, first click test or a tree test) and to website analytics data, if you have any. It’s also not trivial to create a good survey. Furthemore, when they’re often overused, it makes users less and less motivated to participate in your - or any other - survey.

In this guide, we’ll outline when to use surveys, how to construct them to be user friendly and how to motivate participants to engage in your survey. We’ll also discuss the UXtweak Survey tool and how to make the most of it.

Identify your goals 

One of the most important requirements when creating a good survey is a well-defined goal. You need to know why you’re conducting a survey, what information you want to gather, what you want to learn about your users and what the data is going to be used for. It’s also good to always ask yourself the question whether you could get the information in some other, perhaps better way, rather than using a survey (i.e. using other UX tools and metrics or using website analytics data).

Usually when you’re creating a survey, you will have a problem that needs to be solved, a question that needs to be answered. These can come from various sources - from stakeholder meetings, from analytics data, from support and user pain points, from market research or even from the product design process itself. Before you start creating the questions for your survey, always make sure to know what the problem is and whether a survey is the best way to solve it. If it is, focus your questions on this problem.

Open vs. closed questions 

There are generally two kinds of questions that you can ask - open and closed questions (sometimes also referred to as open-ended and closed-ended).

Let’s start with closed questions. These provide the respondent with a restricted number of answers to choose from. In the UXtweak Survey tool, we offer six different formats of closed questions:

1. Checkbox select (the user can select multiple options)
2. Radio button (the user can select only one among a list of options, suitable if you have a limited list of options, e.g. handedness - left-handed, right-handed, ambidextrous)
3. Dropdown select (the user can select only one among a list of options, more suitable when you have a lot of options, e.g. country of residence)
4. Likert scale (user selects a number on a five or seven point scale, often to express the agreement with a statement, i.e. rate your experience using our app, 1 = very unsatisfied; 7 = very satisfied)
5. Net promoter score (user indicates customer experience on a 10 point scale answering questions like: How likely are you to recommend our product to a friend or colleague?)

With closed questions, the data we’re collecting is quantitative. While data from closed questions is thus easier to analyze and represent graphically, as well as quicker and less demanding on the respondents, it only gives us limited information. In checkbox select, radio button and dropdown select questions, it’s important to include all (or most) of the possible responses, so that every respondent can pick an answer that suits them best. This may be a little tricky and sometimes requires doing a small preliminary survey beforehand to see what types of answers are relevant for your users. It’s a good practice to include options like “other”, “don’t know” or “not applicable” to be safe. However, you still need to provide enough other options, otherwise you’re creating a catch-all category and you won't be able to get quality information from the question.

In open questions, you give respondents the ability to type whatever response they want. UXtweak provides two open question formats:

1. Single line text(for when you anticipate shorter answers)
2. Multi-line text (for lengthier answers)

These questions generate qualitative data. While qualitative data can tell you more about your users, give you context and insights into their attitudes and problems and provide you with information you often didn’t expect, they’re also much more time-consuming and complex to analyze. They’re also more demanding on your participants, since they have to formulate the answers and type them. Too many open questions can lower your response rate. This can be the case particularly for mobile users.

To cut a long story short, the best way to go is to find the right ratio of open and closed questions for your survey. If you can create a closed question with a sufficient number of options, use it; don't make your respondents type down the answer if they can just select it. But if you need more complex feedback, don’t be afraid to use open questions and let the respondents express themselves freely. In the next sections, we will provide some tips on how to write good questions for your survey.

Do’s 

When creating your questions, keep your goal in mind and only include the questions necessary for your survey. The longer your survey is, the higher the break-off rate and the more difficult it is to get participants who will attentively complete the entire thing. It may be a good idea to focus on time rather than the number of questions. How long does it take respondents to fill in your survey? How much time will they be probably willing to spend on it? Sometimes, it’s better to have two shorter surveys from two different smaller samples and getting valuable information from both; rather than one longer survey with more participants, most of whom give up before reaching the finish line.

Include a good welcome message and instructions. You should let the participants know why you’re conducting the survey and how it will help you or inform your future policies. People are less likely to feel like they’re wasting time if they know what they’re contributing to. It’s also important to let them know the approximate time it will take to fill in the survey and what kind of data you will collect. Especially if you’re collecting sensitive data, reassure participants about how you will handle the data and how you will protect their privacy.

Keep your language clear. Use a terminology that the respondents will understand. Avoid technical terms or other types of slang. If anything has multiple meanings, is open to interpretation or needs explaining, then explain. Define it; make sure there is a common understanding. This will help you get quality data.

When using closed questions, especially any type of scale or rating, be sure your scales are well balanced. For example, instead of a scale like this:

where on the left we have a very strong statement, but not so on the right and thus the entire scale is skewed towards the left; rather use something like this:

where the semantical difference between the far left option and the middle is approximately the same as between the middle and the far right option.

When you’re using closed questions with a lot of options, be mindful that the order of the options matters. People tend to only scroll so far, until they find an option that seems to suit them well enough. It is also important to think about the wording of the options as the first word in each option is what people will probably be paying most attention to while scanning the list. When having questions with options it’s also important to make sure that the options aren’t overlapping, or the respondents may have difficulty choosing just one.

Before you launch your survey to respondents, it’s always good to do a pilot test. Test it on a small number of people (they don’t necessarily need to be your target audience - although it’s better if they are - they can be your colleagues or acquaintances). This will give you information about how long it takes to fill in the survey. You should also get feedback on the language of the questions and instructions; whether everything is clear and understandable.

Don’ts 

To create a good survey that will generate quality data, it’s very important to avoid questions that are leading or introduce bias. For example, instead of asking how easy or how difficult something was or how much people liked something, rather ask what was your users’ experience with the thing, how they were finding the interaction with the product etc. This way, instead of leading your participants into saying that something was easy or difficult, that they liked or disliked it, you give them the opportunity to express their real authentic feelings and opinions. It is also good to avoid highly subjective terms such as “valuable” or “necessary” - they may mean something else to you and to each of your respondents and hence lower the informative value of your data.

To make your language clear and understandable, avoid double negatives where possible. Especially in closed questions, they can be quite a conundrum for your respondents to decipher and they can make it difficult for them to answer the question.

Another thing to avoid are the so-called double barreled questions, which are questions asking about more than one thing. Always make sure that in a single question, you’re only asking about one thing. For example you shouldn't ask “How do you find our newsletter and customer service? Your user may love your newsletter but have a really bad experience with your customer service. If you need opinion about both, use two questions (but it is worth reiterating here, that you should only ask questions that you really need the answers to and will be able to utilize for improving your product or service).

It’s ok to ask about opinions, attitudes or preferences, but avoid asking about behaviour. You can still ask for feedback about behaviour (i.e. When you did this thing that you did, how was it?), but avoid asking about a future or a hypothetical behaviour. There are two main reasons for this. Firstly, people are not very good at predicting behaviour (yes, even their own) and secondly, there are better tools in UX research to explore behaviour such as usability testing, session recording or tree testing.

Give your survey a structure 

Now that you have your questions, you can start creating your survey by ordering them and deciding on a structure. You want to order the questions in a way that’s the least demanding for your participants. Keep related questions together, don’t jump back and forth between topics. Try starting with easier questions to warm up your respondents, then add more difficult, complex ones and end on a lighter note again with a few easy questions.

It’s good to pay attention to how new concepts are introduced in your questions to try to avoid so-called priming. In the context of surveys, priming is when concepts introduced earlier in the survey can influence responses to later questions.

When it comes to displaying your questions, there are two options currently available in UXtweak’s Survey tool; default option where all questions are on one page and the respondent scrolls through them (the option is known as scrolling) and an additional option where each question is displayed on a separate page (also known as paging). The advantage of scrolling is that it seems to be faster to complete, however there are indications that paging can lead to more quality answers and is more convenient for mobile users.

There’s a possibility to add the “Previous” button to your survey in UXtweak if using paging. This will enable your users to go back to the previous question if they answered incorrectly. Research shows that while the Previous button is generally used rarely, it may mildly decrease the break off rates and increase the quality of responses.

Gather participants and collect data 

Your survey participants should be representative of the larger population for which the survey is intended (by population we mean the pool of individuals from which the sample for a research is drawn. If you’re creating a survey for your users, the population would be all your users and a sample would be the users that actually participated in the survey). It’s especially important to pay attention to your selection of participants if your population is heterogeneous. In this case your participant sample should include all kinds of characteristics that you see in your population.

Using UXtweak, you can approach your participants in three ways. You can use the link to your study, which you can include in your newsletter or post on social media. You can take advantage of the customizable Recruiting Widget - a little widget that will appear directly on your webpages to inform your visitors about the research and allow them to participate with one click. Or, if you want to recruit participants who are not your current visitors or customers, you can use UXtweak User Panel, which allows you to specify what kind of participants you’re looking for and we’ll recruit them for you.

No matter which option you’ll choose, you should be aware of the so-called nonresponse bias. Nonresponse bias means that from all the people you will approach, only part will actually end up participating and these people may be significantly different from the people who chose not to participate. Generally, there is not much you can do about this, it’s just something that’s good to take into consideration.

If you know you want to exclude a group of participants for some reason, you can set up a screening question. This question will be displayed as the first question in your survey and all the participants who answer with the option that you marked for exclusion, will be automatically excluded.

As already mentioned, there are many surveys circulating around and people may not be very motivated to participate in yours. So sometimes it may be good to use an incentive. Offer a coupon or a free trial, a gift card or an early access to those who participate. However, bear in mind that this is also something that may introduce bias in your data. Some people may take the survey just to get the reward and not pay much attention to answers. Others may tend to answer more positively.

In case you’re making changes in your product, it may be useful to collect survey data over a longer period of time and see how the responses change with the changes to your product or a website.

Interpret your results 

The data you’ll get will depend on the types of questions that you ask. As already mentioned, the quantitative data from closed questions is easier to analyze and visualize in comprehensible graphs. In the UXtweak Survey tool, you can also filter the data depending on different conditions (e.g. based on the type of device used by participants to complete the survey).

For the qualitative data from open questions, a good strategy is to group similar questions together and look for common themes. One advantage of such categorization is that you can still provide your shareholders with some qualitative metrics that are easy to understand (i.e. 20% of respondents complained about the website’s menu being confusing).

To sum up, surveys can be a powerful and flexible tool to gather data about your users, regarding their attitudes, preferences and experiences. However, make sure not to overuse surveys (and to take advantage of other UX tools to explore behaviours). Take your time to carefully craft your questions, because only quality questions will get you quality data. With the UXtweak Survey tool you can easily set up a survey and get it to your users.