The definitive guide to Website Testing

Introduction 

User experience is everything when it comes to things on the web. It’s not just a question of usability (whether users can make use of your website or web application). User experience (or UX) encompasses how the user’s brain perceives, processes and memorizes all information that they come across while interacting with your website or web app.

In user-centered design, the goal is not only to make a great product, but also to shape user experience itself in a certain desired way. A good user experience will not only help you keep users happy, it will also help them achieve their goals and help your web site or web app increase its key performance indicators.

User testing is a technique for collecting information about user experience. Because experience is subjective and intangible by its very nature, user testing relies on observing how users interact with the product as a means to learn about the experience. Website Testing is a tool for conducting user tests that also integrates techniques of session replay. This gives you the ability to conduct user testing online and then replay user interactions like a video.

Website Testing can answer the following questions for you:

• Can people use the website/app for important use cases?
• What do interactions between users and the website/app look like? What are users thinking while using the website/app?
• How do users approach their objectives?
• Is there something keeping users from achieving their goals? Did they take a completely different way than they should have? Or did they take a right path but got lost and confused along the way?

With the use of Website Testing, you will be able to find answers to these questions and pinpoint any problems with user experience. Using the gathered insights on how users interact with your website, you can tweak it to aim for best performance.

How it works 

To fully understand Website Testing, you first need to understand the two main concepts that lie behind it:

• Session replay
• Tasks

During a Website Testing study, the job of the respondents is to go through the website/app and find the right solutions to the tasks they were given. Once the respondent arrives at a screen that they think is the answer to the task, they simply have to click a button to mark that screen as their definitive answer.

Let’s say we have a website of a county, which informs citizens and visitors alike about everything the government is doing in the region. A task for this website might go something like this:

You and your friends would like to go camping somewhere in the XY county. Find what are your options for camping locations.

When should I use it? 

One of the charms of Website Testing is that it’s quite flexible with regards to the stage of design when it can be used. You can both apply Website Testing to test simple HTML prototypes and to test main use cases directly on a live website. All that’s important is that you have at least a working piece of HTML that's hosted somewhere public. That way, Website Testing can link to your website, web app or prototype via URL.

User studies in Website Testing can include testing very different subjects on varying scale. You can set up a simple Website Testing study with minimum effort in just a few minutes. You can run tests with tens of tasks that cover most important use cases, or you can do it with just a single task. You can run tests with only a few people, or you can collect results from hundreds of users. Website Testing is quite flexible so you can create studies that suit you best.

Set your objectives 

Awesome, so you’ve decided that you want to use Website Testing in your user testing! Now, I’m sure you want to jump right into setting up your first study so you can begin collecting all of that juicy user data. However, before we do that, it is best that we dedicate a chapter to knowing your objectives. If you’re clear about what you want to learn from the study and if you plan it accordingly, you will always learn more than if you were just shooting blindly. Here’s some of the most important questions you need to ask yourself before you begin your study:

• What do you want to test and improve?
• Who is the user testing going to be aimed at and why?
• When in the project's life cycle are you doing the user testing?

What do you want to test and improve? 

One of the first things you should decide on for planned user testing is the scope. One situation might call for testing all the main use cases of your app or website because you only recently made the switch to user-centered design and you need to know what works and what doesn’t. In a different situation, all you need might be a more focused testing of a few scenarios that surround a single piece of new functionality.

When it comes to large user tests that span across various areas of functionality, the divide and conquer approach is always a good option. In general, you don’t want your user testing to take the respondent more than hour and a half. Not only will people be unwilling to offer you that much time. Long testing sessions also go against how users usually interact with apps and websites for only just a few minutes at most. Testing too many use cases at once can also pollute the study results, as respondents might remember things from earlier in the test that they wouldn’t normally know and use this knowledge to solve tasks later in the test.

User tests with wide functional coverage still have their place, but it’s generally not a good idea to try and test everything at once. Select which parts of your website or app form the essence and then test the main use cases only for those.

Another option that you have is to run more than just one user testing in parallel. Website Testing makes it trivial to set up A/B user testing, so you can test alternative designs in parallel, compare them and choose the design that performs best.

Who is the user testing going to be aimed at and why? 

You cannot have user testing without first stopping to consider the identity of your users. Since you have a website or an app, you’re probably familiar with who your target group is. You can use any of your existing personas, real customer data, or even data from surveys and other sources of feedback.

Knowledge of your user is useful in many ways. Aside from the obvious (to recruit respondents who represent your audience) it will also help you select the right use cases and formulate relatable tasks that represent user’s real way of thinking. This is why you should keep your users in mind from when you start planning user testing.

When in the project's life cycle are you doing the user testing? 

Knowing the right time and place for doing user testing is just as important as knowing what you’re going to be testing and with whom. User testing can be an immensely useful source of feedback in different stages of development. However knowing what’s appropriate to test, when and why, is key.

One very common use for Website Testing is when you already have a running website. To increase the number of people who successfully complete what they come to the website/app to do (e.g. to buy a product or to find a piece of information), set up Website Testing with the right selection of tasks. You will get all the information you need to drive decisions about tweaking user interactions.

You can also use Website Testing while the website/app is still in development. This will help you design user experiences in iterations, along with the rest of the website/app. Using Website Testing with the same tasks in each iterative version will allow you to evaluate changes to your website as they’re being made.

Even during earlier design stages, when you still have no website to speak of, Website Testing can tell you how users interact with HTML prototypes. Just don’t forget that you need to host the prototypes somewhere public, so Website Testing can access them.

Write your tasks 

At its most basic level, a Website Testing study consists of a list of tasks that the respondents are asked to complete in order to complete the study. By carefully preparing your tasks, you can make sure that your user study will yield good results. Well defined tasks lead the respondents to try complete user stories as naturally as they can. This chapter will teach you the basic principles for defining tasks for user studies.

Writing tasks that cover your objectives 

Each one of your tasks should be possible to track back to its source, in the form of one of the objectives that you’ve decided on for your study. After all, it’s the tasks that shape what aspect of the website is going to be tested.

Let’s say we have an e-commerce website with a shopping cart and one of our objectives is to test whether people can add things into said shopping cart. A task aimed at testing this objective could go something like this: “You’re in the market for buying a new TV. Find a TV that you like and buy it.”.

By analyzing the session replay of this task, we will learn things like:

• How many people have successfully bought a TV?
• How long did it take them to complete the order?
• What did the respondents do in order to buy a television?

By analyzing the results of this task, you will be able to fulfill your objective of improving the shopping cart. Look at the success and time taken metrics to quickly find problematic tasks, then replay sessions to discover problems in respondent behavior.

Writing realistic tasks 

Tasks always give the best results when the respondents can identify with the task at hand, approaching it like they were looking to complete it in real life. This is why it is always best to give respondents tasks that they can relate with or conversely - to recruit respondents who can identify with the tasks given to them. For example, if your objective is to test whether users can buy a TV set on your e-commerce website, it is better to recruit people who are likely to personally buy electronics.

To help respondents reach the right mindset for taking on the tasks, you should always formulate your tasks as scenarios. Doing so helps introduce the respondent to the problem and prompts them to seek a solution.

So, instead of

“Buy a TV of your own choosing”

you would say

“You’re in the market for buying a new TV. Find a TV that you like and buy it.”

While the first version directs the respondents to mechanically find something, in the second version we help the respondent understand the problem, to think about it beyond on surface level and to consider how they would approach it themselves.

Another way to help the respondents really identify with the tasks is to allow them a certain degree of freedom to define their goal.

So instead of asking respondents from your target market to

“Buy the least expensive television in the inventory.”

you could recruit respondents who are actually in the process of looking for a new TV and ask them to

“Find a TV that you like and buy it.”

Writing actionable tasks 

When writing your tasks, it’s always better to ask respondents to do something rather than ask them how they would do it. For example, “How would you buy a television that you like?” is a bad task. A much better option would be to write: “Find a TV that you like and buy it.”

Asking the respondent how they would do something triggers a verbal response. What we want to examine are the respondent’s actions. What the respondents say and what they do can be quite quite different. Self-reported claims are often inaccurate. What respondents actually do is much more reliable for discovering UX issues.

If respondents in your Website Testing study begin writing their steps in the comments, it’s a sign that the tasks are not actionable enough.

Writing tasks without giving away the solutions 

The human brain is wired for recognizing patterns and mental shortcuts to finding simple solutions to complex problems. Therefore, if your task contains any hints that might help the respondent resolve the task more easily, the respondent is going to use them. For example:

Goal: To test whether users can find the contact form

Expected: User clicks Contact us

Bad task: "Find a way to contact us."

Better task: "You have a question about our pricing for your company. Find a way to get more information."

Your tasks should avoid using the same language as is found on the tested website or web app. You might bend this rule somewhat in case the word in question is generally well-known in the domain and writing around it might be difficult (e.g. shopping cart in e-commerce, unless your shopping cart is non-standard in some way). The tasks shouldn’t be too roundabout or vague.

What’s more, the wording of your tasks can also introduce bias into how the respondents think about the tasks. In their natural environment, users can be looking for the same content with quite different questions on their mind. How exactly a person would approach a problem can’t be determined in advance. Your tasks shouldn’t encourage a way of thinking that makes finding the solution easier.

Prepare your questionnaire 

Apart from the tasks themselves, you may also have other questions for your respondents. When analyzing the results of a user study, additional information about the users, such as their demographics, their stances, user experiences (or experiences in general) can be useful. You can later use such data for sorting respondents into groups and filtering out anyone who's irrelevant. Or, if you only want to admit a certain kind of people to take part in the study, you can set up a screening question and filter your target group in advance. This is what the questionnaires are for.

In UXtweak Website Testing, you can create questionnaires to be used before the study, after the study or even immediately after each task. The answers to the questions can be either voluntary or required. You can use several types of questions in your questionnaire:

• Single line text
• Multi-line text
• Radio button (single answer multi-choice)
• Checkbox select (multiple answer multi-choice)
• Dropdown select

When asking the respondent to evaluate something (such as their own computer skills or their level of satisfaction with a feature), use multiple choice questions, such as:

• 5 - Great
• 4 - Good
• 3 - Average
• 2 - Bad
• 1 - Awful

How to use a pre-study questionnaire? 

The questionnaire before the user study is usually used to get information about the respondent, such as their age, occupation, level of ICT (information-communication technology) skills, etc. You can also ask about experiences with your company, website or even particular life situations (if you're testing an application for a car renting company, you might ask about their experiences with renting cars).

If you want the respondents to come into the tasks partially uninformed about what exactly it is they're going to be doing, you may want to move the more detailed questions about the domain ("When renting a car online, was the brand of the car important to you?") to the questionnaire at the end of the user study.

How to use a post-study questionnaire? 

The questionnaire after a Website Testing study is a good place for collecting additional feedback. Give your respondents enough space to express themselves and you might get additional feedback that you will find useful during qualitative user study analysis. If the respondents have more to say than what you were originally asking, you may even get feedback that - while totally unrelated to the main objectives of the test - is still be relevant or even enlightening.

How to use per-task questions? 

When you want to gather some detailed feedback about any specific task, it is usually better to ask about it just after said task is finished. (Unless the questions could also infuence the respondent's natural behavior while taking the following tasks.) Respondents tend to quickly forget the specifics of the tasks and of their own user experience. To avoid losing vital information and various mixups, the sooner you ask for details, the better.

Recruit your respondents 

Once your study is ready for launch, all you need now is to recruit some respondent so the data can start flowing in. Sharing a Website Testing study is easy. Simply share a link to the study with your respondents. You can use social media like Facebook, Twitter and LinkedIn to spread your message, or send the link directly to your testers.

However, there is more to recruiting respondents than just sending out a link. A good recruitment strategy can make the difference between learning how to make using your website into smooth sailing, and not discovering something is giving your real users trouble until it’s too late.

Here’s some basic questions that should defined how you recruit people:

• Whom do I want to recruit?
• How many people do I want to recruit?
• What information do I want to communicate to the respondents and how?

Recruiting the right people 

If you’ve been following this guide from the beginning, you should already be familiar with the fact that knowing your user base is the alpha and the omega of user testing. Once you know what sort of people your website is for (which by this point you, already should), you can use this information to set the criteria for respondent recruitment. You want your respondent sample to be representative of the end user so you can learn about actual user behavior.

If you already have an existing audience of people who visit your website of use your web app, you can make use of them as a prime recruitment resource. Reaching out to real users should be easy, given that you have a way to contact them (e.g. a mailing list or a newsletter). Recruiting from your user base, which is already familiar with your website/app and with how they use it has its benefits. They can provide lots of useful feedback, whether it’s for how the changes affect their real use cases, or even real eye-openers that aren’t strictly related to the main goals of the study.

On the other hand, users who are familiar with previous versions also have some undesired bias. While a faithful user could praise your re-design because it’s slicker than the old version, a blank slate respondent might be more critical, pointing out ways for tweaking the design even further. A good way to source general respondents is by sharing the study on social media. Alternatively, you can use an external panel to seek respondents by specific criteria (e.g. 50%, men 50% women, all of whom live in a house and have experience with house insurance).

If your website already has access to your target audience, you can use the perfect way of recruiting respondents. This one is a UXtweak original - UXtweak Recruiting Widget for recruiting respondents directly from your website.

Recruiting enough people 

The goal of user testing - and Website Testing is no different - is to find problems with user experience. It has been proven that 5 respondents is the optimal number of respondents. With 5 respondents, you are going to come across most of the main problems in the tested scenarios. Any number that goes beyond 5 only does very little to help you find anything else and is generally not worth the higher costs.

You might want to gauge the statistical significance of metrics in your study (e.g. the success score). In order to get narrower confidence intervals, the recommended number of respondents is at least 20. However, this is not what you want to do most of the time.

The main aim of user testing is to inform agile design process with insights. For example, you want to find out if people can use the Buy button to buy things without problem. Whether it’s 50% or 70% of people who are being affected by a problem less important. The problem still has to be addressed. If you have the budget to spend on more respondents, great! But it will be much more wisely spent on conducting more different kinds of user studies.

Informing and engaging your respondents 

This might depend a little on who your respondents are and how you recruited them, but the golden rule is that the respondents should be always content and well informed. A happy, informed respondent has much more use for you than someone who’s stressed and has no clue of what’s going on.

Some information that is good to share with your respondents before the user test starts:

• How long it’s going to take (e.g. 45 minutes)
• What is going to be expected of them (e.g. completing some tasks the website of a car renting company)
• What's it all good for (e.g. to make our website easier to use)

Some respondents can feel stressed from having to complete tasks, thinking of it as some sort of test of their IQ or their computer skills. This is why it’s good to remind respondents that no matter what they do, it will be useful to you and there are no good or bad answers. It should be absolutely clear that you’re not testing them, you’re testing your website/web app to make sure that people can use it.

If the respondent was offered some kind of incentive for participating in the study (be it monetary or something else, like a competition or some kind of benefit), you should make sure that they’re well informed about the reward and how they’re going to get it.

Using UXtweak Recruiting Widget 

Good respondents can be a difficult to find and incentives can dig deep into your pocket. With the UXtweak Recruiting Widget, recruiting becomes cheap and simple. Recruiting Widget turns visitors into respondents. Does your website already have existing users? Would you like to test your e-commerce website with real customers? Then add the Recruiting Widget script to your website and let Recruiter handle the recruiting for you.

"Would you like to help us improve our website and get a nice reward for just a few minutes of your time?"

This question (or something else like it) is what visitors are asked when they come to your website and see the Recruiting Widget. Rewards in the form of coupons can be imported into UXtweak and automatically given out to respondents after they complete the study. This direct recruitment between you and your testers cuts out any middlemen, making the process strightforward and beneficial to both sides. Of course, you can also forgo a reward. The Recruiting Widget is fully customizable, including its looks, messages and when and where on the website it appears. You can have the recruiter appear only on certain pages, have it appear immediately or with some delay (after time passes, after scrolling down, etc.).

Interpret your results 

From the moment when you launch your Website Testing study, all respondents who complete the study will begin appearing in the results. You’ll find that the results make it easy to check on your study’s progress - "How many respondents have completed the study?" "How long did it take them? "How successful were they?". From there on, you can delve into the nitty-gritty of task-oriented session replay by analyzing how respondents behaved during the recorded tasks, which is where the real magic happens.

Checking the big picture in results Overview 

The Overview serves you all the information that you need for checking on the state of your user study. The first thing in the summary are the dates when your study was first created when it was launched, along with the number of respondents who participated in it. Check the donut chart to see how many of those people completed the study and whether there were any leavers.

Time Taken is a number that tells you how much time the respondents spent by doing the study. This includes both the respondents whose participation lead to the study’s successful completion and those who abandoned the study at some point.

• The Time Taken is portrayed by a candlestick chart
• The ‘candle’ portrays the range between the upper and lower quartile of Time Taken
• The line inside the candle marks the median value of Time Taken
• The T-shaped wicks mark the maximum and minimum values of Time Taken

Checking whether Time Taken matches the expected completion time is a good way to keep tabs on any unexpected issues so they can be addressed quickly.

The Success score is the percentage of tasks that the respondents completed by submitting the correct page as the answer. Calculated from all tasks from all respondents, it tells you how well the respondents did with the tasks overall.

To get more information about how the respondents fared against the individual tasks, check the Tasks bar chart. This chart displays the Success score calculated for each task individually. Use this chart to get an idea of which tasks were ok and which ones gave the respondents the most trouble.

Just like the Session Recording tool, Website Testing collects further information about your respondents. The overview contains a breakdown of various attributes of your users. The breakdown displays top 3 most common:

• Top Locations - the most common countries where your respondents came from
• Device Breakdown - what type of device (e.g. desktop, mobile) was most commonly used
• OS Breakdown - what operating system was most commonly used
• Browser Breakdown - most commonly used browsers
• Screen Resolution Breakdown - most commonly used monitor resolutions

Checking and filtering Respondents 

In the list of respondents, you’ll find the information about all respondents who participated in your study. It’s here where you can replay completion of tasks, filter your respondents and select which ones should be included in the analysis.

By choosing to view more details about the respondent, you can see their task completion stats, the number of answered questions as well as a number of other information such as the respondent’s location, IP and information about their device. You can check their questionnaire answers and read any comments that they may have left. To replay a respondent’s Website Testing sessions, select the task that you want to analyze and open it in the task player.

While analyzing the session recordings, it might be useful to look into the behavior of various types of users. The respondent list allows you to filter the respondents based on the time it took them to complete the study and on the answers they gave to multi-option questions in the study’s questionnaire. For example, you can use the filter to easily look up whether there was a difference between how the task was solved by new and old users of your web application.

Depending on the method you used to recruit your respondents, you might also need to clean up your study results of outliers and respondents who don’t match your respondent selection criteria. Respondents who didn’t complete the full study are left out of the analysis automatically. However, you can choose to include them if they provided good data in the tasks that they’ve managed to complete.

Replaying tasks 

After you’ve pressed Play on the task that you want to replay, the session player will open, displaying exactly how the interaction between the respondent and the tested website/app went down. What you see is exactly what the respondent was seeing while they were trying to complete the task. You control the session player like you would any video player. Although, if you read our Session Recording guide, you’d know that this is actually not just a simple video, but rather a re-interpretation of how the respondent interacted with your website/app.

Thanks to this, session replay gives us a lot of information that’s useful for efficient analysis, so you don’t have to replay the video to scour for important information. You’ll find a list of all page views and events that happened during the session (clicks, form changes) on the side of the session player. You can immediately see which URLs the respondent visited, which DOM elements they clicked or what text they wrote.

Working with Questionnaire results 

Questionnaires provide you with additional information about your respondents.

While the study is still running, you can use a tabled overview of questionnaire answers to keep watch on the profiling questions so the respondents reflect your target audience and you can adjust the recruiting strategy if such need arises.

After the study concludes, depending on your questionnaire, you can use the filters to analyze your respondents in groups based on their demographics, experience, personality types, etc.