The definitive guide to creating effective Card Sorting

Introduction 

On most websites, it's important that content is organized in accordance with user expectations. To gain understanding of how your website's users think, you can use card sorting. Card sorting is a conceptual activity for determining how people conceptualize and categorize the information found on your website.

The collected insights will help you with labeling and grouping content, to build an information architecture where users can find what they're looking for with maximum efficiency.

Card Sorting is the UXtweak tool that implements card sorting. It makes it easy for you to prepare card sorting studies and to conduct them remotely, just by sharing a link with your respondents. Then, get the most out of your collected data by using the comprehensive set of data analysis tools.

When is a good time to use Card Sorting? 

• When you're creating a website, a section of a website or you're trying to improve an established one
• When you want to make sure that the information on your website is grouped logically for users
• When you want to find out how people understand certain concepts. To find out whether there are groups of people who understand the same concept differently
• When you want people to divide some items into groups for you

As the name suggests, card sorting involves charging respondents with the task of sorting a set of cards, a.k.a. distributing cards into categories. Each card contains a text label which usually represents a piece of content from your website. You want the respondents to sort these cards according to their own opinion and personal sense.

Depending on how people can sort cards into categories, there are three types of card sort: open, closed and hybrid. In the following guide, we will talk about using each of them:

• Open Card Sorting - Respondents have to create and name their own categories to sort cards into.
• Closed Card Sorting - Categories are set by the researcher beforehand. Respondents don't create categories, only fill them with cards.
• Hybrid Card Sorting - Categories are set beforehand. However, if a respondent feels that some cards don't fit inside any of them, they can create their own categories.

The case study we'll use as an example 

In this guide, we'll provide you with various examples of how you can use card sorting for your own project. To showcase Card Sorting, we've prepared a card sorting case study for a website called "Kids' Garden". Kids' Garden provides parents and future parents with all sorts of articles about children, child care and family lifestyle from both professionals and other parents.

Choose the right technique 

Before you rush into your first card sorting, so that you can start using Card Sorting in your design process, you should first know about the different card sorting techniques - what they're for and how to use them. This will help pick the technique that is right for you. All three card sorting techniques (open, closed and hybrid) demand a fundamentally different type of mental activity from respondents. Each one requires that people look at the content in a different manner.

In an open card sort, the groupings that people create show you how people conceptualize the content when they're not constrained by pre-set boundaries. Meanwhile, in a closed card sort, respondents don't have to think about the groups themselves too much. Instead, they can focus more on how to divide up the cards into the groups that you've provided.

Each technique is ideal for use under different conditions and picking the right one is essential for getting the data that is relevant to your testing agenda.

What's open card sorting good for? 

During an open card sort, respondents are given only a set of cards and total freedom to sort them as they see fit, using categories of their own making which are based solely on their own reasoning or intuition. It is the equivalent of an open-ended question on a questionnaire. There are no known answers, so the people can (or are "forced to", depending on your perspective) give an answer which portrays how they actually think, instead of choosing the option that is "the least bad one" among the options they were given.

Among the three card sorting techniques, open card sort is the one best suited for generating new ideas for how the content should be structured. When you don't yet know what groups to use, it's best to begin from a blank slate and with an open card sort. This can be the case either when you're creating a new website or redesigning an old one from scratch and don't want to limit yourself by repeating any bad design decisions.

When should I decide for an open card sort?

• When I lack the knowledge about how people perceive the information on the website.
• When I want to know where on the website users expect certain content to be found.
• When I want to get some inspiration for labeling and grouping content on the website.
• When I want to find out if there are different groups of people who understand the content of the website differently and would look for the same content in different places.

Case study: Kids' Garden open card sort 

Before we create the initial design of our website's site map, we want know how people see the different article topics in their minds, how they profile them into imaginary groups and how they label these groups in their minds. To do this, we can create a card sort where cards contain the titles of articles.

To provide you with some ideas and suggestions on how to use open card sort in practice, see the following examples.

Compare your website structure with how its content is structured in the heads of your users 

Make cards for all of the main content of your website and then let your respondents sort them into their own categories all by themselves. A comparison of the results with your current information structure can give you an idea about which content would be better located in different parts of the website or should be labeled better.

Find out how people group products in your e-shop 

If you own an online shop, your card sort can include cards with images of products from your inventory. The results will show you which products your customers would likely expect to find next to each other. You can use this data to improve your e-commerce conversion rate.

Generate ideas for tagging and categorizing your blog articles intuitively 

An active blog can quickly grow to contain an extensive number of articles, which your audience will want to browse through effectively. You can create a card sort using titles of your articles or their abstracts to discover tags that might be important to your readers. You could also create a card sort with your tags as cards and then categorize your blog according to how your readers form different types of content into logical wholes.

Generate ideas for structuring your help center 

Help centers usually tend to to store a lot of information and it's also critical for them to provide the information fast. This is a good example of when an open card sort can help you sort all help articles so when a person lands in your help center, the right help is always easy to find.

What's closed card sorting good for? 

In a closed cards sort, respondents are tasked with sorting cards into a fixed set of categories. Unlike the open card sort, where respondents have room for expression so they can fully convey their understanding of information represented by the cards, here their options are purposely limited in order to evaluate your own design of how you think the information should be grouped and labeled.

When should I decide for a closed card sort?

• When I want to know if people would sort a piece of information into the same category as I would.
• When I want to find out whether my categories are a good representation their contents, or if they're ambiguous, confusing and people conceptualize them differently.
• When I have different variants of categories and I want decide on which ones to use in my design, depending how intuitive they are.

Case study: Kids' Garden closed card sort 

For the structure of our website, we came up with a list of categories that are well descriptive of the different types of content that we want to provide.

Because we realize there are different points of view at the content of the articles (the age of the child ("baby", "toddler") and the type of content ("tips and tricks", "family life"), we've included both types of categories to ascertain which ones the respondents agree with more.

We've also put in a "No idea" category, in case respondents don't feel any of the categories really matches with a card.

Aside from improving the structure of information on your website, you can also use closed card sort in several less conventional ways.

Do you need to research what priority your customers assign to select features, services or products? Make a Card Sorting study with categories such as "I use this daily" and "I never use this".

Do you want to know what types of articles your audience is most interested in? Let them sort various topics into categories like "I want more like this" and "This doesn't interest me".

To provide you with some ideas and suggestions on how to use closed card sort in practice, see the following examples.

Discover which content of your website is going to be used most 

If your website provides content that is going to be used significantly more often than most other content, a good approach would be to place it somewhere where people won't have to click too many times to find it. If you don't yet have any traction and so no analytics of use available to you, you can use use a card sort with cards such "I use this daily" and "I never use this" to get a sense on which parts of the website should be the most visible.

Research how your customers view your company 

Put together a list of various company values in the form of adjectives (e.g. innovative, friendly, socially responsible). Have your customers sort these values into categories, depending on whether they associate them with your company or not. Give respondents also space to further develop their opinions in the post-test questionnaire. You will see how the results compare with how the company is seen by the employees or stakeholders or when compared with the company mission and vision.

Get fast feedback on your designs from the team or from customers 

Do you have several versions of a design and now you'd like to collect some feedback to see which version is the best? With a closed card sort, that's easy to do - just insert images of the designs into cards and have them sorted into categories such as "This is my favorite (only pick one)", "I like these" and "I don't like these".

Set priorities for features, products and services 

Use a card sort to get a sense of your customers desires and needs. Create a card sort where the different items that you want to prioritize are cards, and the different priority levels are categories. The priority labels should always include a verbal explanation (e.g. "Need this", "Don't care about this").

What's hybrid card sorting good for? 

A hybrid card sort can be explained from the point of view of both open and closed card sort techniques:

• It's like an open card sort where some of the categories are suggested to the respondent from the beginning.
• It's like a closed card sort where the respondent can create their own categories if they find it necessary.

Overall, hybrid card sort is a flexible technique of its own that blurs the line between open and closed card sort, loses some of their specialization, but gains its own perks, uses and benefits.

Whether your hybrid card sort is closer to open or closed depends on the number of categories that you provide to respondents from the start and to what degree you expect those categories to be sufficient.

When to use hybrid card sorting instead of open 

The less your categories encompass all content on the cards, the closer your hybrid card sort comes to open. In a hybrid card sort that leans towards open, respondents have to create more categories in order to fulfill their task. By making some of the categories fixed, you can shift the respondents' focus from the categories, that you already know should be good, to some other (finer) details, like the placement of cards within these existing categories and the creation of categories for the cards that remain.

When should I do an open-style hybrid card sort?

• When I want to generate ideas like in open card sort, but I'd like to give the respondents some inspiration for what a category label should look like.
• When I already did a card sorting study and some categories were very prevalent, but I still need to determine the general consent on their contents as well as ascertain what should be the other categories in the presence of the existing ones.

When to use hybrid card sorting instead of closed 

You get a hybrid card sort that's closer to a closed if you create it with list of categories that are (at least from your own understanding of the information represented by cards) enough to sort all cards without having to resort to creating new categories.

If people can sort a card into an existing category, they are more likely to do so. Unless of course to them, none of the existing categories really fit the perceived concept of what that card represents. This is when they create a new category and the difference between a closed and hybrid card sort manifests itself.

When should I do a closed-style hybrid card sort?

• When I created categories based on a previous open card sort and now I want to evaluate them, while still leaving the respondents some space to express their own perception of concepts.
• When I already have some labels and grouping of content (maybe even from an established website), but I'd like to know if people don't come up with something better.
• When I want to avoid making people sort cards into categories that make little sense to them. To prevent results that happened only because there was no other choice and the cards had to be sorted somewhere.

Case study: Kids' Garden hybrid card sort 

We've drafted a structure of content for our website based on the children's age groups ("baby", "toddler"). However, because not all content exactly fits into this model, we're doing a hybrid card sort instead of a closed one. We're trying to generate some ideas for grouping the remaining articles but we also want to see how the respondents perceive all the cards in the presence of the age group categories.

How to work with hybrid results 

The results of a hybrid card sort are also a combination of the other two techniques. Like in an open card sort, you will get data on all the categories that the respondents created themselves but like in a closed card sort, the results of the fixed categories will be standardized, so you can analyze them just like you would a closed card sort.

Aside from the information that you'd get from an open or a closed card sort, hybrid card sort will help you answer additional questions, such as:

• To what extent do people agree with the labeling and contents of the pre-existing categories?
• Do people create their own categories that are better than the pre-existing ones?
• Did the people use the pre-existing categories as a template when labeling their own categories, or did they use a different way of thinking?

Create cards and categories 

When it comes to the fate of a card sort study, it all depends on what's in the cards. How you choose the labels on your cards, as well as how many cards there are in total. These are both subjects that you should be thinking about while preparing for your card sort research. In general, the ideal number of cards in one card sort is between 30 and 60. Although there can be valid justifications for using either less and more cards, there should always be some solid reasoning behind it. Following are the reasons for keeping the number of cards within the recommended range:

• Avoid having too few cards so you can collect enough data from the respondents.
• Fewer than 30 cards makes it hard for grouping to emerge as fewer cards provide less context for the respondents to form a big picture.
• Having to think about which cards to include in the card sort is a good think exercise for deciding which of the cards are the most relevant.
• The more cards in your card sort, the more time consuming the card sort, making respondents more likely to abandon your study before they're finished.

The open card sort (and potentially the hybrid card sort, see the previous chapter for explanation) is generally the more time-costly technique. The analysis of cards and the creative activity of creating new categories is a more complex exercise than just sorting items into a set number of compartments. Take this into consideration while deciding on the number of your cards. It's best to keep the time scope of a card sort to about 10-15 minutes long (corresponding with 30-50 cards). At the same time, try to preserve enough related cards, so the respondents can see connections between them and form their own conceptions.

Closed card sorts with just a few categories to choose from (e.g. "Like", "Neutral", "Dislike") are less complex and so they usually require lot less time and effort from your respondents. They might be a good way to collect a lot of data quickly, so going even well beyond 60 cards might work out for you. (Although you should probably test the card sort first, to confirm that it's really as straightforward as you think it is.)

If you just have too many cards that you want to sort, Card Sorting also gives you the option to show each respondent a different subset of cards. (It's recommended to use this feature only when you plan on a higher number of respondents, so it's sure that all of the cards get sorted enough times.)

Where do the cards come from? 

Before you jump into making your cards, you should put together a list all items, features, content, etc. that you consider significant for the purposes of this card sort. Once you've gathered all the concepts together, you can start getting more specific, first focusing on the most important aspects from the list. After some refining, you should be able to get cards that evolved from the original concepts.

Some suggestions for where to look for ideas:

• From your colleagues - Brainstorm. Anything that you want to be included on your website can be significant.
• From your stakeholders - Some of your stakeholders might already have a list of everything they want on the website
• From your company - Use any documents available to you, such as: software specification of your website/app/product, business plan, organization chart, product inventory...
• From your website - Do a content audit of your existing website and pick some meaningful items from there.
• From your users - Gather feedback from your users on what kind of content and features they'd like to see.
• From your competition - Study the websites of other similar companies, particularly your competitors.

Making the cards sort-able 

On a real website, information is usually organized hierarchically. Thus, when creating your own card sort, you might be tempted to add cards that are on different conceptual levels. For example, on a website of a ZOO, a lower level card - "pigeons" - mixed with a higher level one - "birds". However, this is a wrong approach.

Card sorting is a conceptual activity, not a usability test. The objective of a card sort is to research how people understand information - not test the navigation of a website. This is the reason why all cards should be on an equal conceptual level, so they can fit into the same mental models that the respondents sort them by.

Aside from all cards needing to be consistent and equal in order to create the possibility for conceptual groups to form, it's also important that enough cards are similar or related to each other. For each card, you should be able to think of at least one other card that it could be categorized with it. Otherwise, if the cards are all too different and people can't see any connection between the cards, people will become unable to create any coherent groups and you won't learn anything useful for grouping items on your website. Also, if the sorting is too hard, respondents will become more likely to abandon the study.

The best way to deal with this problem is check for any cards that are difficult to find relatives for. To do so, share a Card Sorting study preview link with colleagues from your team and let them sort the cards. If some cards seem difficult to sort, but you still consider them important for the study, you can balance them out by adding cards that are unimportant for your study, but are easy to sort and can help you build the respondent's confidence (as suggested in Donna Spencer's book ).

Avoiding the pitfalls of pattern matching 

When conducting a card sort study, you want the respondents to think about the cards on a conceptual level before they categorize them accordingly, based on any pattern they can find or come up with. The human brain is very good at recognizing patterns, which is why card sort is so useful for grouping content in an intuitive manner, according to the patterns that humans observe.

There is, however, another side to this coin. Just as the human brain is well equipped for recognizing patterns, it's also wired for subconsciously using any shortcuts that it can in order to find them - especially when approaching cognitively difficult tasks. If the brain of the respondent can grab onto any lower-level, symbolic or verbal patterns found in your cards, it will focus on them, without even getting a chance to think about your cards on a conceptual level. Nielsen provides a simple example of how something as simple as the wording of cards can change the results of a card sort.

On a website about agriculture, there's supposed to be articles about different types of crops (strawberries, wheat) and the different types of activity related to each of them (planting, growing, harvesting). However, when presented with Card Set A, the respondents - influenced by the order of words on the cards - place these cards in categories named after the crops like "Wheat" and "Strawberries", while with the Card Set B, they are more likely to create categories like "Planting" or "Harvesting".

There are a few recommended ways for preventing such problems:

• Use synonyms - "Picking strawberries", "Harvesting wheat"
• Non-parallel exposition structures - "Strawberry planting", "Planting wheat"

Please note that card sorting serves for testing of concepts, not usability of an interface. The card labels can (and in some cases should) go against rules for good usability. Unlike in interface design, where the aim is to decrease the cognitive load on users, in a card sort we want the users to think about the cards more deeply (without making cards so obfuscated that they become misleading of course).

The labels on the cards should represent the content well for the purposes of the card sort. They won't actually be used as labels in the user interface. Them not being the picture of great usability is completely okay.

Illustrating cards with images 

The saying "one picture is worth a thousand word" has its application even in card sorting. An image can sometime represent a concept much better than words can. That's why Card Sorting gives you the option to illustrate your cards with images, either in an addition to the card label or by replacing it altogether.

Some cases of where adding images to cards might be profitable:

• The items represent physical items, e.g. physical products such as electrical devices or cosmetics
• You want to categorize images for a creative portfolio or an art website
• You want to know which one of your designs or drafts get the best response

Don't forget to still put a good descriptive labels in the cards even when the people only see the pictures. It will make the data analysis easier for you.

Creating intelligible and justifiable categories 

Creation of closed and hybrid card sort involves the additional step of defining pre-existing categories. Just as with the cards, you should think about the purpose of your study and about how you can write your categories so they can help you achieve it. To get most relevant data, you ideally want the respondents to be able to sort most of your cards. (In Card Sorting, you can set sorting all cards as optional or as a requirement).

In closed card sort, this means that you should include enough categories so different kinds of people can find a place for as many cards as possible. Give the respondents a lot of options (e.g. based on different conceptions from a previous open card sort) and you will find which options people agree on the most. Usually, the more categories, the better.

When you define your own categories in closed and hybrid card sort, you can expect your categories to influence how people think of the cards, whether you realize this or not.

For example:
When you do an open card sort with a set of cards representing furniture, some of the people might start sorting them by their design ("modern", "stylish", "rustic") while others are more focused on the room where they belong to ("kitchen", "living room", "garden"). However, if you're preparing a hybrid card sort and you have "garden" as a category that's set in stone, people will become more likely to create other categories based on the room, such as "living room".

When creating a hybrid card sort, the number of categories you create determines how "open-style" or "closed-style" you card sort is:

• If you set only a few categories, the respondents have less options and are more likely to make their own categories based on their own conceptions, leaning the card sort towards open
• If you set a lot of categories, the respondents are more likely to pick from the categories that already exist, instead of creating their own, leaning the card sort towards closed.

Prepare your questionnaire 

Apart from results of the card sorting itself, you may also have other questions for your respondents. When analyzing the results of a card sort study, additional information about the users, such as their demographics, their stances, user experiences (or experiences in general) can be useful. You can later use such data for sorting respondents into groups and filtering out anyone who's irrelevant. Or, if you only want to admit a certain kind of people to take part in the study, you can set up a screening question and filter out your target group in advance. This is what the questionnaires are for.

In UXtweak Card Sorting, you can create questionnaires to be used before the study or after the study. The answers to the questions can be either voluntary or required. You can use several types of questions in your questionnaire:

• Single line text
• Multi-line text
• Radio button (single answer multi-choice)
• Checkbox select (multiple answer multi-choice)
• Dropdown select

When asking the respondent to evaluate something (such as their own computer skills, or their opinion of your company), use multiple choice questions, such as:

• 5 - Great
• 4 - Good
• 3 - Average
• 2 - Bad
• 1 - Awful

How to use a pre-study questionnaire? 

The questionnaire before the Card Sorting is usually used to get information about the respondent, such as their age, occupation, level of ICT (information-communication technology) skills, etc. You can also ask about experiences with your company, website or even particular life situations (if you're testing an application for a car renting company, you might ask about their experiences with renting cars).

If you want the respondents to come into the card sorting partially uninformed about what exactly it is they're going to be doing, you may want to move the more detailed questions about the domain ("When renting a car online, was the brand of the car important to you?") to the questionnaire at the end of the Card Sorting.

How to use a post-study questionnaire? 

The questionnaire after the Card Sorting is a good place for collecting additional feedback. Give your respondents enough space to express themselves and you might get additional feedback that you will find useful during the qualitative Card Sorting analysis. If the respondents have more to say than what you were originally asking, you may even get feedback that - while totally unrelated to the main objectives of the test - is still be relevant or even enlightening.

Manage your respondents 

Because card sorting is all about grasping at people's cognition, it's not only important how you design the card sort, but also how you manage the respondents themselves. You need to think about getting the right number of respondents, source them so they represent a realistic sample of your end users and also keep them well informed, so they can feel comfortable during the study and so they know what's expected of them.

Recruiting the right people 

The selection of respondents reflects whose thoughts you're interested in. This can be your target audience if you're generating ideas for grouping content on your website. It can also be your colleagues if you're looking for some quick feedback on graphical designs.

Depending on who your respondents are, your options for recruiting them might be different. You can send out an email with a link to the study. You can invite your customers through banners, a newsletter or through a mailing list. It's natural to use social media with millions of users like Facebook, LinkedIn and Twitter when you're aiming for a more general audience. You can provide and incentive in the form or a reward or a competition.

If your website already has access to your target audience, you can use the perfect way of recruiting respondents. This one is a UXtweak original - UXtweak Recruiting Widget for recruiting respondents directly from your website.

Recruiting enough people 

You usually want your card sort to have between 30 and 50 respondents in total. Remember, an open card sort is a generative activity. The results will show you the many different ways in which people think, so that you can use this information as an inspiration for organizing the content on your website. That means that in this case, "more" doesn't always equal "better". What you want is enough completed card sorts to give you some ideas for labeling and grouping content that reflect people's perceptions. You also want data that is just statistically reliable enough to tell you which ways of thinking about your cards are more prevalent.

Quantitative metrics can aid you but aren't all that important. Too many ideas might overwhelm you and they'll be hard to process, as you'll have to analyze them all. 30-50 respondents is just enough. Any more than that will just generate more ideas similar to the ones that you already know (if you've recruited respondents who make for a good representation of the different segments of your audience).

Please, don't forget that unless you provide an incentive (be it a financial reward, a competition or some other benefits) no-one has to feel obligated to actually spend their time by doing the card sort for you. This goes for your own users and customers, and even more for random people on social media. Without an incentive, your invite needs to reach a lot more people than you actually need to surpass the required minimum.

Informing and engaging your respondents 

Your respondents should know in advance what they're getting themselves into. Inform them about what will be required of them and how much of their time it's going to take. Don't forget to pepper the study instructions and your invitation with phrases about how important this test is for you and how it will help you to make your website better. Doing this will help you engage the respondents so they can feel comfortable doing the card sort and so they don't abandon it prematurely, before sorting all cards.

Using UXtweak Recruiting Widget 

Good respondents can be a difficult to find and incentives can dig deep into your pocket. With the UXtweak Recruiting Widget, recruiting becomes cheap and simple. Recruiting Widget turns visitors into respondents. Does your website already have existing users? Would you like to do card sorting for your e-commerce website with real customers? Then add the Recruiting Widget script to your website and let Recruiter handle the recruiting for you.

"Would you like to help us improve our website and get a nice reward for just a few minutes of your time?"

This question (or something else like it) is what visitors are asked when they come to your website and see the Recruiting Widget. Rewards in the form of coupons can be imported into UXtweak and automatically given out to respondents after they complete the study. This direct recruitment between you and your testers cuts out any middlemen, making the process strightforward and beneficial to both sides. Of course, you can also forgo a reward. The Recruiting Widget is fully customizable, including its looks, messages and when and where on the website it appears. You can have the recruiter appear only on certain pages, have it appear immediately or with some delay (after time passes, after scrolling down, etc.).

Interpret your results 

As Donna Spencer notes in her book on card sorting, you can analyze card sorting data by two techniques - exploratory analysis (where you "explore" the data, searching for ideas, guided by your intuition and creativity) and statistical analysis (where numerical metrics serve as the backbone for evaluation).

Checking the overview and selecting respondent data 

From the moment when you launch your card sort, you can view its overview and respondent data. In the overview, you can do a summary check of how the study is proceeding, with information on the current number of respondents and the time they spent doing the card sort.

Reviewing and filtering respondents 

In the respondent list, you will find all respondents of your card sort, with all of their data in its purest form. From the moment that you start your study, you can go here, view each respondent individually and select and filter respondents as you deem necessary. Here's some of the useful things that you can do here:

• Quickly check how long it took people to finish the card sort, how many cards they sorted and how many categories they created so you can delete incomplete or outlier card sorts
• Review the respondents by exploring their individual cases and include/exclude them based on their card sort results
• Divide your respondents into groups by their results and analyze them separately

Analyzing open and hybrid card sorts 

The open and hybrid card sort are both a generative exercise that you go into expecting to get an impression for how to label and group the information on your website. Hence, the analysis of results will tend towards exploratory analysis (in the case of hybrid card sort, it depends on how "open-style" it was). Some questions that one might ask of an open or hybrid card sort:

• By what logic do respondents create and label categories?
• Are there any groups of cards that the respondents always agreed belong together?
• Does the division of certain cards into different categories point to them being understood as different concepts?
• How do people label groups that they create? In hybrid sort, do they create labels that are better than the ones we provided?

Categories results in open and hybrid card sort 

The "openness" of the open and hybrid card sort comes from the ability of respondents to create and label their own categories. The categories that the respondents create are the central part of the results that you're interested in. Under the Categories view, you'll find the summary of data about all categories that your respondents created during the card sort. The data presented here will indicate on the kind of concepts people perceive when looking at your cards.

Because your respondents will have a free hand in naming the categories as they wish, they will naturally create a lot of equivalent categories with similar cards, just with labels that contain different expressions, spelling errors, different capitalization, etc.

A good beginning step would be to standardize these categories. (The set categories in a hybrid card sort are all already standardized.) But before you get to standardizing categories all over the place because of similar labels, you should take into account their similarities in depth, otherwise you might miss concepts that seem similar to you at first sight, but are actually different things to your users. Both the techniques of exploratory and statistical analysis are at your disposal to achieve this.

Get to know the categories, which ones are different and which are similar 

When looking at the Categories table, the first step is to scan all category labels to get in the gist of the different kinds of concepts produced by the respondents. You are likely to identify some patterns rather quickly, just from how the categories are named.

Explore the conceptual similarity of categories 

While it would be easy to stop right here and just pronounce the categories with similar names to represent the very same concepts, such a step would be a mistake. Beneath the surface, the thinking of the respondents who are behind those categories could have been quite different. To avert falsely standardizing different concepts, take a look at the cards and make sure that the category creators were really on the same page.

Standardize categories and check the agreement evaluation 

When you standardize all the categories that represent the same concept into one standardized category, the standardized category's representation in the category table will show you just how much consent there was between the combined categories via the measure of agreement. Agreement is an objective value which portrays the consent on a concept by all the respondents who recognized it.

The maximum agreement is 1.0 (100%). It tells you just how much people agreed on the contents of a standardized category, a.k.a. to what extent have people filled it with the same cards. A category that's not standardized always has the agreement of 1.0, because it originates from only one person. After you standardize several categories into one, e.g. an agreement score of 0.9 will tell you, that 90 percent of people agreed on the contents of that category.

Get a bird's eye view on standardization through standardization grid 

In your efforts to achieve an informative standardization, looking at the big picture of the standardized categories and card distributions can prove quite useful. Inspect the standardization grid - a table which shows how many times each card was assigned to all of the standardized categories. This will help you to get a quick pointer to possible missed patterns.

Using the similarity matrix 

When you need to find which cards were most commonly coupled with each other, use the Similarity Matrix. The Similarity Matrix shows you the percentage of respondents who grouped each pair of cards together. A darker shade of blue means that more people perceived the two cards as related to the same concept.

The card groups are clustered along the diagonal edge of the matrix. The darker the shade of blue inside a cluster and the more distinct its edges, the more it represents a concept that was agreed upon by most respondents.

Some uses of the Similarity Matrix:

• For creating information structures for website content based on the card clusters
• For getting quick insight into which cards belong together, because most people paired them together
• For getting quick insight into which cards don't belong together, because they were only rarely paired together

Using dendrograms 

Dendrograms are tree diagrams that provide another representation of the groups created by the respondents. Cards in the dendrogram converge together according to the percentage of respondents who agreed that these cards belong together.

You will find two dendrograms in Card Sorting analysis, created using different methods - Actual Agreement Method (AAM) and Best Match Method (BBM). Which of these diagrams is more useful to you depends on what number of respondents you had participating in your study.

• AAM - "How many people agreed with this exact group?" - Works best with 30 and more respondents

• The AAM algorithm first takes all categories created by all respondents and assigns them a base score of 1.
• Then, for each category that is a superset of another category, the score of the subset category is incremented by 1 (a.k.a. with two categories [card1, card2] and [card1, card2, card3], the score of [card1, card2] would be 2 and the score of [card1, card2, card3] would be 1).
• Categories are then queued to be added into the dendrogram by high score.
• When a category is added into the dendrogram, all categories that are still in the queue and would contradict it are eliminated.

When choosing which category from a list of contradicting categories of same score should be used in the dendrogram, UXtweak Card Sorting uses the category from a respondent that it intelligently judges to have been the 'best'. The best respondent is someone who paid good attention to the card sort and then completed it efficiently.

• BMM - "How many people agreed with parts of this group?" - Helps you extract as much information as possible from less data (less than 30 respondents)

• The BMM algorithm splits all categories from all respondents into sets of their internal pairings (a.k.a. [card1, card2, card3] gets turned into [card1, card2], [card1, card3] and [card2, card3] ).
• All pairings are then scored based on how many times they're found in all respondents' categories and put into a queue based on their score. Cards are added into the dendrogram by taking card pairings from the queue.
• If neither card from a pair is in the dendrogram yet, the two cards form a new cluster.
• If one card from a pair is already in the dendrogram and the other one isn't, the new card is attached to the cluster that contains the other card.
• If both cards from a pair are already in the dendrogram but in different clusters, the clusters are joined.

Using RCA (Respondent-Centric Analysis) 

RCA looks for popular card sorting approches by looking for an answer from a respondent that is similar to as many of the other respondent's answers as possible. Then it also tries to look for other popular approches, which were different the most popular suggestion as well as each other.

The name Respondent-Centric Analysis is derived from the fact that the results are real respondent answers (not aggregations). They're selected based on votes, which are decided by how the answers are similar to each other.

The RCA algorithm compares answers by how many card pairings (cards placed into the same category) they share. If this similarity reaches the minimum similarity threshold (50% and more by default, can be adjusted by you) the two answers are considered as supporting each other. The RCA algorithm then orders answers by how many other answers supported them and selects the ones with most supporters (3 by default, you can show more if you want). If an answer is already a supporter of one of the answers that were selected before it, it drops out of the candidate list. Therefore, all RCA results represent mutual distinct ways of thinking. You can also narrow down the list of RCA result candidates by specifying the number of categories you want the results to have.

The RCA results contain:

• Their support score (how many supporters they had)
• List of categories and the cards sorted into them
• For each category - A list of similar category names (how supporters of this result named similar categories)
• For each category - A similarity matrix showing the percentage of supporters who also placed individual cards together

Analyzing closed card sorts 

The closed card sort is an evaluative activity that you do in order to compare your conceptual models with perception of information structures by your respondents. Some of the questions that one might ask from a closed card sort:

• Which categories were the individual cards sorted into the most?
• Are there any cards that most people agreed on? Or cards that were sorted all over the board?
• Do any of the categories have a different meaning for different groups of people?
• Are any of the categories more popular than others? Were any of them ignored?

You'll find the tools for finding the answers in the Analysis tab.

Categories results in closed card sort 

The "closedness" of the closed card sort comes from the fact that all categories are pre-determined by you, the study owner. The objective of a closed card sort is to evaluate this set of pre-set groupings, so an analysis of how people interacted with the categories from the viewpoint of each category is a central part of the results that you're interested in. Under the Categories view, you'll find the summary of data about all the categories. The data presented here will indicate how the respondents understood the cards, how they understood the categories and how they both matched up in the respondents' perception of them.

In the Categories table, for each category you'll find a list of cards that were sorted into it, the number of respondents who sorted them there, as well as the number of unique cards sorted into the category and the number of respondents who used category in total. A small number of unique cards points to a cohesive and clear category, while a higher number points to a vaguer one.

You can use the information found within the Categories table to:

• Find the top most commonly sorted cards for each category and use them as representatives of the content that belongs there.
• Evaluate which of the categories the respondents disagree on the most and re-label them appropriately.
• Discover whether any of the categories were ignored by the respondents, so you can try to rename them or scrap them altogether.
• Get quick answers if you used the card sort for gathering opinions (e.g. "I like this"/"I'm indifferent to this"/"I dislike this").

Cards results in closed card sort 

Opposite from looking at the results from the viewpoint of the categories, you can also look at them from the viewpoint of the cards. Under the Cards view, you'll find out which categories a card was most commonly placed into. You'll also find the number of unique categories that the card was assigned to. A low value is a good sign of agreement between the respondents while a higher number shows that the card is difficult to sort. (Or, if most cards were sorted into a wide range of categories, there might be a problem with the comprehensibility of the categories themselves).

You can use the information found within the Cards table to:

• Find out which cards fit best inside their categories based on the number of their unique categorizations.
• Pinpoint any vague, hard-to-place cards that people sorted into many different categories, so their location on the website is uncertain.
• Find out which cards are conceptualized in distinct ways by different people.

Results matrix 

A total overview of agreement on all the different cards and categories in one place is useful for evaluating the results of closed card sort. It's also quite handy for presenting to management or stakeholders what the card sort results say.

The Results Matrix is a table which shows results for every card - how many times it was sorted into all of the categories (the total number of respondents, not a percentage). A quick glance at this table immediately tell you where the cards were sorted, so you can pinpoint disagreement or confusion of the respondents, or their preference for certain categories.

Popular placements matrix 

The popular placements matrix displays what percentages of respondents sorted cards into the individual categories. It then suggest a way for grouping the cards by creating clusters of cards where all cards were most popularly sorted into the same category.