9 Types of Survey Bias and How to Handle Them

If you’ve been in the entrepreneurial game for any amount of time, or if you’ve been in marketing and sales for more than a week, you know that getting customer feedback is vital for success.

Surveys, testimonials, ratings, CSAT scores, and anything else you can get your customers to fill out is invaluable in shaping your growth strategy. After all, the customer knows best and their opinions, likes and dislikes, and pain-points can drive your innovation strategy forward.

But what happens when people start giving the wrong answers? What happens when their perception is skewed and their mindset in the wrong place? You get data that ultimately makes no sense and could be dangerous if used to fuel your sales, marketing, or growth in general.

This is called survey bias.

Survey bias when a customer is influenced in such a way that their answers differ from their true beliefs - and it happens way more often than you think. The way you frame your questions, their order and hierarchy, and even collecting data from a specific audience group can prevent you from getting actionable customer feedback.

This can ruin your research and set your sales, product development, and marketing back.

Let's take a look at the common types of survey bias and how to handle them:

1. Acquiescence Bias and Saying Yes to Everything

People say yes for a number of reasons, and they’re not always because they necessarily agree with a statement or question, or because they want something. They say yes to avoid a discussion or a confrontation, and they say yes when the question you posed is framed in a certain way.

When it comes to survey dos and don't's, allowing acquiescence bias is a big no-no.

Consider these two question, both of which will influence a customer’s mindset:

• “Would you like our ice-cream shop to offer more flavors in the future?”

• “Do you like having so many flavors to choose from?”

The answer to both of these questions is, of course, yes. But that might not be because the person agrees with you, but because of the inherent acquiescence bias in the questions. Your goal is not for the customer to agree with you, it’s to give you an honest answer.

No matter the industry you’re in, you have to carefully frame your questions. In healthcare, for example, devising a patient satisfaction survey can be a particularly difficult task, as in other industries where people are inherently biased. After all, we are talking about their health here.

To minimize and potentially eliminate the tendency for the customer to agree with your questions or statements in surveys, you have to eliminate dead-end questions. Simple yes or no questions produce simple answers, and this will usually be yes depending on how you frame the query.

To circle back to ice-cream, here’s what a better question would look like that would generate a more honest answer:

“How would you go about improving our flavor selection and are you happy with the flavors we currently offer?”

This question is two-fold, and notice that the yes or no part is not at the beginning, prompting a more elaborate answer that you can actually use to drive innovation forward.

2. Avoiding the “question Order” Effects

Keeping questions fairly randomized with no particular order or structure might seem like the worst thing ever to your OCD marketing expert, but it’s actually a good thing.

Why? Because structuring your questions to follow a certain pattern or order can cause the dreaded “question order” effect.

But it’s not just about the order of the question, it’s about where, how, and if you choose to mention your brand or any other brand in the survey. For example, just putting your personal logo on the survey or mentioning your brand in the first question can influence the answers in all subsequent questions.

How? Look at it this way, if a person sees the brand name in one question and it influences their answer, then it’s likely that that influence will carry over throughout the survey, especially if you reinforce it by mentioning the brand again. But we’ll talk more about that in the carry-over bias section.

For now, keep in mind that the order of the questions matters, whether you’re selling with quizzes or collecting data via surveys and questionnaires. So, what can you do about it? Simply randomize the questions and jump from one topic to another.

To use the aforementioned ice-cream example, you can ask about a particular flavor first, then about the service, then about the cleanliness of the space, then about the flavor selection again, and so on. This will prevent the customer from developing the “question order” bias.

3. Conformity and Societal Expectations

We all feel the pressures that society puts on us in our everyday lives, both in our personal and professional realms. We feel the need to act, think, and behave a certain way to fit the societal mold, and that’s not necessarily a bad thing. But it could be very bad when you’re trying to collect relevant data and get honest customer opinions.

Here’s the deal. Whether you are using tools to gather customer feedback or if you’re doing it manually through email and DM outreach, you need to be very careful not to pose questions, problems, or statements that people are expected to agree with because of social norms. Why? Because people will simply conform in order to fit the societal expectation and move on.

That’s not what you want. Here are some examples:

• “Do you tuck your kids into bed every night?”

• “Do you think it’s important for a company to go green?”

• “Do you keep natural cleaners in your home or do you use healthy alternatives?”

Yes, yes to all. Of course people will say yes because it is expected of them, and now you’ve wasted time and valuable marketing dollars.

Do this instead - ask more direct questions on specific issues, and use past tense:

• “When was the last time you used brand cleaners in your home? Can you tell us what they were?”

• “Were you tucked into bed every night as a child? How often do you think kids should be tucked into bed and how often should they try to fall asleep on their own?”

Firstly, posing a question in the past tense gives the illusion that you know that the customer is no longer doing the thing that is not exactly socially acceptable. And secondly, by posing another question as a part of the first one, you are getting more valuable responses that put the customer in control.

4. Evoking Traumatic Experiences Deliver Negative Answers

Asking a customer to recall a potentially traumatic experience or an event when they felt uneasy, unsure, afraid, or at risk in any way can produce the “threat and hostility” bias. This is a big part of behavioral targeting for marketing teams, and it means that by recalling such an experience the customer will be primed to give a negative answer.

For example, whether they are surveying their social media audience or your website visitors or any other group, a car insurance company may ask:

• “If you recall the last accident you had, how satisfied were you with our service when you were filing your claim?”

First of all, of course the customer recalls a life-threatening scenario, and secondly, putting your company in the same sentence with the traumatic experience is likely to produce a negative result. This should come as no surprise, because you are evoking the wrong emotions.

You need to avoid negative emotions and set a more neutral playing field when creating your questions. But the same always goes for when you’re building online forms or any other type of data collection tool like a quiz, a rating system, or even when you’re just talking to them in the comment section.

So what can you do to prevent evoking negative emotions? Don’t talk about traumatic experiences. This can be a challenge depending on the type of business you’re running, but there’s alway a way.

Let’s take the most difficult example out there - a funeral company. Instead of asking:

• “Were you satisfied with how we organized and handled the funeral?”

Make sure to rephrase the question into something like:

“Do you feel that we were able to honor the memory of your [loved one] and is there anything more we could have done to make their passing easier for you and your family?”

As you can see, the first question focuses on the funeral and the company, while the other question focuses on the loved one, honoring their memory, and helping the family cope. This type of question turns a traumatic event into a more bitter-sweet memory than a traumatic one - at least for the purpose of your survey.

Okay, that was a grim example, but now you get how to avoid traumatic experiences and the subsequent bias. We promise to use the ice-cream analogy for the rest of this article.

5. Avoiding Prestige Bias

We tend to exaggerate for a number of reasons, but social standing and perception are definitely up there. Whether we are adding an inch to our height or if we are rounding up our monthly income, we tend to exaggerate in order to feel more important, powerful, and influential within our social group. Needless to say, this prestige bias can really prevent you from gathering better data from your customers.

For example, Ecommerce businesses need to constantly gather data and monitor customer trends to stay ahead. To do this, you need surveys, quizzes, scoring tools, the works. While there are many ecommerce strategies that always work in your favor, many leaders in this sector make the mistake of asking questions that involve prestige. This is normal, because you need information like income, social statutes, education levels and more to create detailed buyer personas.

It’s best to avoid questions that involve prestige and social status cues if you can help it, and use third-party databases to get these values indirectly. Always assume that if the question asks about any type of prestige that the answers will be inflated and rounded up, not down.

If you do decide to ask these questions, try to factor in this effect, but keep in mind that you will never be able to fully eliminate it from your results.

6. Stereotype Bias and Its Negative Effects

Stereotypes are everywhere, and you can’t blame a person for holding certain stereotypes if they are unaware of them or if they are under their influence within their social group. But that doesn’t mean that you should ignore it, especially when you’re trying to get high user engagement and inspire people to provide unbiased answers.

While there is not much you can do to eliminate stereotypes, you can make the customer data clean and correct by either staying away from stereotypes altogether, or by trying not to reinforce them in your questions. Here’s an example: There’s an obvious stereotype that seniors are not good with technology. You would be reinforcing this stereotype if you asked:

“Do you think that seniors can adopt new technologies easily?”

A better question that delivers more meaningful answers would be:

“What are some of the technologies that the seniors in your life have managed to adopt in their daily lives?” or “How long would you say it took the seniors in your household to adopt our software/tool/app/solution?”

The latter avoids reinforcing the stereotype simply by shifting the focus o n the results rather than the problem.

7. Minimizing the Effect of Sponsorship Bias

If someone knows that the survey is coming from a particular brand or sponsor, then their answers might be influenced in a negative or positive way. No matter their answers and the overall response rate, you can be sure that they won’t be able to stay unbiased, which is why it’s important to obfuscate the sponsor as much as possible.

Don’t go with sponsored ads if you want people to find you or your survey, don’t promote it or tell them about the sponsor. If you want people to find you organically and take your survey, then every professional SEO company will tell you that you need to focus on inbound marketing and organic search, not sponsorships or paid promotions.

After all, imagine a company sponsoring a survey and then paying for advertising and dissemination as well, do you think people would be able to stay unbiased?

Make sure not to mention any brands or sponsors directly if at all possible, or use a white label service for your surveys.

8. Memory Bias and Motivated Forgetting

We are forgetful creatures and we tend to mold out memories to fit our current beliefs, simply because memories are so malleable and inconsistent. Not only do we mold them, but we tend to overly romanticize them as a natural way of coping with past mistakes, unpleasant events, and even traumatic experiences.

While all of that is a natural process, it can definitely jeopardize your findings. You have to be careful when asking people to recall specific events, and if you’re using a chatbot to gather survey responses or any other automated solution, make sure to prioritize general questions instead of asking for specifics. If you can, you should avoid asking people to recall something and then tell you when the event occurred.

People will unwittingly say that something happened earlier or later than it did, or that something took less or more time, how many people there were, and the like. Specifics within memories are often difficult to recall. Asking more general questions instead without too many specifics will yield more factual, if less detailed data.

9. Mindset Bias and the Carry-over Effect

The mindset bias is when you put a customer in a certain mindset at the beginning, and thus inadvertently influencing their subsequent answers. People can also use this in order to purposefully generate the data they want. But that’s not the way to get high-quality feedback and it would be a waste of valuable resources.

If we circle back to E-commerce, the only way to maximize the potential of the best ecommerce marketing strategies is to put the relevant data into actionable steps and goals, which means that the customer needs to be in the right (neutral) mindset when taking the survey. For E-commerce companies that have a lot of people taking their surveys, it’s important to carefully manage the order of your questions so as not to influence the following answers.

For example, asking a person about your customer service before asking them about your products or brand might spark a flurry of negative responses if their experience with your staff was less than positive. Make sure to structure your questions to create a neutral, if somewhat positive mindset.

Wrapping Up + More Survey Templates

Survey bias is a big problem nowadays in the marketing world, but that doesn’t mean that it should be a problem for your company. Instead of standing around a report scratching your head and wondering what went wrong, be sure to consider these types of survey bias and implement these tips to generate more meaningful insights.