You launch a customer survey, the numbers come back flattering, and you wonder how much of it you can believe. Some of the answers are probably wrong, and most of the time the respondent is not trying to deceive you. Understanding the specific ways answers go wrong is the first step toward a questionnaire that holds up. This article sits within our guide to customer surveys that produce decisions.
Key takeaways
- People lie on customer surveys mostly by accident, not malice, through social desirability, agreeing with everything, rushing, faulty memory or chasing the incentive.
- Four checks catch most bad answers: speeders, straightlining, trap questions and inconsistent answers, such as an age that conflicts with a birth year.
- Remove respondents who fail two or more independent checks, and review single-flag cases by hand, because one miss can be an honest slip.
- Most fixes happen before fielding: neutral wording, short recall windows, anonymity, balanced scales and a short survey all reduce inaccurate answers.
- Rerun your headline numbers with and without flagged respondents: if they barely move you can stop worrying, and if they swing, report it.
Do people really lie on customer surveys?
Some do, but deliberate lying is the smaller part of the problem. A few respondents give false ages or random answers to reach the end of a pop-up or to get an incentive. Far more give answers that are inaccurate for ordinary human reasons: they want to be seen well, they are tired, or they cannot remember.
Older commentary, including the earlier version of this page, quotes figures for how many respondents are dishonest. Estimates vary so widely by audience, topic and survey mode that no single number is worth quoting. Treat the rate as something to measure in your own data.
What are the main reasons survey answers go wrong?
Each pattern below has a name in survey methodology, a typical fingerprint in the data, and a design response.
Social desirability bias
Respondents shade answers toward what looks good or acceptable: overstating healthy habits, understating embarrassing ones, claiming opinions they think are mainstream. It is strongest on questions about behavior, beliefs and belonging, where an answer says something about who the respondent is. The same effect makes customers politely praise a new website or product when asked directly.
Acquiescence bias
Some respondents agree with a statement regardless of its content. If every item in a grid is worded so that “agree” is the positive answer, agreement can pile up that has nothing to do with the real opinion. It is more common when people are tired or unsure.
Satisficing and speeding
Satisficing means giving an acceptable answer instead of an accurate one: picking the first reasonable option, choosing the midpoint, or clicking down a grid in a straight line. Speeding is the extreme version, finishing far faster than anyone could read the questions. Long, repetitive, mobile-unfriendly surveys make both more likely.
Recall bias
People forget, and they also compress or distort time. Ask about a purchase, a service call or a minor event from many months ago and you get confident answers that are partly reconstructed. Underreporting of minor, routine events is the typical result.
Hypothetical bias
Intent questions (“Would you buy this?”, “Would you use this feature?”) measure what people believe about their future selves, and that is often more generous than what they later do. The answer can be sincere and still be wrong.
Professional and incentive-driven respondents
Panels and incentivized links attract some people who complete surveys for the reward. They answer quickly, guess at screening criteria to qualify, and treat open-ended questions as an obstacle. Their data looks tidy and is often empty.
How can you spot bad answers in your data?
Four checks catch most of these patterns, and each works from data you already have.
- Speeders. Compare each completion time with the median. A common rule flags anyone under roughly a third to a half of the median, which you can tune after a soft launch.
- Straightlining. In grids with several items, flag respondents who chose the identical answer on every row. Include at least one reverse-worded item so identical answers can be genuinely inconsistent.
- Trap questions. An instructed item (“Select ‘Somewhat agree’ for this row”) shows whether a respondent is reading. Keep them few and plain, and never treat one miss as proof.
- Inconsistent answers. Compare answers that should agree: age against birth year, “never purchased” against a later brand-usage rating, or a stated income band against an exact figure. Gibberish or copied text in open-ended answers is a related flag, covered in our piece on open-ended survey questions.
What does a data-quality check look like?
Here is an illustrative example, not real data. A 1,000-complete customer survey has a median completion time of 8 minutes, so the speeder cut-off is 40% of the median: 8 x 0.40 = 3.2 minutes.
| Check | Rule | Flagged | Share of 1,000 |
|---|---|---|---|
| Speeding | Under 3.2 minutes | 62 | 6.2% |
| Straightlining | Identical answer on all 10 grid rows | 48 | 4.8% |
| Trap question | Failed instructed item | 31 | 3.1% |
| Inconsistent answers | Age conflicts with birth year | 27 | 2.7% |
| Poor open-ended answer | Gibberish or copied text | 19 | 1.9% |
The flags overlap, because the same careless respondent tends to trip several. Counting each person once, 120 respondents have at least one flag (12.0%). Of those, 38 have two or more flags and 82 have exactly one.
The decision rule: remove the 38 multi-flag respondents (3.8%) and review the 82 single-flag cases by hand. That leaves 1,000 - 38 = 962 completes before any manual removals.
Then rerun your headline numbers with and without the flagged respondents. If they barely move, you can stop worrying. If they swing, you have a data-quality problem worth reporting to stakeholders.
How do you design a survey that minimizes false answers?
Most fixes are made before fielding.
- Ask soon after the event. Shorten recall windows so memory has less to repair.
- Use neutral wording. Remove cues that signal the “right” answer, and avoid leading, loaded or double-barreled questions.
- Make honesty cheap. Anonymity, self-administered modes, and wording that normalizes the answer (“Some people have…”) reduce judgment on sensitive items.
- Balance scales. Mix positive and reverse-worded statements so agreement cannot masquerade as opinion.
- Keep it short and give people time. Tired, rushed respondents satisfice. Cut anything you will not analyze.
- Correct for hypothetical bias. Follow an intent question with “How sure are you?” and count only confident “yes” answers as yes. This works for simple yes or no questions and needs a sensible certainty scale. Warning respondents about the bias can also help, but it may introduce a new bias of its own, so test it.
- Protect the sample. Screen out duplicates, use quality checks from the table above, and read about survey response rate so a low rate does not hide a self-selected group.
Familiarity with your brand may make respondents more candid, but do not assume it. Your most loyal customers are also the most inclined to be polite. Assume some inaccuracy remains and back important findings with other evidence, such as behavioral data or interviews. The reasons people take part in the first place are covered in the psychology of surveying.
How do you do this in practice?
Once flags are defined, run the same screens on every wave and compare results with and without flagged respondents. Halo Reports supports this kind of comparison with crosstabs, weighting and significance testing, so you can see whether removing low-quality completes changes any difference you plan to report. More on Halo Reports. To see what drives satisfaction and loyalty by segment and journey once the data is clean.
What mistakes should you avoid?
- Treating every flag as a lie. Slips and fatigue are common. Use multiple flags before removing anyone.
- Cleaning only at the end. Check quality during a soft launch so you can fix the questionnaire, not just the file.
- Over-using trap questions. Too many insult honest respondents and raise drop-off.
- Removing respondents who disagree with you. Unwelcome answers are not bad data. Define rules before you see results.
- Skipping the sensitivity test. If conclusions only hold with the flagged respondents included, say so.
Before your next survey, write down the four checks above with exact thresholds, add one reverse-worded item and one trap question, and run a 30-person soft launch to see how many people they flag. Between waves, mTab Pulse monitors trackers and studies and surfaces meaningful movement, so you see what changed.