How Crosstab Software Can Help Identify Biased Surveys

Survey BiasOnline surveys have become a staple in many industries across the board, used to gauge everything from customer satisfaction to competitor comparisons. While the surveys themselves have become commonplace, truly accurate results may be far less common.

Case in Point

A prime example of online survey bias was outlined by Information Management contributor Steve Miller, who reviewed one software company’s web-based research study in the analytic software market. One of the most prominent findings was that the company’s platform ranked as the second-most platform in the market, ranking higher, even, than many industry leaders.

Miller found the results a tad suspicious, so he decided to investigate further. Sure enough, he found the web-based survey the company presented was only accessible through the company’s own website. That meant the results were heavily selection-biased, with survey participants far more likely to be existing customers than they were to be representative of the overall analytics population.

The bias, not surprisingly, was in favor of the company conducting the survey. But even if the results made the company happy, they did not present truly accurate results.

Lack of Random Sampling

One of the biggest problems with online surveys is the difficulty of obtaining truly random samples. Unlike telephone polls that can randomly dial telephone numbers, Internet polls don’t have similar means of randomly contacting participants.

They instead rely on a number of other means to obtain their samples, and those samples can consist of people who are already on the company mailing list, use the company products or are otherwise part of a select demographic that does not reflect the total population.

Extensive statistical modeling can be done to fill in the blanks, but the jury is still out on the extent to which newer methods are able to truly reflect a representative sample of the population.

Crosstab Software to the Rescue

One way to illustrate that a representative sample of the population participated in your survey is through the use of crosstab software. Instead of including a small footnote on participants, try including a larger crosstab chart that actually breaks down characteristics of your sample to show how those characteristics match up to comparables from the known population.

You could cross-classify survey businesses by characteristics such as industry, size and geography, and then ascertain how those crosstabs match up with population figures. The closer they fit, the more evidence you have to show your sample aligns with the population and, in turn, reflect a higher degree of accuracy.