5 min read
Survey Error Types: It all starts with a survey
Avoiding survey errors is in everybody’s minds when creating a new survey, yet everybody gets things wrong from time to time. So we took a look at the common survey error types people make when carrying out research and interpreting our results. As for avoiding these mistakes, we’ve written a useful little guide to writing better surveys, the first step to making sure your insights are accurate and precise.
Just because it’s quantitative research, doesn’t mean it’s easy to separate the wheat from the chaff. In real life, hardly anything is black and white. Collecting actionable insights starts with writing surveys that don’t skew your data. It’s an art in itself: you need to consider the comfort of your respondents, so they won’t abandon your survey while making sure you don’t encode mistakes into the datasets you collect.
The most notorious example of this in action, malicious or not is up to you to judge, can be seen in the Trump administration’s Official Presidential Job Performance Poll, released in late 2017. Their survey broke the basic rule of scale questions, by not giving equal options for positive and negative answers. This meant that their final dataset wouldn’t record criticisms of the administration. The controversy even spawned a populated Twitter hashtag, boasting contributions up to one year after the first reports.
Even assuming your survey is perfect (after reading our 10 Tips for pointed survey writing of course), how we interpret results and the weight we give to certain data can also lead us down the wrong path.
Bias: ours, yours, theirs. It’s everywhere
We all want to consider ourselves unbiased and objective judges. We hold ourselves to the humanistic standard and assume we’re, above all, objective. We staunchly believe in our capacity to make fair judgments, consider all the evidence, and use our rational minds to reach rational conclusions.
Unfortunately, human brains are just not built like that. There’s a very well-populated body of research into all the cognitive processes that make us biased in our view of the world. What’s more, the World Bank’s 2015 World Development Report looked to test its own researchers and figure out how far cognitive bias affected their work. Turns out, our post-enlightenment view of ourselves might need some updating.
They tested workers by presenting them with identical datasets on two different issues: gun control and skin cream. They found that depending on their opinion on the subject, they would interpret the data differently, despite being the exact same datasets. They controlled for data literacy skills, which didn’t seem to quench this tendency to find confirmation to one’s previously held beliefs.
Researchers themselves tend to highlight conclusions they agree with, which means that cognitive bias plays a major role in how we judge and evaluate “hard” data. Be careful not to make that mistake when reading through your market research report. In fact, the World Banks’ recommendation is to get a special team of people who are uninvolved in a project to read through research designs, surveys, and their conclusions. We take that into account too, and it’s the reason our research team is on hand at any stage of the research. With the help of an independent perspective, you can be sure you’re not seeing a new consumer market that’s not actually there.
Embrace diversity
One of the key skills of survey writing, which we expand on in our guide, is putting yourself in the shoes of who is answering and trying to see the world as you would. Not doing this has a real effect on the insights you can gather. For example, in some survey error types by not understanding the local vocabulary you’ll create a blind spot in your data, as researchers found when trying to figure out how many people used the internet in Indonesia. Turns out, that nobody uses the internet, but a lot of people use Facebook. Another example is how you’ll evaluate a scale: a score of 7/10 for the quality of a product in Germany is actually a really good result, whereas in the UK you’d be left wondering what went wrong.
Being aware of cultural habits, and remaining sensitive to culture and individuals is essential to creating marketing campaigns that jive with the consumer target. What’s more, this knowledge is incredibly important in designing research and surveys that don’t misrepresent your target consumer. As always, diversity within your research team itself is an important way to get diverse voices and perspectives into your projects.
It’s important to do Market Research right because it can help you get an overview of society as a whole, and take your company, campaigns, and product to another level.
