Defining Panel Quality: Implications of Inadequate Standards

Thanks to technology, today it is much faster and easier to know the consumption preferences of our target audience. What once required countless phone calls and complex information processing can now be carried out with just a few clicks and in a matter of hours. For companies, it is perfect to know in advance if the product we are going to launch will have the expected and desired acceptance, especially in mature markets where both supply and competition are so high. 

But are these quick, automated surveys really reliable? It is the key question that many marketing managers ask, and it makes all the sense in the world.

The key is in the quality of the panels. The process of reaching our consumer to find out what they want is now done much more quickly through digital channels, but the source of the information remains the same: People.   

That is why there are two factors that are essential for the results of our market studies. First of all, we need to know exactly which audience or segment we want to target in order to properly define the parameters of the survey. Secondly, we make sure that the panels to which the survey will be sent are of quality, that is, that there are no fraudulent profiles among the panels to which it is sent, which could significantly bias the results. But how can we know and to what extent can we control it?

Panel Quality and Survey Sample Fraud: The Challenge

In recent years sample fraud has become consolidated in the market research industry, to the point that 20% of panelists are actually false or duplicate profiles. This concern led to the first study on online survey fraud, Online Sample Fraud: Causes, Costs & Cures, being carried out in 2022. This study revealed that 7% of the sample was potentially fraudulent and identified 1% of respondents as duplicates or frauds. An additional 20% of respondents failed traditional post-survey checks, suggesting that around 40% of the sample was problematic.

Ribeiro explains that “the lack of transparency about the origins of their respondents by the sampled suppliers has helped raise the industry’s most recent concerns. “With the entry of several new survey fraud detection companies in recent years using new technology, major brands decided the time was right to look at this previously ignored issue.”

In this sense, what can companies do to guarantee the quality of the panels — and what should we pay attention to as marketing managers?

Factors that contribute to guaranteeing the quality of panels in market studies

One of the first criteria is to ensure that sample quality checks are carried out at various levels, not just during the initial selection of respondents. For example, at Zinklar we also conduct rigorous post-survey quality checks to block questionable respondents and ensure participant engagement and honesty.

Ensure quality and diversity with multiple sample sources

Another of the main criteria to decide if the panels are of quality is the origin of the samples. By having an extensive network of panelists, we can ensure a broader and more diverse representation of the population, which improves results in highly segmented or specialized niche studies. However, close attention must be paid to the selection criteria of these sources:

• Is the company that provides the samples of recognized solvency in the market?
• Do they conduct an aggressive sample hygiene program, including ongoing monitoring, maintenance and evaluation of supplier capacity and sample quality?
• Does it comply with regulatory standards such as GDPR, ISO certification or ESOMAR guidelines?
• Do they have innovative technology and a trained technical team that facilitates intelligent, automated and integrated sampling operations?

The market survey company should be able to reliably answer these questions about the origin of the panels it makes available to us.

Selecting Respondents for Our Surveys

As we mentioned at the beginning, the key to market research is people. But which people among the more than 8 billion inhabitants of the planet? For our survey result to be valid for our purpose, our survey provider must allow us to select the right sample — and if they can’t do this, perhaps the panels they have are not the ones we need.

What do they have to offer? 

• Highly segmented sociodemographic criteria: Parameters such as gender, age range, marital status, children, social class, ethnicity and geographic areas allow us to precisely guide our study.
  
• Predefined profiles: A wide range of predefined profiles adapted to specific consumer categories guarantees availability according to the specific requirements of each country.
• Screening functionality: This feature helps estimate incidence rates and excludes respondents who do not meet specific criteria, optimizing sample quality.
  
• Exclusion of previous respondents: Allows us to avoid repeat participation, ensuring data integrity and avoiding bias in survey results.

Control, control and quality control

Given the problem of fraud in sampling that we have exposed, it is no longer enough to carry out quality controls in the initial selection of the sample, we must establish different controls throughout the process, such as:

• Establishment of quotas: Quotas are established for each country and objective based on the national distribution of the population in terms of age, gender and region, ensuring representativeness.
• Automation: Fully automated processes for feasibility checks, pricing, profile selection, configuration, launch and field work management minimize human errors and streamline operations.

In addition to the two checks above, Zinklar is actively involved in CASE, a nonprofit organization dedicated to improving the quality of online samples. These types of initiatives indicate to us that the supplier is trustworthy, and that they care about the quality of the panels they make available to us.

And what about cases of sample fraud? The market research company must have control measures in place that detect and eliminate fraudulent profiles, such as device analysis, artificial intelligence technology that analyzes survey responses for fraudulent behavior, and a monitoring system I continue with tools like CleanID, which prevent responses from bots or respondents from responding two or more times.

Thus, when in doubt about the lack of transparency or quality of the panels to which we will direct our market study, we can use the criteria presented to our supplier, and ensure that our study will obtain reliable results with guarantees, as well as fast.