Common screening pitfalls

Overview

Even a well-designed screener can go wrong in ways that weaken your study. This article covers the most common mistakes, both in how you write screener questions and in how you plan your data collection around them.

Collecting too many variables

It's tempting to ask about everything that might be useful. But every extra question you collect, whether in the screener or the main study, adds more room for error, more time for participants to spend, and more data for you to manually check later. Focus only on the variables tied directly to what your research actually needs to answer.

Poorly defined variables

If a question or category isn't clearly defined, the answers you get back won't be reliable either. For example, asking someone to self-identify as a "frequent user" means something different to every participant, one person's frequent might be daily, another's might be monthly. Define your variables clearly and specifically, so every participant is answering the same question in the same way.

Sampling bias

Recruiting from whoever is easiest to reach, rather than your actual target population, produces a sample that doesn't represent who you're really trying to study. A well-written screener can't fix this on its own, if your recruitment source is narrow (one social media following, one customer list), your results will reflect that source, not your target audience. See Audience targeting for B2C or Audience targeting for B2B for how to think about this.

Inadequate sample size

A screener that qualifies too few participants for a group you plan to analyse separately weakens that analysis before you even start collecting data. With too few responses, it becomes hard to tell a real pattern apart from random variation, so you risk either missing a real effect or being misled by a result that isn't actually meaningful. See How many participants do I need to plan your numbers before you screen.

Ignoring test assumptions

Some ways of analysing data only work correctly if your data was collected in a certain way. If you skip this check, a result can look valid on the surface while actually being wrong.

For example, comparing the average rating between two groups only makes sense if both groups have enough responses to produce a stable average. If one group has 5 responses and the other has 500, comparing their averages directly can be misleading, the smaller group's average can swing wildly with just one or two unusual answers, while the larger group's average stays steady. Before running any comparison, check that both sides of it have enough data to make the comparison fair. See Statistical confidence in Decode reports for more on this.

Mishandling missing data

When participants skip questions or drop off partway through, don't just ignore the gaps or guess at what the answer might have been. Find out why the data is missing first. If a lot of people are dropping off at the same specific question, that's telling you something, maybe the question is confusing, or something isn't working correctly on that screen, rather than it just being random chance.

Screener-specific mistakes to also check

Beyond the broader data collection issues above, a few mistakes are specific to how you write the screener itself:

• Making the qualifying answer obvious — if it's easy to tell which answer lets someone in, some participants will guess it just to qualify. List a few realistic-sounding options instead of one that clearly stands out.
• Not testing it yourself — a question that's clear to you as the researcher can be genuinely confusing to someone reading it cold, with no other context. Preview your screener as a participant would see it before publishing.
• Getting the qualifying answers wrong in Decode — screener questions mark their correct answers directly on the Multiple Choice block, with an Exact or At Least rule if multiple selections are allowed. Double-check these settings before publishing, since a mismarked answer can let in the wrong participants without you noticing until later.

What to do next

See Audience targeting for B2C or Audience targeting for B2B for guidance on targeting a specific type of audience.