Statistical confidence in Decode reports
Overview
Statistical confidence tells you how much uncertainty there is around a result from your study. A result from a sample is always an estimate of the wider population, and how precise that estimate is depends heavily on your sample size and how your results are broken down.
Why sample size matters
A result based on a larger number of participants gives a more precise estimate than the same result based on a much smaller sample. As a general guide:
- Results based on fewer than 30 responses should be treated as directional only, not statistically reliable
- Results based on 100+ responses start to give you reasonably stable percentages for a single group
- Results based on 385+ responses are generally considered reliable at a 95% confidence level with a 5% margin of error for a large population, the standard benchmark used for most general surveys
Always check the base size behind any percentage before drawing a conclusion from it, a percentage is only as trustworthy as the number of people it's based on.
Margin of error, in practical terms
Margin of error tells you how much a result could realistically shift if you ran the same study again with a different sample. As a rough guide, with a 95% confidence level:
- ~1,000 responses → roughly ±3% margin of error
- ~400 responses → roughly ±5% margin of error
- ~100 responses → roughly ±10% margin of error
Smaller samples have wider margins of error, which means a reported number could realistically be several points higher or lower than what you're seeing.
Confidence vs. practical importance
Statistical confidence and practical importance are not the same thing. A small difference can be statistically detectable with a large enough sample, but still have little real-world importance. Conversely, a large-looking difference from a very small sample may not provide enough evidence to support a strong conclusion. When interpreting a result, ask two separate questions: how reliable is this difference, and is it large enough to actually matter for your decision?
Be careful with small bases
Percentages based on very small groups can be unstable. For example, "70% of participants prefer Concept A" means something very different if it's based on 7 responses versus 700. Always check the number of participants behind the percentage before treating it as a finding.
When comparing groups
Make sure each group you're comparing has enough participants on its own. Comparing a large group against a very small one should be treated cautiously, the smaller group's result is inherently less stable, even if the comparison looks clear at a glance.
Don't over-interpret small differences
A difference in percentage doesn't automatically mean one option is meaningfully better. Concept A at 52% and Concept B at 48% is a 4-point gap, but that gap alone doesn't tell you whether it's statistically meaningful or just noise, that depends on your sample size and margin of error.
Multiple comparisons
The more comparisons you run across concepts or audience segments, the greater the chance that one of them looks different purely by chance. Where possible, decide on your primary comparisons before analysing the data, treat any unexpected findings as exploratory rather than conclusive, and avoid building strong conclusions on a single isolated difference.
Before you launch, check:
- Have I defined what I need to learn?
- Have I identified the groups I need to compare?
- Is my sample size appropriate for those comparisons, based on the benchmarks in How many participants do I need?
- Have I accounted for potential screen-outs or drop-offs?
- Are my quotas aligned with my research objectives, see Quotas and balancing?
- Will my final sample support the conclusions I actually want to make?
Rule of thumb
Plan your sample around the decisions you need to make with confidence, not simply around the largest number you can afford to recruit.