Method · March 2026
Holdout groups without the leftover noise
A holdout is not a percentage in a console. It is a promise that a slice of users will never see the variant — including via cache, deep link, or a family member’s tablet.
In Signal Lab we ask teams to draw their holdout before they draw their funnel. Most diagrams look tidy: 5% never assigned, 95% randomised. Then we ask a dull question — what is the cache key on the screen the variant changes?
If the key is only user_id, a holdout user who once received a treatment screenshot from a shared device, a WebView, or an old build can still render the variant. The baseline is no longer a baseline. It is a slightly quieter treatment group.
Three checks before we trust the slice
First, assignment must be sticky and versioned. A user who is holdout for flag checkout.v3 must remain holdout when checkout.v4 ships, unless you explicitly re-draw and document the break.
Second, the client must key cached UI on flag version, not only on user. We have seen Android disk caches outlive three flag rollouts in Bangkok fintech apps. The warehouse looked clean. The phone did not.
Third, internal users, store review accounts, and “friends and family” builds must be excluded in the same ledger as the holdout. Leaving them in “because they are small” is how a 5% slice quietly becomes 7% with different behaviour.
What we do not claim
A clean holdout does not tell you whether the variant is good. It only tells you whether the comparison is still a comparison. Teams who skip this step often publish confident seasonality stories that are actually leftover pixels.
If you want the worksheet we use in week 4, it lives inside Signal Lab. The public version is this: do not celebrate a baseline you have not proven empty.