Email holdout tests: measure what your campaign actually changed

For a hypothetical product-tips campaign, ask whether one reviewed email increases completion of a specific useful action within fourteen days. That is more precise than ‘does email work?’ Define the alternative: no optional product-tips message during the window, or a reviewed existing message. Name what stays the same in both groups, including necessary service mail and unrelated product changes.

Choose the assignment unit carefully. If coworkers share one workspace, assigning individual members may let the intervention affect people in the holdout through their teammates. Workspace-level assignment may fit better, but changes the sample-size and analysis requirements. A household or shared purchasing account can create similar spillover. Document the unit and reasoning rather than defaulting to whatever identifier the campaign tool makes easiest.

Resolve the eligible population using the applicable policy before randomization. Store experiment identity, assignment unit, group, and assignment time in a durable place. Make repeated enrollment resolve to the same group for the intended test. A retry should not rerandomize someone until they happen to receive the campaign. Use an assignment implementation whose distribution you can inspect and test.

An assigned treatment recipient can become ineligible before dispatch. Honor the new restriction; experiment membership does not override unsubscribe or suppression. Define the primary analysis in advance, commonly including all eligible units as assigned, and report actual delivery separately. Dropping non-delivered people after assignment can introduce bias. Ask a qualified analyst to choose an analysis appropriate to the unit and design, especially when clusters or changing eligibility complicate the comparison.

Microsoft Research describes sample ratio mismatch as a warning that observed group allocation differs unexpectedly from the intended ratio. Investigate it before trusting an effect estimate. A dashboard reporting a large lift cannot repair broken assignment, group-specific telemetry loss, or an enrollment bug. Predefine checks rather than inventing an acceptable imbalance after seeing the result.

Also inspect contamination. Did a second automation send essentially the same intervention to the holdout? Did a manual campaign ignore experiment exclusions? Did support agents expose the offer to only one group? Record overlapping activity, and coordinate the experiment across producers. If the intended contrast disappears, explain that limitation instead of presenting a clean campaign-versus-silence conclusion. A small launch simulation with synthetic identities can catch these wiring errors before real enrollment.

Reference: Microsoft Research: Diagnosing sample ratio mismatch

Suppose 2,000 hypothetical eligible accounts are assigned to each group. In the predeclared window, 180 treatment accounts and 160 holdout accounts complete the useful action. The observed rates are 9% and 8%. The absolute difference is one percentage point; the relative difference is 12.5%. Those are different descriptions of the same observed comparison.

The twenty additional observed completions are not automatically a stable future forecast. Estimate uncertainty with a method suited to the design, and consider the minimum effect worth acting on. A narrow effect can be commercially unimportant; a noisy positive estimate can still be inconclusive. Do not stop as soon as the dashboard turns favorable unless the test uses a valid planned sequential approach. Monitoring operational safety is different from repeatedly deciding significance.

Choose one primary outcome that corresponds to the campaign’s job. Use revenue, contribution, retention, or product progress where appropriate, with a clear observation window and source. Opens and clicks can diagnose the path but do not prove value. Keep outcome collection comparable between groups; measuring treatment purchases through tracked links and holdout purchases through a different system can manufacture a difference.

Add guardrails such as complaints, unsubscribes, support load, and total message burden. Where money is involved, include discounts, refunds, and relevant costs. Avoid slicing dozens of segments after the result to find an apparently winning audience. Exploratory patterns can suggest a later test, but label them as exploration. If the sample is too small for a useful decision, say so; an agent should not convert an uncertain result into a confident rollout recommendation.

• Intervention and comparison are defined before enrollment.
• Assignment is stable at the chosen unit.
• Current eligibility still governs actual dispatch.
• Allocation and contamination checks pass or are explained.
• Primary outcome, uncertainty, and guardrails are reported together.

A useful result states the question, population, assignment unit, dates, intended groups, actual exposure, outcome definition, observed difference, uncertainty, and known limitations. Separate the experiment’s finding from the decision: ‘the estimate is positive but imprecise’ can be a result; ‘run a larger test’ can be the next step. Neither requires pretending the campaign failed or succeeded conclusively.

Use Moosewave templates to keep the intervention itself controlled while you review the test design. Agents can help assemble a report from approved data and flag missing fields, but should not silently choose exclusions or rewrite the hypothesis after seeing outcomes. Save the message revision with the experiment record. A holdout becomes valuable when another person can understand what was tested and decide whether the evidence supports the next action.

• Microsoft Research: Diagnosing sample ratio mismatch. Why allocation mismatch can signal invalid experiment wiring. The email design, arithmetic, and reporting checklist are original applied examples.

Sources checked 11 October 2026. Product behavior and documentation can change, so the linked primary source takes precedence if it differs from this article.