Email A/B testing is uniquely powerful: large sample sizes, fast results, and clear metrics. Yet most marketers only test subject lines. Here's how to build a systematic email testing program that compounds improvements over time.
What to Test (Beyond Subject Lines)
- Subject lines: The most common test, but diminishing returns. Test structure (question vs. statement), personalization, length, and emoji usage
- Send time: Test day of week and time of day. Results vary dramatically by audience and email type
- Content layout: Single-column vs. multi-column, image-heavy vs. text-only, long vs. short copy
- CTA design: Button vs. text link, color, copy, position, number of CTAs
- Personalization depth: Name only vs. behavior-based content vs. AI-generated recommendations
- From name: Company name vs. person name vs. department (e.g., "Sarah from Veltora")
- Preheader text: Often overlooked. This is the second line visible in the inbox after the subject line
Statistical Rigor in Email Testing
Email testing requires the same statistical rigor as website marketing experiments:
- Minimum sample size: At least 1,000 recipients per variant for reliable results. For small lists, use send-to-remainder methodology
- Statistical significance: Target 95% confidence before declaring a winner. Many ESPs call tests too early
- One variable at a time: Test subject line OR send time - not both. Multivariate testing requires 4x the sample size
- Test duration: Most email tests can be called within 24-48 hours. Click and conversion metrics need longer than open metrics
Building a Testing Calendar
Run one to two tests per week. Document every test with hypothesis, variants, sample size, metric, and result. After 50 tests, you'll have a playbook specific to your audience that no competitor can replicate.
Prioritize tests by potential impact × ease of implementation. Subject line tests are easy but low-impact per test. Lifecycle sequence tests are harder but can improve revenue by 20-40%.
Advanced Testing Techniques
- Segment-specific testing: Test different approaches for different audience segments. What works for new subscribers often fails for long-term customers
- Sequential testing: Apply learning from one test to design the next. Build compound improvements over quarters
- Holdout testing: Withhold an entire email from a control group to measure incremental impact
How Xplainr Helps
Xplainr's AI analyzes your email test results in the context of broader campaign performance, helping you understand why variants won and what to test next.
See AI Diagnosis in Action
2-minute setup • No credit card required
Related Resources
- A/B Testing Guide
Apply rigorous testing methodology from web to email.
- Correlation vs Causation
Avoid common statistical pitfalls in email experiments.