Most email marketers make decisions based on gut feel. They write a subject line, hit send, and wonder why results are inconsistent. A/B testing replaces guessing with data. It’s one of the highest-leverage habits you can build as an email marketer, and it requires zero technical skill to start.
This guide walks through what to test, how to set up a clean test, what the data actually means, and how to build a testing habit that compounds over time.
What Is Email A/B Testing?
A/B testing (also called split testing) means sending two versions of an email to different segments of your list to see which performs better. You change one variable between the two versions, measure the results, and roll out the winner to the rest of your list.
Example: You have 2,000 subscribers. You send Version A with the subject line “Your 3-step sales follow-up system” to 500 subscribers. You send Version B with “Stop losing deals because of bad follow-up” to another 500 subscribers. After 4 hours, Version B has a 34% open rate vs. Version A’s 26%. You send Version B to the remaining 1,000 subscribers.
That one test just improved your results by 8 percentage points. Run a test like that once a week for a year and your email performance will be unrecognizable compared to where you started.
What to Test First
Not all email variables are created equal. Some have massive impact, others move the needle marginally. Here’s the priority order for most small business email lists.
1. Subject Lines (Highest Impact)
Subject lines are the single biggest lever in email marketing. A great subject line can double your open rate. Because open rate affects everything downstream (clicks, conversions, revenue), this is where you should spend the most testing energy.
Subject line variables worth testing:
- Question vs. statement: “Ready to fix your open rates?” vs. “Why your open rates are flat”
- Specific vs. curious: “3 subject line formulas that get 30%+ opens” vs. “The subject line trick most people miss”
- With vs. without emoji: “Your Q4 email plan” vs. “Your Q4 email plan”
- Long vs. short: “The re-engagement sequence” vs. “How we win back subscribers who stopped opening (with real examples)”
- Personalization: Using first name in subject vs. no personalization
Real benchmark from Mailchimp’s research: emails with personalized subject lines see open rates 26% higher on average than non-personalized ones. But that doesn’t mean you should always personalize. Test it on your specific list. What works for a B2C list often differs from a B2B one.
2. Preview Text
The preview text (the snippet shown next to or below your subject line in most inboxes) is often ignored. That’s a mistake. It functions as a second subject line and can swing open rates by 5-10% when done well.
Test leaving it blank (some ESPs will default to the first line of email body copy) vs. using a deliberate preview that builds on or subverts the subject line expectation.
3. Send Time and Day
The data on best send times is everywhere and most of it is generic noise. “Tuesday at 10am” is the advice every marketer has heard. It’s also what everyone else is doing, which means inboxes are flooded at that exact time.
Test send times specific to your audience. A newsletter for restaurant owners who start work at 6am hits differently at 7am than 10am. A list of remote freelancers may peak at 9pm. Your own list data, built over 10-15 tests, is worth more than any industry benchmark.
4. Call to Action (CTA)
This affects CTR more than any other in-body variable. Test:
- Button vs. text link
- Single CTA vs. multiple options
- Action-oriented language (“Download the guide”) vs. benefit-oriented (“Get my follow-up templates”)
- CTA placement: above the fold, below the body copy, or both
5. Email Length
There’s no universal rule on email length. Short emails (under 150 words) work well for time-pressed audiences and transactional sequences. Longer emails (500-800 words) can outperform for engaged content lists where subscribers actually read. Test both formats on your list before assuming one is better.
6. From Name
This one surprises people. Testing “Parker at Hustler’s Library” vs. “Hustler’s Library” or just “Parker” can shift open rates noticeably. Personal sender names typically outperform brand names for newsletters and relationship-driven emails. Brand names sometimes outperform for promotional and transactional emails. Test it.
How to Run a Clean A/B Test
The biggest mistake in A/B testing is changing more than one variable at a time. If you test a different subject line AND a different send time simultaneously, you can’t know which drove the difference. One change per test, every time.
Step 1: Set a Clear Hypothesis
Before running any test, write down what you expect to happen and why. “I think a curiosity-gap subject line will outperform a direct one because our audience responds well to storytelling.” This forces clarity and helps you learn whether your assumptions are right over time.
Step 2: Define Your Sample Size
For a test to be statistically meaningful, you need a large enough sample. A general rule of thumb: aim for at least 500 recipients per variation. If your list is under 1,000, your test results will be directional rather than statistically significant. That’s still useful, but treat it as a data point rather than a conclusion.
Most ESPs will calculate statistical significance for you. Klaviyo, Mailchimp, and ActiveCampaign all have built-in A/B testing features that handle sample splitting and winner selection automatically.
Step 3: Choose Your Winning Metric Before You Start
Decide upfront what “winning” means. For subject line tests, that’s usually open rate. For CTA tests, it’s click-through rate. For full campaign tests, it might be conversion rate or revenue. If you decide after the fact, you’ll unconsciously choose the metric that confirms your preferred version.
Step 4: Set Your Test Duration
Most ESPs let you set a test window before the winner is sent to the remainder of your list. A 4-6 hour window is sufficient for most subject line tests if you have an active list. For smaller lists or tests measuring clicks, wait 24-48 hours to allow enough engagement data to accumulate.
Step 5: Document and Archive Results
This step is where most marketers fail. They run the test, note the result briefly, and move on. Six months later, they’re testing the same variable again with no institutional memory.
Keep a simple testing log: date, hypothesis, variable tested, Version A, Version B, winning version, result (open rate, CTR, etc.), and one sentence of interpretation. Over time, this log becomes your list’s proprietary playbook.
Reading the Results
When Version B beats Version A by 2 percentage points, does that matter? It depends on context.
A 2-point open rate improvement on a list of 500 means you got about 10 more opens. A 2-point improvement on a list of 50,000 is 1,000 more opens and potentially hundreds of additional clicks. Scale changes everything.
Don’t over-index on small differences with small lists. Look for patterns across multiple tests. If curiosity-gap subject lines consistently outperform direct ones across six separate tests, that’s a genuine insight about your audience.
Also watch for reversals. Sometimes a test winner in a broadcast context doesn’t hold in an automated sequence. Subscribers in a welcome flow are in a different mental state than people receiving your weekly newsletter. Segment your learnings accordingly.
Building a Testing Cadence
One test per send is achievable for almost any email marketer. At two sends per week, that’s over 100 tests per year. Even if half of them produce inconclusive results, the other half will build a genuine, data-backed picture of what your audience responds to.
A sustainable cadence looks like this:
- Weekly newsletters: Always test subject line or preview text
- Monthly promotional sends: Test CTA copy or email length
- Quarterly: Revisit your automated sequences (welcome emails, re-engagement flows) and test one element in each
For a broader look at how automated sequences fit into your overall email strategy, see our guide on how to build an automated email funnel.
Common A/B Testing Mistakes to Avoid
- Testing too many variables at once. One change per test. Always.
- Ending tests too early. A 4-hour window on a low-engagement list is not enough data. Wait for statistical significance.
- Testing only subject lines. Subject lines matter most for open rate, but your list growth, segmentation quality, and email content quality affect revenue more. Test holistically.
- Ignoring your automated sequences. Most email marketers only test broadcasts. Your welcome sequence runs 365 days a year. Even a 5% improvement in conversion rate from a single CTA change compounds enormously over time.
- Not segmenting your results. A subject line that kills it with your most engaged subscribers might perform differently with cold subscribers. Segment your test analysis when list size allows.
Connecting Tests to Revenue
Testing for its own sake is an academic exercise. The goal is always to connect test results to downstream revenue outcomes. If Version B gets a 30% open rate but the same conversion rate as Version A, your subject line test wasn’t as meaningful as it looked. If Version B’s higher open rate translated to a 15% increase in sales email clicks, that’s a real win.
For a full breakdown of the metrics that connect email engagement to actual revenue, see our guide on email marketing metrics explained.
Campaign Monitor’s A/B testing guide is also one of the best reference resources out there for understanding statistical significance and test design in plain language. You can find it here.
Need Help Setting Up Your Tests?
If you want to move faster, a freelance email strategist can audit your current sequences, identify your highest-leverage test opportunities, and set up your split testing framework in a single day. Find an email marketing specialist on Fiverr and get your testing program off the ground without the agency price tag.
And if you want to understand what you’re actually sending before you start testing variations, make sure your core strategy is solid. Review how to write a sales email that doesn’t get ignored and use that as your control before you start running experiments.
Start Testing This Week
You don’t need a big list, a sophisticated platform, or a marketing team to start A/B testing. You need your next scheduled email and the discipline to change exactly one thing.
Pick your next send. Write two subject lines. Split your list 50/50. See what happens. Then do it again next week.
That’s the whole system. The sophistication comes from repetition, not complexity.
Want more email marketing systems, templates, and playbooks? Join Hustler’s Library free and get full access to the email marketing resource library.
Ready to Know Where You Stand?
The Business Journey dashboard maps your exact position across all 13 stages. Track your progress, unlock resources for each step, and build with a framework used by thousands of founders at Hustler's Library.
No credit card required · Takes 3 minutes · Personalized to your stage
