A/B testing guide for marketers: test, learn, improve

How to run A/B tests on websites, emails and ads that produce trustworthy results, including what to test, sample sizes, avoiding common errors and acting on findings.

By the UK Technology Hub teamPublished 4 minute read

The short answer

A/B testing compares two versions of something, such as a headline, page layout, email subject line or ad, to see which performs better with real users. Start with a clear hypothesis, test one meaningful change, run tests long enough to reach reliable conclusions, avoid stopping early when results look exciting and document what you learn so improvements compound.

Opinions about what will work are cheap; evidence is valuable. A/B testing lets you make changes based on how real people behave, not on the loudest voice in the meeting.

How A/B testing works

  1. Create a variation of a page, email or ad.
  2. Randomly split your audience between the original (A) and the variation (B).
  3. Measure which performs better on a defined goal.
  4. Keep the winner and test the next idea.

Start with a hypothesis

A good hypothesis explains what you'll change, why and what you expect:

"Because visitors ask about pricing in sales calls, adding typical price ranges to the service page will increase enquiry form submissions."

Hypotheses based on research, such as analytics, user testing and customer feedback, produce better tests than random ideas.

What to test

Websites:

  • Headlines and value propositions.
  • Calls to action.
  • Form length.
  • Page layout and content order.
  • Pricing presentation.
  • Social proof placement.

Emails: subject lines, sender names, content length, calls to action and send times. Our guide to email subject lines has ideas.

Ads: creative, headlines, offers and landing pages. Our Google Ads AI campaigns guide covers testing within automated campaigns.

Test changes big enough to matter. Button colour tweaks rarely move results meaningfully.

Sample size and duration

Small samples produce unreliable results. Use a sample size calculator to estimate how many visitors or recipients you need, based on your current conversion rate and the improvement you want to detect. Run tests for at least one full business cycle, often one or two weeks, to account for weekday and weekend differences.

Statistical significance basics

Significance indicates how likely it is that the difference isn't due to chance. Many teams use a 95% confidence threshold. Testing tools calculate this, but understand that:

  • Stopping a test as soon as it looks significant inflates false positives.
  • Testing many variations at once increases the chance of a false winner.
  • Small differences may not be meaningful for the business even if statistically significant.

When you don't have enough traffic

Low-traffic sites can still improve:

  • Test bigger, bolder changes.
  • Use qualitative research: user testing, session recordings and surveys.
  • Run before-and-after comparisons carefully, noting other changes.
  • Test in higher-volume channels, such as email or ads.

Tools

Options include A/B testing features in website builders, e-commerce platforms, email tools, ad platforms and dedicated experimentation platforms. Choose tools that don't slow your site down and that handle consent and privacy properly.

Document everything

Keep a test log: hypothesis, variations, dates, audience, results and decisions. Over time, this becomes a valuable record of what works for your customers.

Using AI

AI can suggest variations, analyse results and spot patterns across tests. Keep human judgement on what's worth testing and whether results make sense.

Common mistakes

  • Testing without a hypothesis.
  • Ending tests too early.
  • Testing tiny changes with low traffic.
  • Ignoring segment differences, such as mobile versus desktop.
  • Not implementing winners.

A worked example

An online booking service tested two versions of its booking page. Version A showed all options at once, while version B introduced a simple step-by-step flow. The team calculated the sample size needed in advance and ran the test for three full weeks to cover weekly patterns. Version B produced more completed bookings, and the team checked that refunds and support questions didn't rise. It then rolled out version B and planned the next test based on what it had learned.

Prioritising tests

Score test ideas by potential impact, confidence and effort. Start with high-impact pages, such as checkout, pricing or sign-up, where small improvements matter most.

Learning from losing tests

A test that doesn't beat the original still teaches you something. Record why you expected it to win and what the result suggests about customers.

Testing in email

Subject lines, send times and content can be tested in email too. Our guide to email subject lines covers practical tests that improve results.

How we help

Our web development team builds testable page templates and sets up experiments without slowing sites down, our e-commerce team tests product and checkout improvements and our AI solutions team helps analyse results and generate test ideas.

Frequently asked questions

How long should an A/B test run?

Long enough to reach your planned sample size and cover full weekly cycles, often one to two weeks or more.

What should I test first?

Changes to high-traffic, high-value pages based on clear hypotheses from research.

Can I A/B test with low traffic?

Yes, with bolder changes and higher-volume channels, supported by qualitative research.

This guide is general information, not legal, tax or financial advice, and details such as prices, features and rules can change. Product and company names mentioned are trademarks of their respective owners and are used only to identify those products; UK Technology Hub is not affiliated with or endorsed by them.

Related guides

All digital marketing guides