A/B testing

In an A/B test, one group might see version A of a page, email or journey while another sees version B. The team then looks at what people actually did, rather than relying only on opinion, preference or whoever was most confident in the meeting.

For content work, A/B testing can be useful when the question is specific, the audience is large enough and the team knows what kind of evidence would be meaningful. It can help compare calls to action, page structures, subject lines, onboarding flows or different ways of explaining a choice.

The trap is thinking that A/B testing automatically tells you what is better. It usually tells you what performed better under particular conditions, for a particular audience, against a particular metric. If the metric is too narrow, the test may reward worse content: more clicks, less understanding; more sign-ups, more regret; more attention, less trust.

Why it matters for content

A/B testing matters because content decisions are often argued from taste. One person likes the short version. Another wants more detail. Someone senior prefers a phrase they saw on a competitor’s website and suddenly everyone is pretending that counts as strategy.

A good test can bring evidence into that argument. It can show whether people notice a message, understand an offer, complete a task or choose one route more often than another. Used carefully, it helps teams learn from behaviour rather than endless internal debate.

But content teams need to be careful about what they are testing. Not every content question can or should be reduced to a conversion rate. Some content exists to reduce anxiety, explain complexity, build confidence, prevent mistakes or help people make better decisions. Those outcomes may not show up neatly in a quick A/B test.

How AI changes the picture

AI makes A/B testing easier to overproduce. It can generate headlines, summaries, calls to action, email variants and page introductions at speed. That can be useful when a team has a clear hypothesis and a disciplined way to review the options.

It can also create a fog of plausible alternatives. If the team tests variations without understanding the purpose of the content, the test becomes theatre: lots of variants, lots of numbers and very little judgement. The danger is not that AI makes bad options. The danger is that it makes too many mediocre ones, quickly enough to make them look intentional.

AI can help with A/B testing when it is used to sharpen hypotheses, draft sensible variants and summarise results. It should not be used as an excuse to test everything or to avoid deciding what good content is supposed to do.