Why A/B Tests Miss What Matters

What A/B tests tell you

A/B testing brought rigour to a discipline that used to run on gut feeling and HiPPOs (Highest Paid Person's Opinion). You can now test hypotheses, measure outcomes, and make decisions on data rather than instinct. That's a genuine improvement.

The constraint: A/B tests tell you what happened. They don't tell you why.

Variant B converted 12% better than Variant A. Was it the headline? The social proof? The CTA colour? The pricing prominence? You know the package performed better. You don't know which element drove it, or whether the result holds for every type of buyer.

The segmentation problem

A B2B SaaS site runs an A/B test. Variant B promotes "Start Free Trial" more prominently. Conversion rises 8%. The team celebrates.

The aggregate conceals this breakdown:

• Individual contributors converted 15% more. The prominent free trial CTA was exactly what they needed.
• Executive buyers converted 20% less. A prominent free trial CTA reads as a self-serve product. They bounced.
• The net effect looked positive because individual contributors outnumber executives. Each lost executive represented a potential six-figure contract.

The test won. Revenue didn't. Nobody knows, because the data stops at the aggregate.

Three things A/B tests miss

Buyer confidence over time

A/B tests measure endpoints: converted or didn't. They miss the confidence arc that precedes conversion. A buyer might arrive at 3/5 confidence, climb to 4.5/5 on the features page, then fall to 1/5 on pricing. The test records "didn't convert." The breakdown point goes unrecorded.

Absent trust signals

You can't test what you haven't built. If your pricing page has no security badges and no testimonials, there's nothing to run a variant against. There's a gap that suppresses conversion, invisible to A/B tooling. Behavioural analysis surfaces what's missing before you have anything to test.

Cross-page effects

Most A/B tests are page-level. Buyer behaviour spans pages. A homepage change might lift homepage conversion while degrading the pricing page, because the homepage now sets an expectation the pricing page doesn't meet. Multi-page tests exist but are slow, complex, and rarely run.

Behavioural forecasting

Behavioural forecasting simulates how specific buyer types experience a site: across all pages, in sequence, with context. A First-Time Customer persona evaluates whether the site is clear, whether it builds trust, whether it answers the right questions. It generates qualitative, explanatory data that A/B tests can't produce.

Run before testing, it identifies what to test and why. Run after, it explains why the winning variant won. Run by buyer segment, it separates results relevant to executives from those relevant to individual contributors.

Better hypotheses, better tests

Testing "blue button vs. green button" is a symptom of not knowing what matters. Understanding how different buyer types experience your site produces hypotheses like: "business-outcome headline vs. feature-list headline for enterprise visitors." That's a test worth running.

A/B tests optimise. Behavioural forecasting tells you what's worth optimising.