Six ways an A/B test lies to you

1. Peeking

Checking results every day and stopping the moment they cross the significance line dramatically raises the chance of declaring a winner that isn’t one. Fix the sample size in advance, or use a method designed for continuous monitoring.

2. Too many metrics

Track twenty metrics and one will move by chance. Name a single primary metric before launch, and treat surprises elsewhere as hypotheses for the next test.

3. Novelty

Returning customers click new things because they are new. A redesigned button can win for a week and lose for a quarter. Run for at least two full weekly cycles and look at the effect on new and returning customers separately.

4. A broken split

If the test was meant to split traffic evenly and one group is noticeably larger, something in the assignment is wrong, and the result can’t be trusted. Check the sample ratio first, every time.

5. The wrong unit

Randomizing by session when the decision is made by an account, or by user when a whole team shares the product, lets the groups contaminate each other. Randomize at the level where the behavior happens.

6. Winning the wrong metric

A test can lift clicks and lower revenue, or lift conversion and raise refunds. Always pair the primary metric with a guardrail — retention, refunds, support contacts — that you are not willing to sacrifice.