Backflip · Real estate lending

Rebuilding a live A/B test that was lying to us

Three variants, two stakeholders who wanted opposite outcomes, and an attribution pipeline quietly discarding about 73% of real mobile sessions. The fix had to happen mid-experiment without invalidating 25 days of data.

Role

Head of Product (previously Senior Product Manager)

Scope

Experiment design, metric definition, attribution pipeline debugging, methodology documentation

The problem

The loan application flow needed a decision about how much rate information to show, and when. Two senior stakeholders had opposing positions and both had a plausible story about which funnel shape proved them right. An experiment run into that situation, with a loosely defined success metric, doesn’t settle anything — it just relocates the argument to after the data comes in.

Designing against the argument

I designed a three-variant split, one third of ad traffic per arm, and spent the design time on the primary metric rather than the variants. The metric had to be funnel-shape-neutral: it could not be gamed by an interpretation that favored a longer or shorter funnel, because that was precisely the axis the two stakeholders disagreed on. Defining it before launch is what made post-hoc result-gaming impossible rather than merely impolite.

I also redirected the build onto a maintainable codebase instead of the fastest path to live, and calibrated leadership expectations on statistical power before data collection started — so “we don’t have enough data yet” was a known checkpoint rather than a stall tactic invented in week two.

The pipeline was wrong

Mid-flight, the entry counts didn’t reconcile. Investigating turned up three overlapping causes, which is the reason it hadn’t been caught earlier — each one alone looked like noise:

• A bot filter tuned too aggressively, sweeping up real traffic
• Event coverage gaps in the instrumentation
• A proxy metric standing in for entry that didn’t track it faithfully

The bot filter was the big one: it was discarding roughly 73% of real mobile sessions. I reversed it and corrected counts across all three funnel arms — entries went from 74 / 41 / 45 to 197 / 127 / 125. Then I documented the full methodology, including what had been wrong and how the correction was derived, so the result would be reproducible and auditable by someone who hadn’t lived through it.

An experiment that produces a number nobody can reproduce is worse than no experiment, because people act on it.

The result held

With corrected attribution, the rates-hidden variant led submission conversion 4.5% vs 2.0% — roughly 2.3× — and it led on every pull across 25 days, not only in the final cut. Consistency across pulls matters more than the endpoint gap: it’s the difference between a real effect and a snapshot that happened to land favorably.