Backflip · Real estate lending

Running the funnel on experiments, not opinions

Ten-plus A/B tests on the acquisition funnel took conversion from 1.5% to 2.4% and loan applications 26% on 10% less marketing spend. The harder one was a three-variant loan application experiment run against two senior stakeholders who wanted opposite answers, where the work was in designing a metric neither of them could reinterpret afterward.

• Conversion 1.5% to 2.4%, +26% applications, on 10% less spend, with no drop in lead quality, one quarter vs a trailing-twelve-month baseline
• Ten-plus A/B tests against measured drop-off points instead of one redesign
• A pre-registered, funnel-shape-neutral metric that let two opposed stakeholders accept one answer

Role

Senior Product Manager for the funnel tests (Q3 2025); Head of Product for the application experiment (June–July 2026)

Scope

Cross-functional pod leadership, funnel analysis, experiment design, metric definition, methodology documentation

Worked with

Design, engineering, sales, marketing

Timeline

Funnel tests over about a quarter; the application experiment ran roughly six weeks from prototype through engineering to the final read

The problem

Everyone could see that people fell out of the funnel. Nobody could say which drop-off actually cost applications. That gap gets filled with conviction: whoever argues best in the room decides what ships, and because nothing is measured against a real baseline, nobody is ever wrong afterward.

The same gap made the biggest open question unresolvable. The loan application flow needed a decision about how much rate information to show, and when. Two senior stakeholders held opposing positions, and each had a plausible story about which funnel shape proved them right: one read a shorter funnel as friction removed, the other read a longer one as qualification working.

An experiment dropped into that situation with a loosely defined success metric doesn’t settle anything. It just relocates the argument to after the data comes in.

Ten experiments instead of a redesign

I led a cross-functional pod through the funnel systematically: find the drop-off points that actually cost applications, then test enhancements against them one at a time. More than ten A/B tests over the quarter, against a trailing-twelve-month baseline rather than against last month. A lending funnel is seasonal enough that a short baseline will happily tell you whatever you want to hear.

Doing it as ten tests rather than one redesign is slower to feel like progress and much faster to produce knowledge. A redesign that lifts conversion tells you the new one is better. Ten tests tell you which changes did it, which is what the next quarter is built on.

Spend is the number that matters most: conversion rose on 10% less marketing spend, so the lift came from the funnel converting better, not from buying more traffic to push through it.

The countermetric mattered as much as the metric. It is easy to lift top-of-funnel conversion by letting weaker applicants through, so lead quality and downstream close rate were tracked the whole way, and marketing held campaign variables constant. Neither moved against us. The extra applications were the same kind of applications.

The experiment two people wanted to win

For the rate-disclosure question I designed a three-arm split: two test variants and a control running the existing application, fed from a single marketing campaign so that swings in campaign performance could not masquerade as conversion changes. I spent the design time on the primary metric rather than on the variants.

The metric had to be funnel-shape-neutral: it could not be gamed by an interpretation that favored a longer or a shorter funnel, because that was precisely the axis the two stakeholders disagreed on. Any metric sitting mid-funnel would have handed one of them the win by construction. Defining it before launch, in writing, is what made post-hoc result-gaming impossible rather than merely impolite.

The decision bar was written down before launch too: a minimum of 300 applications across the test before anyone read the result, and proof that down-funnel conversion had not degraded, because a variant that wins by pushing weaker applications through is a loss dressed as a win. Agreeing that in advance meant “we don’t have enough data yet” was a known checkpoint rather than a stall tactic invented in week two by whoever was losing.

The variants were the easy part. The metric was the experiment.

The result held

The rates-hidden variant led submission conversion 4.5% vs 2.0% over 25 days, and it led on every pull across those 25 days, not only in the final cut. Consistency across pulls matters more than the endpoint gap: it’s the difference between a real effect and a snapshot that happened to land favorably.

The decision closed. Neither stakeholder had a reading left to argue.

Tradeoffs

Ten experiments means accepting that most of them will be flat. The discipline is running them anyway, and not quietly reinterpreting the losers into wins.

A funnel-shape-neutral metric also costs you something real: it deliberately refuses to measure the intermediate steps each stakeholder cared about. That’s a genuine loss of diagnostic detail, traded for a result that could actually end the argument. Worth it here only because the argument was the blocker.

What I’d do differently

Run fewer experiments with more power. Ten tests in a quarter proved the funnel could be run on evidence, but at our volume most of them were underpowered and flat, and flat results teach a team less than clear losses.

Traffic was the constraint, and spreading it across ten tests was the mistake. Five tests, each sized to give a real answer, would have moved us faster, and the saved weeks would have gone to the qualitative work that tells you which five.