Vibe Code Your
A/B Tests

When the numbers from the ads came in and the conversion rate disappointed, I had a hunch: people weren't immediately feeling the value of what they'd get on the other side. The old landing page for the “First Week After Yes” lead magnet showed a static sample plan. So I opened Claude Code and said, more or less, one sentence: create an experiment in PostHog, a fifty-fifty split, control is the old static plan, and the treatment shows a visualization of the conversation with the agent that helps a bride plan her first week – with an example budget.

What Claude did with that one instruction: it set up the experiment in PostHog, then made the actual code change so there aren't two URLs to split traffic between – it's one page, and PostHog handles the fifty-fifty split, so the comparison stays fair. It wired the conversion events that matter: a signup-start (the moment someone clicks the call to action and lands on the signup screen) and a completed signup (they finish and arrive in the app).

The genuinely impressive part was everything I didn't have to think about. It made the tracking conform to our consent policy and stay correct as people move across subdomains – the kind of thing that quietly breaks an experiment. It tested the whole thing in a browser to see which events actually fired. A day into the traffic we even caught a couple of tracking gaps together and fixed them on the fly – none of which I could have done myself. Start to finish I was happy with it in about fifteen minutes. In VWO or Optimize this was a day of work – and honestly, the conversation-style variant couldn't have been built in those visual editors at all.

You don't pick what to test from an opinion in a meeting. You pick it by watching real people. PostHog records the same sessions you already pay it to capture, and I watched a handful of them on the lead-magnet page. The pattern was quiet but consistent: people scrolled to the sample plan, looked at it – and left.

I want to be honest about what that is: a judgment call, not proof. It was pure intuition that the sample plan was the reason they left. That's exactly the right job for the replays though – they're the input, the thing that tells you where to look. The experiment is the output, the thing that tells you whether you were right. And as you'll see in a moment, this particular hunch only turned out to be half the story.

Here's the result, read honestly. The conversation variant came out ahead on every metric we tracked – more clicks, more completed signups. A clear directional lift. And not one bit of it was statistically significant.

So what do you do with a result that's ahead but never crosses the line? My rule is simple and, I'll admit, a bit heretical: if I see a consistent uplift of more than ten percent over several days, I'm confident enough to ship it. Both numbers here cleared that bar – +14% on completed signups, +11% on clicks. You often cannot wait until the ninety-fifth percentile of a p-value at startup traffic – for this test, the honest math was about twenty-six thousand visitors per variant, which at our volume is well over a year away.

This is the pirate way to run it. Early on, before you're saturated on the optimization curve, it is far more effective to throw spaghetti at the wall and see what sticks than to run minute, single-variable tests. Test bold, high-contrast changes. Then, once saturation hits and every fraction of a percent counts, you become more scientific. Rigor scales with you – it's not a gate you have to clear before you're allowed to move.

And you close the test so it doesn't dangle. I nominated the winner, shipped it to a hundred percent of users, and Claude removed the now-unnecessary code and flags. The learning went two places: into the experiment record in PostHog, and into our running A/B-testing journal – so the next test starts from what this one taught us.

Now the deeper read – because the conversation only nudged things, which told me it wasn't the main problem. The dashboard I'd had Claude build for this page made the real leak obvious. People clicked the call to action just fine. But between starting the signup and finishing it, about two-thirds of them dropped. That felt like a bait-and-switch: you click expecting a free plan and instead hit an unannounced wall asking you to create an account. So the next test – running right now, no results to report yet – changes the CTA to name the next step before the click and adds a one-tap Google sign-in. Same loop, bolder swing.

Step back and look at the whole arc. Six weeks, six moves, all in one codebase: the strategy, the content, the funnel, the analytics, the traffic – and now conversion optimization, the move that turns all that traffic into customers.

Your move this week is the one I'd call the Spaghetti Test: run one real A/B test on your highest-traffic page. Vibe-code it behind a feature flag, point it at the conversion that actually matters, make the variant a bold swing rather than a timid tweak, and decide it on the totality of the signal – ship on a consistent multi-day lift, log what you learned, move to the next one. You're done when you have a running experiment, a real conversion event, a written decision rule, and a journal entry.

And here's where it's all heading. Every move in this series has been laying track toward a self-optimizing growth engine. It maps onto the levels of AI skill for founders I've written about: Level 6, where the codebase starts to heal itself and, on the growth side, funnels and campaigns optimize themselves and your knowledge base expands itself – data-driven systems that continuously develop your product and your marketing. This A/B-test loop is the first real brick of that future. Knowing which problem you actually have – the discipline the Growth Codex calls what to improve – is what makes a self-optimizing engine optimize the right thing. That engine is where this whole show goes next season.

If you want to watch the entire setup happen live – the experiment, the session recordings, the honest read of the results – that's exactly what I'm doing at the next Pirate Lab.

Cheers,

Ben