What Optimizely's Live A/B Tests Reveal About How Buyers Actually Decide

First in what i hope becomes a series. i explore brands’ live A/B tests and decode what their variations reveal about how buyers actually decide.

Underneath every B2B software purchase is an identity purchase. The product is the vehicle; the identity is the payload. Every headline you’ve ever read is doing identity work, even when it pretends to be about features.

This is the principle hiding under every B2B headline. The headline pretends to describe a product, but its actual job is offering you an identity. Whether you click depends on whether that identity resonates with the version of yourself you’re trying to become, or more often with the fear you’re trying to avoid being.

Marketers know this. The good ones, the ones running mature optimization programs, will tell you they don’t really test copy. They test theories of the buyer. Each variation is a wager about which fear, hope, or identity is most active when the buyer hits the page.

i went looking for an example of this in the wild and found something better than i expected. Optimizely, the company that sells A/B testing tools, is currently running fifteen different versions of itself.

Their grid looks like this.

Each variation is a different theory of the buyer. Let me walk through what i see, and what the behavioral science says about why each one might work.

Speed: the lever everyone pulls

Every product runs a Speed variation. CMS says “Build an ultra-fast website and engage instantly.” Analytics says “Analytics to power fast, focused action.” CMP says “Scale fast-paced production and deliver campaigns at lightning speed.” Web Experimentation just says “fastest.”

The mechanism here is loss aversion. Daniel Kahneman won a Nobel in 2002 for showing, with Amos Tversky, that humans value avoiding a loss roughly twice as much as they value an equivalent gain (Kahneman & Tversky, “Prospect Theory: An Analysis of Decision Under Risk,” Econometrica, 1979; popularized in Thinking, Fast and Slow, 2011). What the Speed variation activates, beneath the surface promise of velocity, is the fear of falling behind: a competitor shipping first, a launch slipping, a quarter ending without the campaign in market.

That’s why Speed is the universal lever. Every B2B buyer feels behind, regardless of what they’re behind on. The fear translates across roles. Marketers feel behind on campaigns. Engineers feel behind on tooling. CMOs feel behind on quarter goals.

A buyer reading “fast-paced production at lightning speed” is hearing a promise to avoid being the one in the post-mortem whose team was slow.

If you’re picking one variation to test against your own control, Speed is usually the safest place to start. It will rarely produce the most interesting result, but it’s most often the variation that beats the control.

Simplicity: cognitive load and the prevention-focused buyer

Simplicity appears only where the product’s job is to make something complex feel manageable. CMS: “Build an easy-to-manage site and stay in control.” Analytics: “Analytics to power clarity and control.” It doesn’t show up on CMP or Web Experimentation, because Optimizely’s team has decided (probably correctly) that the buyer of a content marketing platform or an A/B testing tool doesn’t feel oppressed by complexity. Those buyers chose into a complex job and made peace with it.

The mechanism is cognitive load. The buyer evaluating a CMS has already been burned by a CMS that ate twelve months and a Confluence wiki. What they’re scanning the headline for is permission to believe the new tool won’t repeat the trauma of the old one.

Notice the word control. Both Simplicity variations use it. Stay in control. Clarity and control. That word is doing double duty. It promises ease, and it speaks to something the behavioral economist Tory Higgins called regulatory focus (Higgins, “Beyond Pleasure and Pain,” American Psychologist, vol. 52, 1997).

Higgins describes two motivational orientations buyers carry. Promotion focus is oriented toward gains, growth, achievements. Prevention focus is oriented toward safety, security, avoiding bad outcomes. Most B2B buyers, especially in IT-adjacent roles, sit closer to prevention focus when they’re shopping. “Stay in control” speaks directly to that. It’s a prevention-focus headline.

The Speed variation, by contrast, is a promotion-focus headline. Engage instantly. Scalable success.

Optimizely’s team is testing whether the buyer in front of them, right now, is currently promotion-oriented or prevention-oriented. They don’t know which. So they let the algorithm match.

This is the single most stealable move in the grid. Write two variations of every important headline on your site, one promotion-framed and one prevention-framed, and run them. You will learn which mode your audience is in. That signal then refracts into the rest of your messaging, your nurture, your pricing page.

Design: identity, not features

Design appears only on the CMS hero. “Build a visually stunning website and leave a lasting impression.”

This is the identity lever. The buyer is being told that using the product will make them the kind of person whose work leaves a lasting impression. Daphna Oyserman’s research on identity-based motivation (Oyserman, “Identity-Based Motivation: Implications for Action-Readiness, Procedural-Readiness, and Consumer Behavior,” Journal of Consumer Psychology, 2009) shows that purchases are heavily mediated by self-concept. People buy what’s congruent with who they think they are, or with who they’re trying to become.

Why does this lever only appear on CMS? Because the CMS buyer’s job is, more than any other product Optimizely sells, public-facing. The Analytics buyer’s work shows up in a dashboard. The CMS buyer’s work shows up on the homepage of the company they work for. The output is visible, judged, and attributed. They have skin in how it looks.

The fear underneath the Design variation has less to do with the site than with the buyer’s own taste. The worry is being the kind of person whose taste isn’t quite there.

The Design variation doesn’t relieve that fear so much as validate it, then offer a route out: pick us, and you’ll be the kind of person who chose well.

Insight, Storytelling, Creativity: professional identity

These variations carry a structurally similar move. Analytics to power data-driven decisions. Scale original storytelling and deliver bold creative campaigns.

Each of these variations is doing identity work.

There’s an old marketing line: nobody wanted a drill, they wanted a hole in the wall. The next layer down: nobody wanted a hole in the wall, they wanted to be the kind of homeowner who hangs their own art.

Optimizely is operating at that next layer. Look at what the Insight variation is actually promising. It’s promising the buyer they’ll get to be the data-driven one in the meeting. The Storytelling variation does parallel work for a different role: the buyer gets to be the creative one, telling the bold original story rather than optimizing thumbnails for click-through.

This works because of self-perception theory (Daryl Bem, “Self-Perception Theory,” in Advances in Experimental Social Psychology, vol. 6, 1972). Bem showed that we infer who we are from what we do. The tools we choose are evidence of which kind of professional we want to be. Buying the creative storytelling CMP means, at some level, declaring yourself the kind of marketer who values storytelling over efficiency. The purchase is a vote for an identity.

If your product has any meaningful identity stakes (most do, even if you’ve forgotten), this is the lever you’re probably under-testing.

The deeper move: there isn’t one buyer

The reason there are fifteen variations and not one shipped winner is that Optimizely is using a feature called CMAB. Contextual Multi-Armed Bandit. An algorithm watches each visitor’s context (referrer, prior page, device, time of day, country, the patterns of who they look like) and routes them to whichever variation the model thinks works for them.

There’s no single winning variation, just a portfolio of variations with the algorithm sorting which one each visitor sees.

The behavioral interpretation: buyers aren’t a single psychology. The CMO landing from a LinkedIn ad at 9am on a Tuesday is in a different fear-and-hope mix than the engineer landing from a Google search at 10pm on a Friday. The same headline cannot serve both. So instead of picking a winner, the team staffs a roster of fears, and the algorithm matches the pitch to the person.

This is the deepest admission in the snippet: there isn’t one buyer.

That insight is licensable. It applies to every page you’ve ever shipped. The notion of a “single best headline” for your product doesn’t survive contact with the data. In reality there are several, each best for a different segment, and the only real question is whether you’re running them simultaneously and routing intelligently, or whether you’ve picked one and are accepting the conversion cost on everyone it doesn’t fit.

What to do with this on your own site

If this piece does one thing for you, let it be this: think of A/B tests as competing theories of the buyer, not as copy experiments.

When you sit down to write your next test, replace the question “which headline gets more clicks?” with “which version of the buyer am I writing for in each variation?” Then write one variation for the time-pressured buyer (Speed, loss aversion). One for the burned-once buyer (Simplicity, prevention focus). One for the identity-shopping buyer (Design, identity-based motivation). One for the ambition-oriented buyer (Growth, promotion focus).

If you run them on a bandit, the algorithm will do the matching for you. If you don’t, you’ll learn something equally useful by hand: which fear your audience is actually carrying.