The visitor decides, not the meeting

Every team that builds websites knows the meeting. A new page is on the screen, and everyone has a view. The designer prefers the calm version, the sales lead wants the price higher up, the founder likes the headline they wrote themselves. We have sat in that meeting for years, as an agency and now as a product company, often next to very good experts.

Then the page goes live and the visitors decide. Often they decide something nobody in the room expected. Split testing is the habit of letting them decide on purpose: two versions of the same page, shown at the same time to two random groups of real visitors, and the one that performs better stays.

The idea nobody wanted

The best known example comes from Microsoft. In 2012 an employee working on Bing proposed a small change to the way ad headlines were displayed. It would take an engineer a few days. It was one of hundreds of ideas, the program managers rated it low, and it sat untouched for more than six months.

When an engineer finally ran a simple test, the new version produced revenue so high that it set off an alert that usually signals a bug. It was not a bug. Ron Kohavi and Stefan Thomke wrote in the Harvard Business Review that the change raised revenue by 12 percent, more than 100 million dollars a year in the US alone, without hurting the measures of user experience. They called it the best revenue generating idea in Bing’s history.

The same article describes the opposite case. Showing content from Facebook and Twitter in a third pane on Bing’s results page cost Microsoft more than 25 million dollars to build and produced negligible gains in engagement and revenue. Small effort, big effect. Big effort, small effect. Nobody could tell which was which in advance.

When the experts were sure

In 2013 Bing tested slight changes to the colours of titles, links and captions on its results page. The differences were barely visible, and design experts were sceptical when the results came back positive: people found what they wanted more often and faster. So the test was run again with 32 million users. The result held, and the change was estimated to be worth more than 10 million dollars a year. For years, the authors note, Microsoft had relied on expert designers rather than the behaviour of real users to set its style guides.

Politics has its own version. In December 2007 the Obama campaign tested its sign up page with 24 combinations of buttons, images and videos on more than 310'000 visitors. Dan Siroker, who ran the analysis, later wrote that the staff loved the videos and would probably have used one. Every video did worse than every image. The winner, a family photo with a button reading Learn More, raised the sign up rate from 8.26 to 11.6 percent, an improvement of 40.6 percent.

Most ideas do not work

The uncomfortable part is the base rate. Kohavi and Thomke report that at Google and Bing only about 10 to 20 percent of experiments produce positive results, and that across Microsoft one third of ideas prove effective, one third are neutral and one third make things worse. Thomke, who studied Booking.com, a company that runs some 25'000 tests a year, puts the general figure at about 10 percent of experiments that improve the bottom line.

That is not a reason for despair. It is a reason for humility. If most of the ideas experienced people are sure about do nothing or do harm, a company that changes its website on opinion alone ships those too, and never finds out. Testing finds the winners. Just as important, it stops the losers quietly.

The companies with the most data take this most seriously. In September 2025 OpenAI bought Statsig, a product testing company, in an all stock deal that TechCrunch put at 1.1 billion dollars.

How to test honestly

A test can mislead as easily as a meeting. Researchers at Wharton studied more than 2'100 experiments run on Optimizely in 2014 and found that 57 percent of experimenters checked their results as they came in and stopped once a test crossed 90 percent significance. By their estimate, that habit raised the share of significant findings that were false from about 30 percent to 42 percent.

A few rules keep a test honest. Decide the goal before the test starts, and make it the one that matters: leads, sales or revenue, not clicks. Decide how long it runs, in full weeks, and do not stop it because the curve looks good on a Tuesday. Split visitors at random and check the split: Microsoft’s platform flags a sample ratio mismatch when the groups differ from the plan, and such results are often void. Change one clear thing at a time, so the cause stays clear.

And be suspicious of big wins. Kohavi and Thomke quote Twyman’s law: “Any figure that looks interesting or different is usually wrong.” Run a surprising result again before you celebrate it, and keep the losers in a log. A test that failed still tells you something about your customers.

What changes when you test

The biggest change is not on the page. It is in the meeting. When every important change can be tested, a disagreement stops being a contest of rank, and the question becomes what to test first. A junior can bring an idea without having to win an argument, and an expert can be wrong without losing face.

What stays human is everything before the numbers: the ideas worth testing, the judgement about what a brand should never try even if it would win, and the responsibility for what goes live. A test can tell you which headline sells. It cannot tell you whether you should say it.

Where Proof fits

We built Proof because the tools we tried were made for testing teams, not for companies. It runs split tests on websites, landing pages, shops, emails and newsletters, splits visitors fairly, and calls a winner only when the result holds, measured on leads, sales or revenue. Its agents propose the next test and write the variants, and a person approves each one. Proof is in early access and runs on our own pages first. The pages themselves can live in Studio, where every publish is checked, tested, made live, checked again and rolled back on its own if anything fails.

Where to start: pick the one page that brings the most leads or sales, write down what you believe would improve it, and test that belief against the current version for at least two full weeks. Whatever the result, you will know something you only assumed before.