The Riskiest Assumption Test: How to Run a £5,000 Experiment Before You Spend £500,000

Every AI experiment carries more assumptions than the hypothesis admits. The data is clean enough to work with. The tool handles the task reliably at the volume required. The team configuring it has the skills to get a fair result. The people whose work changes will use what gets built instead of quietly working around it. Most of these assumptions are safe bets. One of them usually isn't, and it's rarely the one getting attention in the kick-off meeting.

I've watched programmes commit four weeks and most of a six-figure budget before anyone tested the assumption that decided whether the experiment could work at all. The build gets scoped, the vendor gets selected, the success criteria get written up properly, and the team starts configuring the tool against data nobody has checked closely enough. By week three the real problem shows up: the data doesn't join across systems the way the business case assumed, or the people meant to act on the output override it regardless of quality, or the model needs a volume of clean historical records that doesn't exist. None of this is a surprise in hindsight. It was knowable in week one, for a fraction of the cost, if someone had tested it first.

The assumption that decides the outcome before the experiment starts

Section 7: The Experimentation Framework treats this as a distinct step, ahead of designing the minimum viable experiment. Every experiment rests on assumptions about data, about tooling, about the team's skills, and about whether the affected part of the business will adopt what gets built. Not all of them carry equal risk. The riskiest assumption is the one most likely to invalidate the whole experiment, and it's usually one of two kinds.

The first is a data assumption: the model needs data of a certain quality, volume, or structure to produce a useful result, and nobody has confirmed the data meets that bar. The second is a behaviour assumption: the tool only creates value if people change a decision because of it, and nobody has confirmed they will. A recommendation engine can be technically sound and still fail because the customer data behind it doesn't join up. A scheduling tool can produce a better rota and still fail because store managers override it out of habit. Neither failure shows up in a demo. Both show up in week three of a four-week experiment, once real money has gone into finding out.

The Riskiest Assumption Canvas (07a) forces the team to name the assumption most likely to kill the experiment before anyone builds anything, and to design the smallest, cheapest test that would prove it true or false. That test happens before the team commits to running the experiment itself, so the pilot doesn't get built on a foundation nobody checked.

Sorting the assumptions that matter from the ones that don't

Riskiest Assumption Canvas works well when the answer is obvious once someone asks the question. When an experiment carries several live uncertainties at once, the Assumption Map (07b) does the sorting: plot every assumption by how important it is to the outcome and how much evidence already exists for it. Assumptions sitting in the high-importance, low-evidence corner are the ones worth testing before the experiment starts. Assumptions with strong evidence already behind them are operating parameters, not hypotheses, and re-testing them wastes the time-box on questions already answered. The Uncertainty Matrix (07c) does a similar job from a different angle, separating the unknowns the experiment should address from the ones that need separate investigation and the ones the team should simply accept and monitor.

How this looked for a retailer with a recommendation engine on the shelf

Ashworth & Cole, the mid-size UK home and lifestyle retailer used throughout this playbook's worked examples, scored twelve candidate AI use cases against business impact, feasibility, data readiness, risk, and change complexity as part of its Section 4 opportunity assessment. The product recommendation engine scored well on business impact: online conversion sat below the sector average, and a personalised engine using browsing and purchase history had a credible case for improving it. It scored badly on data readiness, because the CRM held duplicate records and incomplete customer profiles, the browsing data from the e-commerce platform wasn't joined to in-store purchase history, and there was no single view of a customer who shopped both channels. The scoring exercise flagged the idea correctly as high-impact and high-risk, and it was earmarked for a second wave, contingent on a customer data integration project the Head of Digital had started scoping.

The instinct, once that project completes, would be to move straight into a four-week recommendation engine experiment. Recommendation engines are a mature, well-understood AI capability, reliable at generating relevant suggestions once the underlying data is clean and joined. The riskiest assumption for Ashworth & Cole's version sits earlier than that: whether a usable single customer view can be assembled from what the business already holds, at a level of completeness good enough to personalise against.

Testing that assumption takes a sample, not a recommendation engine: pull purchase history from the ERP and Shopify for a defined group of customers who have shopped both online and in-store, attempt to join those records into a single view per customer using whatever identifiers currently exist across the CRM and Shopify, and measure what proportion joins cleanly against what proportion doesn't. That's a data engineering task measured in days, not weeks, and it answers the one question the whole recommendation engine experiment depends on. If most of the sample joins cleanly, the recommendation engine experiment is worth designing properly. If only a small minority joins, the customer data integration project the Head of Digital is already scoping is the real next step, and running the recommendation experiment now would only prove what the sample already showed.

What this changes about how experiments get funded

The riskiest assumption test costs little, and its real value is in changing the order in which a programme commits money. Without it, the full experiment budget gets approved on the strength of the hypothesis and the scoring exercise, and the assumption most likely to break it doesn't get tested until the team is already weeks into spending that budget. With it, the assumption gets tested first, cheaply, and the decision to fund the full experiment is made with the answer in hand rather than hoped for.

The riskiest assumption test sits ahead of the kill criteria, the success criteria, and the properly time-boxed experiment that follow it: a cheap check on whether the rest of the process is worth running, taking days rather than the three or four weeks a badly-founded experiment spends finding out the same thing anyway. For a programme running several experiments at once, that ordering compounds. Each riskiest assumption test that flags a genuine problem is a full experiment that never gets funded on a false premise, and a budget that stays available for the one that will.