Run a Pilot That Reaches a Go or No-Go Decision

Most AI pilots succeed and change nothing. They run for a quarter, produce a deck saying the technology is promising, and end without a decision — because they were designed to demonstrate feasibility rather than to answer the question that would have stopped the project.

A pilot is an experiment. Experiments have a hypothesis, a comparison, and a rule written down in advance for what the result means.

Design around the assumption that would kill it

Write the list of things that must be true for this to work, and rank them by how much damage it would do to be wrong. Then design the pilot around the top one.

The riskiest assumption is rarely "can the model do the task" — that is usually answerable in a week and it is why teams gravitate to it. The assumptions that actually kill projects are elsewhere: that users will adopt the workflow, that the accuracy is sufficient for the real decision rather than in aggregate, that the unit economics work at production volume, that the integration is possible, or that the value is worth the change management.

Design accordingly. If adoption is the risk, the pilot needs real users doing real work, not a demo to stakeholders. If economics is the risk, the pilot needs enough volume to measure cost per task properly. A pilot that tests the safe assumption produces a confident answer to a question nobody was worried about.

Scope it tight: one workflow, one user group, a fixed and short duration. Broad pilots produce ambiguous results because too many things vary at once.

Measure against something

"The AI resolved 70 percent of tickets" is uninterpretable. Compared with what?

Get the current-state baseline before the pilot starts, with the same measurement method you will use afterwards. This is the step teams skip, and skipping it makes every subsequent number an argument rather than evidence, because someone will always claim the old process was better or worse than remembered.

Where you can, run a concurrent comparison — a control group working the old way during the same period — because seasonality, a reorganisation, or a hard quarter will otherwise be attributed to your feature.

Measure at least four things: the outcome (was the work done well), time, cost including the human time the workflow still requires, and the experience of the people doing the work. Also track what the pilot broke — the workarounds people invented, the extra checking they started doing. That last one is where the real cost of a promising pilot usually hides.

Be honest about the novelty effect. Usage in the first fortnight is not usage in the tenth week, which is a reason to run long enough to see the curve flatten.

Write the decision rule before the data

This is the discipline that makes a pilot a decision instead of a discussion. Before you see any results, write down and get agreement on:

What result means go. Specific numbers on the measures that matter, not "meaningful improvement."

What result means no-go. Explicitly. Teams write the go criteria and leave this blank, which is how a pilot becomes permanent.

What result means extend, and for how long, with what specific question the extension answers. An extension without a new question is a way of avoiding the decision.

Who decides, by name, and by when.

Then commit. The strongest evidence that a decision rule was needed is how uncomfortable it is to hold to one after a result that is disappointing but hopeful. That discomfort is exactly the bias the rule exists to counter.

A no-go is a good outcome. It saves the year of engineering that a hopeful extension would have consumed, and a team that has never killed a pilot is not running pilots.

Recognise the pilot that has become permanent

Two failure patterns are worth naming.

Perpetual pilot. It runs, nobody decides, it quietly becomes production without ever meeting a production bar — no on-call, no monitoring, no evaluation gate. Users depend on it and no one owns it. The prevention is the decision date, held.

Pilot theatre. It exists to be reported rather than to inform a decision, usually because the decision was already made. If nobody can state what result would stop the project, this is what is happening, and the honest move is to say so and skip to implementation.

Practice

Write your pilot on one page before it starts: the riskiest assumption, the measure that tests it, the baseline and how you will collect it, the four metrics, the go/no-go/extend criteria with numbers, the decision date, and the decider's name.

Circulate that page and get explicit agreement on the criteria. The disagreements that surface at this stage are the ones that would otherwise have surfaced as an argument about the results, when they are much harder to resolve.

After the pilot, hold the meeting on the date, with the page, and decide.

Common mistakes

Testing the assumption you are confident about. It produces a reassuring answer to the wrong question.

No baseline. Every number becomes a matter of opinion.

Only writing go criteria. The pilot cannot fail, so it never ends.

Extending without a new question. That is not an extension, it is a deferral.