Key Takeaway
A stalled AI pilot doesn't need another model or another experiment. Score it against four production gates, rank the top three blockers, and commit to one fixed-scope milestone that gets a first real user on real data, or retire it.
This article is for CTOs and product leaders with an AI pilot that worked in the demo and hasn’t moved since. It gives you a triage you can run in one working session: four production gates, a strict checklist, the blocker patterns we see most, and a way to decide whether to rescue, restart or retire the pilot.
The demo went well. That was six months ago.
Since then the pilot has lived in a sandbox. It still works when the engineer who built it runs it. People still bring it up in planning meetings. And the usual proposal for what to do next is another experiment: try a newer model, add a second use case, run a bigger test.
That proposal is usually wrong. A pilot proves something is possible. Production needs four things a pilot almost never has: an owner, an integration path, a governance sign-off and a measurable outcome. More experimenting doesn’t produce any of them.
Scaling AI adoption is one of the main themes of IDC’s 2026 AI & Data Summit series in Canada, and the Toronto agenda included a session called “Scaling AI from Pilot to Enterprise.” The topic keeps coming up because the pattern keeps repeating. Lots of pilots, few production systems.
We’ve already written about how to run a pilot so it reaches production in the first place. This post covers the other case. The pilot exists, it’s stalled, and you have to decide what to do with it.
Why won’t another experiment get a stalled AI pilot into production?
Because the pilot already answered the only question an experiment can answer: can the model do this task well enough? For most stalled pilots, the answer came back months ago. It was yes, or close enough.
The questions left are a different kind:
- Who’s accountable for this system once it’s live, and who uses it on Monday morning?
- How does it get data from the real system of record, with real permissions, instead of a CSV export from March?
- Has anyone in security, privacy or legal looked at what data it sends where?
- What number should move, and what was that number before the pilot?
Swapping models answers none of them. Teams often keep experimenting because it’s the work they know how to do, and the rest of the work belongs to nobody.
What are the four gates an AI pilot has to pass?
Ownership, integration, governance and outcome. Every AI system that reaches production passes through these four gates, whether anyone names them or not. A stalled pilot is stuck at one or more.
| Gate | The question | Evidence it’s passed | What a stall looks like |
|---|---|---|---|
| Ownership | Who’s accountable for the outcome and the system? | A named business owner and a named technical owner, both with time allocated | The original sponsor moved on. The pilot is “owned” by the innovation team, which doesn’t run production systems. |
| Integration | How does it read from and write to real systems? | A service with defined interfaces, reading live data with production permissions | The pilot runs in a notebook or a standalone app against a data snapshot. Nobody has scoped the API work. |
| Governance | Is it safe and approved to run on real data? | Security review done, data flows documented, failure handling defined, audit logging in place | Security has never seen it. Customer or employee data goes to a third-party API under terms nobody checked. |
| Outcome | What measurable result justifies running it? | A baseline, a target and a way to measure both automatically | Success is “the outputs look good.” There’s an accuracy score but no business metric. |
A pilot can look nearly finished while it’s stuck at a gate that will take as long as the pilot did.
For the governance gate, our post on AI governance for product teams covers the practical guardrails in more depth.
How do you run an AI pilot readiness triage?
A triage is a short, structured review of one stalled pilot. It ends with a ranked list of the three blockers between the pilot and its first production user. It takes one working session plus some prep.
Three is a deliberate limit. List ten problems and nobody acts on any of them. List one and you’ve usually missed something.
Step 1: Reconstruct the original promise
Before scoring anything, write down what the pilot was supposed to do, for whom, and why someone funded it. One paragraph. People in the room will often disagree. A pilot that started as “cut the time it takes to answer customer questionnaires” has quietly turned into “a general assistant for the sales team,” and nobody decided that.
If the original promise no longer matters to the business, stop. That’s a retire decision, and it’s a perfectly good outcome for a triage.
Step 2: Score each gate
Use the checklist below. Each item is yes or no. Be strict. “Sort of” is a no.
Ownership
- A business owner is named, and the outcome is in their goals
- A technical owner is named, and running this system is part of their team’s job
- The first production users are identified by team, not “eventually everyone”
- Someone has agreed to fund running costs (inference, hosting, support)
Integration
- The pilot reads from the system of record, not an export
- It respects the same access controls as the source system
- It runs as a service with a defined interface, deployable through your normal pipeline
- Outputs land where users already work (CRM, ticketing, ERP, chat), not in a separate app
- Failure modes are handled: timeouts, empty results, rate limits, malformed input
Governance
- Security has reviewed data flows, including what goes to third-party model providers
- Privacy and data residency requirements are documented and met
- Inputs, outputs and user actions are logged in a way an auditor could follow
- There’s a defined human control point for consequential actions
- There’s a way to switch it off without a deploy
Outcome
- A business metric is defined (time, cost, revenue, error rate), not only a model metric
- A baseline for that metric was measured before the pilot, or can be measured now
- There’s an evaluation set that reflects real production inputs, messy ones included
- A decision rule exists: what result by what date means expand, adjust or stop
Step 3: Name and rank the blockers
Every “no” is a gap. Not every gap is a blocker. A blocker is a gap that stops the first production user from using the system safely on real data. A missing dashboard is a gap. Missing access controls on customer data is a blocker.
Rank the blockers with one question: if we fixed only this, how much closer would we be to a first production user? The top three go in the triage output, each with a short description of the work to remove it.
Which blockers stall AI pilots most often?
Six patterns cover most of what we see: an orphaned pilot, snapshot data, notebook architecture, an unreviewed data flow, a model-only metric, and the model-of-the-month habit. If you spot one, you’ve probably found at least one of your three.
| Pattern | Symptom | What removing it involves |
|---|---|---|
| Orphaned pilot | The sponsor left or changed roles. The pilot has no business owner. | Find the person whose metric it moves. If nobody will own it, retire it. |
| Snapshot data | It runs on a one-time export. Results look great because the data is clean and frozen. | Connect to the live source with real permissions. Re-run the evaluation on live data, which often drops quality and shows you the real work. |
| Notebook architecture | The logic lives in a notebook or prototype app. No service, no tests, no deployment path. | Extract the logic into a service with an interface, tests and CI/CD (continuous integration and delivery). This is usually the biggest piece of work. |
| Unreviewed data flow | Sensitive data goes to a model provider and nobody has checked the terms or residency. | Map the data flow, get the security review done, and switch providers or add redaction if needed. |
| Model metric only | The team reports 91% accuracy. Nobody knows what that means for the business. | Pick the business metric, measure a baseline, and tie the evaluation set to the cases that matter most for that metric. |
| Model-of-the-month | The team re-runs the pilot on every new model release. | Freeze the model for the first release. Model upgrades become a normal change, evaluated against the baseline. |
Snapshot data and notebook architecture tend to hide each other. The pilot performs well because it never meets production conditions. Our post on AI agent architecture in production describes what those conditions look like for agentic systems.
Should you rescue, restart or retire a stalled AI pilot?
It depends on two things: whether the original promise still matters, and whether the pilot’s approach survives contact with real data. Every triage should end with one of these three recommendations.
| Recommendation | When it fits |
|---|---|
| Rescue | The original promise still matters, the core approach works on real data, and the blockers are engineering and ownership work with a clear scope. |
| Restart | The promise still matters, but the pilot’s architecture or data approach can’t survive production. Keep the learning, rebuild the system. |
| Retire | The promise no longer matters to the business, or nobody will own the outcome. Write down what you learned and stop spending on it. |
Keeping a dead pilot on life support eats engineering attention and makes the next AI proposal harder to approve. Retiring it is a legitimate result.
What should a rescue milestone include?
One milestone with a fixed scope, a fixed date and a fixed price. Not a phase two. Not a retainer to “keep exploring.”
The triage is what makes a fixed price reasonable. Open-ended AI work is hard to price because nobody knows what’s in it. After a triage, the work is bounded by three known blockers.
A rescue milestone has one definition of done: the first production user, on real data, with a named owner, measured against a baseline. In practice that usually means:
- The pilot logic extracted into a deployable service with tests and CI/CD
- Integration with one system of record, using production permissions
- An evaluation harness built from real production inputs, with a regression baseline
- Logging, a human control point and an off switch
- Rollout to one team, with the business metric tracked from day one
What it leaves out on purpose: more use cases, a platform for future AI projects, a model bake-off. Those can come later, once the first production user has produced evidence.
What does a triage look like on a real pilot?
Here’s an illustrative example. It’s a composite of a common pattern, not a specific client.
A 200-person B2B software company piloted an assistant that drafts answers to customer security questionnaires. Sales engineers spend a lot of time on these, and deals wait while they do. The pilot drafted answers from past questionnaires and policy documents, and the sales engineers who tried it liked the drafts.
Eight months later it was still a pilot. The triage scored it like this:
| Gate | Result |
|---|---|
| Ownership | Passed. The head of sales engineering owns it and wants it live. |
| Integration | Failed. It ran on a folder of documents exported once from the knowledge base. Policies had changed since. |
| Governance | Failed. Security had never reviewed it, and customer questionnaire content was going to a model API under default terms. |
| Outcome | Failed. “The drafts are good” was the only measure. Nobody had timed a questionnaire before or after. |
The three blockers, ranked:
- Unreviewed data flow. Nothing else matters if security says no. Map the flow, review provider terms and residency, add redaction where needed.
- Snapshot knowledge base. Connect to the live knowledge base with document-level permissions, so drafts reflect current policy and flag answers pulled from outdated content.
- No baseline. Time the next ten questionnaires done by hand. Set the target, and log time-to-complete and edit distance on every drafted answer.
The recommendation was rescue. The milestone: one sales engineering team using the assistant on live questionnaires, on the live knowledge base, with security sign-off and time per questionnaire tracked against the baseline. Everything else, RFP drafting and a chat interface included, went on a list for later.
Frequently Asked Questions
How long does an AI pilot readiness triage take?
One working session with the people who know the pilot, plus a few hours of prep to reconstruct the original promise and gather evidence for each gate. The output is a ranked list of three blockers and a rescue, restart or retire recommendation.
Why limit the triage to three blockers?
Ten blockers is a backlog nobody acts on, and one blocker usually means you missed something. Three is enough to define a bounded, fixed-price milestone that gets the first production user live.
Is it a failure to retire an AI pilot?
No. If the original promise no longer matters or nobody will own the outcome, retiring the pilot frees engineering time and keeps the next AI proposal credible. Write down what you learned and move on.
Which stalled pilot would you bring?
Most companies we talk to have at least one pilot like this. It worked in the demo, it still has supporters, and it hasn’t moved in months.
We run a no-cost readiness triage on one stalled pilot. You bring the pilot and the people who know it. You leave with its three blockers to production, ranked, and a rescue, restart or retire recommendation. If it’s a rescue, we follow up with a fixed-price milestone to get it to its first production user. Book a pilot triage.