Key Takeaway
Don't scale an AI pilot until one named owner signs a hypothesis with five numbers (baseline, target, adoption, full cost and cost of delay) and a first-milestone scorecard whose thresholds were set before anyone saw the results.
A successful AI pilot proves something can work. It doesn’t prove the rollout will pay. This article is for executives and AI leads facing a request to scale a pilot, and it shows how to write the economic hypothesis that request needs: five numbers, one owner and a scorecard for the first milestone.
The pilot worked. Accuracy beat the target, the twelve people who used it say they’d hate to lose it, and the steering committee saw a good demo. Now there’s a request on the table to roll it out across the department, or the whole business.
That request deserves harder questions than the pilot got. Funding a pilot is a bet that something can work. Funding the rollout is a bet that it will pay: at full volume, with ordinary users, carrying a run cost nobody measured during the pilot. Most of what a pilot produces says nothing about that second bet.
We’ve written before about building the business case for a first AI investment: picking one use case, comparing cost options, modeling value over three years. This post picks up after that. No AI initiative should scale without an economic hypothesis, and that hypothesis comes down to five numbers and one name.
Why don’t AI pilot results transfer to scale?
Because pilots are built to answer “can this work?” and they’re set up in ways that make yes more likely. That’s fine for a pilot. It becomes a problem when the same results get used to justify a rollout.
The volume is selected. Pilot teams pick the cleanest slice of the work, often without meaning to. Invoices from the three biggest suppliers. Support tickets in English. Claims under a dollar threshold. Full volume includes the long tail, and the long tail is where error rates, human review and edge cases live.
The users are volunteers. Early users opted in, got attention from the project team and had a direct line to whoever built the thing. Production users didn’t ask for it, have their own deadlines, and will route around a tool that slows them down once.
The run cost is hidden. Inference often ran on vendor credits. The data scientist who built the model did the human review. Monitoring meant someone glancing at a notebook. None of it shows up as a line item, so the pilot’s cost per unit looks close to zero.
Some costs grow with volume, and some grow faster. Inference cost scales roughly with request volume. Human review scales with volume times the exception rate, and the exception rate usually rises as you move into messier work. Support load grows with the number of users, not the number of transactions.
The metrics measure the model, not the business. Accuracy, precision and user satisfaction are useful engineering signals. None of them is the cost per case, cycle time or revenue figure a CFO answers for. We cover that gap in measuring AI feature success.
IDC’s AI & Data Summit agendas in Canada this year list business value and ROI (return on investment) as a key topic, calling for “clear KPI frameworks, strong business cases, and scalable use cases tied to financial value.” Getting there mostly means writing a few specific numbers down before the rollout starts, and building the system so someone can check them afterward.
What is an economic hypothesis for an AI initiative?
It’s a single statement that could turn out to be false. It names the adoption, the operating metric, the value, the full cost and the person accountable, and it has this shape:
If [adoption] of eligible [work] runs through the system by [date], then [operating metric] moves from [baseline] to [target], worth [$ per month], against a [full cost]. [Owner] is accountable for the result, and we will check it at [first milestone].
Two things make it useful. It’s specific enough that someone can be wrong. And it names the person who’ll be asked about it in two quarters, which changes how carefully everyone fills it in.
Which five numbers does an AI scale-up case need?
Baseline, target, adoption, full cost and cost of delay. Each has a source and a classic way to get it wrong.
| # | Number | What it is | Where it comes from | Common mistake |
|---|---|---|---|---|
| 1 | Baseline | The operating metric the system moves, measured today, in units and dollars | Systems of record: ERP, ticketing, CRM, time tracking. A two-week sample if nothing else exists. | Using the pilot team’s numbers. They were the best performers to begin with. |
| 2 | Target | Where that metric lands across all eligible volume, and by when | Pilot results, discounted for the work the pilot didn’t see | Projecting the pilot’s best week onto the whole population |
| 3 | Adoption | Share of eligible work that flows through the system at each milestone | A ramp plan agreed with the owner: month 2, month 4, month 6 | Assuming 100%, or assuming adoption happens on its own |
| 4 | Full cost | One-time cost to scale plus monthly cost to run | Integration estimate, inference pricing at projected volume, review staffing, monitoring, support | Counting the build and forgetting the run |
| 5 | Cost of delay | Net value lost for each month the scaled system isn’t live | Derived from 1 through 4 | Leaving it out, so every review and procurement step feels free |
Then there’s the owner: one person whose P&L (profit and loss) or operating metric moves. Not the AI lead or the vendor, and never a steering committee. If nobody will put their name next to the target, the initiative isn’t ready to scale, whatever the pilot showed.
Why does adoption need its own plan?
Because adoption is what most often sinks a scale-up, and it’s the number least often written down. A model that saves ten minutes per case saves nothing on cases that never reach it.
Adoption plans that hold up have three parts: which teams come on at which milestone, what changes in their workflow (the tool replaces a step instead of adding a tab), and who in each team is responsible for usage. The owner signs off on the ramp. If the owner won’t commit to one, you have your answer about adoption.
What belongs in the full cost?
Everything that runs, not only the build. For the one-time cost to scale, include integration with the systems of record, an evaluation set that represents full volume rather than the pilot slice, security and privacy review, and training for the teams coming on.
For the monthly run cost, include inference at projected volume (with headroom, since prompts tend to grow), human review of exceptions and low-confidence outputs, monitoring and on-call, and the engineering time to maintain prompts, models and integrations as upstream systems change. For more on production monitoring, see AI agent observability.
How do you calculate the cost of delay?
It’s arithmetic, not rhetoric. Once you have the first four numbers:
monthly gross value = eligible volume × adoption × (baseline unit cost − target unit cost)
monthly net value = monthly gross value − monthly run cost
cost of delay = monthly net value (at the adoption level you'd otherwise have reached)
payback month = one-time scale cost ÷ monthly net valueIt’s a rough number. Its job is to put a visible price on a four-month procurement cycle or a stalled data-access request.
What does an economic hypothesis look like for a real workflow?
Here’s one for invoice exceptions, with illustrative figures. The structure carries over to most process automation work.
A mid-market distributor ran a 10-week pilot on supplier invoice exceptions: invoices that fail automatic three-way matching against the purchase order and receipt. The system reads the invoice, pulls the PO and receiving records, proposes a resolution and drafts the note to the supplier. An AP (accounts payable) clerk approves or corrects it.
The pilot handled about 300 exceptions a month for one AP team. Proposed resolutions were accepted without edits 91% of the time. The team liked it. The request is to roll it out to all of AP.
1. Baseline. The company processes 18,000 invoices a month, and 22% fail matching: 3,960 exceptions. Average handling time is 25 minutes at a loaded cost of $48 an hour, so $20 per exception and about $79,200 a month.
2. Target. Handling time drops to 10 minutes on exceptions routed through the system: $8 each, a $12 saving. The pilot averaged 8 minutes, but it skipped freight and multi-currency invoices, so the target sits above the pilot result on purpose.
3. Adoption. 25% of exceptions routed at month 2, 50% at month 4, 70% at month 6. The other 30% (unusual supplier formats, disputes) stay manual for now.
4. Full cost. A one-time scale cost of $180,000: ERP integration beyond the pilot’s read-only export, an evaluation set covering all supplier types, security review and training. A monthly run cost of $10,000: about $2,500 in inference, about $4,000 for a senior AP specialist reviewing quality half-time, and about $3,500 for monitoring and maintenance.
5. Cost of delay. At 70% adoption, 2,772 exceptions a month × $12 = $33,264 gross. Minus $10,000 run cost, that’s about $23,000 a month net. Each month the rollout slips past month 6 costs roughly that much. Payback on the $180,000 comes about eight months after full adoption, a little longer counting the ramp.
Owner: the corporate controller, who owns AP cost and close timelines.
Written as a hypothesis: If 70% of invoice exceptions run through the system by month 6, handling cost on those exceptions drops from $20 to $8, worth about $33,000 a month gross, against $180,000 to scale and $10,000 a month to run. The controller owns the result, checked at the month-2 milestone.
How do you score the first milestone of an AI rollout?
With a scorecard whose thresholds are agreed before work starts, so the call at the milestone is mechanical rather than political. “Phase 2 complete” isn’t a milestone. Three outcomes are: continue, adjust or stop.
For the example above, at month 2:
| Measure | Continue scaling | Adjust and re-check in 4 weeks | Stop or rescope |
|---|---|---|---|
| Adoption (share of exceptions routed) | ≥ 25% | 15 to 25% | < 15% |
| Handling time on routed exceptions | ≤ 11 min | 11 to 15 min | > 15 min |
| Run cost per routed exception | ≤ $4.50 | $4.50 to $6.00 | > $6.00 |
| Accepted without edits | ≥ 85% | 75 to 85% | < 75% |
| Owner review held, decision recorded | Yes | n/a | No owner review within two weeks of the milestone |
Write down what each column triggers. “Adjust” might mean narrowing the invoice types routed or adding a review step. “Stop” might mean going back to pilot scope while a data issue gets fixed. Nobody should have to invent the criteria after seeing the results.
What does engineering need to build so the numbers can be checked?
Instrumentation, in place before scaling. Most pilots don’t have it, which is one reason so many rollouts can’t answer “did it work?” These are the five pieces we put in first:
- Per-request cost attribution. Tag every model call with the workflow, team and case ID, so run cost per unit is a query, not an estimate.
- Outcome logging. For every case: what the system proposed, what the person did (accepted, edited, rejected) and the handling time. That’s also your labeled data for improving the system.
- A comparison group. During the ramp, teams not yet on the system are your baseline. Compare like-for-like work across the two groups instead of comparing against last year.
- An evaluation set that matches production. Rebuild it from full-volume samples, including the cases the pilot excluded. Accuracy on the pilot slice tells you little about the long tail.
- A metric the owner already watches. Put the scorecard in the report the controller (or whoever owns it) already reviews. A separate AI dashboard gets checked for two weeks.
None of this is exotic. Our post on moving from pilot to production covers the engineering side of that transition in more depth.
Where does an AI economic hypothesis usually break?
In one of three numbers: adoption, full cost or baseline. Adoption misses most often.
Adoption. The fix is rarely more training. It’s usually the workflow: the tool adds a step instead of removing one, or it lives outside the system where the work happens.
Full cost. This one misses next, mostly through human review. As volume moves into messier cases, the share of low-confidence outputs rises and review time grows faster than planned. Narrowing scope to the work the system handles well is often the right call.
Baseline. Sometimes it was wrong. The team was faster than the sample suggested, or the process changed during the pilot. That isn’t a failure of the initiative, but it changes the value, and the hypothesis should be rewritten rather than defended.
A hypothesis that fails at the first milestone has done its job. You found out for the price of a ramp, not a full rollout.
Frequently Asked Questions
How do you know when an AI pilot is ready to scale?
When a named owner has signed a hypothesis with a measured baseline, a target discounted for the work the pilot didn’t see, an adoption ramp, a full run cost and agreed milestone thresholds. Good pilot accuracy alone doesn’t qualify.
What costs do AI pilots usually leave out?
The run costs: inference at full volume, human review of exceptions and low-confidence outputs, monitoring and on-call, and ongoing maintenance of prompts, models and integrations. Pilots often hide these behind vendor credits and the builder’s own time.
What should happen if an AI rollout misses its first milestone?
Follow whatever the scorecard said in advance. Usually that means narrowing scope, fixing the workflow so the tool removes a step, or rewriting the hypothesis around a corrected baseline instead of defending the old one.
Is one of your AI initiatives waiting on a scale-up decision?
Send us a short description: what it does, who uses it today and what it’s supposed to improve. We’ll send back the five numbers its case needs, mark the ones you likely have already, and draft a first-milestone scorecard you can take to your CFO. If the numbers hold up, our engineers embed with your team and build the production version in your stack, with the instrumentation to prove it. Send us an initiative.