Pilot design: what business teams must own in the first 90 days
A pilot measures whether operations changed, not whether the model scored well. Most programs that start without business ownership end quietly by week twelve.
Pilot failure is hard to spot because it rarely shows up on slides. There is an accuracy number, a GPU chart, and a line about scaling in phase two. On the ground, agents have returned to the old spreadsheet, the reviewer queue has grown, and Monday’s meeting runs exactly as before. That is usually a design failure, not a model failure.
However good the engineering team is, the meaning of the pilot lives in business calendar time and authority on the CX or product side. Ninety days is a full learning window when scope is honest and exit criteria are written upfront. Related: architecture review before build vs buy should cite pilot adoption evidence, not only technical scores.
A pilot is not a demo by another name
A demo sells the story that AI can do something impressive. A pilot spends money on a narrower question: does this team, on this journey, under these constraints, actually work differently? Demo success is a screenshot. Pilot success is a behavior change.
| Signal | Demo | Pilot |
|---|---|---|
| Primary metric | Accuracy, latency | Bypass rate, time-to-action |
| Owner | Engineering | Named CX/product lead with calendar time |
| Exit | Slide deck | Continue, pivot, or stop with evidence |
| Failure handling | Deferred | Weekly review with frontline |
At Pivony we learned early that production wins are not model cards. Operations have to make room for the output. If the charter does not say who reviews, when, and what happens when the system is wrong, you are running a demo. See Pivony for how we operationalize enterprise CX feedback.
How ninety days should run
Fix the week-twelve decision date at kickoff or the pilot will drift.
Week zero is charter week. Keep it to one page, but argue every line. Do not close the room until three things are explicit: one cohort, one customer journey, and the week-twelve decision. Pair with VoC governance if customer data feeds the pilot.
Ownership is calendar time, not a title
When a pilot fails despite a named owner, the title usually exists but the calendar does not. The CX or product lead should block at least one day a week for the pilot. Volunteer working groups kill pilots because everyone supports and nobody accounts.
Which metrics matter
Accuracy alone misleads. Ask whether time from insight to action shrank, whether agents bypass the system, and whether the exception queue is growing. Shrink scope until you can measure those on a named cohort. Broader roadmap alignment helps keep the pilot on the portfolio, not as a side quest.
Sign the charter before sprint one
Frequently asked questions
How small should the pilot cohort be?
Small enough to know users by name, invite them to weekly review, and count behavior change at day ninety. Starting with hundreds of users often drowns learning in noise.
What engineering support is reasonable in ninety days?
Enough to keep workflow promises alive: access, logging, bug fixes, and exception routing. Not a parallel product roadmap; the minimum infrastructure the pilot can actually measure.
How do pilot results feed the build vs buy decision?
The architecture memo needs adoption evidence: bypass rate, time-to-action, and exception load. Model accuracy alone is not enough; whether operations actually used the tool drives the decision.
For, CTO and technology leaders
Size engineering support to workflow commitments. Supporting an early stop when business ownership is weak protects trust on both sides.
Paired perspective: Architecture review questions I use before a build vs buy callFurther reading
- McKinsey: The state of AI in early 2024, Gen AI adoption vs value capture
- Gartner: Gen AI deployment survey, 48% reach production; value demonstration barrier
- NIST AI Risk Management Framework, Governance before scale
- Google Cloud: RAG Engine overview, When pilots use retrieval
Running a program on this topic? Describe your context, technology, experience, or both.
Discuss an engagement