What a pilot has to answer
Pilot projects frequently end with a favourable impression and no basis for a decision. How to see that coming beforehand.
A pilot is meant to prepare a decision. Frequently it produces agreement instead: everyone involved has seen something, it worked, and afterwards nobody can say whether the effort is worthwhile.
The question to settle before starting
A pilot with no stopping condition agreed in advance is not a test but a phased rollout. Anyone who does not record, before it begins, which result would argue against continuing will find no such result at the end.
The useful formulation is uncomfortably specific: which measure, on which date, at what level? Anything vaguer can be read afterwards in either direction.
The benchmark is almost always missing
A pilot shows how well a new procedure performs. It says nothing about how well the existing one performs, unless that is measured over the same period on the same material.
Without that comparison, any improvement is an assertion. With it, it frequently becomes clear that the previous process was better than remembered — or that the improvement is real but smaller than the effort.
The pilot's conditions are not the operation's
In a pilot, motivated people work on selected cases with short paths to support. In operation, changing people work on all cases with no special attention.
That difference is not a detail. It is regularly why a successful pilot fails to carry at scale — and it can be narrowed by deliberately running the pilot under ordinary conditions rather than the best available.
What a usable result looks like
A pilot that ends with "it worked well" has delivered nothing. One that ends with "on these cases the error rate was X against Y in the existing process, at this cost in time" has enabled a decision — even if the decision is not to proceed.