How to examine a vendor's claims
Performance figures for AI products are hard to compare. A few checks that can be made without specialist technical knowledge.
Anyone buying technology faces an assessment problem: the vendor knows their product, the buyer knows their requirement, and the figures used in marketing rarely connect the two. A few questions help, without your having to be a specialist.
What was measured?
Performance figures refer to a basis of measurement. Where that basis is not stated, the figure cannot be checked and is therefore worthless for a decision. The question is not whether a number is impressive, but on what it was obtained and whether that basis resembles your own use case.
A system that performs well on a standard task need not perform equally well on your own operational data. The difference between the two is precisely the risk being purchased.
What happens when it is wrong?
Every system produces errors. The relevant question is what an error costs and who notices it. An error that is spotted and corrected is an irritation. An error that passes unnoticed into a downstream process is damage.
Vendors describe the normal case readily. The more revealing question concerns the exception: how does the business learn that the system is wrong, and how quickly?
What does leaving cost?
A purchasing decision is also a commitment decision. Data, workflows and interfaces align themselves to the chosen system over time. Anyone who does not know before entering what leaving would look like has not fully accounted for the cost of the decision.
Concretely: in what format will your own data be held at the end, and how much work would a change involve?
Who is liable for what?
Between the assurance that a function will work and liability for its absence, contracts often leave a considerable gap. Looking at that gap says more about the vendor's own confidence than any product description.
The common thread
All four questions aim at the same thing: the difference between what a system achieves in a demonstration and what it achieves in your own operation. That difference can be narrowed but not negotiated away — and it belongs in the calculation, not in the hope.