Twelve Questions to Ask an AI Vendor Before You Sign

A demo is a performance, and there's nothing wrong with that. The vendor picked the documents, tuned the prompts and rehearsed the flow. You would do the same.

The problem is that your decision depends entirely on the part the demo excluded: the malformed input, the ambiguous case, the fifteen percent where the correct answer requires knowing something about your business that isn't in the file. Twelve questions get you there, and you should ask every vendor the same twelve so the answers are comparable.

Capability

1. Will you run fifty of our real historical cases, including the ones that went wrong?

This is the whole evaluation in one question. Assemble fifty cases weighted toward the difficult ones, with the known correct answers, and make everyone run the same set. The scoreboard writes itself and it rarely matches the demo. A vendor who resists this has told you something more useful than any answer they could have given.

2. What is your accuracy measured against, and who built the test set?

"Ninety-four percent accurate" is meaningless without a denominator. On what data, scored by whom, against what definition of correct? A serious vendor has a held-out test set, a scoring rubric and a number they'll defend. A weak one has a figure from a slide with no provenance.

3. What does the system do when it isn't confident?

Listen for a real mechanism: a threshold, an escalation path, a queue with a human on the other end. The dangerous answer is that it always produces something, because a system that never abstains will be confidently wrong on the exact cases that matter most.

4. Can we speak to a customer at our volume, in our industry?

Not the reference in the case study. A customer with comparable throughput, ideally one who has been live more than a year. Ask them what broke in month three, because something did, and how the vendor handled it.

Substance

5. Which model providers do you depend on, and what happens if one changes pricing?

Nearly every AI product builds on someone's foundation model, and that's fine. What matters is whether they know their exposure and have a plan. A vendor who can't answer this has never thought about the risk you're inheriting, and pricing in this market has moved sharply in both directions.

6. What have you built that a competent team couldn't rebuild in a quarter?

A blunt question and a fair one at the price they're asking. Good answers exist: proprietary data, a hard integration, an evaluation corpus built over years, real workflow depth. A weak answer means you're paying a product multiple for a prompt and an interface, which may still be worth it, at a different price.

7. How do you version and evaluate changes to prompts and models?

This separates engineering organizations from assemblies. If they push a prompt change on Tuesday and your output quality shifts on Wednesday with no notice and no regression test, you have a vendor whose release process is now your operational risk.

Economics

8. What is the total cost per transaction at our volume?

Not the license. Everything: inference, their margin, and the human review your team still performs. Then ask what happens when volume doubles, and when it halves. Contracts written at pilot volume can become unpleasant at production volume, and the reverse is worse.

9. How much human review does the workflow still need?

The honest number is never zero. If a vendor says it is, either they haven't run at scale or they're excluding review that will land on your team anyway. A vendor who says "about fifteen percent of cases, dropping to eight after tuning" is telling you they've actually operated this.

10. What does the integration work cost on our side?

Vendors price their product and are often vague about the work your team does to make it useful. Ask for the last three customers' integration effort in weeks. It's frequently the larger number.

Risk and exit

11. Where does our data go, and may you train on it?

Data residency, retention period, subprocessors, and whether your inputs improve their model. Get it in the contract rather than in an email from a sales engineer. If you've made commitments to your own customers about data handling, check that this vendor's defaults don't quietly break them.

12. If we leave, what do we take?

Your data in a usable format, your configuration, your prompts, your evaluation set. Negotiate it before signature, because afterward you have no leverage at all. A vendor whose exit terms are vague has designed the exit to be painful, and that's a commercial strategy rather than an oversight.

Is an AI automation agency legit?

I get asked this a lot, usually after someone has received a proposal that seems either suspiciously cheap or wildly expensive. The category has a low barrier to entry, so it contains both serious operators and people who configured a low-code workflow last quarter.

Building on top of an existing model is not the test. Almost everyone does, and there's nothing wrong with it. The test is whether they can answer questions 2, 8 and 12: how quality is measured, what it costs to run at your volume, and who owns the thing after launch. An agency with an evaluation process, a per-transaction cost and a named handover plan is legitimate regardless of size. One that judges quality by whether customers complain is selling you a demo with an invoice attached.

One more signal. Ask what they'd tell you not to automate. Anyone who has actually run these systems has a list, usually involving anything irreversible, anything touching a customer commitment, and anything where the exception rate is above about a third. A vendor with no such list will happily build you the thing that shouldn't be built.

Running the process

Set your comparison criteria before you see another demo, or the demos will set them for you. Score the fifty test cases the same way for every vendor. Then read the contract for performance remedies, data export and liability, which is where the difference between vendors is largest and least discussed.

Against a six-figure contract, a two-week evaluation is cheap insurance. If you'd rather not run it internally, that's the vendor diligence engagement, and I hold no vendor relationships and take no referral fees, without which none of this advice would be worth reading.

Send me your shortlist and I'll tell you which questions would separate them fastest.