Choosing an AI Vendor: Telling a Great Demo from Real Delivery

The demo went beautifully: fluid interface, every question answered, charts generated live, smiles around the room. Three months after go-live, the operations team quietly went back to the old way of working. What happened in between?

Most of the time, nobody lied. A demo and production are simply two different things. A demo shows the ceiling under ideal conditions; what you are buying is the floor under everyday conditions. Evaluating a vendor is essentially the art of seeing that floor before you sign.

Why demos are naturally flattering

The usual techniques are not malicious, but you should know them. The data is curated: clean, typical samples with no ambiguity. The questions are rehearsed: every prompt in the script has been tested beforehand. The environment is tuned: demo data volume, concurrency and permission complexity look nothing like your production setup. And the edge cases are steered around: vague inputs, missing fields and out-of-scope questions never make it on stage.

Your business, meanwhile, is made of edge cases: customers who mistype, records with half the fields empty, questions nobody predicted. What a system is worth shows not in how it answers the standard questions, but in how it behaves on messy input.

Five things you can do on the spot

Bring your own data and run it live. Prepare a batch of real business data in advance — anonymised is fine, beautified is not — and ask for it to be run in front of you. "We need to configure first" is a fair answer, but once configured, the demo must run again on your data.

Feed it deliberately bad input. Typos, a ticket missing half its information, an irrelevant question, a prompt built on a wrong premise. Watch whether it fabricates, errors out, or honestly says it is unsure and routes to a human. How a system fails tells you more than how it succeeds.

Ask who runs it after go-live. Who handles daily operations? How fast is incident response? When rules or models need adjusting, is there a defined iteration mechanism, or do you file a ticket and wait? Many projects do not die of technology; they die of "delivered, then orphaned."

Ask where the data lives and how it leaves. Whose servers, in which region? Can everything be exported completely when the contract ends, and in what format? If these two answers are not crisp, discount everything else you hear.

Start with a small contract. One scenario, one short term, expand on evidence. Vendors confident in their delivery rarely mind starting small; a vendor who insists on one big deal up front should be asked why.

Make the POC count: acceptance criteria before work starts

Plenty of companies run a POC and still choose wrong, because the POC becomes an extended demo: a month of activity that ends with impressions instead of conclusions. The right order is to agree written acceptance criteria before work begins: what data to test on, which scenarios, what accuracy or turnaround counts as passing, and who judges. For criteria that can actually be checked, see Writing Pilot Acceptance Criteria That Can Actually Be Checked.

The criteria carry a side benefit: the vendor's reaction to them is itself a signal. One who accepts them and helps sharpen them is a different company from one who keeps urging you to "get it running first and see."

Count the whole cost, not the quote

The software fee on the quote is usually a fraction of the real cost. A complete account has at least four lines: software subscription or licence; implementation — data cleaning, integration, workflow configuration; internal staff time — your best people testing, training and shepherding the rollout, the line most often forgotten and usually the most expensive; and future switching cost — whether the data can leave, and how deeply your processes get bound to one product. Total it over three years and compare again; the ranking often changes.

Red flags

• Promises of full automation with no human involvement — serious AI systems keep a place for human review and fallback
• Committed growth numbers, such as "inquiries up thirty percent after go-live" — outcomes depend on your business, and nobody can promise them in advance
• No trial and no POC, demo only — an admission that the product cannot survive your real data
• No data-export clause in the contract, or a vague one — that is switching cost being welded onto you
• Every request answered with "we can do that", never "this is not our strength" — naming its own boundaries is part of a vendor's competence

Before you pick the vendor, pick the right problem

Step back once, at the end: no amount of vendor diligence rescues a project that should not exist. Before meeting anyone, run the idea through Six Questions to Ask Before Approving an AI Project: what is the business value, is the data ready, who owns it internally. A buyer who knows what they want, doing the five things above, is very hard to sway with a demo — and a vendor who fears those five things has done the screening for you.