Every AI development company you shortlist will show you a working demo. That’s the problem. Demos stopped being a filter sometime around 2024, once the tooling made impressive ones cheap. RAND analyzed 2,400+ enterprise AI initiatives and found 80% failing to deliver their intended value anyway. The demos were fine. What failed was everything a demo doesn’t show, which is most of the job.
So the selection conversation has to happen in questions, and the useful ones aren’t about which model the vendor prefers. Here are twelve, grouped by what they actually reveal. Vague answers to any of them are themselves an answer.
Evidence, Not Vibes
1. Which production system of yours has run the longest, and what broke in its first year? Anyone who’s shipped has a story about drift or an integration that fell over at 2 a.m. A vendor with no war story either hasn’t operated anything long enough to have one or won’t tell you, and both of those are disqualifying in their own way.
2. Can we speak to a client whose project is two years old? Not a launch-week reference. You want the client still living with the maintenance bills, and with how responsive the vendor stayed once the invoice had long since cleared. That reference exists or it doesn’t.
3. What’s a project you turned down or killed, and why? Firms with judgment decline work that won’t produce value. Gartner expects over 40% of agentic AI projects to be canceled by end of 2027 on cost and unclear value grounds. A vendor who has never said no is planning to learn that lesson on your budget.
Engineering Practice
4. How do you evaluate outputs beyond “it sounds right”? There should be a concrete answer involving test sets and retrieval-correctness checks, with regression runs whenever anything changes. No evaluation method means the production bar is vibes, and vibes don’t survive contact with live traffic.
5. What happens when the system isn’t confident? You’re listening for fallback paths and human checkpoints designed in from the start, not bolted on after the first incident.
6. How will this run against our live data, not your curated set? Pilots run on clean data. Your invoices and tickets are not clean, and neither are your sensor feeds. A serious answer talks about data audits before model talk.
7. Who on the team actually builds, and who did we just meet? Ask for the delivery team’s names and roles. Sales engineers build the demo; you’re hiring whoever maintains the thing in month nine. Bait-and-switch staffing is the oldest trick in services, and AI hasn’t retired it.
Money and Terms
8. What does the pricing model charge for change, not just for build? AI systems change constantly by nature: reindexing and retraining, plus the occasional model swap. If every adjustment is a change order, your total cost has no ceiling. If maintenance is bundled, ask what’s actually inside the bundle.
9. What do the six months after launch cost, and what do we get? Monitoring and drift response at minimum, with a retraining cadence and defined support hours. A vendor who quotes launch and goes quiet on operations is selling you the pilot half of a two-part job.
Ownership and Exit
10. Who owns the models, prompts, and pipelines when we part ways? Get it in writing. Fine-tuned weights and embeddings, the evaluation sets, the orchestration code. If the vendor keeps any of it you’re renting, and the rent shows up at renegotiation.
11. Is our data used to train anything beyond our system? The acceptable answer is a flat no, contractually. Anything hedged deserves a follow-up with your legal team in the room.
12. If we hand this to another team in two years, what do they receive? Documentation and runbooks at minimum, plus architecture notes and the evaluation harness itself. Institutional knowledge that lives only in the vendor’s heads is a dependency dressed up as a relationship.
Reading the Answers
Notice what’s absent from the list: which foundation model, which vector database, which framework. Those change yearly and vendors switch them without ceremony. The twelve above probe things that don’t change: whether the firm has operated systems past the honeymoon and priced the full lifecycle, and whether clients end up owning their own stack.
One caution against over-indexing the checklist itself. A smaller AI development company might answer eight of twelve brilliantly and stumble on process questions a large firm handles smoothly, while the large firm’s answers came from a proposal library rather than experience. The follow-up question, “tell us about the last time that happened,” is where rehearsed answers run out. Use it liberally.
Firms like BiztechCS (building AI/ML and generative AI systems for operations-heavy businesses) sit on the receiving end of lists like this one regularly, and the pattern is consistent: the buyers who ask the ownership and post-launch questions end up with systems still running two years later, because they bought the operating half of the job, not the demo. The ai ml development work that compounds is the kind someone is still accountable for in month twenty.
If you’re shortlisting now, send the twelve ahead of the call and watch which vendors welcome them. That reaction is question thirteen, and it’s free. At BiztechCS, we’d rather answer them in writing anyway.
