The Agentic AI Buyer's Checklist: What to Ask Before You Deploy
Read Time 15 mins | Written by: Vinayak Bhagat
Sit through three agentic AI pitches in one quarter and you will notice something: they are the same pitch. The same demo where an agent books the meeting, updates the CRM, and drafts the follow-up. The same architecture slide. The same case study with a logo you cannot call. The demos are real, and they are also the least useful information in the room, because a demo shows you the vendor's best five minutes and tells you nothing about month six.
This agentic AI buyer's checklist is the set of questions we ask when mid-market clients bring us a shortlist. The differences that matter between vendors are almost never visible in the demo. They live in the answers to eight questions, and the pattern we see is consistent: strong vendors enjoy these questions, because they have real answers. Weak vendors change the subject.
If you are still working out what agentic AI is and where it earns its keep, start with our plain-language guide to agentic AI for enterprise. This post assumes you are past that point: the use case is real, budget exists, and vendors are in the building.
What should you ask an agentic AI vendor before you buy? Eight questions: What does the agent decide without a human, exactly? Show me a production customer with my workload, not a demo. How do you prove it works — what do your evaluations measure? What happens when it is wrong, and what is the undo? What does a completed task cost at my volume? Where does our data go, and is it used for training? How do we leave — what exports, what breaks? And who owns the outcome when it fails — contractually, not aspirationally? A vendor with real answers to all eight is rare, and worth shortlisting.
Why Every Agentic Pitch Sounds the Same
Most agentic products are assembled from the same components: frontier models behind an orchestration layer, tool integrations, and a dashboard. That is not a criticism — it is why the demos all work. But it means the demo cannot differentiate vendors, because the demo exercises exactly the part everyone shares. What differs is everything around the model: the boundaries, the evaluation discipline, the failure handling, the economics, and the contract. Those are precisely the things a pitch is designed to keep abstract.
Buying agentic AI is also genuinely different from buying software. Software does what it is configured to do; an agent decides. That single difference moves the risk from "does the feature work" to "what is this system allowed to do in our name, and what happens when it is wrong" — which is why a checklist built for SaaS procurement misses the point. The eight questions below are built for the decision-making part.
The Buyer's Eight
1. What does the agent decide without a human — exactly?
Ask for the specific list of actions the agent takes autonomously, the actions that require approval, and the actions it can never take. A strong vendor hands you a boundary document and walks you through how boundaries are enforced — in the system, not in a prompt. Red flag: "it's configurable" with no default boundaries, which usually means nobody has thought about them until a customer got burned.
2. Show me a production customer with my workload
Not a logo slide — a reference call with a customer of your size, in your kind of workflow, running in production for months. One honest reference outweighs every demo. Ask the reference what broke in month two and how the vendor responded, because something always breaks in month two. Red flag: every named customer turns out to be a pilot, a design partner, or an investor's portfolio company.
3. How do you prove it works — what do your evaluations measure?
Serious agent vendors run evaluation suites: defined tasks, measured completion quality, tracked over releases — and they can show you results for a workload like yours. This is the same discipline we require before any go-live in our production-readiness checks for agentic AI, and a vendor who cannot clear it for their own product will not clear it for your deployment. Red flag: accuracy claims with no definition of what was measured, on what tasks, or by whom.
4. What happens when it is wrong?
Agents will be wrong — the question is whether wrongness is contained. Ask how errors are detected, what the agent does when it is uncertain, whether every action is logged and replayable, and what the undo story is for each action type. A vendor who says "escalates to a human and here is the audit trail" has operated in production. Red flag: the conversation keeps returning to how rarely it is wrong instead of what happens when it is.
5. What does a completed task cost at my volume?
Agents multiply model calls — one task can fan out into dozens of steps — so per-seat or per-call pricing can hide the number that matters: cost per completed task, at your volume, including retries and failures. Ask the vendor to model it, then pressure-test the assumptions with the four dials from our FinOps guide for AI workloads. Red flag: pricing that only makes sense if you do not ask what happens at scale, or a vendor who cannot explain their own unit economics.
6. Where does our data go?
An agent sees more than a form-filling app ever did: it reads records, documents, and conversations to act on them. Ask which models and sub-processors touch your data, whether any of it is used for training, where it is stored, and what the retention terms are — in the contract, not the FAQ. The NIST AI Risk Management Framework is a useful shared vocabulary for this conversation. Red flag: answers that differ between the sales call and the data processing agreement.
7. How do we leave?
Assume you will switch vendors within a few years — the market is young and moving fast. Ask what you can export (configurations, prompts, logs, learned workflows), what formats, and what keeps working the day after you leave. The honest answer is that some value is not portable; a vendor who names what is and is not portable is telling the truth. Red flag: exit only comes up in the contract's termination clause, priced accordingly.
8. Who owns the outcome when it fails?
If the agent mis-handles a customer, ships a wrong order, or leaks a record — what does the vendor owe you, contractually? Look for concrete commitments: incident response times, liability language that survives legal review, and a support model with named humans. Marketing says "partner"; the contract says who pays. Red flag: enterprise-grade claims in the deck and consumer-grade limitation-of-liability in the paper.
Try it against the vendor you met last week. Tap each question they answered with specifics:
Get the full checklist as a printable PDF — every question with the good answer, the red flag, and a pilot worksheet.
| Question | A good answer looks like | Walk away when |
|---|---|---|
| 1 · Autonomy boundaries | A written boundary list, enforced in the system | "It's configurable" and nothing else |
| 2 · Production references | A reference call with your workload, months in production | Every customer is a pilot or design partner |
| 3 · Evaluations | Defined tasks, measured quality, tracked per release | Accuracy claims with no definition |
| 4 · Failure handling | Escalation on uncertainty, full logs, a real undo | "It's almost never wrong" |
| 5 · Unit economics | Cost per completed task modeled at your volume | Pricing that hides what scale costs |
| 6 · Data terms | Named models and sub-processors, no-training terms in the DPA | The sales call and the DPA disagree |
| 7 · Exit path | Honest list of what exports and what does not | Exit exists only as a termination fee |
| 8 · Accountability | Incident SLAs and liability that survive legal review | Enterprise deck, consumer contract |
Prefer it side by side? Flip through each question — the good answer on one face, the walk-away flag on the other:
Get the walk-away table as a one-page reference card — inside the free PDF.
Run Your Cost per Completed Task
Question 5 only has teeth if you bring your own numbers. Set your volume, what the task costs a person today, and the vendor’s pricing — the calculator shows the effective cost per completed task including failures, and what the vendor has to defend:
Download the checklist PDF — it links back to this calculator and the interactive scorecard, so procurement can run the same numbers.
Three Mistakes Buyers Make Under Pitch Pressure
Mistake 1: Letting the demo set the requirements. The demo shows what the vendor built; your checklist should come from your workflow. Write the ten tasks you need automated before the first pitch, and make every vendor run against the same list — otherwise you are comparing each vendor's favorite five minutes.
Mistake 2: Piloting without success criteria. An open-ended pilot always "shows promise" — that is what pilots do. Define the completion rate, the quality bar, and the cost per task that would justify a contract before the pilot starts, and put a calendar end date on it. A pilot without criteria is a sales extension you are paying for with your team's time.
Mistake 3: Buying the agent before the readiness exists. An agent inherits your data quality, your definitions, and your access controls. If those are not in place, the strongest vendor on the shortlist will still stall — and the postmortem will blame the tool. Our AI readiness assessment exists to answer whether the foundation is ready before the contract, not after.
How to Run the Checklist
Use the eight questions as a filter, not a scorecard. Questions 1–4 (boundaries, references, evaluations, failure handling) decide whether a vendor is safe to pilot — send them in writing before you give anyone a second meeting, and read the answers for specificity, not polish. Questions 5–8 (economics, data, exit, accountability) decide whether a pilot is safe to buy — they belong in the pilot period and the contract negotiation. A vendor who resents the questions has answered them.
Download the checklist and send questions 1–4 to your shortlist this week.
Get the Checklist as a PDF
The Agentic AI Buyer's Checklist. One PDF with everything: all eight questions with the good answers and the walk-away red flags, the one-page reference table, and a pilot success-criteria worksheet — plus one-click links to the interactive scorecard and the ROI calculator above.
Fill in the form and the download appears right here. Bring it to the next vendor call, or print one per person on the deal team.
Bring an engineer's eye to the shortlist
Ontrac's agentic AI team runs vendor evaluations and pilots for mid-market companies: the Buyer's Eight against your actual workflow, success criteria that mean something, and a build-vs-buy answer you can defend. We build these systems ourselves, which is exactly why we know what to ask.
Talk to the agentic AI team Explore our agentic AI servicesFrequently Asked Questions
What should be on an agentic AI buyer's checklist?
Eight things: the agent's autonomy boundaries in writing; production references with your kind of workload; evaluation results with defined tasks and metrics; failure handling with logs and an undo; cost per completed task at your volume; data-processing terms that match the sales pitch; a concrete exit path; and contractual accountability for incidents. Boundaries, references, evaluations, and failure handling decide whether to pilot; economics, data, exit, and accountability decide whether to buy.
How is buying agentic AI different from buying normal software?
Software executes configuration; an agent makes decisions and takes actions in your name. That moves the evaluation from features to judgment: what the system is allowed to decide, how its quality is measured, what happens when it is wrong, and who is accountable. It also changes the economics — agents consume model calls per task, so cost scales with usage and loop behavior rather than seats.
What are the red flags in an agentic AI vendor pitch?
The recurring ones: no default autonomy boundaries ("it's configurable"); references that are all pilots or design partners; accuracy claims with no evaluation behind them; deflecting "what happens when it's wrong" back to how rarely it is; pricing that obscures cost at scale; data terms that differ between the call and the contract; and enterprise claims paired with consumer-grade liability language. Any one is a question mark; several together are an answer.
Should we run a pilot before signing an agentic AI contract?
Yes, with two conditions: written success criteria before it starts (completion rate, quality bar, cost per task), and a fixed end date with a decision attached. Run it on real work with the people who would live with the system. An open-ended pilot without criteria will always look promising and never produce a defensible decision.
