Questions to Ask an Agent Vendor Before You Sign

Most agent vendors arrive at the final meeting with the same deck: a workflow demo, a slide on governance, a logo wall, and a reference customer willing to take a call. Shortlisted vendors look interchangeable at that stage. They are not. The gap between a production agent vendor and a demo shop opens up in roughly ninety minutes of pointed questions, and it opens along specific seams: execution semantics, rollback, the tool registry, model portability, and exit.

This is a question set for that meeting. It assumes the commercial terms are mostly settled and the buyer now has to pick one of two or three finalists. It is written for a CIO, head of automation, or platform architect who has already read the SOC 2 report and now needs to know whether the software behind it actually behaves like production infrastructure.

So what do you actually ask?

Start With Execution Semantics, Not Model Choice

Model selection is the easiest question to answer and the least informative. Any serious vendor will swap models in a quarter. What they cannot swap in a quarter is how their runtime handles side effects.

Ask the vendor to walk through a single action end to end. Pick a destructive one: a wire transfer, a ticket close, a vendor payment, a record deletion. Then ask:

• Is this call idempotent, and what is the idempotency key derived from?
• If the agent crashes between the tool call and the state write, what happens on restart?
• What is the exact behavior when the downstream API returns a 5xx after a 200 was already observed?
• Which actions are transactional, which are compensating, and which are fire-and-forget?

A vendor who answers in terms of "the agent will retry" without naming an idempotency strategy is describing a chatbot with hands. The honest answers reference request IDs, dedupe windows, and compensating transactions. If you want a shared vocabulary before the meeting, the site's piece on eventual consistency and the one on race conditions in production cover the failure modes you should be able to name back to them.

The Tool Registry Is the Blast Radius

Every tool the agent can call is a potential wrong wire transfer. The tool registry — the catalog of functions the agent is permitted to invoke, with their scopes, credentials, and approval requirements — is where blast radius is actually bounded. Treat it as the most important artifact in the system.

Useful questions:

• Show us the tool registry for a comparable deployment. Who can add a tool, who can expand its scope, and what change-control approval is required?
• What identity does the agent act under when it calls our systems — a per-agent identity, a shared service account, or the end user's delegated identity?
• How are secrets scoped per tool, and how are they rotated?
• Which tools require a human approval gate by default, and how is that policy expressed — in code, in config, or in a prompt?

Approval policy expressed in a prompt is not policy. It is a suggestion to a stochastic process. The site's view on which actions actually need human approval is a useful cross-check against the tiers the vendor proposes.

Rollback and Kill Switches Are Separate Questions

Buyers often collapse "what happens when it goes wrong" into one question. There are at least three.

First, a kill switch: can an operator halt all in-flight agent runs within seconds, from a console, without a code deploy? Second, a quarantine: can a specific agent version, a specific tool, or a specific tenant be disabled while the rest keeps running? Third, true rollback: when the agent has already written to five downstream systems and the sixth failed, how is the partial state unwound — by compensating actions, by event replay, or not at all?

A vendor who talks about canary deployments, event sourcing, and compensating transactions is describing production software. A vendor who talks about "pausing the agent" is describing a demo. Ask to see a postmortem from a real incident, redacted. The absence of any postmortem is itself an answer.

Audit, Lineage, and the Deployer's Liability

Regulators are moving toward holding deployers responsible for third-party AI, not just builders. In May 2025, the Pennsylvania Attorney General settled with property manager Home365 over allegations that a third-party AI tool it purchased contributed to maintenance delays and unsafe housing conditions. The "we bought it from a vendor" defense did not hold. Your audit trail is your defense, and the vendor's logging schema is what makes that defense possible.

Ask for a sample log line and work backward from it:

• Can you reconstruct, for any action six months ago, the exact prompt, model version, tool version, inputs, outputs, and the human who approved it?
• Are logs tamper-evident, and where are they stored relative to our data residency requirements?
• Which fields map to which clauses in SOC 2, ISO 42001, and the NIST AI RMF?
• How is agent reasoning retained — full trace, summary, or discarded after N days?

If the vendor cannot produce a single log line without caveats, the audit story is theoretical. The site's deeper treatment of SOC 2 and audit trail requirements is worth reading before the meeting so you can press on specifics rather than headlines.

Model Portability and the Lock-In Question

Model prices and capabilities change on a monthly cadence. The vendor's architecture should assume you will want to switch — between Claude, GPT, Gemini, Llama, or an on-prem open-weights model — without rebuilding the agent. Ask directly:

• Which parts of the agent are model-specific: prompts, tool schemas, planner, evaluators?
• What is the effort to run the same workflow on a different foundation model, measured in engineer-weeks?
• Have you done this for a current customer, and can we talk to them?
• Do you support a secondary model for redundancy when the primary provider degrades?

Portability also matters for sovereignty. If the workload may need to move to a VPC or air-gapped deployment later, the question is not whether the vendor supports it today but whether the runtime is designed for it. The trade-offs are covered in the site's piece on air-gapped vs VPC vs SaaS.

Security Testing Should Be Adversarial, Not Theoretical

Prompt-level defenses are weaker than vendor marketing suggests. The Agent Security Bench, presented at ICLR 2025, tested attacks and defenses across 10 agent scenarios, more than 400 tools, 13 LLM backbones, and nearly 90,000 test cases, and found the strongest mixed attacks succeeded 84.3% of the time on average, with no tested defense holding up well.

Push the vendor past "we test for prompt injection." Ask which benchmarks they run against, how often, and whether results are shared with customers. Ask whether their red team targets the tool layer and memory, not just the prompt. Ask what they did the last time an external researcher reported a vulnerability — the timeline, the fix, the disclosure.

Exit, Data, and the Weights You Paid For

Exit is the clause buyers write last and regret first. The questions are mechanical:

• On termination, in what format do we receive our prompts, tool definitions, evaluations, traces, and any fine-tuned weights?
• What is the runtime license on the agent code and orchestration logic after exit — none, read-only, perpetual?
• How long is customer data retained after termination, and how is deletion attested?
• If you are acquired or wind down, what is the escrow arrangement for the artifacts above?

Pair these with the commercial clauses in the site's companion piece on pilot contract terms. The diligence conversation is where you confirm the vendor can actually honor the clauses the contract will require.

What a Good Meeting Looks Like

A production vendor will answer most of these without reaching for a slide. They will quote specific log fields, name their idempotency strategy, pull up a real tool registry, and describe at least one past incident with a blast radius and a fix. They will be comfortable saying what their system does not do.

A demo shop will deflect to the roadmap, cite the model vendor's security posture as if it were their own, and describe approval policy as something the LLM "understands." Both vendors can win a bake-off. Only one can run in production for three years without becoming the subject of your next audit finding. The question set above is how you tell them apart in the room, before the signature.