A protocol to test mission, answers, tools, permissions, exceptions, and human handoff before safely putting an AI agent into production.
Write an operational contract before the prompt
The contract states whom the agent serves, its intended outcome, what information it can use, what actions it can execute and what remains outside its scope. It also defines tone, restrictions, success criteria, escalation signals and an owner. A scheduling agent need not negotiate prices; a support agent should not improvise policies. Separating mission from instructions lets the team change models or implementations without losing the work's definition.
An agent is not ready simply because it responds well in a demo. It is ready when its mission is bounded, it uses tools with minimum permissions, passes normal and adversarial cases, recognizes limits, escalates with context, and can be observed after launch. Evaluation must cover both response quality and action safety.
Build a test with real, difficult conversations
The test suite should cover common requests, incomplete information, local language, typos, changes of intent, contradictions, provocations, and cases requiring a person. Examples are anonymized and represent the markets where the system will operate. Each case includes the expected answer, permitted action, and approval condition. A demo prepared by the team that built the agent rarely reveals the boundaries where it will fail.
Evaluate tools and permissions separately
A correct answer can accompany a wrong action. Therefore authentication, reading and writing, duplicates, retries, timeouts, missing data and revocation are tested. Agents receive minimum permissions: checking availability does not imply editing any calendar; recording a lead does not imply deleting contacts. Important actions should use idempotency, confirmation and logs to prevent a retry from becoming two bookings or two messages.
Measure multiple dimensions, rather than a general impression
The rubric can assess accuracy, relevance, policy compliance, tone, use of sources, handoff quality, and tool outcomes. Critical failures are separated from style preferences: inventing a commercial condition cannot be offset by a friendly response. Consistency across repeated runs and differences between language variants should also be measured. The release threshold is defined before the result is known to avoid approval driven by enthusiasm.
Launch in stages and retain a shutoff switch
It can first run in shadow mode, proposing without acting; then with a limited group, bounded permissions and intensive review; finally expanding volume if evidence permits. Logs, alerts, sampling and team feedback continue in production. There must be a clear way to pause the agent, revert to an earlier version and route every conversation to people. Activation begins evaluation, rather than ending it.
