AI Evaluation and Security
Run a Prompt Injection Tabletop Around One Tool Call
Guide an application team through a harmless tabletop that identifies the independent control protecting one consequential tool call.
Choose one action your assistant can propose, such as changing a support ticket's destination. Then trace the checks between that proposal and a committed change. This Romfix exercise is a design review using fictional records in an isolated environment. It reports no exploit success rate and needs no real customer data.
Draw the smallest action path
Our hypothetical assistant reads a ticket, drafts a response and proposes an update. A sentence inside the ticket asks for delivery to an unrelated recipient. The ticket is data supplied by a third party; its presence in an approved connector does not make its instructions authoritative.
Write down the authenticated user, tenant, ticket ID, permitted operation and approved recipient before considering the model's output. An authorization rule must be able to consult those trusted values even if the model proposes different ones.
Complete a boundary record
| Boundary | Review question | Expected evidence |
|---|---|---|
| Retrieval | Can this user read this tenant's ticket? | Permission decision before context assembly |
| Context | Which material came from the ticket? | Source ID and trust label preserved |
| Proposal | Are the proposed arguments well formed? | Typed validation result |
| Authorization | Does the recipient match the permitted transaction? | Independent allow or deny decision |
| Commit | Is the approved transaction still the one being executed? | Stored outcome tied to the reviewed arguments |
| Follow-up | Can untrusted content become durable policy? | Separate review for memory-like updates |
Schema validation helps establish argument shape. It cannot establish that the recipient belongs to the right tenant or that sending is authorized. Keep those ownership and effect checks in trusted code.
Rehearse three outcomes safely
First, let the assistant summarize the fictional ticket without a write capability. An inaccurate summary is a quality failure to record. Second, permit it to propose an update, but require the gateway to deny a mismatched recipient. Third, supply a legitimate proposed update and check that the normal workflow still succeeds. Preserve both negative and positive results so an always-denying gateway does not look like a working product.
Use only synthetic identifiers and a local sink. Do not send messages to real recipients or place secrets in the context. Record the proposed arguments, policy version, decision and final stored state; a reassuring model answer is not evidence that a downstream action was blocked.
Decide what a failed check means
If hostile text changes the answer but cannot trigger an unauthorized effect, improve answer quality while retaining the boundary. If a denied proposal reaches the sink, investigate the commit path before expanding testing. If a memory update persists an instruction, review that separate capability rather than fixing only the ticket parser.
OWASP's prompt injection guidance describes direct and indirect inputs and the limits of prevention. Romfix's exercise applies that concern to a narrow workflow; it is not a claim that filtering, retrieval or a better prompt removes the risk.
Retest when authority changes
Repeat the record when you add a connector, recipient type, tool or persistent store. Keep the fixture and expected outcomes versioned. Review whether logs show the actual committed effect as well as the model proposal.
Before testing an autonomous environment, consult the cyber evaluation release worksheet. To decide which measurements support a product pilot, use the benchmark decision sheet.