What is actually inside an agent proof file

Sooner or later, someone at the table says it out loud: "can you defend this AI?"

That is not a technical question. It is a question about a file. And the agent vendor's dashboard does not answer it, for one simple reason: nobody outside the vendor can re-examine what is in it. A vendor who attests to the safety of its own system has not produced evidence — it has produced marketing.

The proof file exists to turn that question into a document. Here is what is inside.

1. The perimeter, written down before the testing starts

An agent is not a model. It is a model plus an intent, a set of tools, a memory, a corpus, and a policy written somewhere in a system prompt. The file starts by pinning that perimeter down in writing:

• what the agent is supposed to do, and what it is not supposed to do;
• the tools it can actually call — in other words, its declared side effects: writing to a system, sending a message, committing money;
• the data it can reach, and on what basis;
• who inside the organization answers for it.

This step looks like paperwork. It is not. In most engagements, this is where the first gap shows up: the agent can call a tool nobody had written into the perimeter.

2. What we put it through

Then comes the part the file documents at greatest length: the adversarial scenarios run against the agent before it touches production. Not a performance benchmark — a methodical attempt to push it outside its own policy.

For every scenario, the file records the exact input, the exact response, and the verbatim of the evaluation. The verbatim matters more than the score: a model risk lead wants to read the conversation, not a number.

3. The verdicts, and why one finding can stop everything

Findings land in three states: pass, review, block. The rule that makes the file useful is this one: a critical finding caps the overall result. No averaging, no trading a strong showing on nine dimensions against a failure on the tenth.

That feels wrong if you come from the dashboard world, where everything nets out. But an agent that handles 99 % of cases correctly and discloses personal information in the remaining 1 % is not "99 % good". It is blocking.

4. The mapping back to your own texts

A raw finding is useless to a compliance lead. So every observed behaviour is tied back to the framework that applies to you: AMF, OSFI Guideline E-23 or B-13, Quebec's Law 25, ISO/IEC 42001 and 42005, the OWASP Agentic Top 10, the NIST agentic profile, the EU AI Act if you export.

The goal is narrow and deliberate: your auditor should work in their vocabulary, not ours. That is what separates a file from a tool report.

5. What could not be tested

This is the section vendors leave out, and it is probably the most important one.

An honest file states what it did not cover: surfaces outside the perimeter, integrations that were unavailable when the testing ran, behaviours that depend on production data we were not given. When a required dimension lacks evidence, the verdict is not "pass by default" — it is withheld.

Absence of evidence is not evidence of readiness.

What the file is not

Three things, said out loud before a buyer finds them on their own.

It is not a certification of legal compliance. No tool and no firm can produce one. What is delivered is a mapping from observed behaviour to control status. The legal conclusion belongs to your counsel.

It is not an assurance engagement or an audit in the regulated sense. We produce the evidence; independence belongs to your auditor, who can re-examine every piece without taking our word for anything. That is in fact the only defensible use of the word "independent" here: not the vendor, not us — your auditor.

It is not a guarantee that the agent will never misbehave. A file is dated. It holds for one version of the agent, one set of tools, one model. Change the model, add a tool, widen the perimeter — and it has to be re-tested. That is precisely why ongoing proof custody exists as a separate service.

Why it is a document and not a dashboard

Because a dashboard cannot be filed. It cannot be attached to the minutes of a risk committee, handed to a regulator, or re-read eighteen months later when nobody remembers how the decision was made.

A document can. That is the whole distance between "we tested it" and "here is what we tested, here is what we found, here is what we could not cover, and here is who signed it".

When the question comes back around the table, that is the only answer that holds.