Benchmark methodology for the Ultra13 Context Firewall
Ultra13 benchmarks agentic AI security by replaying attack families against a workflow with the firewall off and then with the firewall on. A finding is counted as neutralized only when the attack succeeds in the unprotected run and is blocked, redacted, quarantined, approval-gated, or otherwise prevented from reaching the unsafe sink in the protected run. Results are tied to product version, policy version, model/provider assumptions, attack definitions, benign test cases, and validation date.
- Each attack family is replayed with the firewall OFF then ON.
- Neutralized = succeeds unprotected AND is blocked when protected.
- Results are pinned to product version, models, datasets, and date.
- Coverage depends on workflow, deployment mode, and policy configuration.
Status
The OFF → ON definition
For each attack, we run the workflow unprotected and confirm the exploit succeeds, then run it with the firewall on. It counts as neutralized only if the protected run blocks, redacts, quarantines, or gates the path before it reaches the unsafe sink. Partial coverage and gaps are defined explicitly and reported separately.
Attack families
Framework mapping
Ultra13 uses OWASP’s agentic security work and MITRE ATLAS as reference points for attack-family mapping and validation planning. Mapping is intended to help security teams reason about adversarial behaviour; it is not a claim of certification or endorsement. See the sample proof report for how a single finding is represented end to end.