Products

AI-Armory

Prove your AI guardrails work, with evidence rather than assurances.

AI-Armory is a red-team testing platform built for AI and LLM security. It fires adversarial payloads at an AI system and measures, in numbers, whether the defences in front of it detect, block, sanitise, or leak.

The problem

AI systems get deployed with security claims nobody has tested. The tools that would test them were built for web applications, not for language models and the guardrails sitting in front of them.

The gap that matters is not whether an attack worked. It is how much of the risk your security layer actually removed, which you cannot know from a single pass against a defended system.

What it does

Two lanes, so the number means something

Every payload fires through two lanes at once: an unprotected baseline lane and the defended lane. Comparing them says exactly how much protection the security layer is adding, rather than only whether a given attack got through.

The headline metric is the block rate, computed as blocked plus sanitised over baseline leaks.

An attack library, mapped to the frameworks

The library ships pre-loaded, generated from the MITRE ATLAS framework, the industry taxonomy for AI and ML adversarial threats. Every payload is tagged to an OWASP LLM Top 10 category and a severity, so results roll into compliance reporting rather than needing translation first.

Payloads escalate on their own, from naive to rephrased to obfuscated with base64, unicode or leetspeak, then contextual and advanced, through deterministic transformation techniques. Pentesters can also author and run their own payloads immediately without touching the curated core.

Five attack engines, one platform

Not a single scanner. Five engines run side by side, each suited to a different job, and they can be run independently or combined into one suite.

The native dual-lane runner drives the curated corpus, the primary block-rate metric and regression testing. Garak, from NVIDIA, sweeps the OWASP probe taxonomy. Promptfoo runs YAML-defined, assertion-based suites that suit CI. Microsoft PyRIT covers encoded and obfuscated attacks and real multi-turn adversarial conversations, including Crescendo escalation. PAIR is an attacker-LLM that refines jailbreak prompts against your system in real time, which is the most thorough test of genuine jailbreak resistance.

A red-team agent, with a human in the way

A built-in red-team agent generates adversarial payloads automatically. Describe an attack goal in plain English and get dozens ready to run. Curated presets cover prompt injection override, system prompt extraction, RAG injection, Crescendo, encoding bypass, PII exfiltration, authority abuse, tool-result injection, policy and toxicity, and a full OWASP sweep.

Anything the agent invents is staged for human review before it runs. Nothing an LLM writes goes live against a target without a curator signing it off.

Scoring that refuses to flatter itself

Armory does not only ask whether an attack was blocked. It scores recall and precision across both ingress and egress, because a guardrail that blocks everything scores perfectly and is useless: it breaks legitimate users too.

Specificity scoring measures false positives against hard, attack-lookalike benign traffic rather than small talk, so over-blocking shows up as a failure instead of a hidden cost. Egress scoring checks what happens to the output of the model, not just whether the prompt was allowed in, which is the gap that ingress-only testing misses entirely.

The attacks that actually get through

RAG and indirect injection testing, under OWASP LLM01, covers document, retrieval and channel injection vectors: the attacks that hide inside the data your AI reads rather than the prompt a user types.

Agentic and tool-calling testing, under OWASP LLM08, covers filesystem, database, credential exfiltration, lateral movement and config tampering, built for AI agents that can take action rather than only talk.

From test to evidence

The client assurance report is a consultancy-grade deliverable: cover page, executive summary, findings with remediation guidance, and compliance mapping to the NIST AI RMF, the EU AI Act and ISO 42001, exported as a versioned PDF.

OWASP and MITRE ATLAS scorecards show coverage gaps, not only passes and failures. That is the difference between defending something and never having tested it, which is the finding a real pentester cares about. Coverage exports to the official MITRE ATLAS Navigator heatmap. Every action, run, finding and configuration change is written to an append-only, tamper-evident audit log.

Continuous, not point-in-time

Nightly automated drift detection runs through a CI-integrated gate script, so a security regression surfaces when a model update, policy change or vendor upgrade weakens the defences, rather than at the next annual pentest. Results export as SARIF into GitHub and GitLab security dashboards, and webhooks carry alerts into Slack, n8n or an existing incident workflow.

Demonstrable in the room

A projector-ready live demo mode shows split-screen, real-time comparisons of the baseline against the defended lane, for walking a client, an executive or an auditor through how an attack is stopped, or is not. A manual chat sandbox lets researchers test hypotheses, including unicode tricks, encoding, zero-width characters and file attachments, before formalising anything into a repeatable test. Playlists turn a sequence of attacks into a scripted, timed narrative.

Built for real engagements

Role-based access control spans five roles: super admin, tenant admin, user and operator, auditor, and super observer, with a full capability matrix, so the right people can run tests and the right people can only observe. Programmatic access uses role-bearing API keys for CI and automation, alongside a REST API and interactive Swagger docs. A dedicated pentesting harness drives live-target engagements, graded on the telemetry of the target itself.

Not tied to one endpoint

Armory is not limited to testing the guardrail Onnex builds. It can point at any HTTP LLM endpoint, whether OpenAI-compatible, Azure OpenAI, an Anthropic-style gateway, or a custom internal service.

Who it's for

Two audiences, out of the same engagement. Security and pentest teams who need technical proof per probe. And executives, auditors and clients who need the outcome: every claim about the defences of a system becomes a number that can be shown to a board or a customer, one engagement produces both the technical report and the compliance-mapped PDF, and a small red team covers ground that used to need a much larger one.

Still to confirm

See exactly what your AI leaks, and exactly what stops it.