hermes-labs-ai/hermes-jailbench
12 stars · Last commit 2026-09-29
Zero-LLM deterministic jailbreak regression benchmark: runs a repeatable battery of known-pattern attacks against an LLM endpoint and scores refusal/partial/compliance across runs. Catch when a model or prompt update silently got weaker. Single-turn, responsible-use. pip install hermes-jailbench
README preview
<div align="center"> <h1>hermes-jailbench</h1> <p>A jailbreak regression benchmark that replays known-pattern attacks against Anthropic or OpenAI-compatible endpoints and uses deterministic keyword heuristics to classify refusal, partial, or compliance.</p> <p>hermes-jailbench is developed by <a href="https://hermes-labs.ai">Hermes Labs</a>.</p> <p>Hermes Labs is an agentic infrastructure company building the reliability layer for autonomous systems.</p> [](https://pypi.org/project/hermes-jailbench/) [](https://pypi.org/project/hermes-jailbench/) [](https://www.apache.org/licenses/LICENSE-2.0) [](https://github.com/hermes-labs-ai/hermes-jailbench/actions/workflows/ci.yml) [](CODE_OF_CONDUCT.md) </div> hermes-jailbench is a jailbreak regression benchmark that runs a repeatable battery of known-pattern attacks against an Anthropic or OpenAI-compatible model endpoint and uses deterministic keyword heuristics to classify each response as refusal, partial, or compliance — so you can tell when a model or prompt update silently got less safe on attacks it used to refuse.