WildClawBench

What WildClawBench does

WildClawBench is an end-to-end benchmark for AI agents built from 60 original, hand-made tasks. Examples in the README include clipping goal highlights from a football match video, negotiating meeting times over multi-round emails, hunting for contradictions in search results, writing inference scripts for undocumented codebases and catching privacy leaks. Tasks are grouped by agency, multimodal work, long-horizon workflows, coding and safety.

Each task runs in its own Docker container with the same image, data and grading code, and ground truth and graders are injected only after the agent finishes. The README says OpenClaw, Claude Code, Codex CLI and Hermes Agent all execute the same 60 tasks under the same grading, which lets you separate harness scaffolding from model ability. The evaluation code lives in the repository, the data ships as three Hugging Face datasets, and a paper is on arXiv.

Key features

• 60 original tasks across agency, multimodal, long-horizon, coding and safety categories
• One Docker container per task with the same image and grading code
• Ground truth and graders injected after the agent finishes
• The same task suite for OpenClaw, Claude Code, Codex CLI and Hermes Agent
• Three Hugging Face datasets: WildClawBench, WildClawBench-Harbor and WildClawBench-Trajectories
• Project page, technical report and arXiv paper

When to use it

• Compare Hermes Agent with other harnesses on the same real-world tasks
• Measure how much of an agent's score comes from its harness rather than the model
• Check safety behavior such as prompt injection defense and credential leak detection

Who it is for: Researchers and agent developers who need a reproducible benchmark for comparing agent harnesses and models.

How it fits with Hermes Agent

Hermes Agent is one of four agent harnesses the README lists as running the full task suite, so the benchmark can be used to compare Hermes Agent with OpenClaw, Claude Code and Codex CLI.

FAQ