Autoreason

What Autoreason does

Autoreason targets three failure modes the README attributes to iterative self-refinement: prompt bias, scope creep and a lack of restraint. Each iteration produces three competing versions, the unchanged incumbent (A), an adversarial revision (B) and a synthesis (AB). Fresh agents with no shared context judge them by blind Borda count, and keeping the incumbent is always a valid outcome. The loop converges when A wins twice.

The repository contains the LaTeX paper and compiled PDF, task prompts for five open-ended and three constrained writing tasks, blinded human evaluation materials, and runners for writing experiments, CodeContests problems, multi-seed replication and ablations. The README reports results across model tiers including Haiku 3.5, Haiku 4.5, Sonnet 4 and Sonnet 4.6, and notes that gains on held-out code problems vanish at Haiku 4.5. Its citation block lists SHL0MS and Hermes Agent as authors.

Key features

• A three-way tournament of incumbent, adversarial revision and synthesis
• Blind Borda-count judging by fresh agents with no shared context
• Do-nothing treated as a first-class option
• Experiment runners for writing tasks and 150 CodeContests problems
• Blinded human evaluation materials with a rubric
• Ablations on judge count, aggregation, components and length control

When to use it

• Reproducing the paper's experiments
• Studying how to stop a self-refinement loop from degrading its output
• Adapting the tournament idea to your own writing or code pipelines

Who it is for: Researchers and agent builders interested in evaluating or stopping iterative self-refinement.

How it fits with Hermes Agent

The README header reads SHL0MS | HERMES AGENT and its citation lists Hermes Agent as a co-author, but the opening part of the README does not describe running the method inside Hermes Agent.

FAQ