Playwright Spec for AI Agent

What Playwright Spec for AI Agent does

The tool reads @qa-scenario annotations on existing Playwright spec files, turns each scenario into a Given/When/Then plan, sends an AI agent to look at the real staging page, and returns pass, fail, manual_review or skip. The tool itself, not the agent, decides whether the evidence supports a verdict. It never runs your Playwright suite against staging, never modifies the specs, and reports ambiguous results as manual_review.

The pipeline runs spec, abstract-ai, judge and review, with optional Slack posting or GitHub issues, and a handoff step addresses a verdict to whoever fixes it. A per-test @qa-live-policy annotation sets how far the judge may go, from readonly to safe-interaction, while policies such as auth-mock are skipped on live. The README example names its judge output dashboard-hermes-judgment.md. The command npx playwright-spec-for-ai-agent demo runs an offline demo with no credentials.

Key features

• Reads @qa-scenario and @qa-live-policy annotations from existing Playwright specs
• Converts scenarios into Given/When/Then plans for live staging checks
• Verdicts of pass, fail, manual_review or skip, decided by the tool rather than the agent
• Optional Slack posting, GitHub issue filing and a handoff report for a coding agent
• Offline demo with a fixture adapter that cannot report a false green
• Zero runtime dependencies, with @playwright/test as an optional peer

When to use it

• Checking a staging dashboard against the intent of Playwright specs that were written for CI mocks
• Filing GitHub issues automatically from a nightly judgment run
• Handing a failed or ambiguous check to a coding agent with a frozen contract

Who it is for: Teams with Playwright specs who want an AI agent to verify live staging pages without rewriting their tests.

How it fits with Hermes Agent

The repository is tagged hermes-agent and its example judgment file is named dashboard-hermes-judgment.md. The README does not spell out the Hermes setup in the portion reviewed.

How to install Playwright Spec for AI Agent

These commands are copied from the project's README. Check the repository for the latest steps before you run them.

npx playwright-spec-for-ai-agent demo

Requirements: Existing Playwright specs with @qa-scenario annotations; @playwright/test is an optional peer needed only for browser session paths and trace or HAR evidence

FAQ