Making AI agents prove they read the manual

What it does

Skillgrade is a CLI test runner for agent skills. You define tasks in an eval.yaml file—along with fixtures, instructions, and graders—and it spins up an agent in a sandbox to see if the agent correctly discovers and uses your skill. It then scores the run with a mix of deterministic checks and LLM-powered rubrics.

The interesting bit

Instead of treating agent evaluation as a vague vibe check, Skillgrade treats it like a CI pipeline: deterministic scripts verify artifacts (did the file get fixed?), while a separate LLM rubric judges the session transcript for workflow quality, and the two are combined into a weighted final score. It also auto-detects which agent to invoke based on whichever API key you have lying around.

Key highlights

• Supports multiple agents out of the box: Gemini, Claude, Codex, OpenCode, and ACP-compatible agents via JSON-RPC over stdio.
• Two grader types: deterministic commands that parse JSON from stdout, and LLM rubrics that evaluate the agent’s session transcript against qualitative criteria.
• Weighted scoring lets you combine “did it work?” checks with “was the approach sensible?” judgments.
• Runs sandboxed in Docker for local safety, or --provider=local for faster CI runs.
• Includes a browser-based report UI and CLI preview, with file-reference shortcuts to keep YAML configs tidy.

Caveats

• Deterministic graders run inside a minimal node:20-slim container that lacks common utilities like bc; the README suggests using awk for arithmetic instead.
• ACP agent support is effectively local-only, since the README notes it “works best with --provider=local” because the agent binary must be present in your environment.
• The tool requires Node.js 20+ and Docker for the default workflow, so it is not a zero-dependency drop-in.

Verdict

Worth a look if you are building reusable agent skills and want regression tests before shipping. Skip it if you are just prompting an agent ad-hoc and do not care whether it consistently follows a structured workflow.