A test runner that grades agents and rewrites its own exams

What it does

skill-up is a CLI tool that evaluates “Agent Skills” — think of them as plugins for AI coding agents — against declarative YAML test cases. It spins up a workspace, installs the skill, invokes one or more agent engines such as Claude Code or Codex, and judges the output using rules, scripts, or even another agent. The result is a structured report in JSON, JUnit, or HTML that tells you exactly where the skill falls short.

The interesting bit

The real trick is the evolution loop: the project ships its own agent skill called skill-upper that reads failure reports, diagnoses whether the skill or the test is wrong, and rewrites the eval cases accordingly. It is essentially using an AI agent to debug the tests for another AI agent, turning evaluation from a static checkpoint into a self-improving conversation.

Key highlights

• Declarative eval configs (eval.yaml plus cases/*.yaml) replace ad-hoc run folders.
• Built-in support for multiple agent engines: claude_code, codex, qodercli, qwen_code, plus custom local transports.
• Flexible judging via rule-based checks, scripts, or an agent_judge.
• Anthropic-compatible evals.json import for migration.
• CI-ready with a root-level GitHub Action and structured reports (benchmark.json, grading.json, JUnit XML).

Caveats

• The README admits to Windows-specific limitations that merit their own guide.
• The skill-upper agent skill is distributed via a Node package workflow, despite the core CLI being Go-native.
• Out-of-the-box configuration defaults are empty; expect to supply your own YAML for telemetry and runtime kwargs.

Verdict

Worth a look if you are building or maintaining agent skills and need to prove they work across different AI clients without drowning in bespoke scripts. Skip it if you are just casually using agents and do not care about structured regression testing or CI gates.