Microsoft’s CLI for stress-testing AI agent skills

What it does Waza is a Go CLI that brings CI discipline to AI agent development. It scaffolds evaluation suites, runs benchmarks against fixtures, and compares results across models so you can catch regressions before shipping. The tool wraps the GitHub Copilot CLI to execute prompts, record behavior, and grade whether your skill actually meets its claims.

The interesting bit The unusual angle is treating LLM unpredictability as a testable surface. Waza can record a live prompt, infer validators from the model’s response, and replay that as a repeatable task against different models or skill versions. It even supports A/B baseline testing and token budget comparisons—exactly the boring operational details that separate demos from production tools.

Key highlights

• Scaffolds entire eval projects with waza init and waza new skill, including GitHub Actions CI pipelines
• Records live Copilot prompts and auto-generates reusable task YAMLs with inferred validators
• Compares results across models and runs A/B baseline tests to measure skill improvement
• Tracks token budgets and suggests optimizations to keep skills lean
• Ships as both a standalone binary and an Azure Developer CLI extension

Caveats

• Template packs for skill scaffolding are marked “coming soon” in the README
• Source builds require Git LFS and cannot use standard go install because of embedded Copilot binaries

Verdict Waza is worth a look if you are building GitHub Copilot custom skills or agent definitions and need to prove they work consistently across models and changes. If you are not working with agent skills or Copilot, this is narrowly scoped tooling that will not offer much.