NVIDIA wants to grade your agent's skills, not just its vibes

What it does

SkillEvaluator evaluates Agent Skills — the instruction-plus-files bundles defined by the agentskills.io spec — through three independent tiers: deterministic validation (schema, PII, license, lint, security scans), semantic deduplication against existing skills, and live evaluation that runs an agent with and without the skill to measure the difference. It’s the engine behind NVIDIA’s Verified Skills pipeline; skills that pass get published to the NVIDIA skills catalog.

The interesting bit

Tier 3 is the research-backed part: paired with/without-skill trials and a “Skill Lift” metric, from a paper on evaluating skills rather than agents. Instead of trusting that a skill file helps, it actually watches the agent behave both ways. Also nice: the tiers are independent entry points, so you can run just the keyless deterministic checks without touching an API key.

Key highlights

• Tier 1 runs fully offline for deterministic checks — no API key, Docker, or clone needed
• Security scanning leans on external tools: Semgrep, Gitleaks, and NVIDIA’s SkillSpector
• Tier 2 detects overlap with existing skills and flags repeated guidance within one skill
• Tier 3 sandboxed agent runs are powered by Harbor, with Docker, local, or cloud sandboxes
• Works with OpenAI, Anthropic, Bedrock, NVIDIA Build, or any OpenAI-compatible endpoint

Caveats

• Explicitly experimental: community-supported, best-effort, no SLA or enterprise support
• Full security coverage requires installing three external scanners; missing them means incomplete validation
• Live evaluation costs real model calls, and local sandbox mode is only for trusted skills and workspaces

Verdict

Worth a look if you’re publishing or curating agent skills and want more than a lint pass. If you just write skills for your own use and never ship them, it’s heavier than you need.