NVIDIA Is Grading the Skills, Not the Agents

SkillEvaluator treats agent skills as artifacts worth testing in their own right — with security gates, deduplication, and paired trials that measure whether a skill actually helps.
The agent-skills ecosystem has a hygiene problem, and it’s not the one you’d guess. The obvious problem is security: a study behind NVIDIA’s SkillSpector scanner analyzed over 42,000 public skills and found 26.1% with vulnerabilities and 5.2% showing clear malicious intent, with the nastiest patterns hidden in comments, invisible text, and disguised toolchains (Inferri). That problem is real, and a July 2026 Hugging Face incident — an autonomous agent-driven attack chain that started with a malicious dataset and ended in credential harvesting — made it impossible to ignore (Generative Programmer).
But there’s a second, quieter problem: nobody knows whether a skill actually works. Skills — those folders of instructions, scripts, and reference files defined by the Agent Skills specification and now supported by 25+ agent frameworks — get written, published, and installed on faith. They look like documentation, so they get reviewed like documentation, which is to say barely. NVIDIA’s SkillEvaluator is the company’s answer: an Apache-licensed, three-tier framework that treats a skill as a testable artifact, running from deterministic linting all the way up to live, sandboxed agent trials that measure what the paper behind the project calls Skill Lift.
It is also the quality-control machinery for a commercial pipeline: SkillEvaluator is part of NVIDIA’s Verified Skills effort, and skills that pass get published to the NVIDIA skills catalog. The tool is open; the funnel it feeds is very much NVIDIA’s.
The boring tier is the load-bearing one
Tier 1 is validation, and it is deliberately unglamorous. Schema checks, PII scans, license checks, quality heuristics, Unicode safety, script linting — the kind of work that makes a security engineer’s eyes glaze over and a supply-chain auditor’s heart sing. The deterministic checks need no API key at all, which is a small design decision with a large implication: the cheapest tier is the one that catches the most obvious garbage, and it runs everywhere, including CI, without credentials or a Docker daemon.
For full coverage, Tier 1 reaches outside itself — Semgrep, Gitleaks, and NVIDIA’s own SkillSpector, which contributes the AI-specific threat models: prompt injection patterns, system-prompt leakage, memory poisoning, MCP tool poisoning, plus traditional AST analysis of dangerous code execution. SkillSpector’s two-phase design (64 static patterns, then optional LLM semantic review pushing accuracy to roughly 87%) slots in as the security conscience of the pipeline. The honest description is that Tier 1 is partly glue — SkillEvaluator orchestrates external scanners rather than reimplementing them — but glue with opinions about what a publishable skill must prove is exactly what the ecosystem lacks. The ten-project survey of the emerging skill-security stack (Generative Programmer) places SkillSpector and Verified Agent Skills alongside Cisco’s scanner, Microsoft’s agent package manager, and sandbox projects like OpenShell; SkillEvaluator is the layer that stitches scanning into a publish-or-reject decision.
Deduplication, or the war on context bloat
Tier 2 is the tier nobody else is building, and it addresses a failure mode the research literature keeps rediscovering. Skills are instructions injected into an agent’s context window. Two skills that say the same thing, or one skill that repeats its own guidance, don’t just waste tokens — they dilute the signal the agent needs to decide when to act on a skill at all. Tessl’s research found that agents often fail to activate available skills even when they’re relevant, with activation rates dropping to around 40% in unforced settings (Tessl). Redundant, bloated skill text is a plausible contributor.
So Tier 2 runs semantic overlap detection — embeddings-based similarity across a skill collection, plus a chat-model pass that checks a single skill for repeated internal guidance. It’s the least flashy idea in the project and possibly the most original. Anyone who has maintained a prompts directory knows the failure mode: guidance accretes, nobody deletes anything, and six months later the agent is being told the same thing four ways. SkillEvaluator makes that measurable, and by default it gates publication on it.
The research core: does the skill actually help?
Tier 3 is where SkillEvaluator earns its arXiv badge. The underlying paper — Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills — proposes paired with/without-skill trials: run the same agent on the same tasks twice, once with the skill loaded and once without, and measure the delta. That delta is Skill Lift. It’s a clean experimental design, and it directly answers the question every other approach dodges: not “is this skill well-written?” but “does this skill change what the agent accomplishes?”
The broader literature suggests the answer is often yes, dramatically. Tessl measured roughly a 20% absolute accuracy improvement when agents had access to a relevant skill, and found that a smaller model with the right skill could match a larger model at 3x lower cost (Tessl). A separate Tessl-authored framework paper, accepted at a KDD workshop, evaluated 500 real-world skills across 1,000 generated tasks and 19 agent–model configurations, reporting that skill access significantly changes model behavior (arXiv).
Tier 3 also includes synthetic eval-dataset generation, which deserves a skeptical eyebrow. Tessl’s own research found that around 30% of generated evaluation tasks exhibit issues like leakage — the answer bleeding into the prompt — which can produce misleadingly optimistic conclusions if unchecked (Tessl). SkillEvaluator’s autopilot mode, which creates an initial eval case when a skill has none, is convenient, and convenience in eval generation is precisely how leakage sneaks in. The framework’s answer is human review of generated datasets before broader runs, which is the right instinct but shifts the burden back onto the skill author — the person the automation was supposed to help.
The live runs themselves are powered by Harbor, the open-source agent evaluation framework, executing in Docker, local-OS, or cloud sandboxes. The local mode is explicitly flagged as experimental and only for trusted skills — a reasonable caveat, given that untrusted skill code running on a host OS is how supply-chain incidents happen. The gating semantics are thoughtfully graduated: Tier 1 always blocks, Tier 2 blocks by default, Tier 3 is advisory by default and can be promoted to blocking. That maps cleanly onto real trust levels — malformed or insecure skills are never acceptable, but a skill that merely fails to help might still be worth shipping while its author iterates.
Everyone is converging on this problem at once
SkillEvaluator is not alone, and the company knows it. Google published a codelab on evaluating agent skills using the Inspect framework, Gemini CLI as the agent under test, and model-graded QA scorers (Codelabs). Langfuse wrote up their own skill-evaluation loop — treating skills like prompts, defining datasets, tracing every tool call, and discovering via trace analysis that agents were substituting curl for the CLI, inventing nonexistent commands, and sometimes ignoring the skill entirely (Langfuse). OpenAI maintains documentation on testing skills systematically with evals (OpenAI). And the academic side is moving too: the SkillEval paper from Xi’an Jiaotong and collaborators decomposes skill quality into document-level dimensions — applicability, content quality, execution guidance, robustness — and scores them via interpretable directions in a model’s hidden representation space (arXiv).
The distinction worth drawing: most of these efforts evaluate either the document (SkillEval) or the skill–task pair (Google’s codelab, Langfuse’s loop). SkillEvaluator’s bet is the full pipeline — security, hygiene, and efficacy in one graduated gate, wired into a catalog. Anthropic’s research on agent evals generally (Anthropic) frames the stakes well: without automated evals, teams ship agents into production “flying blind,” unable to distinguish regressions from noise. Skills amplify that problem, because they’re a second layer of untested behavior stacked on top of the agent — and, as Anthropic’s own τ2-bench anecdote shows, static evals can be outgrown by the systems they measure. A skill that once helped can become dead weight as base models improve; only continuous paired evaluation, re-run over time, would catch that. The paper’s title — continuous evaluation — suggests NVIDIA understands this, though the README says little about how continuously the Verified Skills catalog re-evaluates its inventory. That’s an open question.
The rough edges
The project labels itself Experimental, community-supported, best-effort, no SLA — standard disclaimers, but worth taking seriously given the dependency surface: external scanners, a sandboxing framework, four LLM-provider integrations, and live agent CLIs, each with its own credential dance. The provider matrix is pragmatic (NVIDIA Build, OpenAI, Anthropic, Bedrock, and any OpenAI-compatible endpoint, with a free tier on NVIDIA Build defaulting to a Nemotron model), but Anthropic and Bedrock users need a second provider for embeddings, and full Tier 3 runs cost real money across evaluator, agent, and possibly managed sandbox.
There’s also a mildly awkward data point for NVIDIA specifically: in the Tessl framework paper’s instruction-following rankings, Nemotron models lagged the field badly — Nemotron Nano 30B scored 25.2 against 88.0 for the frontier Opus models (arXiv). SkillEvaluator’s defaults steer newcomers toward Nemotron via NVIDIA Build. A skill-evaluation harness whose default judge is a model that struggles to follow skill instructions is a tension the documentation doesn’t address. Users can override the model; whether they know they should is another matter.
Finally, the ecosystem context cuts both ways. The existence of 155 cataloged NVIDIA-specific skills on third-party directories (Skilled) shows real demand for repo-aware guidance — Megatron training recipes, NCCL timeout tuning, MoE parallelism — and that demand is exactly what Verified Skills wants to certify. But a verification pipeline is only as trustworthy as its independence. When the evaluator, the scanner, the catalog, the default model, and the skills themselves all come from the same vendor, the open-source license is doing important work. Apache 2.0, published methodology, and a citable paper make the machinery inspectable — which is the right design, because the alternative is asking the market to trust a stamp.
The deeper significance is this: the industry spent two years learning to evaluate models, then a year learning to evaluate agents. Skills are the third layer, and they may be the most consequential, because they’re the layer anyone can write and everyone installs. NVIDIA got there early with a complete, if experimental, answer. Whether SkillEvaluator becomes the reference implementation or just one of a dozen converging frameworks depends on the part no architecture diagram can settle — whether skill authors, who currently face zero friction publishing to a credulous market, can be persuaded to measure anything at all.
Sources
- NVIDIA Skills - Skilled
- Evaluate agent skills using open source frameworks - Codelabs
- Demystifying evals for AI agents
- NVIDIA/SkillSpector
- Testing Agent Skills Systematically with Evals
- Evaluating AI Agent Skills
- 10 Open-Source Projects Securing AI Agent Skills
- A Proposed Framework For Evaluating Skills [Research Eng Blog]
- Agent Skills: The Universal Standard Transforming How AI Agents Work
- SkillEval: Decomposing Agent Skill Quality into Interpretable Signals
- A Framework for Evaluating Agentic Skills at Scale
- Behind the scenes: How we build, test, and scale Google Agent Skills