NVIDIA /

SkillEvaluator

AI Agent可试用PythonApache-2.0

★ GitHub 仓库 ↗项目主页 ↗

总 Stars

★ 1,419

今日 +

+133

Forks

178

贡献者

33

未解 Issues

25

能做什么

适用于AI代理技能评估、性能监控。

核心特性

1多级评估框架，全面评估AI代理技能
2支持质量门和语义重叠检测
3生成合成评估数据集，支持实时评估

原文 README

在 GitHub 查看原文 ↗

SkillEvaluator

SkillEvaluator wordmark

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator checks AI agent skills and measures their effect on task results: start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.

Get started

Three ways in, one CLI, same results.

1. Run the CLI directly 2. Add it to your repo via CI 3. Wire it to your agent

1. Run the CLI directly	2. Add it to your repo via CI	3. Wire it to your agent
Install (or `uv tool install`): `pip install skillevaluator` Run all Tier 1 checks, no API key: `skillevaluator validate \ ./my-skill --tiers 1` Or run from a clone: Quickstart.	Check changed skills on every pull request. On GitHub, run: `skillevaluator ci init` It opens a setup PR. On GitLab, copy the CI template.	Add the SkillEvaluator skill to Claude Code, Codex or any skill-aware agent: `npx skills add \ NVIDIA/SkillEvaluator` Then ask: "Is my skill any good?" No Node.js? Copy the folder.

Install (or uv tool install):

pip install skillevaluator

Run all Tier 1 checks, no API key:

skillevaluator validate \
  ./my-skill --tiers 1

Or run from a clone: Quickstart.

Check changed skills on every pull request. On GitHub, run:

skillevaluator ci init

It opens a setup PR. On GitLab, copy the CI template.

Add the SkillEvaluator skill to Claude Code, Codex or any skill-aware agent:

npx skills add \
  NVIDIA/SkillEvaluator

Then ask: "Is my skill any good?" No Node.js? Copy the folder.

Guides: installation, CI setup (GitHub, GitLab, report examples) and agent skill.

Quickstart

Both paths use bundled text-analyzer.

Start here	Requires	First result
Run without an LLM	Git and uv	A deterministic quality score and findings
Use NVIDIA Build	Git, uv, and a Build API key	Model-backed rubric scores and feedback

Install uv and Git, then get the example:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator

uv provisions Python 3.13 and dependencies. For standalone installation, see Installation.

Run without an LLM

uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

No API key, scanner, GPU or Docker is needed. The example scores 86.2/100 (B), exits 0 and saves reports/first-quality/skillevaluator-quality.json. The passing threshold defaults to 70.

This checks static quality; publication evidence stays INCOMPLETE. Security needs scanners; agent impact needs live evaluation. Replace the example path with your skill directory. See the keyless walkthrough.

Use NVIDIA Build

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.

Sign in at build.nvidia.com and complete enrollment.
Open a model page, select Get API Key, and generate your key.
Set the provider and key in the same terminal, then run the rubric check:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build

The command sends skill documentation to the model and saves feedback in reports/first-build/skillevaluator-rubric.json. No agent runtime or Docker is needed. Completed evaluations can still fail rubric thresholds.

The same key supports Tier 2 embeddings and supported Tier 3 agents. See the Build walkthrough or provider configuration.

Three-tier overview

SkillEvaluator three-tier pipeline: Skill → Tier 1 Validation → Tier 2 Deduplication → Tier 3 Live Evaluation → Reports

Each tier is independent. Add its extra to your command:

Tier	Question	Example after setup	Additional setup
Tier 1: Validation	Is the skill safe, well-formed and harness-compatible?	`uv run skillevaluator tier1 ./my-skill --no-llm`	External scanners for full static coverage; a provider and model clients enable LLM checks
Tier 2: Deduplication	Does guidance repeat or overlap?	`uv run --extra tier2 skillevaluator tier2 ./my-skill`	Embeddings and chat; add `--catalog ./skill-catalog.json` for inter-skill comparison
Tier 3: Live Evaluation	Does the skill help the agent?	`uv run --extra tier3 skillevaluator tier3 ./my-skill`	Provider access, a supported agent, and a running sandbox (Docker by default)

Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they are missing; Semgrep is optional. See scanners. --no-llm does not remove scanner requirements; a scanner that cannot run is INCOMPLETE and exits nonzero. SkillSpector provides specialized skill security scanning; Harbor, the open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.

Run the combined pipeline

After setting up the provider, scanners, and agent sandbox, run all three tiers:

uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill

doctor checks readiness, not inference success. validate runs all tiers with autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for static checks. --no-autopilot requires an existing source. Other artifact types retain their applicable stages.

Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage can incur charges; see runtime and cost.

Standalone generation needs substantive evals/EVAL.md guidance, a --prompt file with task requests and outcomes, or matching retained traces. --force below replaces the autopilot starter; --refine refines it. Explicit --grounding skill permits skill-only starters. Review cases; templates omit unauthored negatives.

uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force

Run from a source checkout

Use uv run --python 3.13 with the relevant extra. See Installation for extras and standalone upgrades.

Research and citation

Resources: paper, PDF, DOI, BibTeX, and the NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.

Documentation

See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.

Installation and third-party software

Follow the installation guide to choose the full installation or a smaller per-tier setup.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Contributing

Read CONTRIBUTING.md and run:

make lint && make test && make build

See project governance and the Code of Conduct.

Support

Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.