agent-skill-evaluator

🔍 agent-skill-evaluator

Evaluate agent skills before you install them.
Think npm audit + eslint for SKILL.md files.

---

Why This Exists

There are 1,000+ agent skills on GitHub. Most have no quality signal beyond stars and recency. Before installing a skill into your agent:

• Is it structurally sound? (valid SKILL.md, proper metadata)
• Is it safe? (no prompt injection, credential harvesting, data exfiltration)
• Is it high quality? (decision trees, guardrails, edge cases)
• Is it domain-correct? (does it follow best practices for its domain — statistics, marketing, experiment design, etc.?)
• Is it maintained? (recent updates, tests, documentation)

skill-evaluator answers all five questions with a single command.

---

Quick Start

# Install
cd skill-evaluator
pip install -e .

# Evaluate a local skill
skill-eval ../skills/experiment-designer/

# Evaluate with Markdown output (for CI/READMEs)
skill-eval ../skills/stats-reviewer/ --format md

# Evaluate with JSON output (for pipelines)
skill-eval ../skills/causal-inference-advisor/ --format json

# Force a specific domain for correctness checking
skill-eval ./some-skill/ --domain statistics
skill-eval ./some-skill/ --domain digital-marketing

# CI mode: fail if score is below threshold
skill-eval ./some-skill/ --fail-below 70

---

What You Get

Terminal Output

╭──────────────────────────────────────────────────────────╮
│ Skill Evaluation Report: experiment-designer — A (91/100)│
╰──────────────────────────────────────────────────────────╯

  📋 Excellent — high quality, well-maintained.

  ┌────────────────────┬─────────┬───────┬────────┐
  │ Dimension          │ Score   │ Grade │ Weight │
  ├────────────────────┼─────────┼───────┼────────┤
  │ Structure          │ 95/100  │ A+    │ 15%    │
  │ Security           │ 100/100 │ A+    │ 20%    │
  │ Quality            │ 90/100  │ A     │ 15%    │
  │ Domain Correctness │ 85/100  │ A-    │ 25%    │
  │ Maintenance        │ 80/100  │ B+    │ 15%    │
  └────────────────────┴─────────┴───────┴────────┘

Scoring Dimensions

Each dimension produces a 0–100 score. These are combined into a weighted composite:

Note: Weights are normalized at runtime so they don't need to sum to exactly 100%. If you override weights via --weights, any missing dimensions default to 10%.

Grading Scale

---

Domain Correctness Rules

The novel differentiator. Unlike security/structure checks that any tool can do, domain correctness verifies the guidance itself is correct for its stated domain.

Built-in Domains

The digital marketing domain is organized into 7 sub-domain files:

The finance domain is organized into 7 sub-domain files:

Domain Auto-Detection

When you run skill-eval without --domain, the analyzer auto-detects the most likely domain by counting keyword signals in the skill content. For example, a skill mentioning "attribution," "ROAS," and "landing page" detects as digital-marketing, while one mentioning "p-value," "effect size," and "t-test" detects as statistics.

Adding Custom Domains

Create a YAML file in skill_evaluator/domains/:

domain: my-domain
version: "1.0.0"
rules:
  - name: my-rule
    description: "What this rule checks"
    applicability_patterns:
      - "pattern that makes this rule relevant"
    required_patterns:
      - "pattern that SHOULD be present"
    antipatterns:
      - "pattern that SHOULD NOT be present"
    failure_severity: suspicious  # or "incorrect"
    failure_message: "What went wrong"
    success_message: "What went right"

For domains with many rules, you can organize them into a directory instead of a single file. Create skill_evaluator/domains/my-domain/ and place multiple .yaml files inside — all rules are automatically merged at load time.

We currently ship with 7 domains: Statistics, Causal Inference, Experiment Design, Data Science, Digital Marketing, and Finance — totaling 91 rules. This is meant to be extensible. If you have domain expertise in another area (e.g., healthcare, product management, cybersecurity, NLP evaluation) and want to contribute a rule set, open a PR and we'll review it. The more domains covered, the more useful this tool becomes for everyone.

---

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run against your own skills
skill-eval ../skills/experiment-designer/
skill-eval ../skills/stats-reviewer/

---

How Scoring Works

Each dimension (Structure, Security, Quality, Domain Correctness, Maintenance) produces a 0–100 score. These are combined into a weighted composite, normalized by the total weight of applicable dimensions.

A few things worth knowing:

Security is a gate, not just a weight. If the security analyzer finds critical risks (prompt injection, credential harvesting, destructive shell commands), the overall score is hard-capped regardless of how well the skill scores on other dimensions. A skill with a prompt injection vulnerability gets an F even if the content is otherwise excellent.

Domain correctness uses a two-tier severity model. Rules flagged as incorrect (e.g., recommending last-click attribution as the only model, or computing CLV without a proper probabilistic framework) carry heavier penalties than those flagged as suspicious (best-practice recommendations that may vary by context). This lets the evaluator distinguish hard errors from soft guidance.

Scores are normalized by applicable checks. Quality and maintenance scores are computed as the ratio of passed checks to applicable checks. A concise, well-written skill won't score lower than a verbose one just because it has fewer regex matches. Each check that fires gets an equal vote.

Structural penalties are weighted by severity. Missing your SKILL.md entirely costs more than missing a recommended field. The penalty for each finding reflects how much it actually impacts usability.

What the scores don't tell you. The current system is a static linter — it checks patterns in text. It can't tell you whether the skill actually makes an agent perform better, or whether the domain guidance is semantically correct beyond keyword matching. Those are harder problems (see below).

---

Future Research

These are directions we'd like to explore. Contributions welcome.

Empirical weight calibration. The dimension weights (Security 20%, Domain 25%, etc.) are based on informed judgment, not data. The right approach is to score a labeled corpus of known-good vs. known-bad skills and use the results to find weights that best predict the label. If you have a labeled skill corpus or want to help build one, open an issue.

Semantic domain checks. The domain correctness analyzer currently uses regex patterns — it checks whether certain keywords appear, not whether the guidance is actually correct. Replacing this with embedding-based or lightweight LLM checks (e.g., "does this skill enforce power analysis, or just mention it?") would substantially improve accuracy.

Behavioral evaluation. The ultimate test of a skill is: does an agent using it produce better outputs? A test harness with known-correct answers (e.g., "given this dataset, should the agent refuse to run the test?") would let us score skills by downstream impact rather than surface patterns. This is the direction taken by SkillsBench-style evaluations and would make this tool a performance predictor, not just a linter.

More domains. We're actively expanding domain coverage. Next candidates include healthcare, product management, and cybersecurity. Contributions welcome — each new domain just requires a YAML rule set and test fixtures.

---

CI / CD Integration

GitHub Action

Use agent-skill-evaluator as a reusable GitHub Action to evaluate skills in your CI pipeline:

# .github/workflows/skill-eval.yml
name: Evaluate Skills
on: [push, pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Evaluate skill
        uses: WilliamWJHuang/agent-skill-evaluator@master
        id: eval
        with:
          path: './my-skill/'       # Path to skill directory
          fail-below: '60'          # Fail CI if score < 60
          # domain: 'statistics'    # Optional: force a domain

      - name: Print results
        run: |
          echo "Score: ${{ steps.eval.outputs.score }}"
          echo "Grade: ${{ steps.eval.outputs.grade }}"

Inputs

Outputs

The action also writes a Job Summary with the full report, visible directly in the GitHub Actions UI.

CLI in CI (without the Action)

pip install git+https://github.com/WilliamWJHuang/agent-skill-evaluator.git
skill-eval ./my-skill/ --fail-below 60

The --fail-below flag exits with code 1 if the score is below the threshold.

---

License

MIT — Use freely, attribute kindly.