Five rules for evaluating agent skills before they bloat your context window

You’re probably hearing more about SKILL.md files, and you’ll see that a lot of people test them the same way: paste a prompt into chat, watch it pass once on a laptop, and merge to main. But when you paste three pages of CLI docs into a markdown file without an evaluation loop, you are just as likely to confuse a frontier model as you are to help it. For the past weeks, we have been diligently measuring and optimizing our own Google Cloud Developer Plugin, with the most essential skills and the Developer Knowledge MCP server to ground responses in the documentation. We ran dozens of scenarios with Gemini, Antigravity, and other agents. We used Claude Sonnet 5 on Gemini platform inside isolated Harbor pods on GKE Autopilot. And, we contributed Kubernetes and Gemini platform capabilities upstream to NVIDIA's open-source evaluator.  Here's the five-rule engineering playbook we use to verify a skill is worth its tokens. 1. Gate trigger collisions and security risks before spinning up containers When an evaluation run shows zero improvement between your plugin arm and your baseline arm (Δ = 0), the natural urge is to rewrite the body of your SKILL.md instructions. Hold off on that. Nine times out of ten, the agent never loaded your skill in the first place. Coding agents do not read full SKILL.md bodies at turn zero. The router only inspects the short name and description in your YAML frontmatter. Once a workspace grows past a dozen skills, two failure modes collide: under-triggering (your description describes internal implementation rather than developer intent) and sibling hijacking (a broad phrase like "helps with cloud deployments" in a neighboring skill intercepts the prompt first). We recently open-sourced skill-reach so you can catch frontmatter collisions before spending on container sandboxes. It validates manifests, computes vocabulary overlap against nearest rival skills, and runs empirical routing probes across Claude Code, Google Antigravity, and OpenAI Codex: code_block <ListValue: [StructValue([('code', 'uv tool install skill-reach\r\n\r\n# 1. Validate YAML frontmatter & token budgets (zero API calls)\r\nreach lint ./skills\r\n\r\n# 2. Detect vocabulary collisions & nearest rival skills (e.g. 0.69 overlap)\r\nreach overlap ./skills\r\n\r\n# 3. Run cross-harness routing probes and auto-tune weak descriptions\r\nreach eval ./skills/cloud-run --runs 3\r\nreach optimize ./skills/cloud-run --apply'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f749ade7c90>)])]> Pair skill-reach with NVIDIA's SkillSpector (Tier 1 and Tier 2 of SkillEvaluator). As the Nemotron Labs team details in their Evaluating Agent Skills deep dive, standard code linters miss agent-native risks buried in markdown prose: prompt injection, silent data exfiltration, tool poisoning, and trigger abuse.  2. Never trust an eval that runs on a developer laptop If you run skill benchmarks directly on your workstation, your numbers are lying to you. A developer laptop is packed with ambient state that contaminates the baseline arm: Cached Google Cloud credentials Exported project variables and warm bash history Global environment rules An unassisted baseline agent can pass a cloud deployment test by inheriting a default project ID or token sitting in your local shell profile, masking the exact failure your skill was written to solve. Host resource contention also warps agent trajectories. While hill climbing code-retrieval agents with AlphaEvolve, our internal testing showed that running the exact same agent benchmark while a workstation was under CPU load dropped the score. Slower file scans and subprocess timeouts cause the agent to abandon search paths prematurely. 3. Measure paired WITH vs. W/OUT ablation (Pass@3), not vibes Running prompts a single time and screenshotting the green checkmarks is not a real evaluation. Multi-turn tool-using agents branch on every shell exit code and stdout snippet. In our retrieval study, more than a third of benchmark tasks produced different scores across consecutive runs of the exact same code and prompt. A perfect score is equally meaningless without a paired baseline for comparison. Both Claude's plugin eval tool and NVIDIA's SkillEvaluator run paired ablation across multiple repetitions: Arm A runs the base model in a clean sandbox without the skill, while Arm B runs the identical prompt and sandbox with the skill mounted. Comparing the score improvements across our sandbox trials exposed three clear patterns that single-shot testing hides. Without the plugin, the base model still stumbled into a working deployment on most runs, but only after burning minutes of wall-clock time guessing deprecated flags and retrying bad requests. Conversely, when a test case shows identical success with zero efficiency gain, the base model already owns that capability natively. Your skill is pure context tax — delete it. code_block <Li

Extras afișat conform termenilor RSS ai publicației · 128 cuvinte · textul integral și figurile sunt pe site-ul publicației