Building an agent skill optimizer with AlphaEvolve, Harbor and Gemini 4 Argon

Every day, more and more people are using coding agents like Google Antigravity and OpenCode to complete software engineering tasks. But those tools need instructions to operate and their output is only as good as the inputs they get. By utilizing agent skills that direct the agents through building, testing, and deploying code, developers have a powerful way to automate their workflows.  But most skills are written manually, so it’s hard to know which instructions are effective without assessing their output. To measure the effectiveness of our skills, we ran OpenCode with Gemini 3.8 Flash on tasks from Terminal-Bench 2.0, a public benchmark. The results were inconclusive: with our baseline skill, the agent passed about two-thirds of its attempts, but the outcome was about the same even without the skill. We needed better results. To achieve them, we turned to AlphaEvolve, which optimizes skills the way it optimizes code: Gemini proposes a new version, an evaluator scores it, and the search retains the higher-scoring version. And with Harbor, that evaluator can be a real coding agent working on real tasks. We saw improvements of 5% with just a few iterations with alphaevolve This is how you can build the same loop: Treat the skill as the program: AlphaEvolve can rewrite your skill's text. Let the agent judge: Harbor runs the agent with the skill on real tasks. Make the score trustworthy: Compare each version to the baseline skill and run them both at the same time. Check the winner on unseen tasks: Save some tasks the search never sees for your final check. Treat the skill as the program AlphaEvolve is designed to advance the code between two markers in a seed program. Skills are SKILL.md files with names, descriptions, and markdown instructions, so we put their text between the markers and let Gemini rewrite them: code_block <ListValue: [StructValue([('code', 'def build_skill() -> tuple[str, str]:\r\n """Return the evolvable (description, body) pair for the skill."""\r\n # EVOLVE-BLOCK-START\r\n description = (\r\n "Solve engineering tasks in a Linux terminal: building and compiling "\r\n "software, fixing broken repositories and configuration, ..."\r\n )\r\n body = r"""# Terminal task solver\r\nYou are working in a Linux container. The task has a specific, checkable end\r\nstate, and an automated test suite will decide whether you achieved it. ...\r\n"""\r\n # EVOLVE-BLOCK-END\r\n return description, body'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7ff3ebc68c10>)])]> AlphaEvolve operates in Google Cloud and doesn’t run your code. Its feedback is the scores and notes your evaluator sends back about what "better" means. Image 1. AlphaEvolve proposes a skill, Harbor runs the agent with it next to the baseline skill, and the score and notes go back to AlphaEvolve. Let the agent judge Harbor runs agents on benchmark tasks, with each trial in its own container. We ran it on Google Kubernetes Engine (GKE) with one pod per trial, and a single search ran thousands of agent trials. For each candidate, the evaluator returns a score for passes and failures, as well as  short notes on tasks that failed or timed-out. AlphaEvolve then sends those notes to Gemini for the next version. Make the score trustworthy Agent scores can be noisy. In one of our early runs on the same benchmark, our model’s latency explained much of the variance in candidate scores. With each candidate running next to the baseline skill on the same tasks at the same time,  their scores are the difference. Check the winner on unseen tasks The best skills read differently than the baseline, which is a careful checklist: read first, reproduce the failure, and work in small verified steps. AlphaEvolve's version is a protocol built around the time limit: code_block <ListValue: [StructValue([('code', '## 2. Ship a Working Baseline to the Target Path Early\r\n(...)\r\n- Having a valid artifact on disk protects you from timeouts: if you refine or\r\n golf code (`gpt2.c`, `eigen.py`, `my_warrior.red`), keep the last passing\r\n version written to the target path at all times.\r\n\r\n\r\n(...)\r\n\r\n\r\n## 5. Hard Stop When Green\r\nThe moment `eval.py`, the test suite, or your acceptance check passes and the\r\nrequired files exist with exact formatting (...): **STOP IMMEDIATELY**. (...)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7ff3ebc183d0>)])]> What we learned AlphaEvolve focused on the most common failure: running out of time. About a third of the search's trials hit their time limit, as reported in the evaluator's notes, and AlphaEvolve's best skill is built around that limit. We also found that running the search is harder than building the loop. After about 10 hours, the search scored no new candidate; our latency check held back every evaluation while model responses were slow, and full agent logs filled the

Extras afișat conform termenilor RSS ai publicației · 88 cuvinte · textul integral și figurile sunt pe site-ul publicației