analiza aiqbench
This paper trains a compact classifier to flag malicious agent skills from their SKILL.md files, reporting 99% recall on unseen attack families while flagging 20-25% of benign skills. The notable finding is that the model handles new attacks reasonably well but produces many more false positives on unfamiliar benign skill sources, so recall on known attack types does not guarantee a manageable review load in new deployments. Readers building or auditing agent ecosystems should treat the benchmark results as preliminary until they are tested on real, diverse skill collections.
SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models
arXiv:2610.17727v1 Announce Type: cross Abstract: Agent skills turn natural-language instructions into actions, creating a security challenge: legitimate automation and malicious behaviour can look deceptively similar. A practical screen must catch nearly every attack while keeping the review load manageable. We present SkillScreen, a compact decision model trained to score skill maliciousness directly from SKILL.md in a single forward pass. On MaliciousSkillBench, SkillScreen flags 20-25% of benign skills at 99% recall on unseen attack families, substantially fewer than the word-level classifier and prompted LLM judges tested. Under the same training protocol, it also generalizes better than a fully fine-tuned encoder to unseen sources. Yet unfamiliar benign sources sharply increase false positives, showing that recognizing new attacks does not ensure reliable screening of new skill collections. Our results demonstrate the value of task-trained decision models for high-recall skill screening. Project page: https://jerroldh.github.io/SkillScreen-webpage/
Abstract:Agent skills turn natural-language instructions into actions, creating a security challenge: legitimate automation and malicious behaviour can look deceptively similar. A practical screen must catch nearly every attack while keeping the review load manageable. We present SkillScreen, a compact decision model trained to score skill maliciousness directly from SKILL$.$md in a single forward pass. On MaliciousSkillBench, SkillScreen flags 20-25% of benign skills at 99% recall on unseen attack families, substantially fewer than the word-level classifier and prompted LLM judges tested. Under the same training protocol, it also generalizes better than a fully fine-tuned encoder to unseen sources. Yet unfamiliar benign sources sharply increase false positives, showing that recognizing new attacks does not ensure reliable screening of new skill collections. Our results demonstrate the value of task-trained decision models for high-recall skill screening. Project page: this https URL
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.17727 [cs.CR]
(or arXiv:2610.17727v2 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2610.17727 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 237 cuvinte
Subiecte conexe
I Did This Without Your Knowledge
LessWrong · 17 oct. 2026
Friendly Fire: Hijacking Defensive Cyber AI Agents for Remote Code Execution
AI Now Institute · 8 iul. 2026
The Expensive-Loop Problem
LessWrong · 12 oct. 2026
Gemini 4 Argon: Built to solve your hardest problems
Google DeepMind · 13 oct. 2026