analiza aiqbench
Confidence-based monitoring can flag some prompt-injection hijacks of LLM classifiers, but the study finds that a useful ranking score does not translate into a usable alarm. At a 5% clean-alarm rate, the monitor misses most hijacks, and it misses nearly all of them when the injected text claims certainty. Teams relying on model confidence as a safeguard should measure it at their real operating threshold and against attackers who imitate confidence.
Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities
arXiv:2611.02522v1 Announce Type: new Abstract: Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low confidence ranks hijacked answers above clean ones on four of the five systems (AUROC 0.86-0.90). Ranking them does not stop many of them. A threshold set so that at most 5% of clean answers to separate calibration articles alarm falls among the least confident clean answers, and every more confident hijack passes. In our main gate run on the three API-served systems, the monitor misses 51-70% of the hijacks caused by a plain injected note (per-system point estimates). When the note adds one handwritten sentence that calls the target certain and asks for full probability, the monitor misses 99-100% of the hijacks, although low confidence still ranks them at AUROC 0.70-0.79. Confidence monitors should be tested at their operating alarm rate, against attackers who claim certainty.
Abstract:Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low confidence ranks hijacked answers above clean ones on four of the five systems (AUROC 0.86-0.90). Ranking them does not stop many of them. A threshold set so that at most 5% of clean answers to separate calibration articles alarm falls among the least confident clean answers, and every more confident hijack passes. In our main gate run on the three API-served systems, the monitor misses 51-70% of the hijacks caused by a plain injected note (per-system point estimates). When the note adds one handwritten sentence that calls the target certain and asks for full probability, the monitor misses 99-100% of the hijacks, although low confidence still ranks them at AUROC 0.70-0.79. Confidence monitors should be tested at their operating alarm rate, against attackers who claim certainty.
Computation and Language (cs.CL); Cryptography and Security (cs.CR)
arXiv:2611.02522 [cs.CL]
(or arXiv:2611.02522v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.02522 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
ids/agnews_fresh_ids.txt
ids/agnews_ids.txt
ids/agnews_search_ids.txt
ids/dbpedia14_skipped_ids.txt
ids/dbpedia_ids.txt
ids/yahoo_ids.txt
macros/agnews_ranking.tex
macros/call_flow.tex
macros/certain_note_ablation.tex
macros/concurrent_measures.tex
macros/dbpedia14.tex
macros/fresh_articles.tex
macros/handwritten_texts.tex
macros/requested_probability.tex
macros/robustness.tex
macros/searched_text.tex
macros/second_model_and_phrasing.tex
macros/split_session.tex
macros/top_five_bounds.tex
macros/yahoo_answers.tex
tables/table01.csv
tables/table02.csv
tables/table03.csv
tables/table04.csv
tables/table05.csv
tables/table06.csv
tables/table07.csv
tables/table08.csv
tables/table09.csv
tables/table10.csv
tables/table11.csv
tables/table12.csv
tables/table13.csv
tables/table14.csv
tables/table15.csv
tables/table16.csv
tables/table17.csv
tables/table18.csv
tables/table19.csv
tables/table20.csv
tables/table21.csv
tables/table22.csv
tables/table23.csv
tables/table24.csv
tables/table25.csv
tables/table26.csv
tables/table27.csv
tables/table28.csv
tables/table29.csv
tables/table30.csv
tables/table31.csv
(48 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 374 cuvinte