aiqbench analysis
This research addresses the critical interpretability gap in black-box deep language models by proposing an efficient method that selects linguistically coherent word subsets without requiring access to internal model states. By formulating word selection as an amortized optimization problem trained via policy gradients, the approach achieves high inference-time efficiency while maintaining compatibility with API-only deployments. The method demonstrates superior discriminative power and alignment with human linguistic intuition compared to existing black-box compatible techniques, offering a practical path toward trustworthy AI in high-stakes domains like healthcare.
Explaining Black-Box Language Models: Learning to Optimize Linguistically-Structured Word Subsets
arXiv:2606.08497v1 Announce Type: cross Abstract: As deep language models (DLMs) are increasingly deployed in high-stakes domains such as healthcare, understanding their decision rationale becomes paramount for ensuring trust, safety, and accountability. However, achieving this vital level of interpretability is particularly challenging when these DLMs operate as black-box systems (e.g., via APIs), where access to internal model states (e.g., parameters, gradients) is restricted. Despite numerous efforts, existing explanation methods often fail to concurrently satisfy three key desiderata: (i) inference-time efficiency, (ii) black-box compatibility without inducing out-of-distribution behavior, and (iii) comprehensible explanations grounded in the input's linguistic structure. To address these challenges, we propose a method that explains predictions of DLMs by selecting a small, informative subset of input words. We formulate this as an amortized optimization problem, enabling efficient one-shot inference without the need for input-specific search. Our selection policy is trained via REINFORCE-style policy gradients, allowing discrete word selection in a fully gradient-free setting. To enhance interpretability and align with human linguistic intuition, we integrate graph-structured knowledge into this selection process, fostering linguistically coherent subsets that result in explanations both highly informative and cognitively meaningful to end-users. We evaluated our method on diverse DLM architectures and multiple real-world datasets. It consistently identifies word subsets with enhanced discriminative power and stronger alignment with linguistically salient cues, outperforming both conventional black-box compatible methods and gradient-based approaches that are given oracle access to the black-box model's gradients for a more challenging benchmark. Our code is available at here.
Abstract:As deep language models (DLMs) are increasingly deployed in high-stakes domains such as healthcare, understanding their decision rationale becomes paramount for ensuring trust, safety, and accountability. However, achieving this vital level of interpretability is particularly challenging when these DLMs operate as black-box systems (e.g., via APIs), where access to internal model states (e.g., parameters, gradients) is restricted. Despite numerous efforts, existing explanation methods often fail to concurrently satisfy three key desiderata: (i) inference-time efficiency, (ii) black-box compatibility without inducing out-of-distribution behavior, and (iii) comprehensible explanations grounded in the input's linguistic structure. To address these challenges, we propose a method that explains predictions of DLMs by selecting a small, informative subset of input words. We formulate this as an amortized optimization problem, enabling efficient one-shot inference without the need for input-specific search. Our selection policy is trained via REINFORCE-style policy gradients, allowing discrete word selection in a fully gradient-free setting. To enhance interpretability and align with human linguistic intuition, we integrate graph-structured knowledge into this selection process, fostering linguistically coherent subsets that result in explanations both highly informative and cognitively meaningful to end-users. We evaluated our method on diverse DLM architectures and multiple real-world datasets. It consistently identifies word subsets with enhanced discriminative power and stronger alignment with linguistically salient cues, outperforming both conventional black-box compatible methods and gradient-based approaches that are given oracle access to the black-box model's gradients for a more challenging benchmark. Our code is available at here.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.08497 [cs.AI]
(or arXiv:2606.08497v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.08497 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3770855.3817677 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 348 words
Related coverage
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026
Building and evaluating model diffing agents
AI Alignment Forum · Jun 12, 2026
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · May 21, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026