analiza aiqbench
This study investigates how pre-trained code models internally encode formal type semantics by probing their hidden states across Java and Python, revealing emergent cross-lingual representations even in untyped contexts. The findings demonstrate that these internal structures are partially robust to lexical changes and syntactic variations, offering new insights into the interpretability of large language models for programming tasks.
TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models
arXiv:2607.08339v2 Announce Type: replace Abstract: State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.
Abstract:State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.
18 pages, 12 figures. Accepted at ESSLLI 2026 (StuS; double-blind)
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Programming Languages (cs.PL)
arXiv:2607.08339 [cs.CL]
(or arXiv:2607.08339v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.08339 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 240 cuvinte
Subiecte conexe
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Problems of Proliferation
LessWrong · 29 sept. 2026
When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis
arXiv · 12 iun. 2026