aiqbench analysis
This paper is primarily about LLMs, so it passes the topic gate. It is an empirical benchmark (MolRepBench) testing how 16 LLMs handle molecules under nine text representations, with the finding that results depend heavily on representation and that SMILES, despite its prevalence in pretraining, is rarely the best choice. The mechanistic analysis (tokenization audits, linear probes, attention) adds useful evidence that representations are encoded differently inside models. For AI readers working on scientific or chemistry applications, the practical takeaway is that evaluation and prompt format choices can change conclusions, so representation-aware benchmarking matters.
Rethinking Molecular Text Representations for LLMs: An Empirical Study
arXiv:2606.03057v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.
Abstract:Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.
25 pages, 10 figures, 21 tables. Accepted at the Main Conference of EMNLP 2026
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.03057 [cs.LG]
(or arXiv:2606.03057v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.03057 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 304 words
Related coverage
Rethinking Implicit Spatial Representation in Visuomotor Policy Learning
arXiv · Jun 16, 2026
Rethinking Molecular Text Representations for LLMs: An Empirical Study
arXiv · Jun 3, 2026
From Scalars to Time Series: Rethinking Implicit Neural Representations for Time-Varying Volumetric Data
arXiv · Jul 24, 2026
ScaleMPA: Rethinking Scalable RRT* Acceleration With a Grid-Native Representation
arXiv · Sep 22, 2026