aiqbench analysis
This preprint proposes a symbolic backend redesign using CCG directed types to better handle directional distinctions in linguistic parsing, initially claiming significant performance gains over the previous state-of-the-art. However, the authors have withdrawn these specific claims due to an identified mismatch between their evaluation metric and ground truth data, noting that corrected results are pending. Consequently, this work currently serves as a methodological note on benchmark integrity rather than a validated advancement in machine learning capabilities.
On the Role of Directionality in Structural Generalization
arXiv:2607.02307v2 Announce Type: replace-cross Abstract: Several SLOG test categories explicitly involve directional distinctions (modifier position shifts, argument extraction positions), yet AM-Parser, the previous SOTA, uses an AM algebra whose operations do not encode direction. We redesign the symbolic backend around CCG directed types (deterministic CKY + single linear decoder, 30K learnable parameters). Under the same BERT-base encoder, the system achieves 75.9$\pm$6.4% LF exact match, surpassing AM-Parser (70.8$\pm$4.3%). Per SLOG's own category groupings, gains are highly directional: the CCG system outperforms AM-Parser on all 5 position-shift categories (+29.9pp), while AM-Parser outperforms on all 6 recursive-depth categories. Replacing the encoder with DeBERTa-v3-large yields 90.7$\pm$4.9%, with the largest encoder gains in recursive-depth categories, complementary to directionality's gains. Directional representations shift the bottleneck from the symbolic layer (AM-Parser's 0% category ceiling) to the neural layer, which improves with encoder upgrades.
Abstract:Several SLOG test categories explicitly involve directional distinctions (modifier position shifts, argument extraction positions), yet AM-Parser, the previous SOTA, uses an AM algebra whose operations do not encode direction. We redesign the symbolic backend around CCG directed types (deterministic CKY + single linear decoder, 30K learnable parameters). Under the same BERT-base encoder, the system achieves 75.9$\pm$6.4% LF exact match, surpassing AM-Parser (70.8$\pm$4.3%). Per SLOG's own category groupings, gains are highly directional: the CCG system outperforms AM-Parser on all 5 position-shift categories (+29.9pp), while AM-Parser outperforms on all 6 recursive-depth categories. Replacing the encoder with DeBERTa-v3-large yields 90.7$\pm$4.9%, with the largest encoder gains in recursive-depth categories, complementary to directionality's gains. Directional representations shift the bottleneck from the symbolic layer (AM-Parser's 0% category ceiling) to the neural layer, which improves with encoder upgrades.
We have identified an evaluation-metric mismatch in this preprint (reported LF exact match was computed against a final-state proxy, not against predicted LF edges). We withdraw the claims in this version. A corrected approach with true LF evaluation is in preparation
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2607.02307 [cs.CL]
(or arXiv:2607.02307v2 [cs.CL] for this version)
Full text under the publisher's open licence · 187 words
Related coverage
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · Jul 18, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Game-Theoretic Disempowerment: Loss of Control to Agencyless AI
LessWrong · Sep 27, 2026
The SaaSpocalypse that wasn’t, with Atlassian CEO Mike Cannon-Brookes
The Verge · Sep 28, 2026