aiqbench analysis
This research investigates the reliability of geometric network properties, such as Hessian norms, as proxies for model selection when standard validation data fails in computational biology tasks. The study reveals that while these internal signals can correlate with generalization gaps, they do not consistently identify the optimal model for deployment and may even favor collapsed predictors under certain conditions. These findings highlight critical failure modes in applying AI to biological domains where out-of-distribution performance is paramount.
A Generalisation Signal Need Not Be a Model-Selection Signal
arXiv:2609.39099v1 Announce Type: cross Abstract: Model selection in computational biology often relies on validation data drawn from the training regime, even when deployment lies outside it. When validation no longer preserves which model is best, a natural alternative is to rank candidates using properties of the trained network itself. We test this idea using a novel, forward-only proxy motivated by the norm of the Hessian, alongside common Hessian measures, across molecular property, protein fitness, and drug-response tasks. Contrary to our hypothesis, geometry does not become more useful as validation Spearman correlation deteriorates: augmenting validation helps some shifts but significantly harms others. More surprisingly, the proxy still correlates with generalisation gap on most tasks even when Hessian trace and top-eigenvalue relationships are weak or reversed, yet this signal does not reliably identify the deployment-best model. A curvature bound need not preserve cross-model rankings, and low geometric scores can even favour collapsed predictors. Thus, a generalisation signal need not be a model-selection signal.
Abstract:Model selection in computational biology often relies on validation data drawn from the training regime, even when deployment lies outside it. When validation no longer preserves which model is best, a natural alternative is to rank candidates using properties of the trained network itself. We test this idea using a novel, forward-only proxy motivated by the norm of the Hessian, alongside common Hessian measures, across molecular property, protein fitness, and drug-response tasks. Contrary to our hypothesis, geometry does not become more useful as validation Spearman correlation deteriorates: augmenting validation helps some shifts but significantly harms others. More surprisingly, the proxy still correlates with generalisation gap on most tasks even when Hessian trace and top-eigenvalue relationships are weak or reversed, yet this signal does not reliably identify the deployment-best model. A curvature bound need not preserve cross-model rankings, and low geometric scores can even favour collapsed predictors. Thus, a generalisation signal need not be a model-selection signal.
Accepted (poster) at the NeurIPS 2026 Workshop "I Can't Believe It's Not Better: Failure Modes of AI in Biology" (ICBINB-BIO). 24 pages, 4 figures, 22 tables. Code: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.39099 [cs.LG]
(or arXiv:2609.39099v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.39099 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 207 words
Related coverage
Alignment Midtraining Cracks Under Pressure
LessWrong · Sep 21, 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026
A Post-Mortem for My Goal Crystallisation Project
LessWrong · Jul 17, 2026
Value Generalisation 2: The Missing Hole in AIs’ abilities
AI Alignment Forum · Jul 29, 2026