analiza aiqbench
This research demonstrates that pretraining AI models on abstract, procedurally generated tasks can significantly boost molecular property prediction, offering a viable solution for scenarios where labeled chemical data is scarce. The study reveals that the most effective transfer occurs at an intermediate stage of procedural training, suggesting diminishing returns once the model fully converges on those abstract tasks. By leveraging inductive biases from non-molecular domains, this approach provides a complementary strategy to standard large-scale molecular pretraining for advancing drug discovery capabilities.
Procedural Pretraining for Molecular Property Prediction
arXiv:2609.17831v1 Announce Type: cross Abstract: Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled molecules. In this work, we ask whether useful inductive biases can instead be learned from abstract, procedurally generated data before a model sees any molecular data. We introduce a three-stage training pipeline consisting of procedural pretraining, molecular pretraining on SMILES, and downstream fine-tuning, and evaluate several procedural tasks spanning sequence structure, cellular automata, and graph reasoning. We find that procedural pretraining can improve molecular property prediction even after subsequent molecular pretraining: on Lipophilicity, \textsc{Reverse} reduces test error by 4.8\%. For context, the magnitude of this improvement is roughly 90\% of the performance difference between our 250K-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules. Our analysis shows that the benefit is strongest under downstream data scarcity, depends on the structure of the procedural data rather than only surface-level statistics, and does not increase monotonically with additional procedural training. Instead, transfer typically peaks at an intermediate procedural budget and deteriorates as the model approaches convergence on the procedural task. We further find that, for several tasks, much of the transferable information is localized in the attention layers, while feed-forward layers can contribute to over-specialization. These results show that procedural data can provide transferable structure for molecular learning and offer a complementary route to improving performance when labeled molecular data are limited.
Abstract:Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled molecules. In this work, we ask whether useful inductive biases can instead be learned from abstract, procedurally generated data before a model sees any molecular data. We introduce a three-stage training pipeline consisting of procedural pretraining, molecular pretraining on SMILES, and downstream fine-tuning, and evaluate several procedural tasks spanning sequence structure, cellular automata, and graph reasoning. We find that procedural pretraining can improve molecular property prediction even after subsequent molecular pretraining: on Lipophilicity, \textsc{Reverse} reduces test error by 4.8\%. For context, the magnitude of this improvement is roughly 90\% of the performance difference between our 250K-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules. Our analysis shows that the benefit is strongest under downstream data scarcity, depends on the structure of the procedural data rather than only surface-level statistics, and does not increase monotonically with additional procedural training. Instead, transfer typically peaks at an intermediate procedural budget and deteriorates as the model approaches convergence on the procedural task. We further find that, for several tasks, much of the transferable information is localized in the attention layers, while feed-forward layers can contribute to over-specialization. These results show that procedural data can provide transferable structure for molecular learning and offer a complementary route to improving performance when labeled molecular data are limited.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2609.17831 [cs.LG]
(or arXiv:2609.17831v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.17831 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 332 cuvinte
Subiecte conexe
Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems
arXiv · 7 oct. 2026
Molecular Property Prediction under Structural Shift with Tabular Foundation Models
arXiv · 1 oct. 2026
ED-DiT: Physics-Guided Diffusion Pretraining for Transferable Molecular Representations from Electron Density
arXiv · 5 aug. 2026
Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction
arXiv · 12 iun. 2026