analiza aiqbench
This study rigorously benchmarks 28 pretrained molecular embedding models against classical fingerprints, finding that most neural architectures fail to outperform the standard ECFP count fingerprint in drug discovery tasks. The results challenge the assumed progress in molecular representation learning and suggest that a hybrid approach evaluating both classical and specific neural embeddings is currently more practical for QSAR/QSPR studies. This work underscores the critical need for robust, large-scale statistical testing standards before claiming advancements in chemical AI models.
Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning
arXiv:2508.06199v5 Announce Type: replace-cross Abstract: Pretrained molecular embedding models spanning graph neural networks, text transformers, and hybrid architectures have been widely used in drug discovery, yet no large-scale, rigorous comparison of their representations existed. We evaluated 28 models across 25 classification datasets under a controlled framework using hierarchical Bayesian Bradley-Terry statistical testing. All neural models except CheMeleon failed to outperform the classical ECFP count fingerprint; CheMeleon was the only model rated as better. However, further testing on external validation datasets, based on scaffold cross-validation and a Bayesian hierarchical correlated t-test, revealed no definite advantage of either embedding method. As a practical takeaway, we recommend evaluating both ECFP fingerprints and CheMeleon embeddings in novel QSAR/QSPR studies. Overall, in the context of all neural networks evaluated, these results also challenge the perceived progress in molecular representation learning and highlight the need for rigorous evaluation standards in the field.
Abstract:Pretrained molecular embedding models spanning graph neural networks, text transformers, and hybrid architectures have been widely used in drug discovery, yet no large-scale, rigorous comparison of their representations existed. We evaluated 28 models across 25 classification datasets under a controlled framework using hierarchical Bayesian Bradley-Terry statistical testing. All neural models except CheMeleon failed to outperform the classical ECFP count fingerprint; CheMeleon was the only model rated as better. However, further testing on external validation datasets, based on scaffold cross-validation and a Bayesian hierarchical correlated t-test, revealed no definite advantage of either embedding method. As a practical takeaway, we recommend evaluating both ECFP fingerprints and CheMeleon embeddings in novel QSAR/QSPR studies. Overall, in the context of all neural networks evaluated, these results also challenge the perceived progress in molecular representation learning and highlight the need for rigorous evaluation standards in the field.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2508.06199 [cs.LG]
(or arXiv:2508.06199v5 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2508.06199 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 236 cuvinte
Subiecte conexe
Threat-Preserving Representation Sensitivity in Agent-Security Benchmarks
arXiv · 6 oct. 2026
Uncovering the Computational Ingredients of Human-Like Representations in LLMs
arXiv · 2 sept. 2026
RepBench: Compiling Benchmarks into Capability Representations for Large Language Models
arXiv · 31 iul. 2026
RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling
arXiv · 25 aug. 2026