aiqbench analysis
This arXiv paper is a critical replication of a recent claim that random Gaussian weight perturbations can surface task-specific "experts" in pretrained models. The authors argue that apparent gains from best-of-N selection largely reflect overfitting to the selection set, and that majority voting over random perturbations is often no better than sampling the base model. For ML readers, it is a useful check on how easily selection noise can masquerade as capability gains, and a reminder to use held-out evaluation and run-to-run variance baselines.
Does Random Search Discover Task Experts? Revisiting Neural Thickets
arXiv:2611.01262v1 Announce Type: new Abstract: Pretrained models are typically viewed as generalists, requiring bespoke post-training to achieve task-specific expertise. Gan & Isola (2026) recently proposed a striking alternative: that pretrained models lie in a dense neural thicket of task specialists, which can be surfaced with simple Gaussian weight perturbations. We examine this neural thicket of task-specific experts and identify several key findings that question this perspective. First, we find that the apparent lift of the best-of-N selected perturbation decays quickly with selection-set size m, strongly suggesting overfitting to the selection questions. In fact, for 10 of the 12 model-task pairs we study, a simple null model of pure selection noise predicts more than 75 percent of the top expert's reward lift. Second, the top experts' advantage often vanishes on held-out questions or remains within the run-to-run variability of the unperturbed pretrained model. Third, we find that majority voting over the top-K selected perturbations, termed RandOpt(N, K) by Gan & Isola (2026), is often matched or outperformed by majority voting over K samples from the base model. Overall, our results suggest that random weight perturbations may be ineffective at uncovering task experts around pretrained models, and that task adaptation requires more sophistication than random search. Code and artifacts are available at github.com/asil012/RevisitingNeuralThickets.
Title:Does Random Search Discover Task Experts? Revisiting Neural Thickets
Abstract:Pretrained models are typically viewed as generalists, requiring bespoke post-training to achieve task-specific expertise. Gan & Isola (2026) recently proposed a striking alternative: that pretrained models lie in a dense neural thicket of task specialists, which can be surfaced with simple Gaussian weight perturbations. We examine this neural thicket of task-specific experts and identify several key findings that question this perspective. First, we find that the apparent lift of the best-of-N selected perturbation decays quickly with selection-set size m, strongly suggesting overfitting to the selection questions. In fact, for 10 of the 12 model-task pairs we study, a simple null model of pure selection noise predicts more than 75 percent of the top expert's reward lift. Second, the top experts' advantage often vanishes on held-out questions or remains within the run-to-run variability of the unperturbed pretrained model. Third, we find that majority voting over the top-K selected perturbations, termed RandOpt(N, K) by Gan & Isola (2026), is often matched or outperformed by majority voting over K samples from the base model. Overall, our results suggest that random weight perturbations may be ineffective at uncovering task experts around pretrained models, and that task adaptation requires more sophistication than random search. Code and artifacts are available at this http URL.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2611.01262 [cs.LG]
(or arXiv:2611.01262v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.01262 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 315 words
Related coverage
Forecast: Fun Ahead — 18 Games Join in June to Stream on GeForce NOW
NVIDIA · Jun 4, 2026
GeForce NOW Sets Sail With ‘Path of Exile: Curse of the Allflame’ Joining the Cloud
NVIDIA · Jul 23, 2026
Revisiting Lexicon Evaluation in Unsupervised Word Discovery
arXiv · Jun 5, 2026
Revisiting Positive Samples in Graph Contrastive Learning: From the Perspective of Message Passing
arXiv · Jun 10, 2026