analiza aiqbench
This research provides a geometric explanation for task-aware pruning, demonstrating that it specifically corrects representational distortions caused by out-of-distribution inputs rather than enhancing in-distribution performance. By removing layers that amplify these geometric deviations, the method realigns model representations with the task-adapted geometry, offering a robust mechanism for improving generalization across distribution shifts. The findings hold consistent behavior across different model scales and are supported by causal evidence from controlled distribution shifts.
TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability
arXiv:2605.14738v3 Announce Type: replace Abstract: Recent work has promoted task-aware layer pruning as a way to improve model performance on particular tasks, as shown by TALE. In this paper, we investigate when such improvements occur and why. We show first that, across controlled polynomial regression tasks and large language models, such pruning yields no benefit on in-distribution (ID) data but consistently improves out-of-distribution (OOD) accuracy. We further show empirically that OOD inputs induce layerwise norm and pairwise-distance profiles that deviate from the corresponding ID profiles. This leads to a geometric explanation of task-aware pruning: each task induces a task-adapted geometry, characterized empirically by the representation profiles observed on ID inputs. OOD inputs can introduce a distorted version of the task-adapted geometry. Task-aware pruning identifies layers that create or amplify this distortion; by removing them, it shifts OOD representational norms and pairwise distances toward those observed on the adapted distribution. This realigns OOD inputs with the model's task-adapted geometry and improves performance. We provide causal evidence through controlled distribution shifts and residual-scaling interventions, and demonstrate consistent behavior across model scales.
Abstract:Recent work has promoted task-aware layer pruning as a way to improve model performance on particular tasks, as shown by TALE. In this paper, we investigate when such improvements occur and why. We show first that, across controlled polynomial regression tasks and large language models, such pruning yields no benefit on in-distribution (ID) data but consistently improves out-of-distribution (OOD) accuracy. We further show empirically that OOD inputs induce layerwise norm and pairwise-distance profiles that deviate from the corresponding ID profiles. This leads to a geometric explanation of task-aware pruning: each task induces a task-adapted geometry, characterized empirically by the representation profiles observed on ID inputs. OOD inputs can introduce a distorted version of the task-adapted geometry. Task-aware pruning identifies layers that create or amplify this distortion; by removing them, it shifts OOD representational norms and pairwise distances toward those observed on the adapted distribution. This realigns OOD inputs with the model's task-adapted geometry and improves performance. We provide causal evidence through controlled distribution shifts and residual-scaling interventions, and demonstrate consistent behavior across model scales.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.14738 [cs.LG]
(or arXiv:2605.14738v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 188 cuvinte
Subiecte conexe
Capabilities research expands the safety-usefulness Pareto frontier too
LessWrong · 2 oct. 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · 16 oct. 2026
Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
arXiv · 12 oct. 2026