aiqbench analysis
The paper argues that pruning an LLM should be judged by two things: how many distinct problems it can still solve given enough attempts (pass@k coverage), and how many samples and tokens it takes to get correct answers. Pruned models can lose pass@1 yet keep or exceed the original's coverage, but under heavy pruning they tend to repeat themselves and need many more attempts, which raises inference cost. Readers evaluating compressed models should look beyond single-sample accuracy to sampling efficiency.
Rethinking LLM Pruning via Test-Time Scaling
arXiv:2611.01133v1 Announce Type: new Abstract: Structured pruning reduces the size of large language models, often at the cost of lower accuracy. Whether this decline reflects diminished reasoning capacity or greater difficulty in sampling correct solutions is less clear. We examine this question across expert, FFN and layer pruning on four mathematical and coding benchmarks, comparing original and pruned models through pass@k at matched sampling budgets. Even after substantial pruning, some models match or exceed the original model's reasoning coverage despite lower pass@1, and solve problems left unsolved in the original samples. Retaining broad coverage, however, does not mean that correct solutions remain easy to obtain. Under severe pruning, diverse generated chain-of-thought (CoT) traces can coexist with increased repetition and less reliable generation. As correct answers become harder to sample, the additional attempts can sharply increase total token use even when individual responses become shorter. This separation between coverage and sampling efficiency also clarifies expert-selection gains, as methods with markedly different pass@1 can solve the same problems at large budgets yet require different numbers of attempts to reach a common coverage target. Pruning should therefore be assessed by both the range of problems a model can still solve and the sampling cost of obtaining correct solutions.
Read the full article at arXiv