aiqbench analysis
WildIFEval introduces a new dataset of 7,000 real-world user instructions designed to rigorously benchmark large language models on handling multiple simultaneous constraints. By categorizing these conditions into eight distinct classes, the study reveals significant gaps in current model performance and highlights specific behavioral patterns when faced with complex scenarios. This resource provides an essential baseline for researchers aiming to improve instruction-following capabilities under realistic, multi-faceted conditions rather than simplified synthetic tasks.
WildIFEval: Instruction Following in the Wild
arXiv:2503.06573v3 Announce Type: replace-cross Abstract: Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge. In this work, we introduce WildIFEval - a large-scale dataset of 7K real user instructions with diverse, multi-constraint conditions. Unlike prior datasets, our collection spans a broad lexical and topical spectrum of constraints, extracted from natural user instructions. We categorize these constraints into eight high-level classes to capture their distribution and dynamics in real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments to benchmark the instruction-following capabilities of leading LLMs. WildIFEval clearly differentiates between small and large models, and demonstrates that all models have a large room for improvement on such tasks. We analyze the effects of the number and type of constraints on performance, revealing interesting patterns of model constraint-following behavior. We release our dataset to promote further research on instruction-following under complex, realistic conditions.
Abstract:Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge. In this work, we introduce WildIFEval - a large-scale dataset of 7K real user instructions with diverse, multi-constraint conditions. Unlike prior datasets, our collection spans a broad lexical and topical spectrum of constraints, extracted from natural user instructions. We categorize these constraints into eight high-level classes to capture their distribution and dynamics in real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments to benchmark the instruction-following capabilities of leading LLMs. WildIFEval clearly differentiates between small and large models, and demonstrates that all models have a large room for improvement on such tasks. We analyze the effects of the number and type of constraints on performance, revealing interesting patterns of model constraint-following behavior. We release our dataset to promote further research on instruction-following under complex, realistic conditions.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2503.06573 [cs.CL]
(or arXiv:2503.06573v3 [cs.CL] for this version)
Full text under the publisher's open licence · 160 words