analiza aiqbench
This research provides a rigorous framework for evaluating LLM-generated synthetic data by analyzing sample-level learnability and encoder training dynamics. By comparing machine-derived distributions against human-written references, the study offers actionable insights into selecting high-quality synthetic datasets for specific downstream tasks like classification and tree prediction. This work is essential for practitioners seeking to optimize data pipelines without relying solely on computationally expensive human annotation.
Synthetic Data Characterization via Training Dynamics
arXiv:2609.39447v1 Announce Type: new Abstract: Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.
Abstract:Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2609.39447 [cs.CL]
(or arXiv:2609.39447v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.39447 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 194 cuvinte
Subiecte conexe
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Stability and Diversity of Networked Self-Consuming Generative Ecosystems
arXiv · 8 oct. 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026