A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

Abstract:Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman $\rho$ = -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite ($\Delta$ < 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.

12 pages, 2 figures. Author's accepted manuscript of a paper published in the ICIC 2026 proceedings (oral presentation)

Artificial Intelligence (cs.AI); Software Engineering (cs.SE)

arXiv:2607.17205 [cs.AI]

(or arXiv:2607.17205v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2607.17205 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

In: Huang, D.S., Li, B., Zhang, Q., Bao, W. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2026. Lecture Notes in Computer Science, vol 16669, pp. 39-50. Springer, Singapore (2027)

https://doi.org/10.1007/978-981-92-3492-9_4 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.