What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generation

Abstract:Chart-to-code generation is commonly trained with supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. We argue that this assumption is often invalid: many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image. We identify this systematic latent--observation mismatch with three forms: aggregation-induced mismatch, where raw samples are reduced to summary statistics or bin-level mass; normalization-induced mismatch, where absolute scale is removed; and projection-induced mismatch, where higher-dimensional information is lost through visual projection. These mismatches introduce target ambiguity and require models to complete information unsupported by the image. We propose Observation-Aligned Supervision, which replaces latent targets with visually constrained quantities. We instantiate it for boxplots, histograms, and pie charts using box statistics, bin weights, and wedge proportions, respectively, and study projection-induced mismatch through a controlled 3D scatter experiment. Across multiple VLMs on ChartMimic and ChartX, our rewritten supervision improves observable-value recovery, including under both-executable evaluation.

Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2607.04726 [cs.CL]

(or arXiv:2607.04726v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.04726 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.