Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization

Abstract:Multimodal synthetic datasets combine structured attributes with free text, but are often evaluated separately. Such metrics can remain high after tabular--text pairings are disrupted. We present a projection-based evaluator for tabular--text synthetic data. A fixed sentence encoder maps text to embeddings, \(k\)-means converts them to cluster states, and tabular variables are represented as categorical or quantile-binned states. Real and synthetic contingency tables are compared using Jensen--Shannon divergence (JSD), normalized mutual information (NMI), conditional JSD (cJSD), and joint-state entropy. We also report text-to-attribute (T2A) utility and a holdout-calibrated proximity flag rate (PFR) as a representation-level diagnostic. A text-permutation control preserves both marginal distributions while disrupting their pairing. Experiments on Amazon Reviews, Kiva Loans, and the Employment Scam Aegean Dataset show that modality-specific scores remain high under this control. The projection diagnostics detect disruption when real projected dependence exceeds a permutation baseline, but are less informative for weak or sparse projections. Some conditioned LLM baselines also exhibit stronger measured dependence than the corresponding real-data projections. These results support explicit cross-modal evaluation with permutation baselines and coverage reporting.

Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2609.22149 [cs.CL]

(or arXiv:2609.22149v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2609.22149 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.