Tokenizer-Generator Coupling in Medical Image Generation

Abstract:Latent medical image generators usually treat the tokenizer as fixed preprocessing. We test whether this separation holds in a controlled ChestMNIST study at $64\times64$ that crosses discrete tokenizers, generator families, and sampler settings under a shared latent grid, with continuous-latent reference cells. The rankings depend jointly on the tokenizer, generator, and sampler. The best quantizer changes with the generator, and validation-based sampler selection changes the apparent generator ranking. We interpret this through a rate-distortion-modelability framing in which modelability is conditional on the generator, sampler, and inference budget. The interaction persists when the vocabulary-1024 block is retrained at three seeds, with 4 of 9 pairwise quantizer comparisons exceeding three seed standard deviations, including a reversal between LFQ and FSQ from MaskGIT to D3PM; the rest of the grid was trained at a single seed and is correspondingly less certain. Reconstruction PSNR alone is not a reliable selection criterion. On LFQ-1024, retuning the D3PM and SEDD samplers on a held-out validation split reduces FID-192 from 0.44/0.41 at the default budget to 0.09/0.10 at lower NFE, replicated across seeds, although the continuous references were not given an equivalent sampler sweep. FID-192, our internal ranking metric, ranks consistently with standard FID-2048 (Spearman 0.89) and with a label-free classifier two-sample test (0.86). All experiments are unconditional, use low-resolution $64\times64$ medical-style images and are evaluated with non-clinical FID-based metrics, and our claims are limited to that setting. Implementations are released at this https URL and this https URL.

Accepted at NeurIPS 2026 (poster). Extended version with additional analyses and appendices. Code: this https URL

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

arXiv:2608.07713 [cs.CV]

(or arXiv:2608.07713v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2608.07713 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.