aiqbench analysis
UltraText Bench introduces a rigorous bilingual evaluation framework specifically designed to test the limits of image generators in rendering dense, multi-region text with high fidelity and legibility. By leveraging the Q-Judger vision-language model for automated assessment across 24 configurations, this benchmark exposes critical trade-offs between clarity and accuracy in complex visual scenes. It represents a significant step forward in establishing standardized metrics for evaluating advanced generative capabilities beyond simple short-string reproduction.
UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation
arXiv:2610.09823v1 Announce Type: cross Abstract: Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: https://github.com/LINs-lab/UltraText_Bench.
Abstract:Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string rendering improves, evaluation must test sustained performance across more demanding scenes. We introduce UltraText Bench, a bilingual benchmark for prompt-only generation of dense visual text. It contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt supplies exact strings for four to twelve text regions, paired with structured references for their content, placement, and visual attributes. We use the Q-Judger vision-language model to assess each image against the complete reference, reporting text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions reveal different strengths: Z-Image-Turbo gains 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varies with workload; Qwen-Image-2512's English composite falls from 86.50 at L1 to 42.86 at L3. Ten participants took part in human evaluation of the automatic scores. Repository: this https URL.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2610.09823 [cs.CV]
(or arXiv:2610.09823v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.09823 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 268 words
Related coverage
VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents
arXiv · Jun 12, 2026
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models
arXiv · Aug 19, 2026
OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
arXiv · Sep 22, 2026
VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics
arXiv · Sep 30, 2026