aiqbench analysis
This paper introduces FASHION-RQ, an automated benchmarking framework for fashion retrieval systems that leverages Residual Quantized Variational Autoencoders to derive semantic clusters without manual annotation. By establishing scalable tasks for ranking and relevance classification, it addresses critical gaps in evaluating models against fine-grained, open-ended real-world queries. The work provides a robust dataset and methodology for advancing research in information retrieval and recommendation systems within the AI domain.
FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval
arXiv:2610.17841v1 Announce Type: cross Abstract: Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing benchmarks often rely on manual curation, predefined taxonomies, or synthetic prompts, limiting scalability and semantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical discrete representations learned by Residual Quantized Variational Autoencoders (RQ-VAE) organize products into semantically coherent clusters at progressively finer granularities. We leverage this structure to construct FASHION-RQBENCH, a scalable benchmark without manual query annotation or predefined taxonomies. We propose four complementary tasks for evaluating ranking, relevance classification, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent retrieval models demonstrate that FASHION-RQBENCH differentiates model capabilities across query types and attributes. A human study further shows that benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH dataset will be publicly released upon acceptance.
Abstract:Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing benchmarks often rely on manual curation, predefined taxonomies, or synthetic prompts, limiting scalability and semantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical discrete representations learned by Residual Quantized Variational Autoencoders (RQ-VAE) organize products into semantically coherent clusters at progressively finer granularities. We leverage this structure to construct FASHION-RQBENCH, a scalable benchmark without manual query annotation or predefined taxonomies. We propose four complementary tasks for evaluating ranking, relevance classification, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent retrieval models demonstrate that FASHION-RQBENCH differentiates model capabilities across query types and attributes. A human study further shows that benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH dataset will be publicly released upon acceptance.
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2610.17841 [cs.IR]
(or arXiv:2610.17841v1 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2610.17841 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 291 words
Related coverage
STaR-DRO: Stateful Tsallis Reweighting for Group-Robust Structured Prediction
arXiv · Jun 15, 2026
From code to diagrams: Agentic architecture documentation with Amazon Bedrock AgentCore
AWS · Sep 2, 2026
Automating coherent long-form video generation
Google Research · Sep 24, 2026
HASTE: Hierarchical and Attribute-aware Supervision for Generative Recommendation
arXiv · Oct 15, 2026