aiqbench analysis
This paper introduces Genres, a novel benchmark that evaluates gender bias in Multimodal Large Language Models (MLLMs) specifically within the context of social relationships and interpersonal interactions. By moving beyond isolated prompts to analyze dual-character narrative generation, the study reveals how biases shift and intensify depending on the specific relational dynamic between characters. The work provides critical insights for researchers aiming to improve fairness in socially sensitive AI applications by offering a more nuanced evaluation framework than current standards allow.
From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship
arXiv:2506.23101v2 Announce Type: replace-cross Abstract: Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially sensitive domains. Existing benchmarks primarily assess bias in isolated contexts and are less suited to capturing biases that emerge through interpersonal interactions. We introduce Genres, a benchmark for evaluating gender bias in MLLMs through the lens of social relationships in narrative generation. Grounded in Fiske's relational models theory, Genres comprises 1,440 multimodal narrative elicitation pairs spanning four fundamental relationship types, diverse social scenarios, and three age groups. Its dual-character generation tasks elicit socially situated interpersonal narratives and enable fine-grained assessment across multiple dimensions of gender bias. Experiments on six recent MLLMs reveal persistent yet strongly relationship- and model-dependent gender disparities, with some patterns becoming more pronounced in interpersonal settings. We further evaluate two complementary generation-stage mitigation strategies. Our findings demonstrate the value of relationship-aware evaluation for uncovering subtle, interaction-driven gender bias in MLLMs.
Abstract:Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially sensitive domains. Existing benchmarks primarily assess bias in isolated contexts and are less suited to capturing biases that emerge through interpersonal interactions. We introduce Genres, a benchmark for evaluating gender bias in MLLMs through the lens of social relationships in narrative generation. Grounded in Fiske's relational models theory, Genres comprises 1,440 multimodal narrative elicitation pairs spanning four fundamental relationship types, diverse social scenarios, and three age groups. Its dual-character generation tasks elicit socially situated interpersonal narratives and enable fine-grained assessment across multiple dimensions of gender bias. Experiments on six recent MLLMs reveal persistent yet strongly relationship- and model-dependent gender disparities, with some patterns becoming more pronounced in interpersonal settings. We further evaluate two complementary generation-stage mitigation strategies. Our findings demonstrate the value of relationship-aware evaluation for uncovering subtle, interaction-driven gender bias in MLLMs.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
arXiv:2506.23101 [cs.CL]
(or arXiv:2506.23101v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2506.23101 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 251 words
Related coverage
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · Oct 2, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026