analiza aiqbench
This research introduces FaiRLLM, a new benchmark specifically designed to evaluate fairness in Large Language Model-based recommendation systems across sensitive user attributes. The study reveals that even advanced models like ChatGPT exhibit bias when generating music and movie recommendations, highlighting critical gaps in current evaluation metrics for RecLLMs. This work is essential for developers aiming to mitigate social prejudices inherent in LLMs before deploying them at scale.
Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation
arXiv:2305.07609v4 Announce Type: replace-cross Abstract: The remarkable achievements of Large Language Models (LLMs) have led to the emergence of a novel recommendation paradigm -- Recommendation via LLM (RecLLM). Nevertheless, it is important to note that LLMs may contain social prejudices, and therefore, the fairness of recommendations made by RecLLM requires further investigation. To avoid the potential risks of RecLLM, it is imperative to evaluate the fairness of RecLLM with respect to various sensitive attributes on the user side. Due to the differences between the RecLLM paradigm and the traditional recommendation paradigm, it is problematic to directly use the fairness benchmark of traditional recommendation. To address the dilemma, we propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM). This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes1 in two recommendation scenarios: music and movies. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations. Our code and dataset can be found at https://github.com/jizhi-zhang/FaiRLLM.
Abstract:The remarkable achievements of Large Language Models (LLMs) have led to the emergence of a novel recommendation paradigm -- Recommendation via LLM (RecLLM). Nevertheless, it is important to note that LLMs may contain social prejudices, and therefore, the fairness of recommendations made by RecLLM requires further investigation. To avoid the potential risks of RecLLM, it is imperative to evaluate the fairness of RecLLM with respect to various sensitive attributes on the user side. Due to the differences between the RecLLM paradigm and the traditional recommendation paradigm, it is problematic to directly use the fairness benchmark of traditional recommendation. To address the dilemma, we propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM). This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes1 in two recommendation scenarios: music and movies. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations. Our code and dataset can be found at this https URL.
Accepted by Recsys 2023 (Short). Typo corrections
Information Retrieval (cs.IR); Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2305.07609 [cs.IR]
(or arXiv:2305.07609v4 [cs.IR] for this version)
https://doi.org/10.1145/3604915.3608860 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 204 cuvinte
Subiecte conexe
Evaluating multi-agent systems for explainability and helpfulness with Amazon Bedrock AgentCore
AWS · 5 oct. 2026
MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations
arXiv · 29 oct. 2026
Towards Personalized Bangla Book Recommendation: A Large-Scale Heterogeneous Book Graph Dataset
arXiv · 9 iun. 2026
Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?
arXiv · 30 sept. 2026