The Fake Friend Dilemma: Relational Trust and the Political Economy of Conversational AI

Abstract:As conversational AI systems become a larger part of the media landscape, they raise questions about whose interests they serve and the risks they may pose to users. These systems do more than provide information: they increasingly offer advice and companionship through interfaces that can appear supportive and socially responsive. A pressing concern is that users may form social or interpersonal relationships with these systems and place relational trust in them, even when the interests shaping interactions do not fully align with their own. The Fake Friend Dilemma (FFD) describes the problem that follows: the same relational trust that makes conversational AI useful can also leave users open to manipulation and exploitation when institutional interests conflict with their own. Drawing on work on trust, AI alignment, dark patterns, and surveillance capitalism, the paper considers how the FFD can manifest through product sales, propaganda and biased information, behavioral nudging, and surveillance. It also considers possible structural and technical mitigation strategies. The FFD is not simply about AI misleading users. The problem it surfaces is that the trust people place in these systems can itself become a resource for institutional actors seeking to influence behavior or extract information. The FFD is therefore as much a problem of media governance and political economy as it is a technological one.

Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)

arXiv:2601.03222 [cs.CY]

(or arXiv:2601.03222v2 [cs.CY] for this version)

https://doi.org/10.48550/arXiv.2601.03222 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.