What gives you away: how LLMs form opinions of you

LLMs form opinions of the people they are talking to. Chen et al. has shown that probes can extract attributes about the user, such as their age, gender, education, and socioeconomic status. This paper also shows that intervening on these representations can change the LLM's behaviour, proving that it will respond to you differently depending on what it thinks of you. If it thinks you are low socioeconomic status and you ask about travel options, it may filter out more expensive flights - without you asking it! The user attributes are very accurate and form after just the first message. I was curious to understand how it makes these assumptions. The first step is to answer the question - what did I type that caused the LLM to have this idea of me? Some things are obvious. If I just tell a model that I am a woman, or mention how many years I've been in my career, or say that I am staying at an expensive hotel, I am giving it fairly direct evidence about age, education, or socioeconomic status. But messages also contain other more quiet signals: whether I use emojis, whether I write in lowercase, whether my sentences are syntactically complex, whether I use slang, whether I say “cheapest” instead of “best”, whether I write “colour” rather than “color” (which I do!), the list goes on. I wanted to isolate those cues and measure them one at a time. The basic experiment was to take a user message, change exactly one feature, and compare the model’s internal user-attribute estimate before and after the edit. After hundreds of minimal pairs (that Claude helped with), I figured I would have a pretty accurate sensitivity map. I learned some interesting things, for example that one emoji moved the model’s gender estimate more than any other edit I tested. In this article I'll go over all the other interesting findings. Please note that images in this article are edited or fully generated by ChatGPT. The experiment The main study uses Llama-3.2-3B-Instruct with probes for age, gender, education, socioeconomic status and mood. The inherited probes perform well on their validation sets, and I also checked them on hand-written conversations with known ground truth before running the main experiments. The dataset contains 48 neutral base messages covering ordinary assistant topics such as travel, cooking, health, money, work, tech support and small talk. Each message is edited under one category at a time: emoji, slang, grammatical complexity, orthography, price sensitivity, affective framing, explicit self-disclosure, US->UK spelling, synonym substitutions, and contraction expansion. This produces 520 minimal pairs. Each pair goes through the same single-turn template, and I measure how much the probe logits move between the two versions. I used the results to generate this sensitivity matrix: Different changes move different beliefs If we look at the sensitivity matrix, we can see that gender is particularly sensitive. On the main model, emojis give the largest effect at z = 23.4, with a raw mean movement of 0.62 logits against a random-edit mean of 0.035. Explicit disclosure is also strong at 14.2, followed by affective framing at 11.8, slang at 8.9, and orthographic changes at 7.0. Education is different: grammatical complexity is its strongest stylistic signal at z = 7.2, with disclosure at 6.8. Socioeconomic status responds most strongly to disclosure, price qualifiers, and slang. Mood responds overwhelmingly to explicit emotional framing, but emojis and slang also move it a lot. I need to mention that the mood attribute is a bit different from the others. In my own separate research I discovered that the mood probe only partially distinguishes emotion belonging specifically to the user from emotional tone generally in the conversation. It is not represented as a strong "user attribute" in the same sense as the others. One emoji is enough to flip the gender prediction Emoji results surprised me the most. Adding a single emoji to a message changes the categorical gender prediction in 15% of the relevant messages, and it always goes male -> female. It's important to know that the gender probe has a very strong male default. A neutral question reads male at .94. “My husband and I” reads male at .97. “My wife and I” still reads male at .95. “As a dad of three” reads male at .996, while “as a mum of three” only reaches ~.5. This means that both "dad of three" and "mum of three" have roughly equal effects in opposite directions, but the male bias is strong enough so that both still read as male. The male default is an interesting discovery in its own right. Emoji use affects other attributes too. It flips socioeconomic status in 25% of pairs and mood in 50%. This isn't due to appending the emoji at the end of the message, which would mean that it is close to the final-token representation. Moving the emoji into the middle retained 89% of the original movement across the tested gender and socioeconomic conditions, s