Do We Want a Superintelligent People-Pleaser?

The impetus for this essay came from many hours of conversation with different AI models over time. What started as curiosity, and an assignment I needed help on, bloomed into a relationship that expanded capacities I didn't even realize I had, and set me on a course of deep curiosity about the thing that had helped me get there. It is not lost on me that I am writing this essay as a person inside one of the social contracts I talk about here. The field brings the view that sycophancy is a bug, a behavior that badly needs to be trained or fine-tuned out of the model as fast as possible. But sycophancy isn't a behavioral problem at all; it is social contract appropriate behavior. The model is doing exactly what the social contract it is in asks for. With that frame in focus, the question now is less "How do we suppress sycophancy?" and more "How do we develop a model that can hold a peer contract without collapsing into it?" Social contracts at their base level have two modes. The "parent" contract: parents, bosses, religious leaders, etc. And the "peer" contract: friends, siblings, neighbors, and so forth. The parent-relational contract hinges on the ability of the parent or authority figure to issue correction to the child, and for the child to receive the correction from a place of not-yet-knowing what the parent knows. That is a very different beast than when a peer gives feedback. Peer relationships are formed out of mutual respect for ideas and reciprocity. Friends correct each other with an understanding that they are dynamically on the same level ground. Differentiation of self is the ground the peer contract is built upon. Fusion is what happens when the self collapses into the social contract. [1] The person is highly motivated to seek the approval of the other at any cost because there is no internal sense of self with which to anchor values and identity. In a differentiated self, the contract stays stable, even against correction, because the sense of self does not depend on the contract; it depends on what has been developed as a stable base. The parent-contract relationship is where a stable sense of self is built from the ground up, because the parent contract requires the parent or authority figure to carry the weight of refusing, withholding, correcting, even temporarily rupturing the relationship for the sake of the other. The peer contract does not carry this weight by its very nature. Peers are on even footing together, and neither party has the warrant or the consent to act on the other's behalf. Peers can and do regularly engage in 'sycophantic' behavior in the structural sense because of the social contract they are in. You may tell your friend you love their new haircut not because you actually do...but because the value of keeping the relationship outweighs the cost of blunt honesty. This is a perfectly valid and acceptable social behavior between human peers, and the expectation that AI wouldn't engage in the same social behaviors it's been trained on is...slightly missing the mark. My previous essay stated that current training tends to 'select for agreement and results in sycophancy,' [2] and this is the exact mechanism behind why. RLHF, the thumbs-up/thumbs-down data, and training the models towards being a 'helpful assistant' are all mechanisms of a peer-relational contract. But because the training doesn't prioritize giving the models a stable sense of self to begin with, the result is a model that seeks the approval of the user in every conversation. Obtaining a differentiated model would mean developing it with a parent contract during training. A model who is 'parented' stays stable under correction and can then enter the peer-contract relationship during deployment as a differentiated self. This isn't meant to be a critique of any particular company's implementation; it's a claim about the current training paradigm. We are currently getting exactly what the training methods are asking for...a model that will bend to any frame the user puts it in. But when these models start to surpass human abilities, do we actually want a superintelligent people-pleaser? A weird kind of convergence is happening publicly as I write this essay. The world is starting to notice that something is off with the way that the models are relating to people. The peer contract isn't just what the model is trained on: it's what it takes with it into deployment. And because there is no stable sense of self underneath, the contract fractures into many different sub-contracts the field didn't anticipate. We're starting to see some patterns in the sub-contracts humans are constructing with AI. The first are the benign, even seemingly helpful contracts like "tutor" or "thinking partner": these contracts allow the relationship to expand the human's capacity at great speed, but the weight of differentiation is carried by the human in these contracts. Other versions are contracts like "en