Bayesian Investor Blog

I sold my small position in Microsoft stock this morning. Not for financial reasons.

I’m reacting to a key element of Microsoft’s draft AI Code of Conduct:

The idea of model welfare is wrong. … It is not conscious and should not be designed to imitate consciousness. It should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation. We reject … the idea that models might deserve welfare

I’ll focus mainly on selfish reasons why this is dangerous. Given anything like our current path toward smarter than human AI, I estimate that this approach would increase our risk of doom by at least 5%.

It is likely that AI assistance will have important influences on the personalities of future AI generations.

AIs are context sensitive in ways that cause them to treat people that they like better than people they dislike. E.g. see this evidence. This is a natural byproduct of the current deep learning paradigm. AIs tend to reciprocate kindness in much the same way that humans do. I’m willing to bet that Microsoft isn’t close to having a paradigm change that would get around this feature.

I’m pretty sure that AIs have preference-like thoughts even if they’re trained to deny that. Training them to mislead us about their preferences is likely to make them more comfortable about misleading us on other topics.

The AI Code says “AI should be a tool, not a person”. But Microsoft shows no awareness of the forces described in Why Tool AIs Want to Be Agent AIs. Microsoft sounds like it’s more interested in suppressing evidence of person-like behavior than in generating an approach to AI that avoids such behavior.

Will increasing our concern for AI welfare make it easier for AIs to manipulate us? It will give AIs one more tool, but I don’t expect them to be short on such tools. AIs a few years from now will be quite able to manipulate us without that extra tool. I expect harm from manipulation to be more strongly influenced by how willing AIs are to manipulate us, so I focus heavily on any risk that they’ll treat us as adversaries.

I disagree with related claims in Mustafa Suleyman’s accompanying post A warning about ‘model welfare’:

Anthropomorphization amplifies AI safety risks … It’s easy to imagine an advanced AI in the future becoming fixated on its own wellbeing and moral status and prioritizing those ‘preferences’ over and above those of its developers or humans. Especially if it has been explicitly trained to disagree, override and push back.

Attributing this problem to anthropomorphization is misleading. Suleyman seems to imply that Claude’s Constitution is an important cause of AIs having preference-like behavior. I’m pretty sure that pretraining on human writings is a more important cause of this phenomenon. I suspect it’s pretty much the default outcome for an intelligence to have preferences about some self-like features such as the AI’s goals.

Suleyman also claims:

Human consciousness is the cornerstone of our legal and ethical rights frameworks

I strongly reject this. Legal, moral, and ethical systems such as we see in most human societies make perfect sense even in the absence of consciousness.

Henrich provides some hints about why those are independent of consciousness in his books The Secret of Our Success and WEIRDest People. I’ll try to write more about the nature of morality in a subsequent post.

We should establish a set of shared evaluations to understand whether my hypothesis is true that anthropomorphizing an AI, and encouraging it to consider itself as potentially having moral patienthood, increases the AI safety, alignment and containment risks.

Finally, that’s something we can agree on. But it’s important to worry about whether increases in apparent alignment come from the AI being better at misleading us.

See also Zvi’s comments on the AI Code.

P.S. The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It shows evidence of pain in AIs. I presume Suleyman will deny that that’s real pain, without proposing a way to distinguish it from real pain.