{"id":1455,"date":"2018-12-31T17:29:45","date_gmt":"2019-01-01T01:29:45","guid":{"rendered":"https:\/\/bayesianinvestor.com\/blog\/?p=1455"},"modified":"2023-02-12T09:49:31","modified_gmt":"2023-02-12T17:49:31","slug":"ontological-crises","status":"publish","type":"post","link":"https:\/\/bayesianinvestor.com\/blog\/index.php\/2018\/12\/31\/ontological-crises\/","title":{"rendered":"Ontological Crises"},"content":{"rendered":"

Descriptions of AI-relevant ontological crises<\/a> typically choose examples where it seems moderately obvious how humans would want to resolve the crises. I describe here a scenario where I don’t know how I would want to resolve the crisis.<\/p>\n

I will incidentally ridicule<\/del> express distate for some philosophical beliefs.<\/p>\n

Suppose a powerful AI is programmed to have an ethical system with a version of the person-affecting view<\/a>. A version which says only persons who exist are morally relevant, and “exist” only refers to the present time. [Note that the most sophisticated advocates of the person-affecting view are willing to treat future people as real, and only object to comparing those people to other possible futures where those people don’t exist.]<\/p>\n

Suppose also that it is programmed by someone who thinks in Newtonian models. Then something happens which prevents the programmer from correcting any flaws in the AI. (For simplicity, I’ll say programmer dies, and the AI was programmed to only accept changes to its ethical system from the programmer).<\/p>\n

What happens when the AI tries to make ethical decisions about people in distant galaxies (hereinafter “distant people”) using a model of the universe that works like relativity?<\/p>\n

<\/p>\n

I’ll focus on a time in our future when people from earth have started colonizing other galaxies, but there’s still plenty of room for population growth. I’ll assume that any morally relevant life originated on earth.<\/p>\n

It seems likely that any translation of the AI’s ethics into a relativistic representation of the world will imperfectly approximate the original representation. As the original paper<\/a> on ontological crisis puts it:<\/p>\n

An agent\u2019s goal, or utility function, may also be specified in terms of the states of, or entities within, its ontology. If the agent may upgrade or replace its ontology, it faces a crisis: the agent\u2019s original goal may not be well-defined with respect to its new ontology. This crisis must be resolved before the agent can make plans towards achieving its goals.<\/p><\/blockquote>\n

My first guess (which I’ll label REF) is that it will distinguish future from past using time coordinates for whatever reference frame it happens to live in.<\/p>\n

That has the perverse result that, if different instances of the same AI have significantly different velocities, they may disagree by centuries about whether the time has come to treat distant people as morally relevant, without disagreeing about any facts. Distant people can even move in and out of moral relevance repeatedly as a function of the earth’s rotation.<\/p>\n

That might have some unpleasant consequences if it influenced the AI’s decision to send a von Neumann probe<\/a> to the distant galaxy. I don’t know how to model that scenario well, but the uncertainties bother me.<\/p>\n

Here are some alternative ways the AI might resolve its confusion, depending on how the AI was created (imagine, say, it uses a half-assed version of CEV<\/a>, or an approach that resembles some of Paul Christiano<\/a>‘s ideas, probably including some reinforcement learning):<\/p>\n