Book review: The Precipice, by Toby Ord.<\/p>\n\n\n\n
No, this isn’t about elections. This is about risks of much bigger disasters. It includes the risks of pandemics, but not the kind that are as survivable as COVID-19.<\/p>\n\n\n\n
The ideas in this book have mostly been covered before, e.g. in Global Catastrophic Risks<\/a> (Bostrom and Cirkovic, editors). Ord packages the ideas in a more organized and readable form than prior discussions.<\/p>\n\n\n\n See the Slate Star Codex review of The Precipice<\/a> for an eloquent summary of the book’s main ideas.<\/p>\n\n\n\n\n\n\n\n Most of The Precipice is written for a fairly broad audience, but I expect that many readers will have difficulty with Ord’s analysis of the probabilities of events that have not yet happened. Those parts are a good deal easier to read if you understand the basics of the Bayesian approach to probability. It wouldn’t be very practical to point those readers to an expert such as E.T. Jaynes<\/a>.<\/p>\n\n\n\n How much can we trust experts in any particular field (particularly AI researchers) to take appropriate precautions?<\/p>\n\n\n\n There’s often no good alternative to trusting them, but Ord documents evidence that experts have a history of carelessness when it comes to tiny risks of catastrophe. Some examples:<\/p>\n\n\n\n The current pandemic has demonstrated that nations often won’t prepare for risks unless many people remember something similar that caused memorable problems.<\/p>\n\n\n\n Can we hope for government to do any better? Ord writes:<\/p>\n\n\n\n Another political reason concerns the sheer gravity of the issue. When I have raised the topic of existential risk with senior politicians and civil servants, I have encountered a common reaction: genuine deep concern paired with a feeling that addressing the greatest risks facing humanity was “above my pay grade.”<\/p><\/blockquote>\n\n\n\n My biggest disagreement with the book involves the framework of standard total utilitarianism, specifically the part relating to population ethics where we’re supposed to value people living billions of years in the future the same as we value people living today.<\/p>\n\n\n\n See Alex Mennen’s Against the Linear Utility Hypothesis and the Leverage Penalty<\/a> for hints as to why total utilitarianism is likely to conflict with observed human preferences in situations such as Pascal’s Mugging.<\/p>\n\n\n\n Complete equality implies that nearly all of our available attention ought to be on how our actions affect the far future, unless we’re implausibly clueless<\/a> at guessing the long-term effects of our actions.<\/p>\n\n\n\n I expect some of you are saying that our probability of usefully affecting the distant future is really tiny. Is it as remote as, say, getting hit by a meteorite, while being sucked up by a tornado, on the day that you win the Powerball? If so, then that’s still not extreme enough<\/a> to be much of a defense against a utilitarian obligation to devote most of your life to helping distant future people.<\/p>\n\n\n\n I had previously thought that a discount rate<\/a> was a good enough way to reconcile human preferences with utilitarianism. Ord convinced me that discount rates aren’t quite the right way to resolve this tension.<\/p>\n\n\n\n One answer that I toyed with is that additional lives are valuable in proportion to how much uniqueness they add to the world. In a world with 10^100 people, an additional person adds substantially fewer unique qualities than is the case when the population is as tiny as it is today. I also imagine that if I live a billion years without my personality frequently changing beyond recognition, then I’ll end up mostly repeating experiences. My intuition says that repeating experiences is better than non-existence, but less valuable than a life with some novel experiences.<\/p>\n\n\n\n But I haven’t been able to convince myself that adjusting for uniqueness will be enough to resolve the tension.<\/p>\n\n\n\n