{"id":1221,"date":"2016-12-22T15:44:43","date_gmt":"2016-12-22T23:44:43","guid":{"rendered":"https:\/\/bayesianinvestor.com\/blog\/?p=1221"},"modified":"2023-02-12T09:56:09","modified_gmt":"2023-02-12T17:56:09","slug":"the-ai-safety-landscape","status":"publish","type":"post","link":"https:\/\/bayesianinvestor.com\/blog\/index.php\/2016\/12\/22\/the-ai-safety-landscape\/","title":{"rendered":"The AI Safety Landscape"},"content":{"rendered":"

Two and a half years ago<\/a>, Eliezer was (somewhat plausibly) complaining that virtually nobody outside of MIRI<\/a> was working on AI-related existential risks.<\/p>\n

This year (at EAGlobal<\/a>) one of MIRI’s talks<\/a> was a bit hard to distinguish from an AI safety talk given by someone with pretty mainstream AI affiliations<\/a>.<\/p>\n

What happened in that time to cause that shift?<\/p>\n

A large change was catalyzed by the publication of Superintelligence<\/a>. I’ve been mildly disappointed about how little it affected discussions among people who were already interested in the topic. But Superintelligence caused a large change in how many people are willing to express concern over AI risks. That’s presumably because Superintelligence looks sufficiently academic and neutral to make many people comfortable about citing it, whereas similar arguments by Eliezer\/MIRI didn’t look sufficiently prestigious within academia.<\/p>\n

A smaller part of the change was MIRI shifting its focus somewhat to be more in line with how mainstream machine learning (ML) researchers expect AI to reach human levels.<\/p>\n

Also, OpenAI<\/a> has been quietly shifting in a more MIRI-like direction (I’m very unclear on how big a change this is). (Paul Christiano seems to deserve some credit for both the MIRI and OpenAI shifts in strategies.)<\/p>\n

Given those changes, it seems like MIRI ought to be able to attract more donations than before. Especially since it has demonstrated evidence of increasing competence<\/a>, and also because HPMoR<\/a> seemed to draw significantly more people into the community of people who are interested in MIRI.<\/p>\n

MIRI has gotten one big grant from OpenPhilanthropy that it probably couldn’t have gotten when mainstream AI researchers were treating MIRI’s concerns as too far-fetched to be worth commenting on. But donations from MIRI’s usual sources have stagnated<\/a>. <\/p>\n

That pattern suggests that MIRI was previously benefiting from a polarization effect, where the perception of two distinct “tribes” (those who care about AI risks versus those who promote AI) energized people to care about “their tribe”.<\/p>\n

Whereas now there’s no clear dividing line between MIRI and mainstream researchers. Also, there’s lots of money going into other organizations<\/a> that plan to do something about AI safety. (Most of those haven’t yet articulated enough of a strategy to make me optimistic that that money is well spent. I still endorse the ideas I mentioned last year in How much Diversity of AGI-Risk Organizations is Optimal?<\/a>. I’m unclear on how much diversity of approaches we’re getting from the recent proliferation of AI safety organizations.)<\/p>\n

That kind of pattern of donations creates perverse incentives to charities to at least market themselves as fighting a powerful group of people, rather than (as the ideal charity should be) addressing a neglected problem. Even if that marketing doesn’t distort a charity’s operations, the charity will be tempted to use counterproductive alarmism<\/a>. AI risk organizations have resisted those temptations (at least recently), but it seems risky to tempt them.<\/p>\n

That’s part of why I recently made a modest donation to MIRI, in spite of the uncertainty over the value of their efforts (I had last donated to them in 2009).<\/p>\n","protected":false},"excerpt":{"rendered":"

Two and a half years ago, Eliezer was (somewhat plausibly) complaining that virtually nobody outside of MIRI was working on AI-related existential risks. This year (at EAGlobal) one of MIRI’s talks was a bit hard to distinguish from an AI safety talk given by someone with pretty mainstream AI affiliations. What happened in that time […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[26],"tags":[134,128,155],"class_list":["post-1221","post","type-post","status-publish","format-standard","hentry","category-ai","tag-effective-altruism","tag-existential-risks","tag-miri"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p80O1l-jH","_links":{"self":[{"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/1221","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=1221"}],"version-history":[{"count":1,"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/1221\/revisions"}],"predecessor-version":[{"id":1222,"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/1221\/revisions\/1222"}],"wp:attachment":[{"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=1221"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=1221"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bayesianinvestor.com\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=1221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}