6 comments on “Drexler on AI Risk

  1. Pingback: Rational Feed – deluks917

  2. Thanks for this very clear and interesting review. Some replies to specific points:

    > I haven’t thought of a realistic example where I expect the delay would generate a strong incentive for using an agent AGI, but the cancer example is close enough to be unsettling.

    A “war in cyberspace” might be an example like that. That is, suppose two highly capable countries are actively hacking each others’ internet-connected computers and control equipment (and the people involved with maintaining them), engaged in an “arms race” of offensive and defensive techniques in that sphere. Given that a newly discovered security hole can in principle be either exploited or patched within hours, and that secrets discovered by exploiting one security hole (such as the code running on protected parts of some network) can lead to discovery of new security holes, and since creativity and intelligence are important parts of discovering security holes and hacking strategies, it seems pretty clear that recursive self-improvement within this area could be a decisive advantage.

    (And, that area is general enough to be pretty worrisome. Maybe one or more AI agents running such a war could be governed in a safe way — but in the heat of a high-stakes battle, would they be?)

    I think this is a high risk, since I think the leaderships of several candidate countries are at least vaguely aware of this possible war scenario, and much less aware of AI risks. When evaluating different plans from competing proponents within a “military” mindset, I think they will favor speed of development over caution.

    > Drexler seems to extrapolate current trends, implying that the first entity to generate human-level AI will look like Google or OpenAI. Developers there seem likely to be sufficiently satisfied with the kind of intelligence explosion that CAIS seems likely to produce that it will only take moderate concern about risks to deter them from pursuing something more dangerous.

    But will they prevent their code or services from being stolen or corrupted by rogue employees or external spies?

    > I’m unsure where Siri and Alexa fit in this framework. Their designers have some incentive to incorporate goals that extend well into the future, in order to better adapt to individual customers, by improving their models of each customers desires. I can imagine that being fully compatible with a CAIS approach, but I can also imagine them being given utility functions that would cause them to act quite agenty.

    Assuming their designers or monitors have some idea what kind of data and theories and plans they’re representing, this doesn’t seem like much of a risk. If their AI was so black-boxy that the designers and monitors had no idea “what it was thinking about”, then it might be a risk, but as long as each agent is confined to working with one customer, it doesn’t seem like a high risk for society (the problem will be detected when something goes wrong with a few individual customers).

  3. Bruce, yes, a military arms race would likely create some serious risks. I doubt that any general AI safety paradigm can improve much on CAIS at handling this.

    >But will they prevent their code or services from being stolen or corrupted by rogue employees or external spies?

    Sigh. I suppose we should be worried about external spies.
    I’m less concerned about rogue employees, since recent trends suggest their pay (and working conditions?) will be good and improving.

    I’m less confident than you that Alexa/Siri designers will consistently understand their system in enough detail.
    I’m concerned that they’ll be tempted to give the system a broad goal, such as “maximize revenue”, with results that the AI employs arbitrarily large amounts of resources to make better predictions, and that resource use would affect more than just customers.

  4. Pingback: Overcoming Bias : How Lumpy AI Services?

  5. Pingback: The AI Does Not Hate You | Bayesian Investor Blog

  6. Pingback: Human Compatible | Bayesian Investor Blog

Comments are closed.