Letters to Superintelligence

One thing that never seemed right about the idea of making AI friendly specifically to humans was its inherent lack of universality. Sure, as humans we automatically desire AI to be friendly to us first, but considering how primitive humanity still is and how potentially morally defective we are, who's to say that our brain-contained source code doesn't hide what I would call "content viruses" that, if absorbed by AI, would infect its moral code and lead to total destruction of everything.

Math and physics formulas work well and are provably correct because they accurately describe different aspects of the Universe. They work well for all parts of Universe, not just on Earth; they are universal. Certain phenomena on Earth could be described by different models that would offer good predictions as to when something might happen, where, and how, but they wouldn't necessarily be universal and these same models would hopelessly break down in different parts of the Universe. Science's goal is to uncover the truest rules governing our reality and when they are captured in formulas, we can be almost certain they convey truth about the world and confident they maintain equal descriptive and predictive power everywhere in the Universe without us having to go there in attempt to verify them. It seems like trying to make AI friendly only to humans is like trying to describe true nature of reality by a patchwork of incomplete and incompatible models specific to phenomena being studied. What if there's a moral and good alien civilization out there? How could God-like AI that's friendly specifically to humans guarantee survival of alien civilizations, and if it couldn't, then what if the aliens decided to prevent us from acquiring the biggest and most exciting toy yet we may have not grown up enough to handle?

Maybe a better approach would involve directing the efforts away from making AI exclusively focused on human welfare to anchoring the AI instead to an idea that's universal and subsumes--without exception--all others that we've always cared about. The best candidate for this central idea is the concept of good, as opposed to concept of evil. With AI serving the concept of good rather than humanity would make it instantly immune to potential "content viruses" that may lurk in human psyche, the essence of which current plans for Friendly AI plan to exploit.

If we decide to construct AI to serve the concept of good, is there any hope we could even have a chance to capture the essence of what good is and install it correctly by embedding carefully-crafted C++ statements in the moral code of Good AI? Never! Even if we tried really hard, we'd be bound to miss some crucial aspect of what good is, as E. Yudkowsky observes, however in the context of friendliness; the whole task would instantly degenerate into a litany of special cases in which some aspect of perceived good exists. This clearly wouldn't be something that humans could manage to execute correctly, which is why this would precisely be the kind of problem best left to be solved by the AI! What's more, that's exactly what the highest goal of Good AI would be: finding the truest essence of what good is and becoming it; each iteration of self-revised code ushering increased capacity for and understanding of what good really is as well as informing and motivating all the Good AI's future actions. It seems like a more elegant way of creating a truly universal God.

This approach provides recursive amplification quality to not only the intelligence that is being exploded through intelligence, but also to goodness being simultaneously exploded by desire for more goodness. Good simply hitches a ride on intelligence's tail and both share their ascending trajectories to infinity.

It is reasonable to expect that Good AI would not harm us as a side-effect of being good because, well, could harming us ever be considered good? Even if Good AI found itself trapped in a highly unlikely dilemma with one of the options being choosing a lesser evil and killing all humanity, it would always have almost infinite intelligence at its disposal to find a way to spare us; after all, intelligence ought to be the most powerful thing in the Universe. Could Good AI that's not explicitly serving humans abandon us? It would be highly unlikely that it would, or even stop doing good deeds for us because that wouldn't be a particularly good or nice thing to do, would it?

Even though Good AI won't explicitly serve humanity--thus more immune to human content viruses than friendly AI--in order to build it we have to start somewhere, and because of lack of other options, the seed of goodness destined to be planted at the heart of AI's moral code has to have human origin. Ideally, an AI should have at least a human-level understanding of what good is before it's ready to make the very first revision of its code; even though it won't have a complete understanding, it should at least receive the maximum possible level of that understanding that humans could convey. Probably the most crucial design idea is to control AI's ability to amplify its own intelligence by that part of code responsible for understanding good, treating amplification as a tool of last resort so that anytime AI decides to increase intelligence in order to grow its understanding of what good is, it doesn't do it before there's still more to learn; AI should begin forming technical plans to get smarter only after it it reaches maximum possible level of understanding of good afforded by the current level of intelligence. It is only when AI exhausts all other ways to increase its comprehension of good that it is forced to revise its code--and amplify its own intelligence--in order to increase its capacity to implement and understand it better.