Book Byte #153 "Superintelligence" by Nick Bostrom

Paths, Dangers, Strategies

📣 Curious Quote from the Author

“Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization - a niche we filled because we got there first, not because we are in any sense optimally adapted to it.”

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”

“The computer scientist Donald Knuth was struck that “AI has by now succeeded in doing essentially everything that requires ‘thinking’ but has failed to do most of what people and animals do ‘without thinking’—that, somehow, is much harder!”

“One can speculate that the tardiness and wobbliness of humanity's progress on many of the "eternal problems" of philosophy are due to the unsuitability of the human cortex for philosophical work. On this view, our most celebrated philosophers are like dogs walking on their hind legs - just barely attaining the threshold level of performance required for engaging in the activity at all.”

“It might not be immediately obvious to some readers why the ability to perform 10^85 computational operations is a big deal. So it's useful to put it in context. [I]t may take about 10^31-10^44 operations to simulate all neuronal operations that have occurred in the history of life on Earth. Alternatively, let us suppose that the computers are used to run human whole brain emulations that live rich and happy lives while interacting with one another in virtual environments. A typical estimate of the computational requirements for running one emulation is 10^18 operations per second. To run an emulation for 100 subjective years would then require some 10^27 operations. This would be mean that at least 10^58 human lives could be created in emulation even with quite conservative assumptions about the efficiency of computronium. In other words, assuming that the observable universe is void of extraterrestrial civilizations, then what hangs in the balance is at least 10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 human lives. If we represent all the happiness experienced during one entire such life with a single teardrop of joy, then the happiness of these souls could fill and refill the Earth's oceans every second, and keep doing so for a hundred billion billion millennia. It is really important that we make sure these truly are tears of joy.”

“Consider an AI that has hedonism as its final goal, and which would therefore like to tile the universe with “hedonium” (matter organized in a configuration that is optimal for the generation of pleasurable experience). To this end, the AI might produce computronium (matter organized in a configuration that is optimal for computation) and use it to implement digital minds in states of euphoria. In order to maximize efficiency, the AI omits from the implementation any mental faculties that are not essential for the experience of pleasure, and exploits any computational shortcuts that according to its definition of pleasure do not vitiate the generation of pleasure. For instance, the AI might confine its simulation to reward circuitry, eliding faculties such as a memory, sensory perception, executive function, and language; it might simulate minds at a relatively coarse-grained level of functionality, omitting lower-level neuronal processes; it might replace commonly repeated computations with calls to a lookup table; or it might put in place some arrangement whereby multiple minds would share most parts of their underlying computational machinery (their “supervenience bases” in philosophical parlance). Such tricks could greatly increase the quantity of pleasure producible with a given amount of resources.”

“The gap between a dumb and a clever person may appear large from an anthropocentric perspective, yet in a less parochial view the two have nearly indistinguishable minds.”

“(On one estimate, the adult human brain stores about one billion bits—a couple of orders of magnitude less than a low-end smartphone.”

Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~ 2 GHz).”

“The cognitive functioning of a human brain depends on a delicate orchestration of many factors, especially during the critical stages of embryo development—and it is much more likely that this self-organizing structure, to be enhanced, needs to be carefully balanced, tuned, and cultivated rather than simply flooded with some extraneous potion.”

“Some little idiot is bound to press the ignite button just to see what happens.”

📚 Cognition of the Book’s Big Idea:

The idea of creating a superintelligent computer that is capable of feats that are impossible for a human being to accomplish is both exciting and dangerous. Our first priority must be safety over unrestrained technological advancement if we are to guarantee that such technology advances in a responsible and safe manner.

🛠️Fixing the Tech Industry

Artificial General Intelligence is just around the corner. With it, our Jobs will change, our interactions will change, our ways of communication will change.

What won’t change are our desires. Our desires to learn, our desires to be successful, our desires to be good humans towards each other in the world we live in.

Information won’t matter as much when everything is already at our fingertips…

But experience, that’s the actual Final Frontier…

🤝Collaborate with others with this Social Media Prompt:

When do you feel AGI will occur?

---

My Software Stack: I use Skool for my Online Community Platform and ClickFunnels for my Landing Pages, Payments, and Email Sequencing. I use Substack for my Newsletter and Taskade for AI Note Taking/Second Brain/Project Management. I use my Personal Amazon Store for Tech and Book Recommendations.

Try out the "Think and Grow Rich Challenge" by Russell Brunson and Learn more about the First Self Help Author Napoleon Hill

---