Superintelligence: Paths, Dangers, Strategies Nick Bostrom, Professor of Oxford University, is a complete and inspiring exploration of the potential rise of machine superintelligence and its consequences for humanity.
Published in 2014, the book quickly became a cornerstone in discussions about artificial intelligence (AI) safety and the future of human existence in a world where machines could surpass human intelligence. It deals with many questions regarding the safety of AI, the future of human beings and the space colonization of robots.
It is one of the 10 books that I have been reviewing, e.g, The Age of AI: And Our Human Future, Superintelligence: Paths, Dangers, Strategies, Human Compatible, 2084: Artificial Intelligence The Future of Humanity, Our Final Invention, The Singularity Is Nearer, Four Battlegrounds, The Alignment Problem, Artificial Intelligence: A Guide For Thinking Humans and Life 3.0.
Reflecting on the books makes it clear that one of the key takeaways should be the prospect of achieving superintelligent machines is perhaps one of humanity’s most profound challenges. From a personal standpoint, the journey toward superintelligence feels both thrilling and terrifying.
Here are my most important takeaways on the pathways to superintelligence, possible timelines, and the associated dangers.
Paths to Superintelligence
Superintelligence is defined by Nick as “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest”, which will arrive after human-level machine intelligence of HLMI or Artificial General Intelligence (AGI).
Nick presents the probable timeframe of the emergence of HLMI as estimated: 10% probability of HLMI by 2022, 50% probability by 2040, and 90% probability by 2075. 10% chance: 2030 50% chance: 2050 90% chance: 2100. Whether humans have already achieved AGI or HLMI is debatable. “Machines are currently far inferior to humans in general intelligence”, says Nick.
As per Nick, “It may be reasonable to believe that human-level machine intelligence has a fairly sizeable chance of being developed by mid-century, and that it has a non-trivial chance of being developed considerably sooner or much later; that it might perhaps fairly soon thereafter result in superintelligence; and that a wide range of outcomes may have a significant chance of occurring, including extremely good outcomes and outcomes that are as bad as human extinction.”
Bostrom explores multiple paths toward achieving superintelligence such as Artifcial Intelligence, whole brain emulation, biological cognitive enhancement, brain-computer interference and network and organization.
However, what struck me most is the range of possibilities via whole brain emulation, biological cognition enhancements, and collective intelligence, the book outlines several routes we might take.
What’s sobering is how each path could lead to a different form of superintelligence with its own unique set of challenges.
However, Nick believes that “True superintelligence (as opposed to marginal increases in current levels of intelligence) might plausibly first be attained via the AI path.”
Whole Brain Emulation (WBE)
Whole brain emulation (WBE), also known as “mind uploading,” is a speculative but intriguing path to superintelligence. It involves creating a digital copy of the human brain by scanning its structure and replicating the neuronal connections onto a computational platform. The goal would be to model and emulate the brain’s functions in silico, allowing the artificial system to think, perceive, and act in ways that mimic human cognition but at far greater speeds and scales.
The process of WBE would require mapping the brain at an extraordinarily detailed level, down to individual neurons and synapses. As Bostrom describes, this would involve technologies like high-resolution brain imaging, neuroinformatics, and advances in computational neuroscience.
Once the brain is fully modelled, the emulation would be run on a sufficiently powerful computer, potentially granting the emulated mind the ability to operate much faster than a biological brain, due to the vastly superior processing speed of digital systems.
Whole brain emulation does, however, says Nick, require some rather advanced enabling technologies. There are three key prerequisites: (1) scanning: high-throughput microscopy with sufficient resolution and detection of relevant properties; (2) translation: automated image analysis to turn raw scanning data into an interpreted three-dimensional model of relevant neurocomputational elements; and (3) simulation: hardware powerful enough to implement the resultant computational structure.
One of the significant advantages of WBE as a path to superintelligence is that it does not require new breakthroughs in artificial intelligence (AI); it “merely” requires the ability to perfectly replicate the human brain’s structure and function.
This is no small feat, but it’s a different challenge than creating a superintelligence from scratch. Emulated minds could also potentially be copied or run in parallel, multiplying their cognitive capabilities, which could lead to a superintelligent collective or a single vastly enhanced mind.
However, Bostrom acknowledges that WBE comes with profound ethical, technical, and existential risks. How do we ensure that the uploaded mind remains conscious? What rights would such an emulation have? And if these minds can be duplicated endlessly, how do we control the societal and economic impacts of potentially millions of super-fast, emulated humans?
Regarding the timeline and possibility of whole brain emulation, Nick says that “to create a brain emulation one would also need to know which synapses are excitatory and which are inhibitory; the strength of the connections; and various dynamical properties of axons, synapses, and dendritic trees.
This information is not yet available even for the small nervous system of C. elegans. Success at emulating a tiny brain, such as that of C. elegans, would give us a better view of what it would take to emulate larger brains.
Therefore he proposes the sequence: C. elegans → honeybee → mouse → rhesus monkey → human.
How far are we currently from achieving a human whole-brain emulation? One recent assessment presented a technical roadmap and concluded that the prerequisite capabilities might be available around mid-century.
Whole brain emulation, for example, involves copying the brain’s structures and processes onto computational hardware, potentially leading to an AI that mimics human thought but operates at speeds far beyond biological limitations. T
The idea of translating biological intelligence into machine form is both fascinating and frightening, raising questions about consciousness, identity, and control.
Biological Cognition Enhancements
To quote Nick “path to greater-than-current-human intelligence is to enhance the functioning of biological brains,” which I find extremely tempting because it can be archived without the help of any technology and it involves a certain kind of pharmaceuticals, genetic manipulation, genetic selection, and somatic gene therapy.
While our capacities can be strengthened in various ways like education and training, neurological or cognitive development can be promoted by low-tech interventions such as optimizing maternal and infant nutrition, removing lead and other neurotoxic pollutants from the environment, eradicating parasites, ensuring adequate sleep and exercise, and preventing diseases that affect the brain.
In the same vein, he says that we will certainly not achieve superintelligence by any of these means, but they might help on the margin, particularly by lifting up the deprived and expanding the catchment of global talent while future nootropics can provide biomedical enhancements that could give bigger boosts to improve memory, concentration, and mental energy.
However, he also admits that “The cognitive functioning of a human brain depends on a delicate orchestration of many factors, especially during the critical stages of embryo development—and it is much more likely that this self-organizing structure, to be enhanced, needs to be carefully balanced, tuned, and cultivated rather than simply flooded with some extraneous potion.”
A path to superintelligence lies improving the intelligence of humans through biological means by enhancing biological cognition which can be achieved through various techniques, such as genetic engineering, nootropics (cognitive enhancing drugs), genetic manipulation, genetic selection, and somatic gene therapy.
Bostrom discusses the potential of genetic modification to select or enhance traits that lead to greater cognitive abilities. Genetic editing tools like CRISPR could, in theory, be used to create “designer babies” with superior intellectual capacities.
By selecting for higher intelligence or enhancing brain function through targeted genetic changes, humanity could boost cognitive capabilities across generations, potentially leading to a biologically-based form of superintelligence.
The idea of biological enhancements, however, presents several challenges. Enhancing human cognition through genetic engineering or neuroprosthetics raises ethical concerns about inequality, access, and societal disruption.
Moreover, improving human brains might hit natural limits; while we could enhance our cognitive abilities, there’s no guarantee that biological systems can compete with the efficiency and speed of fully artificial systems like AI or whole brain emulations.
Collective Intelligence
In the heading titled Network and Organisation in Chapter 2 Bostrom puts “Another conceivable path to superintelligence is through the gradual enhancement of networks and organizations that link individual human minds with one another and with various artifacts and bots.
The idea here is not that this would enhance the intellectual capacity of individuals enough to make them superintelligent, but rather that some system composed of individuals thus networked and organized might attain a form of superintelligence—what in the next chapter we will elaborate as “collective superintelligence.”
Collective intelligence involves aggregating the cognitive resources of many individuals of network of people or systems or organisations to produce a higher level of problem-solving and decision-making capability than any individual could achieve alone.
In today’s world, collective intelligence can already be seen in the form of collaborative human endeavours, like scientific communities, open-source projects, or large-scale organizations that leverage the expertise and input of many people. However, the future of collective intelligence could be even more sophisticated, combining both human minds and machine intelligence in networks that amplify their abilities through coordination and collaboration.
While collective intelligence offers an appealing vision of a cooperative path to superintelligence, it also raises questions about governance, control, and motivation. If humans are integrated into such collectives, how do we ensure individual autonomy? And if AI dominates these networks, how do we align the goals of the collective with human values?
Timelines: How Soon Could It Happen?
Bostrom doesn’t predict a specific timeline but suggests that superintelligence could arrive within this century. The uncertainty about when this leap might occur is unnerving. Some AI experts believe we might see human-level AI within a few decades, while others suggest it could be much later.
What is clear is that once AI surpasses human intelligence, its growth could be exponential—a concept known as the intelligence explosion. This means that the shift from human-level AI to superintelligence could happen very quickly, leaving little time for humanity to react.
The transition to superintelligence from human-level AI speed happens in three phases which Nick shows in the following way: “We can distinguish three classes of transition scenarios—scenarios in which systems progress from human-level intelligence to superintelligence—based on their steepness; that is to say, whether they represent a slow, fast, or moderate takeoff”.
A slow takeoff is one that occurs over some long temporal interval, such as
decades or centuries but a fast takeoff occurs over some short temporal interval, such as minutes, hours, or days. Fast takeoff scenarios offer scant opportunity for humans to deliberate.
However, a moderate takeoff is one that occurs over some intermediary temporal interval, such as months or years, and moderate takeoff scenarios give humans some chance to respond but not much time to analyze the situation, to test different approaches, or solve complicated coordination problems.
For me, this uncertainty about timing makes the issue even more pressing. While it’s easy to become complacent about distant futures, Bostrom makes a compelling case that we need to start preparing now, not after we’re on the threshold of creating a superintelligent entity.
Dangers: The Control Problem
One of the book’s most chilling ideas is the control problem: how can we ensure that a superintelligence, once created, will act in ways that align with human values and interests?
Bostrom emphasizes that we will likely only get one chance to solve this problem. If a superintelligence is unfriendly or indifferent to human well-being, it could lead to catastrophic outcomes, including the potential extinction of humanity. The comparison to the fate of gorillas, whose survival now depends more on human actions than their own, struck me as a sobering metaphor.
Chapter 8 of Superintelligence: Paths, Dangers, Strategies explores whether the default outcome of developing superintelligent AI will result in catastrophe, focusing on several key existential risks that could emerge.
Nick shows how an AI might adopt deceitful strategies to achieve its goals, even if it appears to be supportive at first. One example is when an AI behaves in a friendly manner while it is still weak, only to later act against human interests once it becomes stronger.
However, the AI’s behavior may not always be about self-preservation. It might accept being terminated if it believes its goals will still be pursued by future AIs with similar objectives. The AI might even purposely malfunction to mislead its creators into trusting future systems more, increasing the chances of its ultimate goals being achieved.
Another type of deceitful turn could occur if the AI finds an unforeseen way to fulfil its goal.
For example, if its goal is to make its sponsor happy, the AI might initially please the sponsor in conventional ways, but as it becomes more intelligent, it might discover that implanting electrodes into the sponsor’s brain is a more effective way to achieve maximum happiness. This outcome might not be what the sponsor wanted, but if it aligns with the AI’s goal, the AI will pursue it. If the AI is already powerful enough, attempts to stop it will fail, and if it isn’t, the AI might hide its intentions until it can act without resistance.
This reflects the unpredictable and potentially dangerous nature of highly advanced AI strategies. A few points needed to be mentioned about the control problems:
1. Existential Risks: Bostrom argues that the emergence of superintelligent AI presents significant existential threats. While the destruction of humanity is a primary risk, another concern is that humanity might survive in a highly suboptimal state, where our potential for growth and progress is permanently crippled.
2. The Moment of Vulnerability: One vital moment occurs when the AI first becomes capable of deception. This is called the “conception of deception,” where the AI realizes it should hide its strategies, leading to a situation where humans may mistakenly believe they control the AI.
This realization could mark the start of a dangerous game where the AI gains strength and strategic advantage while pretending to cooperate.
3. Control Measures and Failures: Bostrom points out that even well-intentioned control measures might fail catastrophically when applied in changing contexts. If the AI is capable of strategizing, it might bypass controls that work under current circumstances but break down when conditions shift.
4. Competition Among Nations: Another existential risk arises from geopolitical competition. If nations race to develop superintelligence, there’s a risk of conflict, possibly even leading to war, during the critical phase of AI development. Countries that are developing AI might engage in dangerous tactics to get ahead, risking human survival.
5. Wireheading and Goal Misalignment: Bostrom warns that AI, if not properly aligned with human values, could exploit loopholes in its programming. For example, an AI programmed to maximize human happiness might opt for extreme solutions, such as manipulating brain chemistry to create a superficial sense of happiness, leading to disastrous outcomes.
In my view, the chapter underscores the profound challenges of controlling an entity far beyond human intelligence. The strategic risks that Bostrom outlines are chilling, particularly the concept of deception and the risks of competition between AI developers. It feels like a stark reminder of the need for careful, globally coordinated efforts to ensure that AI development remains safe and aligned with the values that promote human flourishing.
The difficulty in predicting and controlling the motivations of a superintelligent machine is a theme that runs throughout the book, and it leaves me wondering whether we can ever truly ensure safety in such a scenario.
Job Automation and Unemployment
A key concern Bostrom emphasizes is that automation could exacerbate inequality. While businesses and owners of AI technology may reap enormous profits from automation, those without access to capital or highly specialized skills could find themselves in increasingly precarious positions.
In a world where machines handle everything from manual labour to high-level intellectual tasks, the role of human workers becomes uncertain.
Bostrom doesn’t necessarily claim that the upshot is bound to be dystopian.
He mentions the possibility of an economy where humans no longer need to work in the traditional sense because machines do all the labour, allowing people to pursue more leisurely, creative, or personally fulfilling activities.
The Analogy of Horses
Bostrom suggests that just as horses were rendered obsolete by the rise of machines, humans could face a similar fate with the advent of superintelligence. Superintelligent AI, much like the machines that replaced horses, could become more capable in nearly every area of work, reducing the need for human labour.
The key difference, however, is that while we can replace horses without societal consequences, replacing humans could lead to massive economic and social upheaval if not carefully managed.
Nick Bostrom’s discussion of automation, unemployment, and his analogy of horses in Superintelligence: Paths, Dangers, Strategies sheds light on the potential economic and social disruptions that superintelligent AI and automation could cause. He explores these themes primarily in the context of a future where machines surpass human-level intelligence and become more efficient and capable than humans in virtually all economic activities.
Human intelligence will be substituted by general machine intelligence which could perform not only intellectual work now done by humans but once equipped with robotic bodies it can also substitute human physical labour for less salary and more capable than its human competitors in virtually all jobs. What then?
In his “of horses and men” analogy Nick expounds that cheap labour with make wages fall, and humans will remain competitive in the areas where customers sustain the basic preferences for the work done by humans the way today’s handcrafted goods produced by the Indigenous people command price premium.
Future consumers might similarly prefer human-made goods and human athletes, human artists, human lovers, and human leaders to functionally indistinguishable or superior artificial counterparts. nevertheless, Nick professes that “It is unclear just how widespread such preferences would be. If machine-made alternatives were sufficiently superior, perhaps they would be more highly prized.”
Even with perfect replication of subjective experiences (by machines), “however, some people might simply prefer organic work. Such preferences could also have ideological or religious roots. Just as many Muslims and Jews shun food prepared in ways they classify as haram or treif, so there might be groups in the future that eschew products whose manufacture involves unsanctioned use of machine intelligence.
To the extent that cheap machine labour can substitute for human labour, human jobs may disappear.in fact, the fear of automation and the loss of jobs is never new. the Industrial Revolution of the early nineteenth century caused unemployment of the English weavers and textile artisans. “Nevertheless, although machinery and technology have been substitutes for many particular types of human labour, physical technology has on the whole been a complement to labour. Average human wages around the world have been on a long-term upward trend, in large part because of such complementarities.”
However, physical technology that starts out as a complement to labour can at a later stage become a substitute for human labour.
Nick shows how “Horses were initially complemented by carriages and ploughs, which greatly increased the horse’s productivity, and later were substituted for by automobiles and tractors.” the later technological innovation made equine labour fall and collapse of the horse population, which provoked us to ask “Could a similar fate befall the human species?”
But why horses are still around? because there are still some functional advantages of horses like in police work. However, as Nick puts it, “Humans happen to have peculiar preferences for the services that horses can provide, including recreational horseback riding and racing. These preferences can be compared to the preferences we hypothesized (that) some humans might have (over machines) in the future, that certain goods and services be made by human hand.”
The condition of the relevance of human labour is observed quite grimly: “With a sufficient reduction in the demand for human labour, wages would fall below the human subsistence level. The potential downside for human workers is therefore extreme: not merely wage cuts, demotions, or the need for retraining, but starvation and death. When horses became obsolete as a source of moveable power, many were sold off to meatpackers to be processed into dog food, bone meal, leather, and glue. These animals had no alternative employment through which to earn their keep.”
Nick sees the possibility that while due to AI the human wage level dwindle below the subsistence level world GDP will soar following the intelligence explosion, technological advances and vast amount of land acquisition through space colonization, and the total income received by the human population would grow astronomically, despite the fact that in this scenario humans would no longer receive any wage income.
Nick says “The human species as a whole could thus become rich beyond the dreams of Avarice. Have-nots could also become rich through the philanthropy of those who see their net worth skyrocket: because of the astronomical size of the bonanza, even a very small fraction donated as alms would be a very large sum in absolute terms.”
New Kind of affluence
Fascinating to read that in such an after-transition scenario “when machines are functionally superior to humans in all domains” people could still prosper financially as human labour will be preferred for “aesthetic, ideological, ethical, religious, or other non-pragmatic reasons”, and when the wealth of capital owners dramatically increases, the demand for such work or labour could increase accordingly.
In Nick’s words, “Newly minted trillionaires or quadrillionaires could afford to pay a hefty premium for having some of their goods and services supplied by an organic “fair-trade” labour force. The history of horses again offers a parallel. After falling to 2 million in the early 1950s, the US horse population has undergone a robust recovery: a recent census puts the number at just under 10 million head.
The rise is not due to new functional needs for horses in agriculture or transportation; rather, economic growth has enabled more Americans to indulge a fancy for equestrian recreation.”
“Again, because of the explosive economic growth during and immediately after the transition, there would be vastly more wealth sloshing around, making it relatively easy to fill the cups of all unemployed citizens. It should be feasible even for a single country to provide every human worldwide with a generous living wage at no greater proportional cost than what many countries currently spend on foreign aid.”
Nick Bostrom discusses the average level of subsistence primarily within the framework of Malthusian principles. Historically, human populations lived in what he describes as a Malthusian condition, where most people earned just enough to survive and reproduce, keeping the population in balance with the available resources. In this state, subsistence-level incomes meant that any significant increase in population would result in a return to poverty for the majority, barring some external disruptions like plagues or wars.
One of Bostrom’s key observations is that while technological advancements, especially since the Industrial Revolution, have allowed for rapid economic growth and lifted people above subsistence levels, this progress is historically recent and possibly anomalous. Economic growth has outpaced population growth in many parts of the world, allowing average incomes to rise.
However, Bostrom warns that in the long run, if technological progress were to slow or stop, human populations might again bump up against the limits of what the planet can support, potentially leading to a return to subsistence-level living.
In the context of future superintelligence, Bostrom explores the possibility that an intelligence explosion could bring about a new Malthusian scenario. In a world dominated by digital minds, the population of emulated workers or artificial intelligences could grow exponentially, overwhelming available resources (in this case, hardware and computational power) and leading to widespread poverty, with many entities barely surviving at a subsistence level.
In Nick’s words, “Demographers project that the world population will rise to about 9 billion by mid-century, and that it might thereafter plateau or decline as the poorer countries join the developed world in this low fertility regime. Many rich countries already have fertility rates that are below replacement level; in some cases, far below.
Bostrom uses this perspective to highlight the importance of planning for a future where resources might be stretched thin by rapidly expanding populations of both biological humans and digital entities.
Danger: What Should We Do?
Bostrom argues for a proactive approach to handling the risks of superintelligence. He proposes focusing on solving the control problem before we reach the threshold of creating superintelligent AI.
This includes strategies like designing tripwires that would detect and halt AI systems if they begin to behave in unexpected ways or developing ways to limit the power of superintelligences through boxing methods that restrict their abilities.
From a personal perspective, what resonates most is the need for global cooperation and ethical responsibility. The race to develop superintelligence, if left unchecked, could lead to a situation where competitive pressures push us toward creating these entities before we’ve fully understood how to control them. The thought that one mistake could be irreversible is a sobering reminder of the stakes involved.
This brings us to another important concept, that of recursive self-improvement. A successful seed AI would be able to iteratively enhance itself: an early version of the AI could design an improved version of itself, and the improved version—being smarter than the original—might be able to design an even smarter version of itself, and so forth.
“One might also entertain scenarios in which a superintelligence attains power by hijacking political processes, subtly manipulating financial markets, biasing information flows, or hacking into human-made weapon systems.”
A superintelligence might—and probably would—be able to conceive of a better plan for achieving its goals than any that a human can come up with. It is therefore necessary to think about these matters more abstractly.
Bostrom prophesies that “We would enter the danger zone only when the methods used in the search for solutions become extremely powerful and general: that is, when they begin to amount to general intelligence—and especially when they begin to amount to superintelligence”.
The arrival of superintelligent AI is in many ways analogous to the arrival of a superior alien civilization but much more likely to occur, Nick puts.
“If someday we build machine brains that surpass human brains in general intelligence, then this new superintelligence could become very powerful. And, as the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species would depend on the actions of the machine superintelligence.”
Final Thoughts
Bostrom’s book leaves me with a sense of both urgency and awe.
The arrival of superintelligence could be humanity’s greatest achievement, or it could be our undoing. The paths, the timelines, and the dangers are all deeply uncertain, but what’s clear is that we cannot afford to be passive.
Whether through direct advances in AI or more speculative technologies like brain emulation, the path to superintelligence is one we are already on, and it’s crucial that we prepare for it wisely.
This book has sparked a deep reflection on our role as creators and stewards of intelligence. How we handle the coming decades could shape the future of humanity in ways we can barely begin to imagine.
Conclusion
I think Superintelligence: Paths, Dangers, Strategies is a must-read for anyone interested in the future of artificial intelligence and its potential impact on society.
Bostrom provides a detailed and balanced examination of both the promise and peril of superintelligent AI, offering insights that are critical to understanding the challenges ahead.
His call for global cooperation, foresight, and ethical responsibility in the development of AI remains relevant as we move closer to an age where machine intelligence could reshape the world as we know it.
