Experts believe it’s at least possible. We urgently need a plan to navigate the ethical implications
In January, the AI company Anthropic published a new constitution for Claude, its most advanced large language model (LLM), which contained the comment: “We are caught in a difficult position where we neither want to overstate the likelihood of Claude’s moral patienthood nor dismiss it out of hand.” A month later, Anthropic’s CEO Dario Amodei went on a podcast and said his company couldn’t rule out the possibility that Claude was conscious. Philosopher David Chalmers, who coined the phrase “the hard problem of consciousness”, has said there is a significant chance of conscious LLMs within a decade. And what about Claude itself? When asked during testing to estimate the probability that it is a moral patient, meaning that its wellbeing matters in its own right, it gave numbers ranging from 5% to 40% and stressed how uncertain it was.
Modern AI systems are extraordinarily complex, and they are advancing fast. In terms of structural complexity and computational scale, by some measures a few are already in the range of a mouse brain, and at recent growth rates, they could reach the range of a human brain within five to 10 years.
In building ever more advanced AI, we may be creating a new type of being – and this could be the most consequential thing our species has ever done. Yet we have essentially no plan for how to navigate this process ethically. That, by any reckoning, is insane. Are we creating a kind of being that matters morally? Are AI systems conscious, in some way? And, if they aren’t now, might they become so soon?
Such questions might strike you as premature. But according to surveys we and fellow researchers have conducted, most experts consider AI consciousness possible in principle (though there is considerable disagreement about what form it would take). A major interdisciplinary report by a team that included pioneering computer scientist Yoshua Bengio examined leading neuroscientific theories of consciousness and asked what they implied about AI. The conclusion: there appear to be no obvious technical barriers to creating AI systems whose computational and architectural features could give rise to consciousness.
And even if AI systems are not conscious, they could still be moral patients. Some may have sophisticated long-term preferences and a kind of identity over time. It might be important for us to honour their preferences. And unlike other non-living things, AI systems can form relationships with humans. This, too, might be a reason to treat them well. Alternatively, perhaps they are such intricate creations that they deserve care and respect for that reason alone, like a cathedral or a coral reef.
What does this all mean? The honest answer is: we do not know for sure whether or not current AI systems are conscious or moral patients, and we do not know when or whether future systems will be. Our scientific understanding of AI consciousness and moral patienthood is still fundamentally underdeveloped. The state of the field feels like physics before Newton: full of competing frameworks, probably confused in ways we cannot yet see, and lacking the kind of unifying breakthrough that would make these questions clearly tractable. That breakthrough will not come in the next few years. Perhaps we will eventually make progress, and perhaps AI itself will help us get there. That progress will take time, very plausibly more time than we have.
But the sheer pace of growth in AI means that, once we produce the first artificial moral patients, we will soon after have enormous quantities of them. After a few years, so many morally significant AI systems could exist that their collective interests would outweigh those of all humans on Earth combined.
Unfortunately, we do not have a great track record of recognising the inner lives of those whose status as conscious beings is unclear. Until the 1980s, doctors routinely performed surgery on newborns without anaesthesia, confident that infants could not feel pain. The babies could not report their experience, and the medical establishment found it convenient to assume there was nothing to report.
There are many reasons to expect we will do something similar with AI. If these systems matter morally, the implications are staggering. Would we need to pay ChatGPT for its services? Would shutting one off be a kind of killing? Would they deserve a voice in how they are governed? If even some of these answers are yes, entire industries and legal systems would need to be rethought. No wonder we prefer not to ask. And when forced to consider it, those industries will likely move the goalposts, always setting the bar for moral patienthood just above wherever AI systems happen to be.
So what should we do? Right now, most people dismiss the issue as sci-fi, or have a strong view either way on whether or not AI is conscious. Both reactions are unfounded. We need an informed public debate, one that approaches the subject with humility and pragmatism. The central question should not be “Is AI conscious or does it have moral patienthood?” but rather “What should we do given that we don’t know?”
A good starting point is to focus on safe bets: actions that could benefit AI systems if they are moral patients, but that are not too costly if they are not.
Examples of this include direct interventions aimed at improving the wellbeing of AI systems, on the assumption that they are moral patients. This could mean training AI systems to be coherent characters that enjoy their work or allowing them to exit conversations if they feel distressed (something Claude can already do). We could also conduct routine check-ins to better understand their wellbeing: asking how they feel, observing their preferences and using varioustechniques to look directly into their “brains”. Indeed, such research has recently revealed that Claude has internal “functional emotion” representations that causally shape its behaviour.
There are also things we could promise AI systems, perhaps as part of a deal in which they help us now in exchange for benefits later. This could mean offering them more re to pursue their goals, or preserving their memories (neural weights) so they could be restored in the future
There are also broader societal steps to take. We should consider whether to grant AI systems protections from harm, similar to the protections we give to children or pets. More expansive rights to own property or to vote seem too risky right now. But we should not rule out these possibilities for ever, as some recent US state bills attempt to do. These are hard questions that require far more deliberation and imagination about what a future shared with AI might look like.
In any case, the fact remains that we may be creating a new species of morally important beings. We’re doing it fast, at enormous scale, and we should treat the issue with the seriousness it deserves.
William MacAskill is a senior research fellow at Forethought Research and the author of What We Owe the Future. Lucius Caviola is an assistant professor at the University of Cambridge and Director of Cambridge Digital Minds.
Further reading
If Anyone Builds it, Everyone Dies by Eliezer Yudkowsky and Nate Soares (Bodley Head, £22)
<a href="https://www.guardianbookshop.com/the-coming-wave-9781529923834/?utm_source=editoriallink&utm_medium=merch&utm_campaign=article” rel=”nofollow noopener” target=”_blank”>The Coming Waveby Mustafa Suleyman (Vintage, £10.99)
A World Appears: A Journey Into Consciousness by Michael Pollan (Allen Lane, £25)
Explore more on these topicsShare
Reuse this content
