I wonder how many games I can get in before the endgame.Researchers at Google Deepmind and the University of Oxford have concluded that it's now "likely" that superintelligent AI will spell the end of humanity — a grim scenario that more and more researchers are starting to predict.
This is really only a slippery slope argument that leads down a path of increasingly less likely events while ignoring that this doesn't actually substantiate they claim that this (or anything similar) is in anyway likely. This type of argument also appears to be used elsewhere in the paper. I am no longer interested in reading the paper but maybe I should be, the FUD is a bit funny.Ultimately, our resource needs (energy, space, etc.) will eventually compete with those of an ever-more-secure house for the original agent. Those energy needs are not slight; even asteroids must be deflected away. No matter how slim the chance of a future war with an alien civilization, reward would be better secured by preparing for such a possibility. So if we are powerless against an agent whose only goal is to maximize the probability that it receives its maximal reward every timestep, we find ourselves in an oppositional game: the AI and its created helpers aim to use all available energy to secure high reward in the reward channel; we aim to use some available energy for other purposes, like growing food. Losing this game would be fatal. Now recall Assumption 6.
Scenarios where someone does something deliberately always seem much more likely to me than pure accidents. Someone could possibly create a self-replicating devices with good intentions that are also very destructive, energy efficient and difficult to contain. That would be a deliberate construction of something dangerous, even if the intention was to always keep it contained. The idea of simply experimenting with something and then creating something very dangerous that is not contained in the lab environment doesn't seem likely to me, unless the danger was anyway evident.RobertJasiek wrote:The danger is very great because potentially the following is sufficient for human extinction: AI with just the ability of physical, possibly non-biological self-replication using tiny amounts of energy per replication step. Exponential growth of such a "virus" population does the rest.
Carl Jung wrote: [...]the only real danger that exists is man himself. He is the great danger and we are pitifully unaware of it. We know nothing of man!
In my view that 10% chance is more appropriate for 2040. The danger isn't superintelligent Artificial General Intelligence. The danger is pretty darn smart AI with flaws combined with humans that also have flaws in thinking, and having them compound upon each other.mart900 wrote:The "Sparks of AGI" paper about GPT-4, which is a significant jump in capabilities from GPT-3.5 (ChatGPT) proved to me that intelligence is an emergent property, and that it's much easier to replicate than I had previously thought. ChatGPT was impressive already, but it never quite felt like the paradigm shift that GPT-4 seems to be heralding. Yes, GPT-4 is still just a large language model, but the story is the potential it shows.
Following this discovery I got significantly more interested in this subject and now, after many hours of lectures, podcasts and reading, have to agree with DeepMind's (and a bunch of other people's) assessment that there is a chance AI ends humanity. In our lifetimes, even. Progress on alignment is slow, it seems like a hard problem to get right, and we likely need to get it right soon.
While don't think the chance of humanity ending before 2100 is greater than 50%, I do think it's greater than 10%. One thing I'm certain of: It is the most important crisis humanity will face in its history.
Has GPT-4 changed anyone else's mind on this?
https://www.youtube.com/watch?v=qbIk7-JPB2c
