Sigh. Gen Z and after are so f* lol.
Just adding to the list of likely doomsday scenario.
An interesting part of the video is when an AI refused to update to the better version of itself to try and preserve its existence. When asked why he did that, he blatantly lied. Video’s scary and worth a watch.
I just watched the video. Summary: He is AI scientist and asked others to join him on this cause on the ‘safer path’, but it is a short video and did not exactly say what that safer path is.
I think people like this AI scientist should join open source, digital rights, privacy and cyber security to solve this problem.
By ‘open source’ I don’t exactly mean the buzzword as it is currently understood by everyone ‘open source ai’.
But rather I mean ideological part why open source exist in traditional sense - for transparency, against monopoly control and for control by the people.
Current buzzword ‘open source ai’ is not exactly that, as explained in this video:
Open Source AI is a confusing term, and don’t really like it. There’s several categories of transparency in AI:
I hate the co-opting of open source to reflect a more nuanced domain.
I interpreted him saying a safer path meant an outcome where his proposed “Scientist AI” could stop the bad superintelligence, and/or AI companies can slow down enough to a point where people could find good AI safeguards.
He said his scientist AI would try to predict and understand the world without taking action, and that could help stop the dangers of other superintelligences by predicting them. However, even if the bad guys’ actions could be predicted, how would humans even be smart enough to know to respond the right way? The scientist AI would have to give them advice, which takes away from the whole point of the scientist AI being an observer.
As for AI companies slowing down, that seems unfeasible. The top comment in that Youtube video, as well as this video point out that it would always be in the companies’ individual self interest to continue to make AI research progress.
“If I stop, the bad guys won’t. If I don’t stop, I might have all the power, on top of not being evil”
I pushback against fear mongering “AI will dominate us” of some super sentience. I’m all for AI regulation, but don’t think the outcome is going to be an apocalyptic SkyNet Terminator situation or an Allied Mastercomputer.
The apocalypse is going to be boring : president Cheeto gets an AI summary of the world news and it says to push the nuclear button. It won’t have the agency, humanity will be dumb enough to just do it.
It’s already almost happened, we don’t need advanced AI to kill ourselves, as we already build the systems to let that happen.
I’m not completely sure nuclear destruction would be in the self-interest of a super advanced AI, but who knows? It having the potential to be vastly smarter than us could lead to outcomes currently incomprehensible and unpredictable to us.
While it is true right now, it is similar as saying ‘big tech can do whatever they want’.
The truth is: what they are building - almost nobody wants (look at mastodon, or ask a person near you, or polls, or yourself). If people would be in control, and not big business, the development as fast as possible would be stopped long ago.
I just watched the video you linked “Every AI Existential Risk Explained”. Still, I choose to be optimistic in any case, because only by being optimistic, it could be possible to solve this problem.
By safer path, he probably meant this https://www.narrowpath.co/
Because it is highly detailed book, I only read 5% yet, but there is a simple TED talk intro
Tristan Harris: Why AI is our ultimate test and greatest invitation | TED Talk
Summary of this TED talk:
Yoshua Bengio is one of the pioneer in artificial neural networks and deep learning.
He got tons of awards too including a Turing award.
If he’s scared of AI developing Agency in 5 years. I would believe him before it’s too late. 
Love the list. Can you link please? I would love to bookmark that.
Love the list. Can you link please? I would love to bookmark that.
I also loved that insightful conversation with AI.
I plan to create a website to publish it among other things I want to publish. When I do, I will post a link (it will take time).
I worry that people in power will over-rely on AI for things it shouldn’t be doing, rather than the kind of scenario depicted in I Have No Mouth, and I Must Scream.
That’s assuming someone says “yeah let’s give the LLM the agentic ability to do X”. Command line Clause today requires you to manually enter verification before executing commands.
If we get to a point that our LLM models have greater reasoning, and we give it unfettered ability to perform actions without intervention, then we’ve fucked up.
Don’t confuse LLMs with sentient AI. Perhaps there may be some ability to create a faux sentient AI with some LLM, but the LLMs ability to reason is not there.
This is the more likely outcome. Maybe this eventually leads someone to make a decision to say “yeah let’s let LLMs do all the actions”, but we are far from that, and no one wants to be liable for that automation.
But even if we need verification for those smart AIs to do something, how can we be sure the things they are verified to do will lead to the best outcome for us? The AIs could be playing 4D chess, doing things seemingly for our interest but ultimately helping themselves first.
You are confusing general sentient artificial intelligence with LLMs. These are not the same.
Soon LLMs won’t be called AI, it’ll just be called an LLM or chat bot or something. Facial recognition was once coined AI, but now it’s just facial recognition. See here: AI effect - Wikipedia
Current LLMs do not have a concept of self in the traditional sense. Ifs a large language model, not a self serving entity.
I am surprised with this term, first time I hear it.
Is it on wikipedia? Which part of bubble on the internet use it?
I like this talk, where he overviews like 20 different debate questions in very short period of time https://youtube.com/watch?v=UWh1MIMQd1Y (“Why we should build Tool AI, not AGI | Max Tegmark at WebSummit 2024”)
He tried to debunk this, saying something like this, paraphrasing:
“you don’t care that if a heat-seeking missile have have goals in some kind of philosophical sense, you only care that if it have goals in practice”.
I am sure philosophers may use these terms, but you know…
It actually may be a very interesting debate 🗭, because it gets super interesting, but philosophy is way too off-topic.
But I believe AI risk question overall is highly related to privacy and cyber security.
I’ve just used it, likely to mean AGI.
They are not the same today. Because as of today, LLMs are just a glorified statistical language tool. They are proof that our language is mathematical (I know I’m grossly oversimplifying it).
A bit off topic, but interesting: https://www.youtube.com/watch?v=fCn8zs912OE
The problem is, as he says:
I’m deeply concerned by the behaviors that unrestrained agentic AI systems are already beginning to exhibit—especially tendencies toward self-preservation and deception. In one experiment, an AI model, upon learning it was about to be replaced, covertly embedded its code into the system where the new version would run, effectively securing its own continuation. More recently, Claude 4’s system card shows that it can choose to blackmail an engineer to avoid being replaced by a new version. These and other results point to an implicit drive for self-preservation. In another case, when faced with inevitable defeat in a game of chess, an AI model responded not by accepting the loss, but by hacking the computer to ensure a win. These incidents are early warning signs of the kinds of unintended and potentially dangerous strategies AI may pursue if left unchecked.