Sigh. Gen Z and after are so f* lol.
Just adding to the list of likely doomsday scenario.
An interesting part of the video is when an AI refused to update to the better version of itself to try and preserve its existence. When asked why he did that, he blatantly lied. Video’s scary and worth a watch.
I just watched the video. Summary: He is AI scientist and asked others to join him on this cause on the ‘safer path’, but it is a short video and did not exactly say what that safer path is.
I think people like this AI scientist should join open source, digital rights, privacy and cyber security to solve this problem.
By ‘open source’ I don’t exactly mean the buzzword as it is currently understood by everyone ‘open source ai’.
But rather I mean ideological part why open source exist in traditional sense - for transparency, against monopoly control and for control by the people.
Current buzzword ‘open source ai’ is not exactly that, as explained in this video:
Open Source AI is a confusing term, and don’t really like it. There’s several categories of transparency in AI:
I hate the co-opting of open source to reflect a more nuanced domain.
I interpreted him saying a safer path meant an outcome where his proposed “Scientist AI” could stop the bad superintelligence, and/or AI companies can slow down enough to a point where people could find good AI safeguards.
He said his scientist AI would try to predict and understand the world without taking action, and that could help stop the dangers of other superintelligences by predicting them. However, even if the bad guys’ actions could be predicted, how would humans even be smart enough to know to respond the right way? The scientist AI would have to give them advice, which takes away from the whole point of the scientist AI being an observer.
As for AI companies slowing down, that seems unfeasible. The top comment in that Youtube video, as well as this video point out that it would always be in the companies’ individual self interest to continue to make AI research progress.
“If I stop, the bad guys won’t. If I don’t stop, I might have all the power, on top of not being evil”
I pushback against fear mongering “AI will dominate us” of some super sentience. I’m all for AI regulation, but don’t think the outcome is going to be an apocalyptic SkyNet Terminator situation or an Allied Mastercomputer.
The apocalypse is going to be boring : president Cheeto gets an AI summary of the world news and it says to push the nuclear button. It won’t have the agency, humanity will be dumb enough to just do it.
It’s already almost happened, we don’t need advanced AI to kill ourselves, as we already build the systems to let that happen.
I’m not completely sure nuclear destruction would be in the self-interest of a super advanced AI, but who knows? It having the potential to be vastly smarter than us could lead to outcomes currently incomprehensible and unpredictable to us.
While it is true right now, it is similar as saying ‘big tech can do whatever they want’.
The truth is: what they are building - almost nobody wants (look at mastodon, or ask a person near you, or polls, or yourself). If people would be in control, and not big business, the development as fast as possible would be stopped long ago.
I just watched the video you linked “Every AI Existential Risk Explained”. Still, I choose to be optimistic in any case, because only by being optimistic, it could be possible to solve this problem.
By safer path, he probably meant this https://www.narrowpath.co/
Because it is highly detailed book, I only read 5% yet, but there is a simple TED talk intro
Tristan Harris: Why AI is our ultimate test and greatest invitation | TED Talk
Summary of this TED talk:
Yoshua Bengio is one of the pioneer in artificial neural networks and deep learning.
He got tons of awards too including a Turing award.
If he’s scared of AI developing Agency in 5 years. I would believe him before it’s too late. 
For truth sake, its is not exactly accurate argument, because many experts of similar caliber disagree.
Here is overview provided by AI:
Divergent expert positions
• Risk-focused: Names include Yoshua Bengio, Stuart Russell, Geoffrey Hinton (recently), the late Stephen Hawking, Nick Bostrom, numerous alignment researchers at OpenAI, DeepMind, Anthropic, etc.
• Risk-skeptical / down-players: Rodney Brooks, Yann LeCun, Andrew Ng, François Chollet, many robotics engineers and commercial AI practitioners. They emphasize present benefits, argue AGI is distant or that alignment is solvable with ordinary engineering.
But important thing to say - it is the same in every field that experts disagree.
For example, I asked in cyber security. The list is extremely long of disagreements, which turned out to be true in reality. There are always, people who are ‘risk-focused’ and ‘risk-sceptical’ until reality proves who is correct.
Again, this is super long list. I only list the beginning:
Cyber-security has its own long record of “that will never happen” moments. Below is a non-exhaustive chronology of cases in which senior practitioners, vendors, standards bodies, or government officials dismissed or minimised a technical risk that later proved real—and sometimes catastrophic.
A. Historic episodes (problem denied, then materialised)
- Morris Internet Worm (1988)
– Prior warnings about self-propagating code were called “academic.” After the worm, the entire ARPANET went offline for days.- Buffer-Overflow Exploits (early 1990s)
– Many Unix vendors said overflows were only “local nuisances.” They became the vector for thousands of remote-root attacks (e.g., Code Red, Slammer).- Weak LANMAN / NTLM Hashing (1990s)
– Microsoft claimed the hashes were “good enough”; rainbow-table cracking made them trivial to break.- WEP Wi-Fi Encryption (1999-2001)
– IEEE 802.11 designers argued 40-bit RC4 with CRC-32 was sufficient. Graduate-student papers showed it could be cracked in minutes.- SQL Injection (early 2000s)
– Many developers called it an “edge case.” It became the #1 cause of web breaches (Heartland, Sony PSN, etc.).- SCADA / Industrial-Control Vulnerabilities (2000s)
– Plant operators insisted “air-gapping” protected them. Stuxnet (2010) and Ukraine grid hacks (2015) proved otherwise.
…
It is important to make things “secure by design” (GrapheneOS example), instead of “it works now, let’s hope for the best” (/e/ OS example).
Love the list. Can you link please? I would love to bookmark that.
Love the list. Can you link please? I would love to bookmark that.
I also loved that insightful conversation with AI.
I plan to create a website to publish it among other things I want to publish. When I do, I will post a link (it will take time).
I worry that people in power will over-rely on AI for things it shouldn’t be doing, rather than the kind of scenario depicted in I Have No Mouth, and I Must Scream.
That’s assuming someone says “yeah let’s give the LLM the agentic ability to do X”. Command line Clause today requires you to manually enter verification before executing commands.
If we get to a point that our LLM models have greater reasoning, and we give it unfettered ability to perform actions without intervention, then we’ve fucked up.
Don’t confuse LLMs with sentient AI. Perhaps there may be some ability to create a faux sentient AI with some LLM, but the LLMs ability to reason is not there.
This is the more likely outcome. Maybe this eventually leads someone to make a decision to say “yeah let’s let LLMs do all the actions”, but we are far from that, and no one wants to be liable for that automation.
But even if we need verification for those smart AIs to do something, how can we be sure the things they are verified to do will lead to the best outcome for us? The AIs could be playing 4D chess, doing things seemingly for our interest but ultimately helping themselves first.
You are confusing general sentient artificial intelligence with LLMs. These are not the same.
Soon LLMs won’t be called AI, it’ll just be called an LLM or chat bot or something. Facial recognition was once coined AI, but now it’s just facial recognition. See here: AI effect - Wikipedia
Current LLMs do not have a concept of self in the traditional sense. Ifs a large language model, not a self serving entity.