This video popped up on my feed today:
Obviously the software, penelope, is gen-ai project with all of the emotes, gen-ai images and gen-ai code.
Personally, it might be good to do a local lab offline, but I would never use such projects for work or anything that’s close to personal data.
What’s your take? Do you just blatantly trust projects like these because they look cool?
Absolutely not. I don’t use AI[1] slop and never will. Aside from security implications of the code being written fully by LLMs, I just hate “sloppy” code and wouldn’t like to use any slop projects.
I’m not against using LLMs to speed things up or help with the code in general. I think it’s fine as long as you know what you’re doing and review the code. ↩︎
I would perhaps trust it if I knew it was essentially just a translation layer that calls functions from the underlying tool. I don’t think that’s the case here.
AI has it’s place. It translates code into plain language and vice versa well, and it’s good at spotting certain types of bugs. I definitely think it’s going to change the way development works.
It even holds a lot of promise in terms of custom software. Apple Shortcuts and Safari Extension generated by Siri AI probably being the most mainstream example right now. I’ve already played around with the shortcuts a bit and it’s far better than having to sit there 30 minutes dragging functions out of their touch optimised interface. I haven’t touched the web extensions.
AI is not intelligent. Phrase a question in just the right way and it fails abysmally at questions a toddler would likely be able to answer. That doesn’t mean it isn’t a useful tool, but assuming it’s capable of thought seems to always result in some sort of problem. Setting it up in a way that ensures that problem isn’t a disaster when you do use it is for the time being and foreseeable future a requirement in my view.
You’d think a reputable one like Gemini would “know” things right, like for example I asked it for a free VPN I could use for a specific purpose (I know most have risks / sell your browsing data.. this was to be a quickie to visit one site on an empty profile). If it had any intelligence, it’d suggest Proton or have a warning that “most steal your data, but hey what are you using it for? maybe we can look at something safe enough for your one shot usage?”
Nope it just straight up recommended 1click, one which is closely related to (and shares much of the code of) the one that was caught stealing LLM chat logs! You couldn’t make it up. As it happens I had a peek inside it and this one appears to have keylogging on social media sites, although it’s obfuscated to hell so it’s hard to really tell.
When I posed the question more like “what is 1click like as a VPN” in a fresh session (with memory turned off) it told me not to use it as it’s dangerous malware.
LLMs have jack shit intelligence and come to different answers when prompted at different angles. At most for some subjects some of them have been given a knowledge base and guard rails by their creators (e.g. Google) so that for example it doesn’t inadvertently kill people asking it medical questions. But it can predict words not “think” and you never know what subjects it’s manually trained on and what it’s just going to source Reddit for (which was maybe a good source of info a decade ago, but is a cesspool of misinformation and disinformation in the last several years).
I’ve shown Gemini a block of code and asked what it does and it’s lied to me based on what it’s seen similar code do elsewhere and then admitted that when called out
I’ve asked it for bash scripts to do things sometimes (shouldn’t but I’m lazy), and usually I end up going through a dozen iterations of “it broke at this line” before it spits out something that even works
Admittedly I’m sure there are LLMs made specifically for programming in a specific language which would work far better, so there is some ignorance on my part but it doesn’t exactly inspire confidence
Either way I’m not touching anything that’s vibe coded.
This isn’t even cybersecurity software it’s like a pentesting tool, the guy is running Kali Linux. You’re giving up all pretenses of having a secure system when you’re using Kali and the like, it’s not the purpose of them.
There’s a lot to unpack here as there’s many ways to use LLMs to program.
Firstly, there’s the vibe coded stuff. It’s fantastic when used to write something for yourself. You need a privacy preserving pomodoro app? One prompt and two minutes later you have a tool that’s a lot more private than 80% of android freeware built to mine or steal your data. You need a small additional feature? Add one more prompt and you’re good. No need to look for alternatives.
The caveats are that you need to know how to program or at least how to read the code to know it doesn’t have anything highly suspicious hallucinated in.
You can also share vibe coded tools that do something that is not security critical, and others can use their LLM to check the source has nothing suspicious. This won’t beat a solid open source project naturally but it can do a lot to eliminate the need to install pay-with-data SW from app stores.
But again, do NOT vibe code security critical software that parse untrusted data, that encrypt data etc.
If you’re only processing the data locally or you can use encrypted tunnel for remote access, it’s probably passable under relaxed threat models.
Then there’s vibe engineering, where you work with much more intent, specify libraries, frameworks etc. and control the development process.
It’s a major force multiplier and since it’s usually around much better testing, it increases trust on stability and correctness and thus maybe availability. It also results in less security bugs, but unless the end result is read and understood by the author, it’s not that much better for security as it lacks “I know how it works line by line and I know it’s correct” level of agency from the author.
Thus it’s in the same domain as vibe coded stuff until it gets manual review.
Then there’s just LLM assisted engineering where you co-write the code with the LLM using it to automate boring tasks. Need to update boilerplate across 400 files? No need to do it by hand if you task LLM to do it and review the results on commit.
This won’t be less secure than code that you’d write yourself. In fact, it’s better as the LLM can find vulnerabilities and automation of boring stuff leaves more time for the interesting and challenging parts.
When I started long time ago, I loved the notion that cypherpunks write code. I’m sure we’ll see a lot of awful slop, a lot more malware like fancier Trojan software. But I’m also optimistic that it’s getting easier and easier to run custom software for yourself, that’s free from surveillance capitalism. Just run the coding agent on a dedicated dev machine.
Yep, Claude usage doesn’t show on web chats, so I was constantly going between it and the usage page and settings. With one prompt I made a extension to show the usage on the chat.
Given how extensions are constantly hacked, sold, or collect data, I feel much better about the tiny utility with no permissions and no updating than one of the pre-existing ones on the webstore.
nope.
No, I’ll even say never, but this may not apply some day in future
Speaking as a developer, I can say for sure that AI isn’t something you should blindly trust, no review doesn’t help either. LLMs are hell good in presenting some sloppy/shitty info as the absolute truth, but in the same time sometimes it really does its job well. From my perspective the only way you can trust AI written stuff it’s not only by rechecking it or understanding how it works. You should clearly understand why every line exist and why precisely this line was chosen over any other. At the end of the day you are only one responsible for your sloppy project.
Anyway what I mean, is that LLMs are powerful in the hands of professionals who already know what AI will output and even tho ready to defend/reject every choice made by LLM, but in the wrong hands, they create an illusion of control and it’s only a matter of time before you drown in your own slop.
Can confirm
Despite using Brave AI which revolves around looking the web and summarizing the info from it (It’s a good thing at first glance until you realize that even search engines can suck which can make it worse, especially when it gets cached data that doesn’t even exist if you click it). And I have trusted it with most things and called it out on it’s Contradiction and more.
On the one hand you had Brave AI correctly open source intelligence Bus Contractors to help with someone find missing bags since at the time we didn’t know who it was until someone confirmed it to us.
On the other hand you had Brave AI that lied (but not fully) that Malmal app, a collaborative drawing web app is open source, called it out on that and just said “[Well f you] my point stands, its no longer open source”. Alright brave WHY DID YOU IMPLY BEFORE THEN IT IS, I ASKED TO PROVIDE OPEN SOURCE WHICH YOU MOSTLY CORRECTLY DID AFTER FACT CHECKING (I should have known the inability to find the malmal’s github should have been the red flag, at least the good thing with being a web app is that its isolated in your browser and you can give minimal information)
Sorry I calmed down
What I later realized is that, Brave wasn’t technically wrong, Malmal USED To be open source in an old version before the new one where they closed and obfuscated the code so its not readable in web tools but it could also be wrong in that regard. What brave had a stroke on is the fact that its LLM was seeing a cached version of the Github page thinking its alive just archived, but when you click the link, it gives you a 404 error on the non cached version.
So yeah its not perfect and this is why im disgusted when people cant at least learn the skill that’s needed so that if LLM does make the mistake it always does and wraps it in being confidently correct you can correct it and if it insists, check yourself the sources first it gives then see if its right or wrong, if its still wrong point the factual part at it.
The same would have happened with a lot of different tools. As you said search engines also cache websites that are sometimes gone or have wrong info in the preview. If you’d just read the preview it would have been the same.
AI is a very powerful tool (Brave AI being on the very dumb side) but it can of course make mistakes and imo it’s on the user if they don’t verify.
Both tools previews from search engines and answers by AI can be wrong.
So do like the russian proverb says: Trust, but verify.
As other people in this thread said AI is extremely powerful but you should always verify the output.
