This is more of an opinion post than usual, but something I have been thinking of that will be relevant at least until the AI bubble bursts, if not even longer.
I think it is a terrible idea from a security perspective not just to use AI in software development, but to use even partially AI-developed software. In a nutshell, it makes programmers less capable of spotting potential security holes and introduces security vulnerabilities itself.
I believe that the best way to approach this is, as the title suggests, deliberately avoid even partly AI-generated software, let alone 100% AI-generated software. In addition, if you are a developer, I strongly encourage you to reevaluate your decisions around this as especially if your application is handling sensitive data, this can put users of your software at significantly more risk. If your software doesnât have generative-AI use at all already, I highly recommend making it clear by stating so in the README file or wherever it would be most obvious for users, as more and more software is becoming less secure by the day due to the use of AI.
I think it really depends on the approach. It is not so black and white. Vibe coding is surely a terrible idea especially in lanaguages that the AI model is not trained on.
However to quickly set up things and to do basically analysis and suggestions it can actually make code more robust too. For example copilot is very good at spotting bugs or context issues when writing code and makes mostly useful suggestions.
The real risk is where a non-skilled user is crafting code without understanding the workings. That will surely lead to significant vulnerabilities. The risk in my opinion is not much different from letting a junior (in terms of knowledge level) write an authentication system. It is an inherritly bad idea. It needs guidance and propper review.
That being said i will say that it will be extremely hard if not impossible to find any software soon that is not for some part written by AI. All software these days uses, frameworks, libraries, compilers, so it will all be connected.
I think like the rest of AI it is better to find defenses and mitigate the risks than to avoid and deny itâs existance. The latter is not a durable strategy in my opinion.
Disagree. AI is a tool. Same as a hammer or knife. Boycotting all knives from the kitchen because some jerks use them for crime is silly
For the specific case of security in AI-generated software, without touching on ethics or the morality of supporting the AI industry:
IMO, the problem isnât AI, not remotely. Itâs poor developer, pushing out weak & insecure products. Same as it ever was, now they just have AI
You can absolutely build solid code with AI assistance - AI doesnât automatically reduce the quality of a software product
I saw a quote over on the Qubes forum that expressed this well:
If automated work is good (which most often requires human curation or at least direction), there is no reason to refuse it, if the work is bad, it should be discarded regardless of its origin, and if you accept bad job thinking it is good (or reject good job thinking it is bad) - your review process sucks
Edit: additional thoughts
if you accept bad job thinking it is good⌠your review process sucks
Most PG recommendation categories include a third-party audit as basic criteria. Software must pass a strong review process, conducted by competent professionals, to be considered trustworthy. This is intended to weed out any poor tools with vulnerabilities. It doesnt really matter whether the vulnerability was introduced by AI or an underpaid intern
Yes in theory, in practice your employers would like you to SHIP CRAZY FAST
So you canât just do it old-school and hand-craft your thing.
For some use cases it also is not a big deal to skip over the scaffolding.
Saying that as a fulltime Web dev too.
Yes.
People should just stop giving away their life to OpenClaw or similar nonsense (especially if non-tech savvy) under the promise of greener fields tho.
Also, stop shoving it into your end-users mouths, nobody wants nor needs AI in the tools they use daily.
Yes, just increases the risk of atrophy just like you could forget how to do divisions if not used to it anymore.
I have to agree with others that it really depends. The risk for insecure code is higher for sure.
But personally Iâm very confident that Iâm putting out more secure code than I was without AI. I still review the generated code, I still understand why things are built the way they are. But now I can do a lot more analysis on that code, become aware of security issues that I wouldnât have caught before (and even learn about types of issues I wasnât even aware of), and also do much more comprehensive testing on that code.
To be clear, I understand the sentiment. There is a LOT of bad AI code out there and its hard to know what you can trust. But I trust good security review, good values, and good development systems more than I care about whether the code had AI involved or not.
Also I agree that itâs going to be near impossible very soon to guarantee there was no AI used anywhere in a given piece of software. IMO what will be more important is verifiability and validation of security guarantees because who knows how the code was generated.
That being said i will say that it will be extremely hard if not impossible to find any software soon that is not for some part written by AI. All software these days uses, frameworks, libraries, compilers, so it will all be connected.
This is specifically why I advocate for developers who donât use AI to add a clear label (you should also do that if you do use AI in development so that itâs easier to make an informed decision but realistically most wonât). Realistically due to the high output of AI-generated software (along with most other AI-generated content) itâs going to be easier to label what isnât AI-generated than what is. I hate that I have to by-default assume that everything is AI-generated unless explicitly stated otherwise, but that is as you put it, extremely hard if not impossible to not do.
Unfortunately youâre right that in a more enterprise environment itâs not really possible to be slow and methodical in development and so AI-generated code is gonna be expected, and in many workplaces is already expected. That ultimately does make the problems Iâm describing even worse as thereâs little to no time for proper code reviews because it has to be on the bossâ desk by Monday.
How would this work? Wouldnt you need a third-party authority to verify this claim is truthful?
If we, for the sake of argument, do believe the claim that âAI is a security riskâ, then a âNo AIâ label equates to a security feature. Bad faith actors would certainly abuse this label if its claims sold more software
We could take some cues from existing security practices and have multiple third-party auditors who specialize in manually reviewing code to determine if AI was used to generate it (or even ask existing security auditing firms to include that in their audits), but admittedly Iâm not sure how feasible this is especially for smaller projects, which are more likely to not use AI in the first place, and thus might want such an auditor. I donât really have a perfect solution here but I think having something is better than no label at all. Itâs still up to each person to either trust the label/auditor or do their own audits to verify.
I do also wonder are you going to avoid the Linux kernel too? I mean what are you suggesting here? All these projects have AI generated code in them already.
Vulnerabilities have almost nothing to do with âwithout understanding the workingsâ, but with the fact that writing secure code is hard, keeping systems secure is hard. Thereâs just too many variables and too many ways to pwn system/code. I doubt majority professionals (let alone vibe coders) would claim they know how to. It is a specialized field in on itself. In fact, LLMs may even help some of us; ex:
Weâve been using Copilot (for Rethink) since it first came out (in mid 2021)! And it writes like 40% of the code. Though, I must say that for the more sensitive parts, I routinely discard the LLM output and prefer to hand-roll my own (but thatâs on me; and it isnât to do anything with LLMâs capabilities but my impatience and perhaps inability to review the generated globs).
You have a point.
Developers working in information security are well aware, and usually competent enough to make the right calls; but that doesnât mean they wonât slip up from time to time. Regardless, software in general, due to its malleability (code as data / data as code), has always been a wild beast to tame.
For instance, hereâs Edsger Dijkstra mulling the so-called âSoftware Crisisâ back in the day:
⌠I would like to insert a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem. Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity.
The entire essay is worth a read for some perspective (on new tools / languages / techniques resulting in more issues that continue to humble the profession), if nothing else: E.W.Dijkstra Archive: The Humble Programmer (EWD 340) (mirror).
We donât disagree here. Just if code is not reviewed by somebody who understands but only by an LLM that does prediction it can introduce them is what I was trying to say.
Idk I feel like all projects would get that label. What I find actually way more useful is the way copilot and claude can handle pull requests on GitHub where they are the contributor/author and you can review the code and ask/make changes. It will be per line have a clear answer on git blame :D. Although you should also blame the reviewers of PRs if something goes south.
At the very least, there is a plan to do something about this. This would at least make it relatively easier to create a fork that removes AI-generated commits and perhaps replaces them with human-written equivalents. But overall, yes, it is a serious problem that AI usage isnât properly disclosed in the first place, and having a no-AI label is more of a band-aid solution that will be imperfect, but even a partial solution is imo better than no solution at all.
I have also recently found out about a project to keep track of AI-generated contributions, as well as listing alternatives to software which contains them. This could be useful for me as well as others here who may not want AI slop in their software.
AI coding fits our environment in the same way self-lawyers fit.
Yes, you can look up stuff to argue your case properly. But most people can be expected to do so only while consequences are slim.
Nobody goes to argue their criminal case by themselves. For all the time and effort to self prepare for a criminal case, you could hire a lawyer instead.
Itâs going to be understood that AI coded software cannot be trusted. Youâll get a good script and all, but youâll be paying a programmer to verify it before production about the same amount of man hours as if you hired the programmer in the first place.
And investors are not going to be okay with losing money because the CEO didnât want to pay for proper engineers. Same goes for programming.
You canât AI your way out of tougher problems like is this software secure? Is this database protected?
I think AI security would need to be normalized in todayâs conversation or else we get more hardcoded enviromental variables leaking to the public.
AI (or in other words, algorithmic) driven static analysis has always been a product developers use to catch bugs. The issue now becomes whether the vibecoder in question is aware of basic security practices that used to be drilled into the heads of every freshman CS student.
Personally, I would never use software that is almost 100% vibecoded with no formal security review process, but it is pretty difficult to avoid software with AI-generated snippets or code as companies really want to normalize LLMs into their workflows. All hail efficiency and endless profits /s!
If I come across software that is obviously vibe-coded or has obviously created its website with AI tooling I tend to skip it. I have no issues with âAI-augmentedâ workflows, as a human still ends up being the arbiter for every line of code in the project. In fact, being able to write a blob of code then go âHey Claude, can you review this diff for any security issues?â is really nice.
With that being said, one of the underlying things woven in the fabric of this privacy movement is the building of more ethical & equitable software/platforms. As individuals, we are overwhelmingly propping up small developers/teams and have a vested interest in doing so. Even this forum is a business and makes some money!
We (usually) pay for our e-mail. A lot of us even pay separately for e-mail aliasing. We donate & otherwise contribute to open source projects, probably at a far greater rate than the general public. As a result of the cottage industry weâve inadvertently built, we have extremely smart people coming in to solve really cool, niche problems.
I donât think the same could be built off the backs of AI generated software. We have a vested interest in keeping things human, as Claude has no need for privacy.
