SSH backdoored by xz compression library in Fedora 40 Beta, Fedora Rawhide, Debian Sid, openSUSE Tumbleweed (+ likely other distros)

Yesterday, Red Hat Information Risk and Security and Red Hat Product Security learned that the latest versions of the “xz” tools and libraries contain malicious code that appears to be intended to allow unauthorized access. Specifically, this code is present in versions 5.6.0 and 5.6.1 of the libraries. Fedora Linux 40 users may have received version 5.6.0, depending on the timing of system updates. Fedora Rawhide users may have received version 5.6.0 or 5.6.1. This vulnerability was assigned CVE-2024-3094.

PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity. Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be redeployed. Note that Fedora Rawhide is the development distribution of Fedora Linux, and serves as the basis for future Fedora Linux builds (in this case, the yet-to-be-released Fedora Linux 41).

At this time the Fedora Linux 40 builds have not been shown to be compromised. We believe the malicious code injection did not take effect in these builds. However, Fedora Linux 40 users should still downgrade to a 5.4 build to be safe. An update that reverts xz to 5.4.x has recently been published and is becoming available to Fedora Linux 40 users through the normal update system. Concerned users can force the update by following the instructions at FEDORA-2024-d02c7bb266 — unspecified update for perl-Compress-Raw-Lzma and xz — Fedora Updates System.

The fix can be found here:
https://bodhi.fedoraproject.org/updates/FEDORA-2024-d02c7bb266

sudo dnf upgrade --refresh --advisory=FEDORA-2024-d02c7bb266

Other leading edge, or rolling distro are probably/possibly affected as well. If you use OpenSUSE Tumbleweed, Arch, etc, you should check to see if you are affected.

4 Likes

Arch was probably not affected but they recommend upgrading to latest version nonetheless to be entirely sure that you aren’t affected: [ASA-202403-1] xz: arbitrary code execution - Arch Linux

1 Like

This appears to be the original discovery/disclosure of the malicious code, with some preliminary analysis.

I see Debian and Ubuntu have both taken steps to address this in their development branches as well.

Hacker News has a good bit of discussion on the topic as well

I think it’s important to note that the only reason we can safely avoid this issue today is because Linux is open-source. Some non-security developer doing some non-security-related benchmarks led him down a rabbit hole of code checks which would have only been possible with an open-source codebase like this.

The simple truth of the matter is that had a backdoor like this (likely of nation-state origin tbh) been added to Windows, macOS, iOS, etc. it would never have been caught in this manner. And no, it is not “easier” to add a backdoor like this to an open source project: If this person could have infiltrated an open-source project for 2+ years just to add a backdoor now, they could have just as easily got hired at Microsoft/Apple/etc. to do the same thing. Intelligence agencies can and will do all of those things because they can afford to make long-term plays like that: You bet this is happening with all systems, and yet it was uncovered here :slight_smile:

Obviously Linux teams will need to evaluate why this code made it this far past regular checks, etc., but I’d consider this discovery to be a pretty big W for the Linux/open-source community overall.

12 Likes

As some articles mentioned: if the backdoor devs hadn’t been sloppy implementing their script (the dev that found it noticed his SSH connections were unusually slow) it would have probably gone unnoticed for quite a bit longer.

1 Like

Also main discussion thread on HN where this was.

2 Likes

https://www.openwall.com/lists/oss-security/2024/03/29/4

https://twitter.com/thegrugq/status/1773827035976405003

Edit 1 -

FAQ on the xz-utils backdoor

1 Like

Updated topic title just to reflect that this isn’t Fedora-specific.

1 Like

https://twitter.com/GrapheneOS/status/1773851003063853146

There was a recent pull request for the Linux kernel officially listing the person who added the backdoor as a maintainer with a bunch of changes. It’s currently in linux-next.
[PATCH 01/11] MAINTAINERS: Add XZ Embedded maintainers - Lasse Collin

It is also scary even though it’s not this person submitting it since it implies they’ve already been working on the Linux kernel code as a trusted maintainer too. It’s not listing them as a new maintainer but rather documenting that they are already one.

1 Like

Thanks for that

From what I’ve read it seems that the version found to be malicious was present in at least these distros:

  • Fedora
    • Rawhide
    • 41
    • 40
  • OpenSUSE Tumbleweed
  • Arch
  • Debian
    • Unstable/Sid
    • Testing
  • Ubuntu 24.04 (pre-release)
  • NixOS (unstable)

All of these distros did have, at least briefly, the version of XZ which was found to be malicious. But it is unclear if every distro on the list shipped aversion that was exploitable/vulnerable, the belief is that some distros that had the version found to be malicious were not affected by it. (I don’t quite understand what is meant by that, but that is what is being reported).

Only distros which patched sshd to add systemd support were impacted, so Arch was not impacted by this specific SSH vulnerability, even though they shipped the compromised package.

If the distribution built the package from the git repo then it was unaffected. If it used the tarball on the downloads page then it was.

As far as I know the only distribution there which actually shipped the package was Archlinux

I suppose it’s possible Tumbleweed did as well. The other distributions are frozen so it would have only been in unstable/not final releases. (Fedora 40 for example isn’t currently yet released).

You can download Fedora 40 beta builds, they were released 3 days ago.

1 Like

That we know of. This maintainer is responsible for over 750 commits in a timeframe measured in years. There could be other backdoors or malicious code hiding in xz, though not necessarily related to SSH.

As an Arch user myself, I’m concerned, but can only wait for more information. My Mac has also been impacted due to installing a newer xz package as a dependency several days ago in homebrew. I immediately downgraded it and removed the old versions as suggested here: brew install xz installs the outdated version 5.4.6 instead of 5.6.1 · Homebrew · Discussion #5243 · GitHub

To be clear, what Arch has done is not downgrade to a known previously good version before the bad actor took over a year ago. They have simply decided to build xz 5.6.1 directly from the github repository instead of using the maliciously-prepared tarballs: upgpkg: 5.6.1-2 (88138575) · Commits · Arch Linux / Packaging / Packages / xz · GitLab

True, but it is in beta/soon to be released. Many people in the Fedora community are already using it (myself included). Likewise, with Debian, while testing and unstable are technically development branches, tons of desktop users and hobbyists prefer those versions to stable.

@dngray as far as actually released releases go, Kali Linux, openSUSE Tumbleweed, and Arch are the three I know of so far that shipped the backdoored package, and the first two also shipped impacted SSH packages.

But yes plenty of people use Debian Sid, and probably many use Fedora 40 too given the official beta release.

Lucky i main linux mint xfce which is based on Ubuntu 22.04 and not recommemded by PG :grin: dodged that bullet.

2 Likes

https://twitter.com/hackerfantastic/status/1773864354439417983

The author of the ‘xz’ backdoor commit history and activity shows that they kept office hours mostly. Mon-Fri, every other Saturday, I would imagine some of these would correlate with public holidays as this was clearly not a hobbyist.
JiaT75 (Jia Tan) · GitHub

On the one hand, it is harder because if you are sloppy, people will find it, like they did here. But on the other, many open source projects have sloppy QA (if they even have it) that just waves through changes if it doesn’t break the build, which kinda makes it easier if you’re smart about it.

as far as actually released releases go, Kali Linux, openSUSE Tumbleweed, and Arch are the three I know of so far that shipped the backdoored package, and the first two also shipped impacted SSH packages

It is also on Nix unstable. Affected package was on my system. Discussion at Revert "xz: 5.4.6 -> 5.6.1" by mweinelt · Pull Request #300028 · NixOS/nixpkgs · GitHub
Appears to only “activate” on debian-based systems though, but who knows what else is hiding given the commit history.

1 Like