Mozilla's abysmal handling of the eMudhra compromise

TLDR: Some Firefox users on desktop haven’t received the OneCRL revocations. Mozilla knows this and doesn’t seem to care.

I assume most people are aware of the eMudhra Intermediate CA compromise.

An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.

This has already been remediated by Mozilla, Microsoft, Apple, and Chrome for their users through their tools for distributing revocation information such as OneCRL, CRLite and CRLSets.

Mozilla even posted a blog entry boasting about their response:

Mozilla believes everyone has a fundamental right to security and privacy online. We remain committed to defending those rights and acting quickly to protect users.

One would think trusting Mozilla’s statements should be no problem. Any blatant false claim about the “market leader in security” would eventually be noticed, right?

Well, someone noticed, and it isn’t pretty.

Firefox has an external system called Remote Settings that fetches data which needs to be updated regularly without requiring a full browser update. It is used for many things: translation models, wallpapers, overrides for fingerprinting protections, and, most importantly, OneCRL for fast certificate revocations/blocklisting.

For most users this works seamlessly; they receive updates automatically. But what about users on a spotty network, or whose connections are restricted, or who have the entire Remote Settings system blocked? Firefox still ships the OneCRL data with the browser release, so that users aren’t left unprotected. Mozilla claims:

These releases will distribute this protection for a minority of users whose custom settings may have prevented them from receiving the OneCRL update.

That would be fine if it weren’t a blatant lie. The bundled OneCRL data still requires an external sync event to be loaded and that sync may never happen when the user has networking problems or the servers are blocked. The sync also never occurs when Remote Settings is disabled by policy, even though Mozilla’s 158.0.1 release notes claim otherwise:

This update contains the updated OneCRL block list, ensuring clients will block the compromised certificates even if RemoteSettings is disabled by policy or otherwise unavailable in the user’s network environment.

If this were simply a testing mistake, I would give Mozilla the benefit of the doubt. Instead, they clearly know that the bundled OneCRL data needs that external sync. They have already committed a fix, but they decided it isn’t important enough for desktop and will only release it for Android. Desktop users are now potentially left without the revocations for the eMudhra Intermediate CA certificates, even though Mozilla claims the protection is bundled with the build and does not require Remote Settings or external network connections.

It is absolutely inexcusable and makes you wonder whether other parts of Firefox are handled just as shoddily. Shame on you, Mozilla.

3 Likes

I think the statement should be narrower. Desktop users who changed the policy that is literally called “DisableRemoteSettingsAndAcceptSecurityConsequences”.

I wasn’t even aware it existed, and from my understanding it’s used mostly in enterprise setups. I wouldn’t say Firefox left its users — the users in this case are corporate administrators who opted into this setup and whose job is to handle things like this.

but they decided it isn’t important enough for desktop and will only release it for Android

Android doesn’t even have a way to change the RemoteSettings policy. The Android fix is more about the fact that on mobile, the browser often isn’t running long enough to poll for updates, so unlike desktop it makes more sense to check on startup rather than eventually. Good lesson for all of us, by the way.

And again, Firefox for enterprise simply believes desktop isn’t affected by what was fixed for Android in 158.0.2:

1 Like

Did you even read my entire post?

First of all, Mozilla claims in their releases notes that even when Remote Settings is disabled, it would still revoke the certificates, which is a blatant lie. It also claims that you do not need an external network connection for them to load, which is also a lie. If you are on a network which blocks those connections, you never load the revocations. There is also a period of time on new profiles/installs before it ever does the sync, even if everything is working correctly, you are still vulnerable during that window.

Not sure why you are trying to downplay this, when I have clearly provided links that shows that Mozilla is lying.

https://www.reddit.com/r/firefox/comments/1xatekz/comment/19rfd66qp74wx/ to show that this actually is causing issues in the real world.

Edit: https://librewolf.dev/librewolf/issues/issues/3305#issuecomment-21865 which also confirms that Firefox still has not revoked the certs, but my backport has fixed it for them.

1 Like

I did. I’d write it slightly differently to make it easier to follow, but I did.

Not sure why you are trying to downplay this

Because “Mozilla is lying” is an overreaction worth downplaying. No signs of Mozilla intentionally making untrue statements, i.e. lying. We are not a tabloid authors, are we?

screenshot

Eventually, right.

It also claims that you do not need an external network connection for them to load,

When you push an update, it sometimes doesn’t work in all cases as intended. That doesn’t mean someone is lying to you.

Finally, my claim was that “desktop users” was too broad: only opted-in advanced users are affected. Others were never affected to begin with, before 158.0.1 was even released, so the 158.0.1 release note is irrelevant here.

So this is not a lie?

Mozilla claimed you get the revocations bundled with the browser, which clearly is not the case, as the user says that they updated to 158.0.1 and still did not have the revocations. You also ignored all the replies claiming the same issue.

They claimed multiple times that it works for scenarios it doesn’t, fixed the exact issue for Android, but did not bother fixing it for Desktop users. How can you even defend this?

If that is the case, why did the users in the Reddit post I linked not have them revoked already? You do realize there are infinite number of reasons someone might never receive the sync and Remote Settings being disabled is only one of them.

2 Likes

Look, I feel weird arguing with you, because you clearly understand the Firefox code way better than I do. Acknowledging that:

So this is not a lie?

It’s not, it’s a bug. Correct me if I’m wrong, but from what I see comparing 158.0.1 and 158.0.2: they introduced a timestamp check to force a OneCRL update, but that check isn’t triggered when Remote Settings is disabled, while bypassing Remote Settings was the whole point. Looks like a typical bug where a protective mechanism gets added but doesn’t actually fire. Cast the first stone if you’ve never done the same.

Being wrong or introducing a bug does not mean lying.

They claimed multiple times that it works for scenarios it doesn’t, fixed the exact issue for Android, but did not bother fixing it for Desktop users. How can you even defend this?

I’m not defending it or state there is no issue. I’m stating that the affected audience is not desktop users in general, but users who explicitly said they’re ready to deal with the security consequences of their custom setup. Not sure why you want me to repeat that a third time.

If that is the case, why did the users in the Reddit post I linked not have them revoked already?

The fact that the users you’re talking about received the update, as shown by the highlighted comment, and got it gradually (6 certs, then 6 more) means their Remote Settings is not turned off. The 158.0.1 update doesn’t seem to even target those users check next comment

While it looks like a bug after reading the release notes, it could also be intentional if we read code only. For example, if Remote Settings is on but the remote server is unavailable, in that case 158.0.1 seems to update certs anyway, which looks like the main idea behind the code. So it respects user settings and doesn’t forcefully bypass DisableRemoteSettings, which makes sense.

That also could exlpain why 158.0.2 works differently, as there is no DisableRemoteSettings on android. Not because “desktop is not important enough”

It’s possible that bypassing DisableRemoteSettings would break something for users who opted into that setup and don’t expect such behavior. Still, Firefox has a wide audience and has to think carefully about whether changes break things for some of them

Oh just saw this post but, correct me if i am wrong, starting a blank Librewolf profile was able to capture DNS lookups for “firefox.settings.services.mozilla.com”.