ProtonVPN Additional note: Killswitch failure and IP Leakage on Linux

[jonah (Jonah Aragon)]

@Encounter5729

I would agree that this issue in itself obviously doesn’t in itself merits delistment.

I’m going to mark this topic (which is specifically about adding a warning to Proton VPN about this issue) as approved, because I think it is fairly reasonable for us to do, and it seems like the maximum that people in this thread are willing to do about this issue alone.

So, (and I’m of course requesting this of myself or the rest of the team or anyone here, not you specifically) a PR adding this warning would be appreciated.

@Encounter5729

The question is whether the entirety of those issue does IMO.

I think the question of whether we should have additional guidance/warnings about software kill switches in general probably belongs in a different topic or somewhere in Site Development > Guide Suggestions, and I think your question of whether all these minor issues should add up belongs in another thread. I’m not sure if that’s a question for a new thread where we ask this question about services across the site in general and not specifically about Proton VPN exclusively, or if it belongs in How should we handle Proton’s misleading marketing?, but it doesn’t seem to fit here.

[Encounter5729]

@jonah

think the question of whether we should have additional guidance/warnings about software kill switches in general probably belongs in a different topic or

One already exists for that

[ignoramous]

@fiqiluvo.epileto

Qubes approach makes it possible to have a messy user facing OS while having a tight and predictable network VM. With Qubes, in the worst case scenario a user facing VM will loose connection.

We can mimick this by disallowing the host OS to manage wwan and connect to the VPN router directly.

Yes, but these are what amount to ‘software killswitches’, which some here argue aren’t enough. It is my understanding that an external hardware gateway cannot act like a ‘killswitch’ if there’s no corresponding software-related killswitches installed / setup on the devices.

@fiqiluvo.epileto

Like wwan manager instructing the component to connect elsewhere or are implying firmware glitches? In both cases, it can be negated by connecting wired.

Possibly, but I was thinking of mobile networks that bypass “broadband routers”. How do you cut off an esim / mobile data on a smartphone? Very feasible with software killswitches (like turning off the baseband radio, and absent remote backdoors, it would stay off).

---

@jonah

I think we are getting caught up in minor client details here.

I feel, you’re being way too dismissive of the concerns, here. One simply cannot provide client-side guarantees (“hide all traffic from ISPs”, “hide from piracy organizations” like PG does on behalf of the public VPN providers) without client-side software to match. Client details, for VPN providers, for the kind of guarantee they claim, isn’t “minor”.

[jonah (Jonah Aragon)]

@ignoramous

It is my understanding that an external hardware gateway cannot act like a ‘killswitch’ if there’s no corresponding software-related killswitches installed / setup on the devices.

I feel this is true but the point being brought up is that the external device cannot be interfered with by software running locally on your computer, resolving interface binding issues like this. You’d also need kill switch functionality on the router to protect against unintentional disconnects, but this is an easier(?) problem to solve and (probably more importantly) seems to be typically more robust on routers compared to normal operating systems.

[ignoramous]

@jonah

the point being brought up is that the external device cannot be interfered with by software running locally on your computer, resolving interface binding issues like this.

Binding to an interface is the first thing ALL sockets (in the BSD APIs that Linux and BSDs and macOS use) do. If anyone’s ‘(software) killswitch’ is bypassed by any process merely binding to another network interface, then what they’ve setup isn’t really a ‘killswitch’. It is erroneous to call it a ‘killswitch.’

@jonah

You’d also need kill switch functionality on the router to protect against unintentional disconnects, but this is an easier(?) problem to solve and (probably more importantly) seems to be typically more robust on routers compared to normal operating systems

OSes do have a larger attack surface than a router might (especially if it isn’t also running an OS-grade kernel like Linux). That said, non-open source router firmware/software is usually not nearly as scrutinized as Linux is. Besides the upsides (you point out) of physical isolation (of the router and the device), the downside is routers also have no control over the device itself; meaning, a killswitch on the router is going to be far less effective if corresponding software killswitches are not setup properly on the device. All of that is to say, external routers are a defense in depth, but cannot overcome all limitations in/of software killswitches. The threat model that demands such a setup, imo, is out-of-scope for day-to-day use of public VPN providers (aka overkill), but probably in-scope for those using Tor.

For public VPNs, it is enough to demand that VPN clients solely rely on / compatible with the OS-provided / OS-assisted killswitch. I don’t get why there’s a sense here that Linux-based distros don’t provide anyway to setup a leak-proof killswitch (when the Linux kernel does via various sandboxing / filtering / in-kernel mechanisms). Big Cloud Security providers literally rely on these for network observability, incidence monitoring, and management.

[fiqiluvo.epileto]

@ignoramous

I don’t get why there’s a sense here that Linux-based distros don’t provide anyway to setup a leak-proof killswitch (when the Linux kernel does via various sandboxing / filtering / in-kernel mechanisms)

That is, if a user doesn’t supposedly tinker with routing in any way. Even if you run a seemingly innocent software, a leak will occur. Like docker listening on 0.0.0.0, deliberately overriding every other firewall rule, having a misleading -p 8080:80 that had lead to countless users believing it binds to localhost…

Not to mention the fact there are preexisting privileged programs, network managers, that screw VPN setups 1 2 3 all the time. For networkmanager in particular, some distros set ipv*.dns, ipv4.dns-priority to -1 by default for any new connection, making it impossible to maintain wg0 exclusivity without resorting to scripts.

Dnsmasq, systemd, but in most cases multiple privileged components all handle dns caching and forwarding on typical a Linux Desktop OS and usually forward queries elsewhere.

I can make it very comical by reciting a short list of things that do influence resolvconf alone. systemd-resolved with stub-resolved, dhclient, systemd-networkd, resolvconf/openresolv, both musl and glibc NSS, nscd, networkmanager, unbound via systemd or networkmanager or directly, dnsmasq….

Pair this with the fact Debian, Ubuntu and RHEL flavors all heavily drift from upstream and go out of their way to configure networking in the most ridiculous non standardized ways possible, and you have one of the hardest landscape to maintain VPN kill switches.

@ignoramous

Big Cloud Security providers literally rely on these for network observability, incidence monitoring, and management.

I haven’t seen any VPN client utilize caps to limit other privileged apps from interfering with crucial bits of routing rules. As of today, prebundled network daemons contribute to the majority of VPN leakage on Desktop Linux.

Businesses have a privilege of maintaining their SBOM. No k8s/traditional VPS provider/PaaS manages networking in userspace amongst their customers. I just don’t get how we can compare a server environment with desktop chaos. Terraform/cloud-init comes with a static routing on a bridge and customer just can’t screw anything. Devops and SRE engineers i know are allergic to Linux networking and want nothing to do with it unless it’s eBPF

It wasn’t uncommon to see a hardware “VPN“ blackbox being the only certified solution in the Enterprise world because a technician physically couldn’t screw terminating a network into a designated physical port just until 6-8 years ago.

A standalone router with slim and predictable dnsmasq, wireguard and firewall rules is waaay easier to manage and trust than a vpn client on a Desktop Linux OS.

[fiqiluvo.epileto]

I guess we can shift the discussion towards decoupling VPN services and VPN clients. With the existing killswitch criteria, PG couldn’t recommend service providers that rely on censorship evasion protocols as they were historically implemented as proxies due to different proxying strategy needs. Various projects began experimenting with gVisor tun stacking, TProxy by the means of netfilters and eBPF recently and its possible to make a “killswitch” by routing all traffic through it, but this isn’t available out of the box.

That is to say users do have different censorship evasion strategies across the globe and typically route traffic using complex rules to avoid VPN usage discovery by the adversary.

Some of the best VPN providers in China invest in vendor agnostic solutions instead of developing their own clients. As far as my reading of the criteria goes, PG can’t recommend a provider that doesn’t develop their own client (think of wasted duplicated hours of work) that implements a specific routing strategy.

Proton’s VPN clients are messy, but their VPN servers are quite bulletproof and are competitive with what Mullvad or iVPN offers. Mullvad might be the coolest kid on the block, but their censorship evasion protocol implementations (QUIC, LWO) aren’t vendor agnostic while iVPN offers universally accessible VLESS and so on.

[dngray (Daniel Gray)]

@jonah

@ignoramous

It is my understanding that an external hardware gateway cannot act like a ‘killswitch’ if there’s no corresponding software-related killswitches installed / setup on the devices.

@ignoramous

OSes do have a larger attack surface than a router might (especially if it isn’t also running an OS-grade kernel like Linux). That said, non-open source router firmware/software is usually not nearly as scrutinized as Linux is.

I feel this is true but the point being brought up is that the external device cannot be interfered with by software running locally on your computer, resolving interface binding issues like this. You’d also need kill switch functionality on the router to protect against unintentional disconnects, but this is an easier(?) problem to solve and (probably more importantly) seems to be typically more robust on routers compared to normal operating systems.

I think you’re all missing the main point, the hardware on a router is fundamentally different in the way it works. It has multiple network interfaces ie IN and OUT. The VPN tunnel is just another exit, and can be treated as another OUT interface. This makes developing a ruleset a lot easier.

The actual router itself doesn’t have a kill switch, it is allowed to route to wherever it likes, and that’s okay because the router doesn’t generate much traffic by itself. It might check an NTP server or system update server but that’s about it.

The decision logic then comes down to “if the packet comes in on this interface” then “i put it OUT on XXX”. A rule then will then select on the IN interface as well as VLAN ID.

The VLAN ID can either be tagged on the device (advanced network setting) or by the port on the switch, or WLAN access point. The packet has already left the macbook so the VLAN tag cannot be stripped by any software running on it.

The problem when you try build a kill switch into an application on the client using it is, in the case of rule based firewalls, multiple things might mess with it. The VPN software has no way to know.

The second issue is that still has to be exceptions, a kill switch must allow access to the VPN provider’s API servers otherwise when a user logs in it, won’t be able to pull the latest wireguard configuration files (those have the keys in them).

The third issue is that because there’s exceptions and nothing really enforcing anything besides the firewall should something change, the kill switch might simply be broken. That is also where you get issues with software directly binding to an interface its not supposed to.

A fourth issue is that when changing servers I doubt any of these clients actually clear out all existing established connections from the routing table. That I haven’t tested.

The TLDR of that is there’s many more variables and things that can go awry, that its really difficult to test for.

What it does sound like is Proton is developing a method which based on the description:

@Proton_Team

What we are doing instead is baking the protections into the network stack itself to ensure that the protected tunnel must always be retained even when the connection is interrupted - either accidentally, or during server switches. This new cross-platform network architecture was the main thing that we were working on during our Autumn/Winter roadmap cycle

is actually the correct way to do this. It means it will work on iOS properly too. I would not be at all surprised if Apple is not involved behind the scenes, as there may have needed to be changes on their end as well - that is speculation, we really can’t know until we see it.

[ignoramous]

@dngray

think you’re all missing the main point

@fiqiluvo.epileto

A standalone router with slim and predictable dnsmasq, wireguard and firewall rules is waaay easier to manage and trust than a vpn client on a Desktop Linux OS.

@fiqiluvo.epileto

I haven’t seen any VPN client utilize caps to limit other privileged apps from interfering with crucial bits of routing rules. As of today, prebundled network daemons contribute to the majority of VPN leakage on Desktop Linux.

Both of you are right; just that I don’t agree that an external gateway is the only acceptable solution to on-device ‘software killswitch’ implementation limitations (in the context of public VPN providers and the threat model they operate under). For Tor, sure.

@fiqiluvo.epileto

hardest landscape to maintain VPN kill switches.

Well, any VPN provider that wants to put money where its marketing is (“hide from ISPs”) should figure this out, instead of putting the burden on the end-user.

@dngray

The decision logic then comes down to “if the packet comes in on this interface” then “i put it OUT on XXX”.

You may know this already, but I must point out that on Linuxes and BSD-based OSes like macOS, WireGuard VPNs are (expected to be) setup exactly like this (only difference is that the “interface” is virtual not physical).

@dngray

The packet has already left the macbook so the VLAN tag cannot be stripped by any software running on it.

As @fiqiluvo.epileto notes, Linux supports such a setup?

@dngray

in the case of rule based firewalls, multiple things might mess with it. The VPN software has no way to know.

The “VPN software”…

• doesn’t need to “know” if the OS provides an API to setup a ‘killswitch’.
• will have to inevitably use firewall functionality if there’s no such API (ie, it will have to “know”).

Hence the point here… What should we require of VPN providers on macOS? - #69 by jonah

@dngray

… wireguard configuration files (those have the keys in them).

Not sure why keys are relevant here? Anyhow, I must note that these keys (being sent around) are not (supposed to be) private/secret.

@dngray

That is also where you get issues with software directly binding to an interface its not supposed to.

There’s nothing “not supposed to” here. Sockets, as the BSD APIs are, will have to bind (before connect / send) to some interface (or all interfaces)?

A “killswitch” that cannot deal with this normal / very usual scenario cannot be called a “killswitch”.

@dngray

A fourth issue is that when changing servers I doubt any of these clients actually clear out all existing established connections from the routing table.

tbf, the issues you are pointing out are implementation considerations (of which there are likely to be many more than just four), not limitations.

@dngray

is actually the correct way to do this.

I am not sure. Proton’s post hasn’t gone into any technical details for me to make any judgement call.

@dngray

that is speculation

@Overall-Bet3743 pulled in an Apple engineer’s post that helped me in getting clarity on much of the speculative bits (cc: @ph00lt0 too) from the previous thread … that speculation may not have been entirely correct: What should we require of VPN providers on macOS? - #77 by Overall-Bet3743

[dngray (Daniel Gray)]

@ignoramous

There’s nothing “not supposed to” here. Sockets, as the BSD APIs are, will have to bind (before connect / send) to some interface (or all interfaces)?

I meant in the case of some miss configured software.

@ignoramous

tbf, the issues you are pointing out are implementation considerations (of which there are likely to be many more than just four), not limitations.

Probably, those were just the ones I could think of.

@ignoramous

@dngray

is actually the correct way to do this.

I am not sure.

Proton’s post hasn’t gone into any technical details for me to make any judgement call.

If I were to guess I’d say there was some collaboration with Apple. What they described was:

@Proton_Team

baking the protections into the network stack itself to ensure that the protected tunnel must always be retained even when the connection is interrupted - either accidentally, or during server switches

Baking anything in that spot sounds pretty privileged. They also stated they want it to essentially work rootless by stating that they want it to work on iOS. If it was an easy solution it would have been done.

Apple has all sorts of things like System Integrity Protection (SIP/CSR), AMFI which probably prevent you from really installing anything like that and/or using privileged kernel modules etc. That is what they’re clearly referring to by this comment:

@Proton_Team

An alternative used by other VPNs involves a tunnel interface set up by a daemon that runs without a sandbox on the user’s machine with elevated privileges. We did consider this solution, but rejected it. The legacy APIs that are used are ones that Apple has heavily signalled that they intend to deprecate, so this was not going to be a long-term, reliable solution.

They know any method that worked around the sandbox would never make it into the Appstore as that requires AMFI Compliance and sandboxing.

[lyricism]

@dngray

If I were to guess I’d say there was some collaboration with Apple. What they described was:

@Proton_Team

baking the protections into the network stack itself to ensure that the protected tunnel must always be retained even when the connection is interrupted - either accidentally, or during server switches

There is literally zero chance Apple is working with Proton on modifications to the macOS network stack. Much more likely to just be poor wording.

[dngray (Daniel Gray)]

I don’t know about that,

@lyricism

literally zero chance Apple is working with Proton on modifications to the macOS network stack

It could be something to do with Network Extensions, Apple does say quite clearly in TN3165. “If your product needs to do something that’s not covered by one of these providers, use Feedback Assistant to let us know what’s missing.” so I would not rule it out.

If there are any changes they won’t be Proton specific, and likely need further testing, and proper design. Things like that can’t be allowed to be bad.

[lyricism]

I mean Proton has already clarified and confirmed they were talking about their “network stack”, so no, it was just poor wording.

@Proton_Team

We’ve been rebuilding our network stack with a native WireGuard implementation, developed in-house, that includes a native kill switch maintaining the tunnel through server switches

[dngray (Daniel Gray)]

I guess that’s possibly, true but doesn’t really explain much about how a kill switch would function any better, unless these “deeper changes” has something to do with a Network Extension.

[lyricism]

I mean writing a Network Extension isn’t changing the system network stack and doesn’t require any assistance from Apple, so even if there is a Network Extension involved that doesn’t mean they’re working with Apple to change macOS components.

I’m also unsure how using a different wireguard implementation makes their kill switch any better, but I think assuming that means they are working with Apple to change macOS’s networking stack is a bit of a leap.

I think either way beyond just the wireguard implementation and the apparent fact that they didn’t mean the OS network stack discussing anything additional they might be doing is pure speculation and not worth going back and forth about.

[dngray (Daniel Gray)]

@lyricism

working with Apple to change macOS’s networking stack is a bit of a leap.

Not so much that, but adding something that is missing, not change macOS in any signficant way though.

Given they changed over to this method in 2019 with the relaunch of their app and it’s taken considerable time, I would expect there is more to it. There is not really enough specifics, but you can get from there replies there will be some immutable requirements

1. It doesn’t rely on pf, (that’s not on iOS), nor is root
2. It doesn’t get around the sandbox in any way, which would be required to be in Appstore.
3. They will most certainly want to not have a massively different implementation on macOS to iOS.

[fiqiluvo.epileto]

@Proton_Team

A scenario highlighted in this discussion involves an advanced user manually binding an application with the appropriate privileges to a physical network interface. Such bindings intentionally bypass the system routing table and therefore bypass any routing-based firewall or kill-switch mechanism.

[byte]

Actually, i don’t know why do you need to use app itself.

I moved to native NetworkManager client (especially perfectly working on “user friendly” distros).

This eliminates many issues (including this), because when using native tool, you can draft a FW rules yourself.

On other systems there is reason to use app, on Linux - there is already a built-in tool for OpenVPN and Wireguard.