This is Post/Suggestion 4: Auditing in General and Rationale for AirVPN and Cryptostorm not Having Audits
In my previous three posts, I talk about various criteria/options I think should be added to Privacy Guide, including anonymous payments/registration, and including some VPN options that have port forwarding and that support both OpenVPN and Wireguard. I come up with Cryptostorm and Air. However, it’s been shot down before because of a lack of audits AirVPN (VPN Services) - #4 by anon28734771 . But, as has been discussed in various circles online, auditing doesn’t guarantee anything because 1.) there’s nothing stopping a VPN service from violating its own logging policies as soon as an audit is over or before one begins; 2.) there’s LOTS of VPNs that have audits that you won’t find on any privacy list like NordVPN, SurfShark, etc. (not that these are bad, but they don’t scream privacy and anonymity rights, either), meaning audits don’t add any sort of credibility that isn’t added by having good-faith security and anonymity criteria in the first place; Everything else equal, is a VPN with 100% anonymous registrations and payments more or less likely to have logs than one that doesn’t? I think that offers more credibility that one wouldn’t log because what would be the point vs. if you’re “audited” to have no logs but aren’t as privacy/anonymity focused like a Nord (or even a Proton VPN) compared to those on the other extreme end of the security/anonymity spectrum such as Cryptostorm (which is even advertised for the security/anonymity “paranoid”) or AirVPN.
Therefore, I think having auditing as a requirement to forbid adding other VPNs, even those with very good privacy/anonymity standards like CryptoStorm and AirVPN (among others, see other posts), shouldn’t be held against them.
I do think there is some value in evaluating if audits are worthy as a minimum criteria for VPNs given the seemingly majority view that audits do not guarantee anything and, that most people cannot understand the audits findings (outside of reading the summary). Plus there is no criteria to evaluate the auditor (which also vary in quality).
We do this though, just to be clear, as well as evaluate the audit’s scope because some are very limited.
However, I agree it might be a problem that this isn’t done with strict definitions, and is moreso done by gathering consensus about how the community feels about a specific audit based on posts here, and by various team members & volunteers reading the audits and sharing their thoughts internally and here on the forum. I don’t know how to define a better process here personally, but yes it is not the most transparent.
Audits don’t just ensure no logs at the time. They also search for security holes and bugs. This is of major value when considering the safety of a system. It’s also the main reason why audits are so expensive.
Sorry, I don’t agree that these VPNs you mention (that I personally have never heard of) should be added. They need to clear the proper criteria.
Also, any list that recommends NordVPN for privacy purposes should not be taken seriously.
Another option I will bring up in this discussion:
If we feel like audits are somewhat important, but not strictly necessary, we could move audits to the best-case scenario criteria and prioritize providers that have them, but keep the door open to listing other providers as well.
This gets us dangerously close to “worth mentioning” territory which a lot of people here and on the team want to avoid on the site, but it is not completely unprecedented and definitely could be done in very specific categories like VPN providers, so I wouldn’t be opposed to this.
I am interested to know what everyone thinks about this possibility.
For some context, we previously did not have many criteria when it comes to trustworthiness and only really focused on technical capabilities, but in 2019 we added requirements for audits and public-facing leadership that were not really meant to be technical criteria, but criteria for trustworthiness and commitment to privacy, based on a fairly extensive VPN review performed by Wirecutter at the time.
Third-party security audits aren’t a guarantee that a VPN has no security flaws, but they are a sign of trustworthiness, especially if the reports are easily accessible to the public and outside security experts.
That is true, but I would argue the main reason why people historically want audits is to verify logs. But yes, that is true. Assuming the implementation/back end of the VPN is minimal and/or open-source (or at least forked from a generally trusted/open source one), how high the actual chance there is one is debatable. Wireguard/OpenVPN is doing much of the heavy lifting. I suppose auditing can provide at least some value in this regard, but whether that should be the value by which to negate an option if it does not is shaky to me. As in, I don’t think it should be a requirement.
Yeah, I’m not saying they shouldn’t be held to a similar standard. In fact, that’s precisely what I’m proposing.
I recommended NordVPN? I suggest you reread what I said.
I like this idea. I think it depends what the “worth mentioning” actually sacrifices. I don’t think denying an exemplar (for any category, not just VPNs) on the basis that it doesn’t meet a single criteria is apt UNLESS that criteria is determined to be “essential.” (And I’m assuming most of the current criteria fall under this category, but like you said, if some need to be tweaked. )Meaning, I think the criteria ought to be graded on importance. Maybe “criteria” as essential vs. maybe “guidelines” or “suggestions” as non-essential. Something like “recommendations” feels too in the direction of being too willing to compromise your own criteria a la the “worth mentioning” concept. But something like guidelines or suggestions I think qualifies that it doesn’t actually compromise the important criteria and just offer more options.
Also consider for example on the topic of port-forwarding: In order to be secure as possible, it’s recommended to keep all ports you don’t need open to be closed. So a “guideline” could be to, if you don’t know what you need, use a VPN that doesn’t have port forwarding. However, as this may compromise your ability to use certain applications or games or connect to some networks/in some countries, it is not a “criteria” that a VPN has to not have port forwarding, perhaps only a “guideline.” Perhaps the same thing with OpenVPN, etc.
Btw, you can just highlight another person’s comment or a portion of it and click “Quote” on the pop up that appears - you don’t have to or need to add the quotations like you are in these comments/posts. It’s a little jarring to read it like that.
FYI in case you didn’t know how others are doing it.
Well this is what we currently do, is say that every minimum criteria is essential. But if we move audits from mininum to best-case criteria it would be no longer essential.
Port forwarding is an example of a current best-case criteria we have. Mullvad doesn’t have port forwarding, but that does not prevent it from being listed. However, we pretty prominently call them out on not having it.
To me, an equally important part of why audits are generally done (to prove their proper implementation of the service and to find issues with it) is that a VPN company is serious about their product, believe in their product and its legitimacy, and spend their funds on their product first before asking others to trust and use their products for a long time. It’s a thing I value equally but more as a gesture. Perhaps it is a less stringent way to look at it.
I think that is what I was saying was the reason we added it as a criteria in the first place (trustworthiness indicator), but if that wasn’t clear then yes, I definitely agree
I am all for this, which i think was the original intent of this topic. Kind of wild how things have changed on this topic since then, as OP was grilled pretty hard by staff, having the idea called “mad” and being asked if there was an ulterior motive.
I personally don’t see audits as having enough value added to being a minimum requirement (ie it should not be used to discount other VPN options on that basis alone) but its also not nothing.
I think this is the problem with audits as a criteria in general. The process around audits and who performs them are very nebulous to evaluate objectively. Which I think makes it a tough thing to be a minimum requirement.
Most users of PG are not going to understand the technical side of the audit, know anything about the auditor, or how to differentiate a good audit from one purchased for marketing. That to me seems like a slippery slope.
That may be true. But it also doesn’t mean audits don’t establish a sense of trust and transparency like what I implied in my comment above. They do.
I agree, just because PG officially may not include it in their list doesn’t mean they are discounting or disparaging it. It just didn’t make the cut, as good as the service or product may be.
–
I do see what you’re saying but audits are the cherry on top. Not all sundae’s have them or need to but it does round off the desert well. And like I said, having audits is a fantastic gesture for the most part for most people that may end up tipping the scale toward one over the other (as it shows the seriousness with which a company takes its service and also shows their confidence in their claims)
Are we agreeing? This sounds like an argument for making audits as a best-case instead of a minimum requirement. Its a nice to have on a sundae, but if its missing it doesn’t stop you from ordering one.
If you can’t understand what you are trusting, that’s just blind faith. I don’t think that’s what we should be encouraging. I would also point out there is already a trust criteria.
People keep bringing up audits as an example of adding trust but, this is a security criteria. Trust is already evaluated with its own criteria.
Well, it’s both trust and security. The other thing is that security audits can (potentially) validate the competency of people doing the security work. I think people underestimate how easy it would be for a no logging provider to (truly) accidentally log some data, and so having someone from the outside take a look is valuable from a security perspective.
We could arguably list public security audits under either section of the criteria, and maybe it would be a good idea to move it to trust to avoid confusion, and also hammer down the idea that audits are kind of that company putting their money where their mouth is in some way, which is a trustworthiness factor.
I think the idea is that we just wouldn’t list audits that we don’t feel meet these criteria, so I don’t think potential user confusion is a huge concern, although there is certainly some room for confusion since we are also telling people to look for audits on their own.
Well I think the context is important, because at the time there were not many legitimate complaints about the providers we recommended, and having 3 already seemed like a lot. However, more legitimate complaints about the 3 have begun cropping up as their services have changed over time, particularly with port forwarding.
Totally agree. I am not trying to say there isn’t value in audits, I just question if its worthy of being a min requirement.
I feel people take this view when the criteria is questioned but whenever audits seemingly come up in other threads, it seems people tend to use the GrapheneOS quote to dismiss their value.
Auditing and code review cannot be done properly as a one time thing but rather need to be done continuously as the code changes.
In that vein, it might be worthwhile to add in a qualifier like “annual audits”. At this point its a bit vague of a requirement of “published audits”.
Published security audits from a reputable third-party firm.
My nit-pick would be, even on the PG side, there does not seem to be much transparency about how that is evaluated. Its just “the team evaluated it”.
Fair enough. To me, looking back at that topic it seems like people were a bit harsh to OP on something that doesn’t seem like that “mad” of an idea, then or now. It was surprising to me that the first comment is from you and you immediately question the users motives. To your point though, I don’t have the full context.
