Method for verifying product criteria claims

Site Development
1

The Proton VPN saga illustrated that despite our general exigency for VPNs, we hadn’t properly verified it met our criteria on all platforms.

This could be due to us trusting Proton claims, and verifying in one or two platforms, but not all.

I think in the future we need to be much more careful, and essentially do our own “audit” of new recommendations, and perhaps we should also start checking existing ones.

For another reason on why this is important, see this thread about OpenSuse Tumbleweed not meetiing one of our criteria

Ideas on how we could strengthen the process are welcome.

6 Likes
2

If we start checking existing recommendations then we should either clearly document the underlying OS and steps to reproduce the disqualifying behaviour if possible or it should be based on a credible source (audits, professional security researchers, proof of concept, etc.) which attest that a recommended product has some kind of disqualifying behaviour. If some news outlet posts an article that there are VPN leaks on XYZ or that program ABC has issue DEF we should always have such a credible source or the steps to reproduce ready before thinking about changing anything on the site. This includes not opening a “Remove …” thread on this forum, before you can state in this post on which OS this behaviour happens, if it is by default or by (mis)configuration, what the implications are, the steps to reproduce/credible source and if you tried to/could verify the same behaviour on other platforms.

2 Likes
3

A third-party mentions policy:

For example, here is a no liability statement:

4

That would make recommendations less useful.

2 Likes
5

Recommendations do not become more or less useful with a no liability statement. Instead, despite recommendations being made with or without supporting citations and/or evidence, third-party products/services may introduce changes or experience issues in practice that were not present during the review process, so a no liability statement addresses the limitations of reviews stemming from a live snapshot or living document.

6

Threads calling out tools not meeting a criteria is the audit. Its not new reccomendations that have these issue. The problem is, and always will be, these tools are not static. They change and can evolve into something that then does not meet the criteria.

To me, speed is the big issue. Personally I think focusing on updating the proccess of how these issues are raised such as

would help. I also think a lot of resources are taken up just having to verify user claims. Whether users are right or wrong there is a lack of focus on being evidence based in the thread.

2 Likes