A new comparaison draft

Context

Continuing the discussion from 2026 Instant Messenger Tier List.
The video made a lot of people react to it which could be good but most of the time adds confusion or false belief.
As a comparison summary, tier lists have become popular for their simplicity.
After watching the video and reading the posts, here are my impressions.

Problems

  1. Tier lists are highly subjective
  2. Simplicity and completeness aren’t best friends
  3. Comparisons need methodology, completeness and sources
  4. Every day or so, users ask to add another app/tool
  5. Team of PG (Privacy Guides) lack resources to fully address points 3 & 4

Solution

A mix of Wikipedia and a comparative table like this to enable traceable open contributions that are simple and fast to make; while centralizing and clarify information to let people choose what’s best for them based on facts.

Personal thoughts

First thing I notice was that there is no standard on which to test messaging apps compared to browsers (see “Solution” section example of comparative table) that all depend on the same standards (Javascript API, HTTP headers, DNS queries, …). It would be impossible to create a similar comparison in that sense. BUT, what can be done is a scenario comparison.

Second thing, while looking at the 9 listed threats on PG, each of them needs user data to succeed even if they don’t all aim for data retrieval (ransomware, infrastructure deterioration, …). This means that we should focus on data.

So I thought to compare, for each functionality in messaging app, which data is necessary and to whom it is shared. This is the “completeness”.
This implies that we should define what is a messaging app (what functionalities enable peoples to exchange) and bound this comparison to the main players that are parts of the process.
In that sense, features like usage of Tor that can increase privacy should be reserved for another comparison as it is not a “messaging app” functionality (even if some apps offer it) and because it can be configured outside the app while still providing what it intends to. Following this example, yes ISPs play a role and may gather data but as for any other usage needing internet access so it shouldn’t be part of this comparison. This is the “methodology”.

Prototype

Tried to prototype it, thought to be a 5 minutes hop… Fast-forward many days, tears and sweat: here it is! (not accessible via Tor)

What’s next

Some thoughts on what could be improved:

  • Because everything is a question of trust (I was fine before realizing it, damn it! See this list of assumptions as an example), maybe a “trust indicator” should be shown for the entities able to access user data. No idea how to gather the data, how to weight it or what the indicator should look like.
  • Metadata are data and should be part of the comparison.
  • While trying to keep it simple, those comparisons can quickly become hard to read. To be frank, this prototype isn’t a good way to compare apps because it doesn’t present multiple apps side by side. It could still be useful for completeness and sources management in order to help synthesize information. What could be done to inform people nicely would be two tables. One for the mandatory data usage of each app with their default settings and another one but with hardened settings. Example:

Insert toggle between default settings and hardened settings table

Default settings comparison table

Data Signal SimpleX Briar …
Phone number . . . …
… . . . …

One caveat being having to define what are, for each app, default and hardened settings? (example: is Signal phone numberless registration a default or a hardened setting?) This could reintroduce debates and thus undermine the point of the proposed solution.

Disclaimers

Multiple things to keep in mind:

  • I’m not a specialist in security nor a jurist nor a dev…
  • …so, information gathered aren’t complete or 100% factual nor the app is coded, or the data arranged in the best possible way
  • Everything is hand-made.

Conclusion

This is a draft that should serve as a base for a larger discussion. Indeed, messaging apps are just one type of tool but this could be applied to VPNs, OSes, …
But more importantly, it asks how comparisons made on PG could be improved.
Last point of “What’s next” is, at the time of publishing this topic, where I end up being.

Well said! And I agree with your points.

The problem with tier lists in general are that they are one-dimensional ranking systems and therefore cannot account for differences in rank based on threat model. Tier lists (must) assume a singular, generalized threat model to base their rankings on (to be coherent). The issue, therefore, is that multiple people have different threat models, and no singular tier list will be of help to all people!

However, it’s quite clear as day that Internet videos are marketed towards beginners, which are those who likely have no explicit threat model. They are not supposed to be of help to “all people”. Tier lists assume not just any singular, generalized threat model, but one that most people are likely to adopt. And they serve not necessarily as educational videos on the technical aspects of privsec or the evaluation of tools based on one’s threat model. Rather, they socialize the beginner privacy-interested individual into the social norms of what we dub the privacy community. They allow beginners to internalize what the community accepts as generally acceptable beliefs. It is generally acceptable to believe, for example, that Signal is better than Discord, which the video reflects.

What you propose should not be a replacement for these kinds of Internet videos. In spite of their flaws, they have a clear purpose: attract audiences to the Privacy Guides organization.

This does not mean your project should be abandoned. On the contrary, I hope others will join in your efforts and that this will become as informative and useful as PrivacyTests.org.

I was initially going to direct you to the community wikis page so that you could port your project to markdown. However, I think the website you used has a lot more functionality and therefore cannot be ported.

You’d probably like this project

https://softwarecompare.org/messaging-apps

Thank you for your reply and your encouragement!

Indeed, everyone has a different threat model. But as I wrote, aren’t we all primarily trying to protect our data?

Sorry if I wasn’t clear, but this topic is not about whether:

  • tier list should be used or not
  • PG’s video was good or bad
  • this draft should or shouldn’t be a replacement for this kind of videos (it’s not)

What I meant was that a lot of people reacted to the video which is good as it created discussions and means that people were interested in the subject. But I also meant after reading comments (here on this forum and not only on the video topic) that people:

  • were contradicting each others
  • asking why this or that app wasn’t part of the comparative

So, as a user, this is adding confusion and makes decision harder. In the end, the Problems raised aren’t about the video but how, as a group of people, we can gather, verify and present information so that these guides don’t make any place for interpretation, lack of trust or incompleteness. And therefore help people, beginners or not.

Indeed, a markdown port isn’t possible as is, but that’s ok, it’s not something that can’t change! The Prototype is only one representation of what the Solution may be.

Not related to this topic

it’s quite clear as day that Internet videos are marketed towards beginners

In my opinion, this is a generalization and examples could be opposed to it.

Well I guess all of it was for nothing :upside_down_face:

More seriously, I will edit the topic to add those information. Than you!

It’s been awhile since I’ve worked on Software Compare, but I’m definitely down to update the site if you’re interested, I’m sure some info needs to be changed (I have a test branch with a lot of changes like an email provider comparison)

Ok so apparently it’s not possible to edit our own topic after some times?

EDIT

Existing solutions

Other projects offering their comparisons:

  1. A new comparaison draft - #4 by Tally
  2. Secure Messaging Apps Comparison | Privacy Matters
  3. Comparison of Instant Messengers

All of them are good tools but don’t fully address mentioned problems as they:

  • lack the easiness of Wikipedia contributions as they all rely on GitHub and thus needs accounts. (which adds friction)
  • lack the decentralization of Wikipedia contributions as they all have a mandatory reviewing process that depends on small teams (which put too much work on them) and that slows updates.
  • all have gathered information that other could have reused instead of doing it all over again.
  • lacks “publicity”.
  • aren’t “complete”. (Full completeness isn’t achievable though)

The EFF also developed their tool but abandoned it.
Rtings made one for VPNs.
The list goes on and on and while they don’t suffer from all the above-mentioned points, they still face some limitations.

Conclusion

What if the information were gathered in a single place and accessible for each site to presents them as wanted?

The draft could be reformulated with two questions:

  1. How to federate players inside and outside security related spaces on a shared database?
  2. How PG can use and present data from the database in a meaningful way to its users?

We could probably add a “report incorrect data” button to Software Compare

This is a reply to your first and second comments.

Before anything else, thank you for having created Software Compare (SC), working on privacy-oriented projects and offering me to participate in SC.

Then, I hope that you didn’t take personally the remarks I’ve made in the precedent post nor that you will take personally the ones from this post but as written, SC in its current form can’t tackle the issues mentioned.

This being said, SC’s modifications shouldn’t be discussed here unless PG’s team is ok with using SC as one of their data source, but I can’t talk for them.

It might also be worth adding a date/version of the product/service that is being compared, similar to the way https://privacytests.org/ has it.