I just got a chance to read this blog post, and I love it so I wanted to share in case someone missed it. It really sums up what I have been saying about these sorts of sites in offhand comments myself for a while, but @soatok actually takes the time to write down his thoughts in detail which is far more useful
I literally just yesterday saw someone (on YouTube, not here) point to a useless āVPN comparison tableā as an example of someone else supposedly taking the time to āthoroughly testā and be āobjectiveā about VPNs lol
It also makes me glad we (Privacy Guides) never pursued this whole making-a-checklist thing. At some point I wanted to do sort of a comparison table based on our recommendations (which to be fair is the opposite of what most people do: make a comparison table and then draw their recommendations from it), because people fucking love their comparison tables and I knew itād be shared widely. Unfortunately, Privacy Guides is still a very small community compared to others in this space, and I always want to change that.
Ultimately, I could not come up with a way to make one that communicates our recommendations in a way thatās both accurate and doesnāt appear deceptive. If thereās one thing crappy companies are good at, itās checking boxes (and getting people accustomed to expect certain boxes via marketing, as he alludes to with the ājurisdictionā point in the post).
One final thing I will add. I do think there is one exception: when a comparison table compares a single metric. Hereās a beautiful example of this from @yaelwrites, where the comparison table exclusively compares āwhat sites does this service remove data fromā and not much else:
(I know he was not really referring to lists like this with his blog post in the first place, but I wanted an excuse to share it anyways lol)
As a somewhat related sidenote, I noticed that deceptive checklists can consist of entire websites. Like, scammy versions of our own recommendations with little to no criteria on why they should be included. Since many checklist-esque websites are essentially paid advertising to shady services, I canāt agree with this sentiment enough. Bonus points if the website has āhackerā graphics and neon green headers
Shady VPN services and crypto exchanges asides, these websites often take advantage of SEO to get their website spread around folks new to the community. I donāt know how much website traffic they get but it must be substantial enough.
Iām glad we avoided this reputation but it is SO difficult to gain community trust. Even minor things like not including comparison tables can help so much.
Yeah and what they decide to include or leave out is a big area where their subjective opinion bleeds through. If you look at almost any product page they have a heavily biased checklist of cherry-picked features that their app supports and other apps donāt, which are almost always technically accurate but they donāt tell the full picture.
I think weāre pretty open that we arenāt an objective source of info and we try to show what criteria weāre going by and what we value in the different categories, hopefully weāre open enough about our biases. Thereās never really going to be a way to say one thing is objectively better than something else.
I donāt think checklists (or comparison tables) are inherently bad, just the ones used for self-promotion. They could be used as a tool for people to make decisions but never something to be relied on.
I think comparison tables could be okay if theyā¦
Are made by a neutral entity not affiliated with anything included in the comparison
Include explanations of each metric
There arenāt any unnecessary metrics or too many metrics
One product doesnāt dominate all other products included, either nothing meets all the criteria, or more than one product meets all the criteria
Remain consistently up-to-date
An example of what I mean (this may be inaccurate)
Unfortunately, the criteria presented in this way is still biased.
Not all columns (in your example) should have equal weight.
There are (practically speaking) nearly infinite ways to measure something. That there are a discrete, finite number selected is an expression of the values of the person or organization creating the checklist.
If you want to make one, I cannot stop you, but nobody should ever trust them for anything else but the summarization of an authorās opinion.
Theyāre objectivity theater (in the same way that VPNs are security theater in many threat models).
I mostly agree with the article but I think part of the problem is that comparison tables are one of the easiest ways to crowdsource contributions and compile information about related projects in a relatively egalitarian way. Personally, I can really sympathise with someone who is curious and wants to leverage community interest to improve the knowledge base for everyone (like a lifting tide).
I think it would be cool to think of ways to surface this crowdsourced information in a way that informs (no coloured ranking) or opens the path to learning more (well cited/source code linked).
The long-form essay is a foundation for capturing nuance, but suffer from the bias of a single author.
Wikis are a collaborative version of a long-form essay, and can be refined over time to reduce the effect of bias. You wonāt ever hit zero, but thatās the closest weāll get as a society, Iād wager.
As for communicating effectively to people who donāt have the patience to read large blocks of text? Make one consensus recommendation with experts and front-load the wiki pege with the easy answer for people in a hurry.
People who are interested in the details will actually read it.
The reasons itās being used so much by so many websites is that it is effective to convey information quickly. Of course, there are many examples where it is used for marketing purpose in a deceptive manner and not to be trusted.
Sorry, I have to disagree. Checklists are not a great way to summarize information. Theyāre a great way to lie by omission and exacerbate the base rate fallacy.
I think it entirely depends on what is in the table has so many columns there are. A personās brain simply is not able to parse a 50 column table and get anything meaningful from it. Not to mention that it requires constant updates as services change or improve often then meaning the information may simply not be correct.
A simple table like the one for recommended email providers is intended to make a comparison on very specific common use cases.
For example accessibility, anonymity and compatibility. I think that can work if the reader is like āWell i want to use PGP because friends i have use that and I want to use Thunderbird because i have work accounts on that and i may need to be anonymous for this accountā.
I mean, thatās not a real comparison table to be fair, the introductions of these pages are just a summary of the content.
Usually tables are presented without context and those are universally extremely problematic. I see one commonly thrown around about Android OSās, I see many, many VPN tables like the one just linked above, I see, well, exactly the tables that were talked about in the original blog post None of these tables are suitable for use, and yet even āexpertsā cite them as gospel all the time, which is why I wanted to share this blog in the first place.
IMO the biggest issue almost always comes down to:
I can assure you that many tables are used for decision making for instance. Search for one pagers, dashboard metrics, the Kaizen approach (I can name many more) and youāll find all sort of ways where tables are super efficient into communicating information quickly.
But yes, for all the tables meant for marketing purposes, it is obviously meant to scam people off
