How to audit proprietary software

Questions
1

I use open source software as much as possible. This notwithstanding, I can’t only use open source software all the time, although I do not use any proprietary apps.

I always see that open source doesn’t equal privacy, but it does mean you can check the code. I’m sure there’s also privacy-respecting proprietary software, and websites. But how can you tell?

I have pretty strict ad and tracker blocking by way of uBlock origin, so when the number of blocked elements it shows is really high, I’d consider that an indicator. This happens when I login to my Microsoft 365 for school.

Is there any other way to determine this though? As it’s websites I’m most asking about, I’m most interested in a way to tell if they’re privacy-respecting or not. Any ideas on how to accomplish this would be welcome!

2

You can read their privacy policy, it normally tells you what you can expect they do with your data, and what 3rd parties they share it with.

It’s also your only option with many open source projects, unless you host everything yourself, you don’t know if the public code is what they are running on production servers.

3

It depends on the scope, but generally audits are broken up into smaller pieces.

Looking at the privacy policy is a start, for larger more trustworthy companies, that’s probably hold more value that what is written there is actually true. The reason for that is, breaches of their own privacy policy could result in lawsuits. For software as a service products, where you don’t have physical access to production systems, its really the only thing you have.

When looking at systems you do have access to they can be easier to easier to audit if the code is available, ie in terms of looking at how it works, and how it is put together, - that can give you a pretty good idea if the team behind the software is competent, (did they do stupid things?) it requires a high level of expertise and a lot of time, as essentially you have to what is right from wrong, in the implementation. If the product uses cryptography then you should understand that too.

As this is mostly unattainable, even if you had the competency to perform such audits, due to time required, we recommend, sticking to products which have, been around a while, had professional third party audits, and not jump on everything that is “new” and “shiny”.

There are a couple of examples I can think here, of products we’ve purposefully not added to Privacy Guides for this reason, while other privacy sites have blindly added them because “they sound cool” in the product description.

Just because something isn’t open source though doesn’t mean it can’t be reverse engineered. This may be as simple as monitoring network traffic, such as using a packet analysis tool like wireshark, or tcpdump.

More in depth methods might involve decompiling (the process to try to get some code generated out of binary instructions), fuzzing (Eg. Trying to trigger buffer overflows - often used in discovering firmware vulnerabilities, using analysis tools and disassemblers like IDA Pro, Ghidra - one of the common methods for figuring out what malware does.

This article is quite good and discusses that: The right thing for the wrong reasons: FLOSS doesn't imply security.

TLDR it’s not just one thing, and it depends on the situation.

3 Likes
4

Thank you for your long and detailed answer!

As you say it depends on the situation, would any of the tools or techniques you mentioned apply to websites?

5

Unless the source code for the site is hosted somewhere (eg github) that can’t be audited. The page you see is often dynamically generated with some sort of application on the server. However, it is still possible to blackbox test, various things such as looking for cross site scripting vulnerabilities, data validation or other issues etc.

1 Like