It depends on the scope, but generally audits are broken up into smaller pieces.
When looking at systems you do have access to they can be easier to easier to audit if the code is available, ie in terms of looking at how it works, and how it is put together, - that can give you a pretty good idea if the team behind the software is competent, (did they do stupid things?) it requires a high level of expertise and a lot of time, as essentially you have to what is right from wrong, in the implementation. If the product uses cryptography then you should understand that too.
As this is mostly unattainable, even if you had the competency to perform such audits, due to time required, we recommend, sticking to products which have, been around a while, had professional third party audits, and not jump on everything that is “new” and “shiny”.
There are a couple of examples I can think here, of products we’ve purposefully not added to Privacy Guides for this reason, while other privacy sites have blindly added them because “they sound cool” in the product description.
Just because something isn’t open source though doesn’t mean it can’t be reverse engineered. This may be as simple as monitoring network traffic, such as using a packet analysis tool like wireshark, or tcpdump.
More in depth methods might involve decompiling (the process to try to get some code generated out of binary instructions), fuzzing (Eg. Trying to trigger buffer overflows - often used in discovering firmware vulnerabilities, using analysis tools and disassemblers like IDA Pro, Ghidra - one of the common methods for figuring out what malware does.
This article is quite good and discusses that: The right thing for the wrong reasons: FLOSS doesn't imply security.
TLDR it’s not just one thing, and it depends on the situation.