Great work again from Mayrhofer et al: “A Data-Driven Evaluation of the Current Security State of Android Devices”. Contains a research paper, crowd-sourced table of hardware and an app for evaluation and contributing:
According to this table, Google Pixels beat other devices listed there by far. Even Samsung S-series doesn’t look good.
What a surprise! /s
I downloaded their app and it looks like they require devices to be Google certified, yikes.
They probably need attestation so they can be sure that results haven’t been tampered with.
I’m not talking about MEETS_BASIC_INTEGRITY, MEETS_STRONG_INTEGRITY or them using hardware attestation like GrapheneOS recommends. They’re asking for MEETS_DEVICE_INTEGRITY which is basically a blessing from Google and nothing more.
Feel free to contact Rene, e.g. on Mastodon. I wrote with him in the past and he seems like a nice guy and is a great expert in Android security. Maybe he can do something about changing this requirement.
Just sent him a message.
Hi, this app requires devices to pass MEETS_DEVICE_INTEGRITY. To pass it, your OS needs to be certified by Google which makes this check just a blessing from Google and nothing more.
Other two checks are fine, but the proper way to audit the integrity of the devices is docummented here: Attestation compatibility guide | Articles | GrapheneOS
Android’s hardware attestation API is a lot more robust form of attestation than Play Integrity API and doesn’t require OSs to be certified by Google, which is anti-competitive.
This wasn’t my experience. The app worked perfectly fine for me on a GrapheneOS device (with sandboxed play services), although I did not attempt to upload the results.
They require certification to upload the results.
Did you know he is not only the ‘head of the Institute for Networks and Security’ at a university in Austria, but also the ‘Director of Android Platform Security at Google’?
I hope you don’t mind, but considering this, the way you phrased your message, made me giggle a bit 
Thanks for clarifying, that is indeed unfortunate.
That check is just circus, and I think he knows it, at least I hope so.
Anyway, GrapheneOS is in contact with EU regulators and with a company that would help them file a lawsuit to put this Play Integrity circus to an end.
From GOS tweets it would seem Android Security Team is friendly to GOS, it is the bean counters that are the issue
At first, this seemed like a cool project, but unfortunately, many of the results are plain nonsense. The Pixel 6 is given a score of 94.91 as a ‘secure device,’ while the Pixel 9 Pro gets a 79 and is considered ‘insecure’?
After some testing, the issue is due to outdated information pertaining to devices in their database, particularly relating to patch level, which (rightfully) weighs very heavily.
I really wouldn’t recommend this tool to anyone who is trying to evaluate and/or compare device security, since their database is outdated, their site is unhelpful at best and misleading at worst.
Outdated patch level information is indeed a problem. Nevertheless it can still be useful, for example to lookup and compare hardware security features of different devices.