We use Cloudflare’s CSAM detection APIs on unencrypted content on Matrix.org.
We use the IWF Hash, URL and Keyword lists from the IWF on unencrypted content on Matrix.org.
Tbf to alot of people that will say 'Wow, matrix scans messages? Matrix bad!"
First things first, as it says, it’s on unencrypted content
Second of all, CSAM Was a problem in Matrix for a long time and it was about time they deployed a solution to this ever raging problem, GrapheneOS and many many other if not even former matrix communities were disbanded or are otherwise suffering because of this problem. I am not going to claim this is a perfect solution, no, no it isn’t but a solution was needed to be honest considering the situation.
So do yourself a favour and think about it before posting anything like what I said above
6 Likes
Couldn’t SimpleX’s approach to moderation be more privacy respecting?
they could very well indeed been inspired from SimpleX instead of deploying Cloudflare CSAM, as I stated:
I am not going to claim this is a perfect solution, no, no it isn’t but a solution was needed to be honest considering the situation.
maybe I’m hoping it’s temporary so they can provide a more privacy friendly solution but that’s about as far as I’ll put it if hopefully understandable
For context, random Matrix users would be added to group chats that contain illicit content, including CSAM unfortunately. I never had to encounter that but I heard that happening to quite a few people.
Why would using Cloudflare’s API on unencrypted content excessive? I don’t personally have a problem with it since they are making a good faith effort to mitigate the problem without undermining E2EE. I only have an issue when these platform try to break the entire security model itself, endangering everyone.
Like you said, I am quite intrigued to learn more about SimpleX’s approach to mitigating CSAM. Wouldn’t a bot joining these group chats be another way of scanning for illegal content?
3 Likes
The CSAM is a problem, and this shit must never see the light of day, some are the solutions, I refuse to have the content I post scanned, child pornography is a decoy, it will only be used for mass surveillance and research into political criticism (Israel and its genocide), manifestation etc., don’t be naive!
2 Likes
For context, random Matrix users would be added to group chats
This seems like the kind of capability you don’t want clients to support by default. Even without the CSAM element, this screams “poor design that doesn’t take user consent into consideration” to me.
4 Likes
I believe he meant “invited to” not “added to.” What drives people to accept random invites I’m not quite sure, but my personal “deny all requests” policy has had a 100% success rate against this sort of thing on Matrix lol
Although what I will say is that Matrix doesn’t blur out profile pictures of unknown contacts like Signal does, so there is that vector remaining I suppose.
2 Likes
Many would argue Cloudflare does do this as a general rule 
But I agree, I have no issue with server-side scanning of content tbh
3 Likes
Right. It’s the client-side scanning that I object to as well.
3 Likes
Yeah sorry but this is nothing, every unencrypted service scans everything for CSAM. You should assume unencrypted = public for literally everyone including authorities.
5 Likes
And those “many” have no idea what a WAF is and how it works because they’ve never spent a day working an actual cybersecurity job 
Yeah that’s a valid position, but in general offloading any sort of service to “the cloud” is not ideal, which brings Cloudflare from “useful service” down to just neutral to me.
This was much more of a problem back in the day when people almost exclusively used Cloudflare as their SSL termination and just had an unencrypted Cloudflare to Origin connection, but now that has mostly been solved by either origin SSL or Cloudflare tunnels. I do think Cloudflare still technically allows admins to choose that unencrypted footgun though.
Automatic CP detection is a cybercrime disaster waiting to happen.
Unfortunately, its also a good vector for remotely disposing of people, by both governments and just malicious people alike. Instead of trying to plant drugs, it might be safer and more effect to plant CP, which can be done remotely through a variety of ways. Matrix will be particularly vulnerable to CP-planting due to a usual lack of multifactor authentication.
Imagine if a troll, Mallory, obtains a victim’s (Bob) matrix login credentials from a leaked password database or by a fake login page. Mallory logs into Bob’s account from Tor, sends an unencrypted PM with a bunch of teen nudes to a controlled account, gets the AI to report Bob to the police, and Mallory quickly deletes the messages to hide his tracks. Bob is arrested within a week, faces years in prison, and his name is plastered on the newspaper as a child sex abuser.
How does Bob defend himself?
Imagine if the troll Mallory wasn’t just an amateur troll, but a well-resourced Chernarussian state-sponsored hacker, with access to a 0-click Android exploit. Mallory is tasked to politically assassinate Alice, a Mallory targets finds out Alice’s phone number, sends Alice a corrupted message that installs a keylogger, and exports Alice’s password credentials. Mallory then sends an old video of lesbian teenagers having sex through Alice’s account, and Matrix’s AI reports Alice, and she gets sent to prison.
How does Alice defend herself?
What if this happens to you? What can you do to prevent it?
This isn’t just a Matrix problem, its a problem inherent with any laws that punish for internet speech or looking at content. Because computers are inherently insecure, and users are inherently insecure, a user can be forced or tricked into downloading or posting illegal content. Matrix will just be more vulnerable than most to this type of attack since all thats needed is user account access.
In my experience, I haven’t seen planting CP isn’t a widely used tactic among sophisticated hackers, though I have seen unsophisticated criminals with physical access, like jealous spouses, attempt to plant CP plenty of times - and sometimes they succeed. Oftentimes the courts ignore any possible evidence of foul play - they hear “child sex” and immediately turn their brains off). But let’s hope the day never comes that remotely planting CP becomes a common issue.
1 Like
Even if not, Cloudflare still sees all the unencrypted traffic so it’s a massive concern (IMHO). ~20% of all web traffic is visible, unencrypted, by a single centralized entity. This includes passwords, PII, TOTP tokens when enrolling MFA, PHI, etc…
From a technical perspective, I get where you’re coming from, and I largely agree even if I like and use Cloudflare services myself.
However, from a legal perspective, any “serious” customer of Cloudflare will drag their ass through court if they mishandle particular classes of data, which is a decently strong incentive for CF to be as secure as needed and not rock the boat too hard on using the data they process
2 Likes