SimpleX Chat - Evolving content moderation

jerm · January 12, 2025, 7:57pm

simplex-chat/simplex-chat/blob/821f034d18930c5c32d1c010a72bff15775b9337/docs/rfcs/2024-12-30-content-moderation.md

# Evolving content moderation

## Problem

As the users and groups grow, and particularly given that we are planning to make large (10-100k members) groups work, the abuse will inevitably grow as well.

Our current approach to content moderation is the following:
- receive a user complaints about the group that violates content guidelines (e.g., most users who send complaints, send them about relatively rare cases of CSAM distribution). This complaint contains the link to join the group, so it is a public group that anybody can join, and there is no expectation of privacy of communications in this group.
- we forward this complaint to our automatic bot joins this group and validates the complaint.
- if the complaint is valid, and the link is hosted on one of the pre-configured servers, then we can disable the link to join the group.
- in addition to that, the bot automatically deletes all files sent to the group, in case they are uploaded to our servers, via secure SSH connection directly to server control port (we don't expose shell access in this way, only to a limited set of server control port commands).

The problem of CSAM is small at the moment, compared with the network size, but without moderation it would grow, and we need to be ahead of this problem, so this solution was in place since early 2024 - we wrote about it on social media.

The limitation of this approach is that nothing prevents users who created such group to create a new one, and communicate the link to the new group to the existing members so they can migrate there. While this whack-a-mole game has been working so far, it will not be sustainable once we add support for large groups, so we need to be ahead of this problem again, and implement more efficient solutions.

At the same time, the advantage of both this solution and of the proposed one is that it achieves removal of CSAM without compromising privacy in any way. Most CSAM distribution in all communication networks happens in publicly accessible channels, and it's the same for SimpleX network. So while as server operators we cannot access any content, as users, anybody can access it, and we, acting as users can use available information to remove this content without any compromise to privacy in security.

This is covered in our [Privacy Policy](https://simplex.chat/privacy/).

This file has been truncated. show original

yipii · January 13, 2025, 5:19am

As usual, excellent communication from simplex chat, and innovative ideas.

I recommend opening the github link for proper mobile view word wrapprd

nihilist · January 13, 2025, 6:21pm

before the fud gets spread here aswell, moderation improvements doesn’t mean privacy / anonymity reductions, the SimpleX context is different from traditional chat apps due to the E2EE by default, and decentralisation of the network.

In any case, it still is a pleasure to see SimpleX achieve more of what the rest of other chat apps didn’t achieve yet (Yes i am talking about Matrix’s CSAM problem that they are intentionally ignoring on the main matrix.org instance)

maqp · January 19, 2025, 3:36pm

Not complaining about the basic principle of this, that is, doing what you can when you can. Public chats should be moderated.

But does this allow DoS attacks? E.g. former group member joins the group with throw-away account via Tor, posts CSAM at a time when the mods are asleep, and reports it with another account using Tor.

How well does the bot scrutinize the entire channel? Hopefully it’s not a knee-jerk reaction if the disciplinary action is indeed

“the bot automatically deletes all files sent to the group”