Due Process: Why Fair Moderation Builds Healthier Communities

By Audrey Hingle and Julia Kamin. Originally published on the Prosocial Design Network blog.

In the offline world, "due process," or more broadly procedural justice, is the idea that before someone faces a consequence, they're entitled to fair treatment: clear rules, notice of what they did wrong, and a chance to respond. But its principles apply online as well. As Matt Katsaros put it in a conversation with the Prosocial Design Network, the core insight is simple: “people care about how they are treated,” not just about the outcome. Transparency and fairness are the principles that run through all of it.

There is a straightforward ethical case for this. Treating people with dignity and fairness is the right thing to do. But this primer is mostly about a second kind of argument. Most platforms today lean on a deterrence model, where rule-breaking content is removed, and offenders move up an escalating ladder of suspensions and bans, on the theory that punishment alone changes behavior. The Yale Justice Collaboratory's framework for tech professionals argues that procedural justice offers a better path. And as we hope this primer shows, designing for due process can actually reduce harm and foster healthier engagement. In practice, it comes down to three things: being clear about the rules up front, telling people why when you enforce them, and giving people a way to appeal.

Be clear about the rules and the consequences

Due process starts before anyone breaks a rule. It starts with making the rules, and how they're enforced, visible. The practical guidance here is to state plainly what behavior is encouraged and what's discouraged or prohibited, keep the rules and guidelines prominent, and remind people of them. That means surfacing rules where people will see them: pinned to the top of a thread, or shown as a prompt when someone enters a forum – not tucked into the terms of service that no one reads.

This is one of the better-evidenced interventions in the prosocial research space. We at the Prosocial Design Network rate rules and norms setting as a validated intervention. A key study is J. Nathan Matias's 2019 field experiment on Reddit's r/science, where some posts were randomly assigned to display a reminder of the community's rules along with a note about how they're enforced. New users who saw the reminder were about 8% more likely to follow the rules and 70% more likely to comment. This is a serendipitous result: the intervention boosted both compliance and participation

A few other studies point the same direction, which is why we feel the evidence that rules and norms work is so strong. A collaboration with Nextdoor randomized 312 new groups to either show group guidelines before people joined, or show nothing. The groups that posted guidelines saw noticeably fewer comments reported for abuse (about 0.3% versus 0.7%). And a Facebook field experiment found that people who'd had content removed re-offended less when they were afterward reminded of the specific rule, or simply that rules are enforced. One detail worth remembering from the Matias work: the reminder didn't just list the rules, it signaled that they'd be enforced. That enforcement cue appears to play a significant role. 

Tell users why their content was removed

The right to know the charge against you isn't only a courtroom idea. When you remove someone's post or comment, the due-process move is to tell them why – which rule it broke and what happened. The good news is this can be lightweight: explanations can be short, and they can be delivered by a bot rather than a human moderator.

The evidence here is a little less strong, which is why we rate removal explanations as likely rather than validated. The central study by Shagun Jhaver and colleagues in 2019 is observational: it tracked Reddit users who had their posts removed across Reddit’s thousands of communities, each with its own policies for removal explanations. They found that those who received an explanation were less likely to have another post removed later. Explanations delivered as detailed comments posted below the removed submission worked better than short labels attached to a post. And usefully for anyone worried about moderator workload, it made no difference whether the message came from a bot or a person. Because it's observational, we can't fully rule out that the kinds of communities offering explanations also happen to attract more receptive users, but related removal experiments (more on those below) point in the same direction.

A separate question is whether public explanations, e.g. "this comment was removed because it violated our rule on X," do anything for bystanders. A later, additional study by Jhaver and colleagues looked exactly at that. They found that bystanders who saw removal explanations significantly increased their activity afterward: they posted more frequently and commented more on others' threads (higher interactivity). However, witnessing an explanation did not reduce bystanders' own rule-breaking. 

Give users a way to appeal

When you remove a comment or restrict someone's access, tell them why. Due process means also giving someone a route to challenge the decision. A well-designed appeal asks the user to explain why they think the action was unfair or mistaken, which puts a bit of the burden back on them to make their case.

To be candid, the evidence that an appeals process encourages prosocial behavior is the thinnest and most mixed of any section in this primer. We have not reviewed a study that isolates an appeals process and tests whether it, on its own, encourages people to behave better. What we have is indirect – and it comes from two quasi-experiments that showed that removing rule-breaking comments reduces repeat offending: one on Facebook (Ribeiro et al., 2022) and one on Reddit's ChangeMyView (Srinivasan et al., 2019). In both, the removal came bundled with an explanation and a way for the user to contest the decision. So the appeal was baked into the interventions that work, and its individual contribution is implied rather than measured. 

In our interpretation, consistent with procedural-justice theory on fairness and recidivism, is that the chance to appeal may be a necessary ingredient for a removal to land constructively. But that is a reasonable inference, not a proven mechanism. There's firmer evidence on the moderator side: a field experiment on a tool (AppealMod) that asked users to explain their appeal before it reached a human cut the volume of appeals moderators handled by roughly 60–70% and their exposure to toxic appeals by around 90%, with no drop in successful appeals. Even there, though, the study measured effects on moderators, not on whether the wider community behaved more prosocially – so the prosocial case for appeals is promising, but not yet directly demonstrated.

Why it's worth it

Put the pieces together and we hope we’ve made the case: due process isn't only the "right" thing to do, it also produces better outcomes, lowering recidivism while keeping (or even increasing!) healthy interaction and engagement. And the appeals research suggests you can build a process that respects users and lightens the load on the moderators who keep communities running.

There's one more reason that the research can't (yet) prove with a randomized trial: trust. A central claim of procedural-justice theory is that treating people fairly builds their willingness to accept a system's authority and stay engaged with it. We can't point to a controlled experiment that nails this down for online platforms, so it's fair to label it theory rather than settled finding. But it's the through-line beneath everything above, and it brings us back to where we started: people care about how they're treated, and platforms that demonstrate respect in their moderation policies are more prosocial places to be.

From the Group Chat 👥 💬

This week in our Signal community, we got talking about:

The UK government's headline pledge to make it "impossible" for children to take, share or view nude images. The plan relies on device-level scanning, technology ministers say already exists on Apple and Google phones and merely needs switching on. But by the government's own admission, the existing Apple and Google features only blur suspected nude images inside messaging apps and don't cover the camera, third-party messengers, or search, which is the much larger, always-on, whole-device capability ministers are actually demanding:

"Despite this, the nudity detection is not applied to the camera or broader apps, third-party messaging services, or search functions, meaning children can still take, view, share and save nude images. The government therefore wants Apple and Google to block nudity across the whole device by default, so they can only be deactivated via age assurance."

The big winner in this is SafeToNet, a technology provider cited in the government’s press release that IX contributor Heather Burns has written about, and that has previously been accused of spying on children after claiming in 2020 that its software had caught girls as young as 10 using the names of UK restaurant chain Nando's peri-peri sauces as coded sexual slang, a claim built on its screening of tens of millions of children's messages. A cybersecurity researcher cited in that article also found the app contained a whitelist allowing advertisers to monitor children's web browsing habits.

The firm's product today, HarmBlock, performs the client-side scanning the government now wants nationwide. Client-side scanning is widely criticized by privacy and security experts because it sidesteps end-to-end encryption, inspecting content on the device before it is encrypted. Signal, opposing the plan, warns that surveillance is not safety: scanning every device on the presumption of nudity will not keep children safe. Instead it entrenches Apple, Google and Microsoft’s market dominance and their control over our most personal information, and builds infrastructure that history shows never stays narrowly scoped. While today it detects nudity, tomorrow it may be political speech… or our conversations about peri-peri chicken.

This Week's Links