7 Ways Startup Apps Define Moderation Categories (Harassment to PII)

Here is the story summarized in my own words:

A startup app should define moderation categories based on seven key risk checks. These categories are harassment, sexual content, self-harm, violence, illegal activity, spam, and PII (personal identifiable information). However, each category should not be allowed to decide the outcome of a customer relationship management (CRM) action on its own.

Instead, the categories should be combined with severity, confidence level, and the proposed CRM action. The end result is a two-step process that combines synchronous checks for content that could cause problems with deferred checks for quality issues that can wait.

Each risk check produces a finding that includes a category, severity, confidence level, and evidence of where in the transcript the finding was made. Those findings are then evaluated against potential CRM actions. For example, a summary entering a restricted trust queue has a different exposure level than one being copied into a widely visible account note. The key is that the content itself stays the same, but the consequences change based on the final decision.

The seven top-level categories are enough to keep the moderation system easy to understand and use. However, teams can add their own internal subcategories if needed, as long as they help clarify decisions rather than just repeating every possible phrase a classifier might recognize. The design should avoid becoming cluttered with too many options.

The transcript of a sales call or other input should be traced through the moderation process before any risk categories are named. The primary tradeoff is between quality and latency. Running all possible checks for every CRM write can improve accuracy but slows things down. Running everything later feels faster, but could allow harmful actions to proceed.

Instead, the checks should be split between synchronous checks that can block dangerous actions immediately, and deferred checks for quality issues that can be handled later. Self-harm, credible violence, and illegal facilitation should go through the synchronous lane to prevent amplifying harm. Spam and lower-severity record-quality checks are better suited for deferred processing. Latency needs to be measured explicitly for each stage of the process.

Ultimately, the seven risk checks are mapped down to four CRM actions: allow, redact, review, or block. Having a smaller set of defined actions makes it easier to document, monitor, and maintain the moderation system even as the underlying classifier changes over time. Each category maps to one of these final decisions based on the severity, confidence level, and evidence provided by the seven checks. The goal is to keep the system simple, yet powerful enough to handle a wide range of moderation scenarios.