Upload Moderation: Node.js NSFW, Violence, Hate-Symbol Classification + JSON Fallback
Short answer: for media support tickets that include an image, keep classification, policy enforcement, and tenant cost accounting as three separate steps. Send the image to a multimodal chat model with a strict JSON Schema, validate the returned object locally, and send invalid or uncertain cases to review. The fallback is a queue, not a guess. That design matters because a support agent is…
The article outlines a method for moderating media uploads in Node.js applications. The process involves three key steps: classification, policy enforcement, and tenant cost accounting. Images are sent to a multimodal chat model using a strict JSON Schema for validation. If the model's response is invalid or uncertain, it is sent to review.
This design ensures that a support agent is triaging the complaint, not publishing a photo. The upload might be evidence of a violent broadcast, a screenshot containing a hate symbol, or an ordinary account avatar. A boolean called "safe" is discouraged as it discards important context that the agent needs. The article emphasizes the importance of keeping the system boring and reliable.
To classify risky uploads without relying on JSON, the article suggests starting with a versioned taxonomy. The taxonomy includes separate observations for NSFW, violence, and hate symbols, as well as an uncertain category and a short evidence string. The model describes what it sees, and the application code decides whether the ticket is visible, blocked, or needs human review. This boundary allows for useful evaluation harness testing, where a prompt change can be tested independently from the enforcement policy.
The article also cautions against using a sentence as input and searching it for words, as this approach becomes difficult to replay due to punctuation changes, missing categories, and tenant policy reconstruction. Instead, it advocates for typed output, which provides a stable input for the rest of the pipeline. The article provides an example adapter in Node.js that can be called from any language through an OpenAI-compatible chat client.
The model call is kept behind an OpenAI-compatible chat client, and environment variables are used for the concrete base URL and model. The adapter does not publish an upload merely because the response parses, ensuring a more reliable moderation process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
