SHARE
Recently, we worked on an AI-assisted anti-fraud moderation system for marketplace chats. The goal was not simply to “add AI” to an existing moderation process. We needed a system that could identify risky messages in real conversations while accounting for each dialogue's context.
The system had to detect different types of fraud and spam, including phishing links, platform impersonation, advance payment scams, requests for banking or payment details, attempts to move conversations off the platform, and unsolicited commercial spam.
The main challenge was that anti-fraud rules were not fully defined from day one. Many cases could only be evaluated in context.
A message that looks suspicious on its own is not always fraudulent.
For example, a buyer asking for a seller’s location, a car photo, or a VIN can be completely legitimate. The same is true for payment-related messages: a normal discussion between a buyer and a seller should not automatically be treated as a scam.
The important questions are often:
For this reason, we didn't build the system as a black box that simply blocks messages. Instead, we designed an iterative moderation loop that combines automated analysis with human review.
Eligible chat messages are analyzed asynchronously by a spam-analysis pipeline.
The first step uses deterministic rules and behavioral or contextual signals to collect initial risk indicators. These signals can identify known patterns before the message reaches the AI classifier.
Next, an LLM classifier analyzes the message together with a limited amount of conversation context. Instead of returning only a simple yes/no result, the model produces a structured response:
A separate policy layer then combines the rule-based score with the LLM verdict and assigns the message to a final moderation tier: regular, flagged, or spam.
This separation matters because an explicit policy layer translates deterministic signals and LLM output into the final moderation tier. The policy layer controls how different signals are combined and makes the behavior easier to tune.
Selected review-worthy cases are flagged for moderator review.
Authorized moderators can review the message, available context, moderation signals, score, and review status before making a final decision.
Reviewer feedback can inform future human-controlled updates to rules, prompts, policy logic, thresholds, and evaluation cases.
In this way, moderation becomes a continuous loop:
signal detection → AI analysis → policy decision → human review → feedback → policy improvement
Running an AI model in production requires more than connecting an API and waiting for predictions.
We added production controls such as provider fallback, timeout and cooldown behavior, token, usage, and estimated-cost tracking, audit events, retention for LLM moderation events, and dashboard counters.
The moderation dashboard also provides an operational view of the pipeline. During the quarter, it processed 929 641 messages, with 2 flagged, 2 blocked, and 9 121 pending review.
We also created a curated, production-derived challenge set for offline comparison. This lets us compare different versions of the system before changing the production policy.
The goal is to evaluate more than detection quality. We also need to understand cost, latency, and false-positive risk. A system that catches more spam but incorrectly blocks legitimate conversations is not necessarily a better anti-fraud system.
The main value of AI here isn't just automatic spam detection.
LLMs help turn complex and sometimes unclear anti-fraud expertise into a more structured process. Instead of relying only on rules that cover known patterns, the system can analyze message meaning and conversation context, provide structured reasons for its decision, and send uncertain cases to human reviewers.
This creates a process that combines signal detection, structured explanations, human review, moderation-event records, and continuous evaluation.
AI-assisted moderation works best when it is treated as part of a broader decision-making system, not as a standalone classifier.
For marketplace chats, context matters most because legitimate conversations can include the same words and actions as fraudulent ones. A payment request, external link, or request for personal information does not automatically mean fraud.
By combining deterministic rules, contextual signals, LLM analysis, policy logic, and human review, we built a moderation loop that evolves as new fraud patterns emerge and the anti-fraud policy sharpens.
The result is not simply an AI model in production. It is a repeatable process for detecting risk, explaining decisions, reviewing uncertain cases, measuring quality, and continuously improving the moderation policy.