Designing a Spam & Phishing Link Detection System for Direct Messages at Scale
A deep, interview-ready walkthrough of how large messaging platforms detect and stop spam and phishing links in direct messages — built to handle millions of messages per minute with low latency, low false positives, and a continuously-adapting learning loop against adversaries who never stop evolving.
Introduction & History — Foundations
Every messaging platform that lets one stranger send a link to another stranger eventually has to answer an uncomfortable question: how do you let billions of harmless links through in real time while catching the malicious ones fast enough that nobody gets hurt? This is the exact problem we are going to design a system for today — a spam and phishing link detection pipeline for direct messages (DMs), built for a platform the size of a large social network or chat app, handling millions of messages every minute.
To understand why this system looks the way it does, it helps to understand how the problem evolved. In the early days of email, spam filtering was mostly about keyword matching — if a message contained words like “viagra” or “lottery winner,” it was flagged. Spammers adapted quickly, using misspellings, image-based text, and obfuscation. Email providers responded with Bayesian filters, reputation-based blocklists (like sender IP reputation), and later, machine learning models trained on millions of labeled examples.
Direct messaging platforms inherited this arms race but with new constraints. Unlike email, DMs are expected to feel instantaneous — a delay of even a second or two before a message appears feels broken to users. DMs are also far more private by nature; users expect a reasonable level of confidentiality, which limits how deeply a platform can inspect message content compared to, say, scanning public posts. And DMs are a favorite channel for phishing precisely because they feel personal — a message that appears to come from a friend, a customer support account, or a giveaway page is far more convincing than a cold email.
Phishing links sent over DMs typically follow recognizable patterns: fake login pages that mimic the platform itself (“your account will be suspended, verify here”), fake giveaways or crypto scams, romance-scam links, fake job offers, and compromised-account chains where one hijacked account messages the victim’s entire friend list. Because these campaigns often spread through automated or semi-automated accounts, they tend to arrive in bursts — thousands of nearly identical messages within minutes — which is both the danger (fast-moving damage) and the opportunity (a detectable signature) that our system design will exploit.
Modern platforms — think of how Meta, LinkedIn, Discord, Telegram, and Slack all deal with this — converged on a similar shape of solution: a layered pipeline that combines fast, cheap checks (blocklists, rate limits, hashing) with progressively more expensive checks (URL reputation lookups, content classification, behavioral graph analysis), so that the vast majority of legitimate messages are cleared in single-digit milliseconds, while only a small fraction of suspicious messages pay the cost of deeper analysis. This is the core design idea we will build out in detail.
Spam and phishing detection is one of the most commonly asked system design interview problems for senior and staff engineering roles because it combines almost every hard distributed systems topic at once: real-time stream processing, machine learning inference at scale, graph analysis, caching, rate limiting, and strict latency budgets — all under adversarial conditions where the “input” is actively trying to evade you.
- Why is this problem different from spam detection in public posts or email?
- What are the competing goals the system needs to balance, and how would you quantify the trade-off?
- How would your design change if the platform were end-to-end encrypted?
Problem Framing & Requirements — What Makes It Hard
Before diving into architecture, it is worth naming the constraints explicitly, because every design decision later traces back to one of these. The interesting engineering here is not a single hard problem — it is the collision between latency, adversarial input, privacy, and asymmetric error cost, all at once.
2.1 The Six Load-bearing Constraints
Extreme scale
A platform with hundreds of millions of daily active users can generate tens of millions of direct messages per minute during peak hours — every one a candidate for inspection, and every one carrying a budget for the whole detection pipeline that is measured in milliseconds, not seconds.
Tight latency budget
Users expect messages to appear near-instantly. The detection system typically has a budget of 50–150 milliseconds to make a send/block/hold decision without materially harming user experience. Anything slower gets perceived as “the app is broken.”
Adaptive attackers
Spammers actively test the system, rotate domains, use URL shorteners, and change wording to evade filters — the system must be able to update its rules and models faster than attackers can adapt, which pushes toward continuous, not periodic, learning.
Asymmetric cost of errors
A false negative (missed phishing link) can lead to real financial or safety harm to a user. A false positive (blocking a legitimate message) erodes user trust and can be a support and PR problem. The system needs different thresholds for “silently monitor,” “warn user,” and “hard block.”
Privacy expectations
DMs are semi-private. The design must minimize what humans (as opposed to automated systems) ever read, and must be clear about what is logged, for how long, and under what governance — automation is not just a scale requirement, it is a privacy requirement.
Multilingual, multi-script content
A global platform needs to detect scams in dozens of languages and scripts, including messages that mix scripts to evade keyword filters — homoglyph attacks and Unicode obfuscation are first-class threats, not edge cases.
2.2 Turning Constraints into Requirements
- Functional: for every DM containing a link (and, in practice, for every DM period), classify as allow, allow with friction, hold for review, or block, then take the corresponding action.
- Non-functional: p99 detection latency under ~150 ms; false-positive rate low enough to preserve trust; false-negative rate low enough to keep users safe; observable enough to detect silent drift.
- Operational: support hot-patched blocklist updates propagated across regions within seconds; support full model retrains on a daily cadence; support human review at a rate the moderation workforce can actually sustain.
“If you had to name the single most important non-functional requirement here, what would it be, and why?” A strong answer: it is the latency budget on the synchronous path, because it is what forces the whole design to be layered. If you had unlimited time per message, you could run every deep model and every graph query in-band and detection would be a much simpler system. Everything interesting in this design — the fast rules engine in front of the ML, the aggressive local caching, the asynchronous graph and retraining loops behind the scenes — exists specifically because you have to answer within roughly 100 milliseconds while still catching adversaries who have unlimited time to attack you.
Architecture & Components — Blueprint
At a high level, the system sits directly in the critical path of every direct message send, plus a set of asynchronous pipelines that continuously improve detection quality. Think of it as two loops working together: a fast synchronous loop that makes the immediate allow/hold/block decision, and a slower asynchronous loop that mines patterns, retrains models, and updates blocklists based on what happened across the whole platform.
Figure 1 — Full architecture. Solid blue lines are synchronous control flow, green lines are decision and persistence paths, red lines are enforcement/blocking paths, purple dashed lines are the asynchronous learning loop. The bulkhead invariant at the bottom is what keeps a Detection Service outage from becoming a messaging outage.
3.1 Core Components
| Component | Responsibility |
|---|---|
| API Gateway | Handles authentication, basic request validation, coarse rate limiting per user and per IP, and routes to the Message Service. First, cheapest layer of protection lives here: connection-level throttling that stops an obviously automated client from opening thousands of connections per second. |
| Message Service | Core service that owns the lifecycle of a direct message: receiving the send request, orchestrating the spam check, persisting the message, and triggering delivery. Treats the Detection Service as a synchronous dependency with a strict timeout. |
| Spam & Phishing Detection Service | The heart of the design. A dedicated service (or a small family of services) whose only job is to answer one question extremely fast: should this message be delivered as-is, delivered with a warning, held for review, or blocked outright? Fans out to several sub-components in parallel and aggregates their signals. |
| Decision Aggregator | Combines parallel check outputs into a single decision using a weighted scoring model or small decision tree. Intentionally simple and fast, because it sits on the critical path — the heavy lifting has already happened in the components that feed it. |
| Event Stream (Kafka) | Every decision, along with the underlying signals, is published here. Powers everything that does not need to happen in the synchronous path: feature computation for future messages, graph analysis, and continuous retraining data for the ML models. |
| Human Review Queue | Messages that are ambiguous — where the model’s confidence is in a middle band, not clearly safe and not clearly malicious — are routed to trained human moderators. Prioritized so high-risk or high-volume patterns are reviewed first, and feeds labeled examples back into the training pipeline. |
3.2 The Four Detection Sub-components
Rules Engine
Low-latency engine evaluating deterministic rules: known-bad domains, regex patterns for common scam phrasing, sender velocity limits, and account-age based restrictions. Designed to answer in under 5 milliseconds using in-memory data structures.
URL Reputation Cache
A distributed cache (backed by Redis or a similar in-memory store) holding reputation scores for URLs and domains seen recently, fed by internal crawls and third-party threat intelligence feeds (like Google Safe Browsing or PhishTank-style feeds).
ML Classification Service
A model-serving layer that scores message text, URL structure, and metadata using trained classifiers — typically a gradient-boosted tree model for structured features plus a lightweight transformer-based text model for content, both optimized for low-latency inference.
User & Account Risk
Maintains a rolling risk score per account based on account age, verification status, past violations, device and IP reputation, and behavioral signals such as sudden bursts of outbound messages.
- Why split detection into a fast rules engine and a slower ML service instead of just always calling the ML model?
- What happens if the ML Classification Service is slow or down — what is the fallback?
- How would you avoid the Decision Aggregator becoming a single point of failure?
Internal Working — Under the Hood
Now let’s go one level deeper into how each piece actually makes its decision, because the interesting engineering is in the details of each sub-component, not just the boxes and arrows.
4.1 The Rules Engine in Detail
The rules engine is intentionally the first line of defense because it is the cheapest to run and catches the most obvious cases — known-bad domains, previously reported phishing kits, and accounts that are already flagged. It typically holds its data in memory using structures like a compressed trie or a Bloom filter for domain lookups, so that checking “has this domain been seen as malicious before” is a near-constant-time operation, not a database round trip.
A Bloom filter is especially useful here: it can tell you with certainty that a domain is not on the blocklist (no false negatives for membership), while occasionally giving a false positive that gets confirmed by a secondary, more precise lookup (like a hash map or a small key-value store) only when the Bloom filter says “maybe.” This two-step check keeps memory usage low while keeping the fast path fast for the overwhelming majority of clean traffic.
Rules also cover structural red flags that do not require reputation data at all: a URL using an IP address instead of a domain name, a domain registered within the last 24–48 hours (a strong phishing signal, since attackers frequently burn through freshly registered domains), excessive use of URL shorteners stacked on top of each other, and lookalike domains that use homoglyphs (like a zero instead of the letter O) to impersonate a trusted brand.
4.2 URL Expansion and Normalization
Before any reputation check can be meaningful, a shortened URL (like those from generic link-shortening services) needs to be expanded to its final destination. This is done asynchronously wherever possible, using a dedicated URL-resolution service that follows redirects in a sandboxed environment, with strict timeouts and a cap on redirect depth to avoid redirect loops designed to stall the crawler. Because expansion can take time, the system typically caches the resolved destination aggressively — once a shortened URL has been resolved, that result is reused for all future messages containing the same short link, dramatically reducing repeated work.
4.3 The ML Classification Service in Detail
The machine learning layer typically runs two complementary models rather than one monolithic model, because the two failure modes they catch are different:
- A structured-feature model (gradient boosted trees, such as XGBoost or LightGBM): consumes numeric and categorical features — domain age, sender account age, number of recipients in the last hour, ratio of links to text, whether the sender and recipient have interacted before, and reputation scores from the URL cache. These models are extremely fast at inference time (often under a millisecond) and are easy to retrain frequently.
- A lightweight text model: a distilled transformer or a simpler embedding-based classifier that looks at the actual message text for phishing language patterns — urgency cues (“your account will be locked”), impersonation cues, and scam vocabulary — while being small enough to run within the latency budget. Larger, more accurate models are reserved for the asynchronous review path rather than the synchronous send path.
Both models output a probability score, which is combined (often via a simple logistic combination or a small stacking model) into a single spam/phishing likelihood score between 0 and 1.
4.4 The User & Account Risk Service in Detail
Content alone is not enough — attackers often rotate their wording. The account and behavioral signal is frequently the strongest one. This service maintains a rolling risk score per account, updated using a sliding-window approach (for example, messages sent in the last 60 seconds, 10 minutes, and 24 hours), tracking things like: sudden spikes in outbound message volume, a high fraction of first-time recipients (a legitimate user mostly messages the same people repeatedly; a spam account blasts many strangers), device and IP churn, and prior moderation history.
This score is computed incrementally using a streaming aggregation framework (such as Flink or Kafka Streams) rather than recomputed from scratch on every message, which keeps it cheap enough to check on every single send.
4.5 Graph Analysis for Coordinated Campaigns
Some of the most dangerous spam campaigns are not detectable by looking at a single message in isolation — they are only visible when you look at the graph of who is messaging whom. If a thousand new accounts, all created in the last hour, each message a different group of unrelated users with a nearly identical link, that pattern is a campaign, even if the individual message content looks unremarkable. This is handled asynchronously by a graph analysis service that builds a sender-recipient interaction graph and looks for dense, suspicious subgraphs — clusters of accounts with unusually similar behavior and low historical interaction with their recipients.
When the graph service detects a coordinated campaign in progress, it can push an emergency update back into the fast path — for example, temporarily raising the risk score for all accounts in the cluster, or blocklisting the specific URL pattern — closing the loop between slow analysis and fast enforcement within seconds to minutes, rather than waiting for the next full model retrain.
Figure 2 — Synchronous sequence. All parallel checks fan out from the Detection Service inside a shared 100 ms timeout window, then the Aggregator produces one of three verdicts. Any check that misses the timeout returns “no signal” rather than blocking the aggregation.
- Why use two separate ML models instead of one combined model?
- How would you detect a coordinated spam campaign that no single message reveals?
- How do you keep the account risk score cheap to check on every message send?
Data Flow & Lifecycle — Journey of a Message
It helps to trace a single message from the moment a user hits send to the moment (or non-moment) it appears in the recipient’s inbox, and then to trace what happens afterward, because detection does not stop once a decision is made — every decision becomes training data for tomorrow’s detection.
5.1 Stage 1: Ingestion
The client sends the message payload — recipient ID, message text, and any attached link — to the API gateway over an encrypted connection. The gateway authenticates the sender, applies basic request-shape validation, and forwards it to the Message Service.
5.2 Stage 2: Synchronous Detection
The Message Service calls the Detection Service, which fans out to the rules engine, the URL reputation cache, the ML classifier, and the account risk service, all in parallel, with a hard timeout (commonly 100–150 milliseconds total). Each sub-component returns either a verdict or “no signal” if it cannot complete in time.
5.3 Stage 3: Decision and Enforcement
The Decision Aggregator combines the signals into one of typically four actions:
- Allow: message is delivered normally — this is the outcome for the overwhelming majority of traffic.
- Allow with friction: message is delivered but the recipient sees an interstitial warning before the link is clickable (common for links with moderate risk scores).
- Hold for review: message is not delivered immediately; it is queued for automated deeper analysis or human review, and the sender may see a “message sending” state rather than an explicit block (to avoid tipping off attackers that they were caught).
- Block: message is rejected outright, and depending on severity, the sending account may be rate-limited, challenged with a verification step (like a CAPTCHA or phone re-verification), or suspended.
5.4 Stage 4: Delivery
Allowed messages are handed to the Message Delivery Service, which fans out to the recipient’s active devices via push notification and persistent connection (such as a WebSocket), and persists the message in the message store.
5.5 Stage 5: Asynchronous Learning Loop
Regardless of the outcome, the full record of signals and the final decision is published onto the event stream. From there:
- The feature store pipeline updates aggregate features (like a sender’s rolling message volume) so the next message from that sender has fresh context.
- The graph analysis service incorporates the new sender-recipient edge into its graph for campaign detection.
- The model training pipeline collects labeled outcomes (from human review, user reports, and confirmed campaigns) and periodically retrains the ML models — commonly on a daily cadence for the main models, with faster, lighter “hot patch” updates (like adding a domain to a blocklist) deployable within minutes.
5.6 Stage 6: Feedback from User Reports
Users reporting a message as spam or phishing is one of the highest-quality labels a system can get, and this feedback loop is intentionally kept short: a report updates the sender’s risk score quickly, flags the specific URL for re-review, and, if enough reports accumulate for the same URL or campaign pattern within a short time window, can trigger the same emergency blocklist update path used by the graph analysis service.
Figure 3 — Post-decision data flow. Solid arrows are the enforcement path; dashed purple lines are the asynchronous learning loop; the dashed red line is the “hot patch” blocklist update that skips the daily retrain and hits the rules engine within seconds.
- Why not just block a message outright instead of using a “hold for review” state?
- How would you make sure the feedback loop from user reports cannot itself be abused (for example, attackers mass-reporting a legitimate account)?
- What is the trade-off between fast “hot patch” blocklist updates and full model retraining?
Advantages, Disadvantages & Trade-offs — Balancing Act
No detection system is free — every design choice trades something against something else. Being explicit about these trade-offs is exactly what separates a strong system design answer from a superficial one.
| Design Choice | Advantage | Disadvantage / Trade-off |
|---|---|---|
| Layered detection (fast rules before slow ML) | Keeps median latency very low; saves compute for the minority of risky traffic | Adds pipeline complexity; a bug in an early layer can shadow later layers |
| Synchronous inline blocking | Stops harm before it reaches the recipient | Every message pays a latency cost; strict timeout budgets constrain model complexity |
| “Hold for review” instead of instant block | Reduces false-positive damage to legitimate senders; gathers human-verified labels | Introduces delivery delay; requires a review workforce or automation to avoid backlog |
| Aggressive URL reputation caching | Massively reduces repeated lookups and latency for common domains | Stale cache entries can let a newly-compromised domain slip through briefly |
| Graph-based campaign detection | Catches coordinated attacks invisible at the single-message level | Runs asynchronously, so detection lags the first wave of messages by minutes |
| Account risk scoring | Strong signal that is hard for attackers to fake quickly | New, legitimate accounts can be unfairly penalized (“cold start” problem) |
6.1 The Precision vs. Recall Trade-off
This is the single most important tuning decision in the whole system. High recall (catching almost every phishing link) tends to come with lower precision (more false positives, meaning legitimate messages get blocked or delayed). Because the cost of these two error types is different — a missed phishing link can lead to real financial harm, while a false positive is an annoyance — most production systems deliberately use different thresholds for different actions: a very low threshold to trigger a soft warning interstitial (favoring recall), and a much higher, more confident threshold to trigger a hard block (favoring precision), with an in-between band routed to human review rather than forced into either extreme.
6.2 Centralized vs. Federated Detection Logic
A single Detection Service that owns all logic is simpler to reason about and keeps signals consistent, but it becomes a critical dependency for every message send on the platform. An alternative is to embed lightweight checks (like the Bloom filter blocklist) directly into the Message Service itself, so a portion of decisions can be made even if the full Detection Service is degraded — trading some architectural purity for resilience.
Section takeaway
Every trade above swaps one axis for another — latency for accuracy, granularity for scale, freshness for stability. The design is a portfolio of small, bounded, tunable choices rather than one big absolute decision, and the thresholds themselves should be treated as product decisions with owners, dashboards, and change logs — not implicit consequences of whatever the model happened to output on its last training run.
- If you had to pick one metric to optimize first, would you choose precision or recall, and why?
- How would you handle the cold-start problem for brand-new accounts?
- Would you ever accept higher latency in exchange for higher accuracy, and for which class of messages?
Performance & Scalability — Scale
Let’s ground this in real numbers. Assume a platform with 500 million daily active users, where 10% send at least one direct message with a link on a given day, and message volume peaks at roughly 5x the daily average during a two-hour peak window. That can realistically translate into a peak load of several million detection checks per minute, each of which needs an answer within roughly 100 milliseconds.
7.1 Horizontal Scaling of the Detection Service
The Detection Service is designed to be stateless at the request-handling layer — all the data it needs (blocklists, reputation cache, model weights) is either held in local memory (refreshed periodically) or fetched from a fast external store. This statelessness is what allows it to scale horizontally: adding more instances behind a load balancer increases throughput close to linearly, since there is no shared mutable state that instances need to coordinate on for a single request.
7.2 Keeping the Fast Path Fast
The single biggest lever for performance is making sure the overwhelming majority of traffic (well above 95% in a healthy system) is resolved by the cheapest checks — the in-memory rules engine and cache lookups — without ever needing a full ML inference call or a network round trip to an external reputation service. This is achieved by:
- Local, replicated caches: each Detection Service instance keeps a local, periodically refreshed copy of the hottest blocklist and allowlist entries in memory, avoiding a network hop to Redis for the most common lookups.
- Batched model inference: where possible, the ML service batches multiple concurrent scoring requests together to make more efficient use of GPU or CPU vector instructions, trading a few milliseconds of queuing delay for significantly higher throughput per machine.
- Feature pre-computation: expensive-to-compute features (like a sender’s rolling message count) are maintained continuously by the streaming pipeline rather than computed on-demand at request time.
7.3 Sharding and Partitioning
The URL reputation cache and account risk store are partitioned (sharded) by a hash of the URL domain and account ID respectively, spreading load evenly across many cache and database nodes. Consistent hashing is used so that adding or removing nodes causes minimal reshuffling of keys, which matters a great deal at this scale since a full rehash under peak load could itself cause a latency spike.
7.4 Handling Burst Traffic and Campaigns
A large phishing campaign can itself look like a traffic spike — thousands of messages referencing the same URL within seconds. The system needs to absorb this without falling over. Rate limiting at the API gateway (per account and per IP) prevents any single source from overwhelming the pipeline, while the URL reputation cache is designed so that a burst of lookups for the same newly-seen URL results in a single upstream reputation check (using a request-coalescing pattern, sometimes called “single-flight”), with all concurrent requests for that URL sharing the one result rather than each triggering a separate expensive lookup.
7.5 Back-of-the-envelope Sizing
At 3 million detection checks per minute (50,000 per second) and a target of under 5 milliseconds average time spent in the synchronous fast path per check, a single Detection Service instance handling roughly 2,000 requests per second per core would need on the order of 25–30 cores’ worth of capacity at peak, scaled out across many smaller instances for resilience rather than a few large ones. That is the capacity for the fast path only; the ML serving fleet is provisioned separately and generously (typically 2–3x expected load) because model inference latency is much more sensitive to under-provisioning than the rules engine is.
- How would you estimate the number of Detection Service instances needed at peak load?
- What is request coalescing, and why does it matter here?
- How would consistent hashing help you scale the URL reputation cache?
High Availability & Reliability — Resilience
Because the Detection Service sits directly in the path of every message send, its availability directly affects whether users can send messages at all — which makes reliability engineering here just as important as detection accuracy.
Fail-open vs. fail-closed
If the Detection Service is unreachable or times out, should the Message Service allow the message through (fail-open) or block it (fail-closed)? Most production platforms fail open for low-risk-looking traffic and fail closed for high-risk-looking traffic — the Message Service keeps a small amount of locally cached rule data (like the Bloom filter blocklist) so it can still make a coarse decision even when the full Detection Service is down.
Redundancy and replication
Every stateful component — the URL reputation cache, the account risk store, the model-serving fleet — is replicated across multiple availability zones. The reputation cache uses a primary-replica setup with automatic failover, so a single node failure does not cascade lookup misses into full ML fallback for a large slice of traffic.
Circuit breakers & graceful degradation
Each sub-component call is wrapped in a circuit breaker. If the ML Classification Service starts timing out repeatedly, the breaker trips, and the Aggregator temporarily relies more heavily on the rules engine and cached reputation data alone, rather than continuing to send requests to a struggling dependency and making the problem worse.
Multi-region deployment
The entire detection pipeline is deployed in multiple regions, with users routed to their nearest region for latency reasons. Reputation and blocklist data is replicated across regions (often via the same event stream used for the learning loop), so a domain flagged as malicious in one region becomes known everywhere within seconds.
Disaster recovery
Regular backups of the account risk store and model artifacts are taken with a documented RTO (typically minutes for the Detection Service itself, since it can restart from replicated caches and the latest deployed model) and RPO (a few minutes for the risk store, acceptable because losing a few minutes of behavioral history rarely causes serious harm on its own).
Figure 4 — Multi-region HA layout. Green dashed lines are asynchronous replication; solid blue lines are normal-path calls to shared ML serving; the red dashed line is the circuit-breaker fallback that switches the Aggregator to rules-only mode when the ML fleet is degraded.
- Should this system fail open or fail closed, and does the answer change by message risk level?
- How does a circuit breaker prevent a cascading failure here?
- How quickly should a newly discovered malicious domain propagate across regions, and how would you achieve that?
Security & Privacy — Protection
Security here has two distinct dimensions: securing the detection system itself against tampering or evasion, and protecting the privacy of the very messages the system is inspecting. Both matter, and they occasionally pull in different directions.
9.1 Protecting Message Privacy
Because DMs are semi-private, the system is designed so that message content is analyzed by automated systems by default, with human review reserved for a small, carefully governed subset of cases (typically messages already flagged as high-risk, or ones a user has explicitly reported). Access to raw message content in the review queue is logged, access-controlled, and time-limited, and content used for model training is anonymized and stripped of personally identifiable information wherever possible, often replaced with abstracted features rather than raw text once a model has learned from it.
9.2 Encryption
Messages are encrypted in transit (TLS) between client and server, and at rest in the message store. If the platform supports end-to-end encryption for DMs, the detection strategy has to shift meaningfully — since the server cannot read message content at all, detection leans much more heavily on metadata signals (sender behavior, recipient graph, link domain shared by the client before encryption, or a client-side, on-device scanning step that flags known-bad links before sending).
9.3 Defending Against Evasion
Because the “adversary” here is actively trying to defeat the system, several specific attack patterns need defenses:
- Domain rotation: attackers register many disposable domains. Defense: treat domain age, registration patterns, and hosting-provider reputation as strong features rather than relying solely on static blocklists.
- Character obfuscation and homoglyphs: replacing letters with visually similar characters to evade keyword and domain matching. Defense: normalize text and URLs (Unicode normalization, homoglyph mapping) before running any pattern match.
- Slow-drip campaigns: sending just below rate-limit thresholds to avoid triggering velocity-based rules. Defense: use longer-window behavioral features (hours to days) in addition to short-window ones (seconds to minutes).
- Model probing: attackers testing many message variants against the live system to find ones that slip through. Defense: rate-limit and monitor for accounts that repeatedly send messages that get held or blocked, since probing itself is a strong risk signal.
- Poisoning the feedback loop: mass false-reporting of a legitimate account, or coordinated “this is fine” signals to normalize a malicious pattern. Defense: weight feedback by reporter trust score and look for coordination patterns in who is reporting.
9.4 Securing the Detection Pipeline Itself
Internal services authenticate to each other using mutual TLS or signed service tokens, so an attacker cannot simply call the Decision Aggregator directly and inject a fake “allow” verdict. Model artifacts and blocklist updates are signed and verified before being loaded, preventing a compromised deployment pipeline from silently weakening detection.
Treating the detection pipeline as purely a content-analysis problem and under-investing in account and network-level signals is one of the most common mistakes in real deployments — attackers can rewrite content far more easily than they can fake years of consistent, trusted account behavior.
- How would detection change if the platform moved to end-to-end encrypted messaging?
- How would you prevent an attacker from using the system’s own responses to reverse-engineer its rules?
- How do you protect the feedback loop from being gamed by coordinated false reporting?
Monitoring, Logging & Metrics — Visibility
A detection system that nobody is watching will silently degrade — attackers change tactics constantly, and a model that was accurate last month can quietly lose effectiveness as new campaign patterns emerge. Observability here is not optional polish; it is core to the system staying effective over time.
10.1 Key Metrics to Track
| Metric | Why it matters |
|---|---|
| Detection latency (p50, p95, p99) | Ensures the fast path stays fast; a creeping p99 often signals a degraded dependency |
| False positive rate (from appeals and human review overturns) | Directly measures harm to legitimate users |
| False negative rate (from confirmed post-hoc reports) | Measures how much malicious content is slipping through |
| Block/hold/allow rate over time | Sudden shifts often indicate a new campaign or a misbehaving model |
| Model score distribution drift | Detects when incoming traffic patterns are shifting away from what the model was trained on |
| Human review queue depth and age | Backlogs here directly delay legitimate messages stuck in “hold” |
| Cache hit rate for URL reputation | A dropping hit rate increases load on slower downstream checks |
10.2 Logging Strategy
Every decision is logged with the signals that contributed to it (rule matches, model scores, cache hit/miss), but message content itself is logged separately, with stricter access controls and shorter retention, to balance debuggability against privacy. Structured logging (consistent, machine-parseable fields) makes it possible to build dashboards and alerts directly from log data rather than requiring bespoke instrumentation for every new question.
10.3 Alerting and Anomaly Detection
Beyond static thresholds (like “page if p99 latency exceeds 200ms”), the system benefits from anomaly detection on its own metrics — a sudden spike in block rate for a particular geographic region, or a sharp rise in messages containing a specific new domain, is itself a signal worth alerting a human on, since it often means a fresh campaign is underway before the automated systems have fully adapted.
10.4 Distributed Tracing
Because a single detection decision fans out across multiple services (rules engine, cache, ML service, risk service), distributed tracing (using a standard like OpenTelemetry) is essential for debugging latency issues — without it, a slow decision is hard to attribute to any one of the four or five parallel calls that contributed to it.
- How would you detect that your ML model’s accuracy is silently degrading in production?
- What would you alert on immediately versus review the next morning?
- How do you balance detailed logging with user privacy expectations?
Deployment & Cloud Strategy — Rollout
The Detection Service and its dependencies are deployed as a set of independently scalable microservices, typically on a container orchestration platform such as Kubernetes, which allows each component (rules engine, ML serving, risk service) to be scaled based on its own resource profile — the ML serving layer often needs GPU or high-memory instances, while the rules engine is CPU-light and memory-bound.
11.1 Progressive Rollout of Model and Rule Updates
Because a bad model update or an overly aggressive blocklist push can block legitimate traffic at massive scale within seconds, updates are never deployed globally all at once. Instead, the platform uses canary deployments — rolling a new model or ruleset out to a small percentage of traffic first, comparing its decisions against the current production system (often in shadow mode, where the new logic runs alongside the old one but its verdict is only logged, not enforced), and only promoting it to full traffic once its metrics look healthy.
11.2 Infrastructure as Code and Multi-region Setup
All infrastructure is defined declaratively (using tools like Terraform), enabling consistent deployment across regions and making disaster recovery drills repeatable. Regions are deployed with enough independent capacity that a failure in one region does not require the others to instantly absorb its full load without warning — traffic shifting is done gradually with health checks guiding the pace.
11.3 Blue-green and Shadow Deployments for the ML Layer
Model updates are particularly risky because their behavior is harder to predict from code review alone. A shadow deployment — running the new model on live traffic without letting its decisions take effect — lets the team compare precision and recall against the current model on real, current data before it ever affects a real user’s message.
- Why is shadow deployment particularly important for a spam detection model compared to a typical feature rollout?
- How would you structure a canary rollout for a new blocklist rule that might be too aggressive?
- What is your rollback strategy if a newly deployed model starts blocking too much legitimate traffic?
Databases, Caching & Load Balancing — Storage Layer
12.1 URL Reputation Store
This needs extremely fast reads and can tolerate slightly stale data, making an in-memory key-value store like Redis (or a similar distributed cache) the natural fit, backed by a more durable store (such as a wide-column database) for the full historical record used by offline analysis. Reputation entries carry a time-to-live, so stale, unconfirmed data ages out automatically rather than persisting indefinitely.
12.2 Account Risk Store
This needs both fast point lookups (by account ID) and support for range queries over time windows (for the sliding-window behavioral features). A combination of a fast key-value store for the current rolling score and a time-series-friendly store for the underlying event history works well, with the streaming pipeline responsible for keeping the rolling score current.
12.3 Message and Decision Log Store
This is a high-write-volume, append-mostly workload, well suited to a distributed, horizontally partitioned database (partitioned by a combination of time and sender/recipient ID), since queries against this data are typically either “recent activity for this account” or “everything from this time window,” both of which align naturally with time-based partitioning.
12.4 Caching Strategy
Caching happens at multiple layers: a local, in-process cache inside each Detection Service instance for the hottest data (refreshed every few seconds to a minute), backed by a shared distributed cache for everything else. A cache-aside pattern is typical — check the cache first, fall back to the source of truth on a miss, and populate the cache with the result — combined with request coalescing to prevent a “thundering herd” of simultaneous lookups for a URL that just went viral (in the bad sense).
12.5 Load Balancing
Traffic is distributed across Detection Service instances using a load balancer that supports health-check-based routing, so instances that are slow or failing are automatically pulled out of rotation. Within the ML serving layer specifically, load balancing is often combined with request batching, routing concurrent requests to the same instance briefly so they can be scored together for efficiency, rather than always routing purely round-robin.
- Why is a cache-aside pattern a good fit for URL reputation data specifically?
- How would you partition the message decision log store to keep both write throughput and query performance healthy?
- What is a thundering herd problem, and how does request coalescing prevent it here?
APIs & Microservices — Interfaces
The system is naturally decomposed into microservices along the boundaries we’ve already described, each independently deployable and scalable, communicating primarily over internal gRPC (chosen for its low latency and strong typing) for the synchronous path, and over the event stream for the asynchronous path.
13.1 Service Surface
Detection Service API
Exposes a single low-latency “evaluate message” endpoint, accepting message metadata and returning a decision plus the contributing signal scores, primarily for downstream logging and explainability.
URL Reputation API
Exposes lookup and bulk-lookup endpoints, plus an internal write path used by the crawling and threat-intel ingestion pipelines to update scores.
Account Risk API
Exposes a fast “get current risk score” endpoint, and a separate, slower endpoint for detailed behavioral history used by human reviewers.
Review Queue API
Exposes endpoints for moderators to fetch prioritized cases, submit decisions, and for those decisions to flow back into the training pipeline as labeled data.
13.2 Why Microservices Instead of a Monolith Here
Splitting detection into separate services makes sense specifically because each piece has a very different scaling and resourcing profile: the ML serving layer benefits from specialized (and expensive) hardware and should scale independently of the lightweight rules engine, and the review queue’s traffic pattern (bursty, human-paced) is entirely different from the message send path (constant, machine-paced). A monolith would force all of these to scale together, wasting resources.
13.3 Internal API Design Considerations
Because the Detection Service call sits on the critical path, its API is designed to degrade gracefully — a partial response (some signals present, others missing due to a timeout) is a valid, expected response shape, not an error condition, and the Decision Aggregator is built to make a reasonable decision even with incomplete signals rather than treating a partial timeout as a hard failure.
- Why choose gRPC over REST for the internal synchronous calls here?
- How would you design the Detection Service API to handle partial signal availability gracefully?
- What are the risks of over-decomposing this system into too many microservices?
Design Patterns & Anti-patterns — Reusable Wisdom
14.1 Patterns That Fit Well Here
Circuit Breaker
Protects the Detection Service from cascading failure when a downstream dependency (like the ML service) degrades — when the breaker trips, the Aggregator temporarily falls back to rules + cached reputation only.
Cache-Aside
Used throughout for reputation and risk lookups, keeping the common case fast while tolerating a source of truth that changes independently. Combined with request coalescing to prevent thundering-herd storms on newly-viral URLs.
Command Query Responsibility Segregation
Writes to the account risk store (from the streaming pipeline) and reads (from the synchronous detection path) are handled through different, optimized paths rather than a single shared model.
Event Sourcing (partial)
The event stream of decisions and signals acts as the append-only source of truth that downstream systems (feature store, graph analysis, training pipeline) replay and build their own views from.
Bulkhead
Isolating resource pools per dependency (separate thread or connection pools for the ML service call versus the reputation cache call) so a slowdown in one does not starve the others.
Strangler Pattern
When replacing an older rules engine with a newer one, both run in shadow mode side by side until the new one is trusted, then traffic is gradually shifted.
14.2 Anti-patterns to Avoid
- Single monolithic model: relying on one large model for every signal type makes retraining slow and risky, and makes it hard to reason about which feature caused a false positive.
- Synchronous calls to slow external threat-intel APIs: calling a third-party reputation API directly on every message send, without a local cache in front of it, introduces both latency risk and a hard external dependency for a critical-path decision.
- Static, rarely-updated blocklists: treating the blocklist as a “set it and forget it” artifact rather than a continuously updated feed lets attackers simply wait out old entries and rotate domains freely.
- Binary allow/block with no middle ground: skipping the “hold for review” and “warn” states forces every decision into two harsh extremes, increasing both user-visible false positives and moderation review complexity.
- Ignoring the feedback loop: building a detection system without a clear, fast path for confirmed outcomes (human review, user reports) to feed back into retraining data means the system never actually improves, no matter how good the initial model is.
- Where would a circuit breaker specifically help in this design, and what happens when it trips?
- Why might a single large ML model be an anti-pattern here compared to a layered approach?
- What is the risk of a blocklist that updates too slowly, versus one that updates too aggressively?
Best Practices & Common Mistakes — Doing It Right
Best practices
- Set an explicit, enforced latency budget for the synchronous detection path, and design every component to degrade gracefully within that budget rather than blocking indefinitely.
- Treat precision and recall as tunable, business-level decisions with documented thresholds, not implicit side effects of whatever the model happens to output.
- Invest as much in account and behavioral signals as in content analysis — attackers can rewrite text instantly but cannot instantly fake trusted account history.
- Use canary and shadow deployments for every model and ruleset change, since a bad update can cause outsized, instantaneous harm at this scale.
- Build the human review loop as a first-class part of the system, not an afterthought — it is often the highest-quality source of training labels available.
- Design for multi-region propagation of newly discovered threats within seconds, since attackers do not respect regional boundaries.
- Log signals and decisions separately from raw message content, with different retention and access policies, to balance debuggability with privacy.
Common mistakes
- Over-optimizing for recall early on, leading to a flood of false positives that erodes user trust and generates a support burden that can overwhelm the review team.
- Under-investing in URL expansion and normalization, allowing simple shortener chains or homoglyph tricks to bypass otherwise solid detection.
- Letting the account risk cold-start problem block or over-restrict brand-new legitimate users, without a clear, fast path for them to build trust.
- Treating the detection system as “done” after launch rather than as a continuously adversarial, continuously retrained system — attackers adapt, and static systems decay in effectiveness within weeks to months.
- Not rate-limiting the review-queue and reporting pathways themselves, leaving them open to being gamed by coordinated abuse.
- What would you do in the first 90 days after this system launches to validate it is working as intended?
- How would you handle a sudden spike in false positive reports after a model update?
- How do you keep a detection system effective as attacker behavior evolves over months and years?
Real-World Industry Examples — In Practice
While exact internal architectures are proprietary, the publicly discussed approaches from large platforms echo the same layered pattern described in this tutorial — each tuned with domain-specific signals appropriate to its own product surface.
Meta — Messenger & Instagram DMs
Meta has publicly discussed using a combination of behavioral graph analysis, account-level trust scoring, and machine learning classifiers to detect coordinated inauthentic behavior and scam campaigns across its messaging surfaces, with particular emphasis on catching networks of fake or compromised accounts operating together rather than evaluating messages purely in isolation — directly mirroring the graph analysis layer in our design.
LinkedIn’s trust and safety approach for InMail and messaging emphasizes account reputation and professional-network context (such as whether the sender and recipient share connections or industry overlap) as a strong signal, since fake job-offer and recruitment scams are a dominant phishing pattern on a professional network specifically — an example of how the general architecture gets tuned with domain-specific signals.
Discord
Discord operates at very high message volume across both public servers and DMs, and has discussed layered automated detection combined with community-driven reporting and dedicated safety teams for review, plus rate limiting and account verification challenges (like phone or CAPTCHA verification) for accounts exhibiting spam-like send patterns — a direct real-world instance of the “hold for review” and “challenge” actions in our decision aggregator.
Telegram & E2E Platforms
Platforms with strong end-to-end encryption face a fundamentally different constraint: server-side content inspection is not possible for encrypted chats. This pushes detection toward metadata and behavioral signals (message frequency, account age, forwarding patterns) and, in some designs, optional client-side scanning against known-bad link lists before a message is even sent — illustrating the earlier point about how end-to-end encryption reshapes the entire design.
Google Safe Browsing
Many platforms, rather than building URL reputation entirely from scratch, integrate with shared threat-intelligence services like Google Safe Browsing, which aggregates malicious URL reports across a huge portion of the web. This reflects a broader real-world pattern: reputation data often benefits from being pooled across organizations, since no single platform sees the entire scope of an attacker’s infrastructure alone.
Industry Takeaway
Across every platform, the same shape recurs: fast layered decisioning on the send path, asynchronous graph and behavioral analysis behind the scenes, a human review loop for hard cases, and pooled threat intelligence for reputation. What differs is the mix of signals and thresholds, not the underlying architecture — which is exactly why this design generalizes across products.
The contrast between LinkedIn (professional-network graph as a first-class signal) and Telegram-style E2E platforms (metadata and client-side signals only, because message content is off-limits) is a clean illustration that the same architectural skeleton can host very different signal portfolios — the design does not need to change to accommodate encryption; the signal mix does.
- How might a professional networking platform’s detection signals differ from a general messaging app’s?
- Why would a platform choose to integrate a third-party threat intelligence feed instead of building reputation data entirely in-house?
- How does end-to-end encryption change what signals are even available to a detection system?
Frequently Asked Questions — Quick Answers
Why not just run every message through the full ML model — why bother with a rules engine at all?
Running a full model on every message at this scale would be prohibitively expensive in both latency and compute cost. The rules engine resolves the vast majority of clearly-safe and clearly-malicious traffic in a few milliseconds using cheap, deterministic checks, reserving the more expensive model inference for the smaller slice of ambiguous traffic where it actually adds value.
How does the system avoid punishing brand-new, legitimate users?
New accounts are given a moderate default risk level rather than the most restrictive one, with restrictions (like lower message-send limits or link-sending delays) that loosen automatically as the account builds a track record of normal behavior, rather than a system that treats “new” as equivalent to “suspicious.”
What happens when the ML Classification Service goes down?
The circuit breaker trips, and the Decision Aggregator falls back to the rules engine and cached reputation data alone, accepting a temporary reduction in nuanced detection in exchange for keeping the system available and responsive.
How quickly can the system react to a brand-new phishing campaign?
Graph analysis and reporting-driven signals can identify a coordinated campaign within minutes of it starting, at which point an emergency blocklist or risk-score update can propagate to the fast path across all regions within seconds, well before the full daily model retraining cycle would have caught it.
Is a message ever read by a human?
Only a small, carefully governed subset of held or reported messages are reviewed by trained moderators, with strict access controls and logging, specifically to balance detection accuracy against the platform’s privacy commitments to its users.
Should the system fail open or fail closed if the Detection Service is unreachable?
Most production platforms do both, depending on the residual risk visible to the Message Service: fail open for traffic that looks low-risk based on a locally cached Bloom filter blocklist and a coarse account risk snapshot, and fail closed for traffic that hits that local blocklist directly, so the total absence of the Detection Service does not become the total absence of enforcement.
How is the feedback loop protected from mass false-reporting attacks?
Every report is weighted by the reporter’s own trust score, and coordinated reporting patterns (many reports from newly created or clustered accounts within a short window) are themselves treated as a suspicious signal rather than automatically believed — the same graph-analysis tooling used to catch coordinated spam campaigns is used to catch coordinated abuse of the reporting system itself.
Summary & Key Takeaways — Wrap-Up
A spam and phishing link detection system for direct messages at scale is fundamentally a layered pipeline: cheap, deterministic checks catch the obvious cases in milliseconds; caching and reputation data handle the common cases efficiently; machine learning models handle the nuanced cases within a tight latency budget; and an asynchronous learning loop — powered by graph analysis, human review, and user feedback — continuously improves the system’s ability to catch tomorrow’s attacks, which will inevitably look different from today’s.
The system succeeds not because any single component is perfect, but because the layers compensate for each other’s weaknesses: rules catch what models might miss on brand-new patterns copied from known attacks, models catch what rules cannot express, behavioral and graph signals catch coordinated attacks invisible in any single message, and the human-in-the-loop review process provides both a safety net for hard cases and a continuous stream of high-quality training data.
Key takeaways
- Layer detection from cheapest to most expensive, so most traffic never touches the costly checks.
- Treat precision and recall as deliberate, tunable business trade-offs, not accidents of model output.
- Account and behavioral signals are often more durable than content signals, since attackers can rewrite text far faster than they can fake trust.
- Design explicitly for graceful degradation — fail-open versus fail-closed decisions, circuit breakers, and partial-signal handling all matter because this system sits on the critical path of every message.
- The asynchronous learning loop — graph analysis, human review, retraining — is what keeps the system effective over time against an adversary that never stops adapting.
- Treat the whole pipeline as a living system, not a launch artifact: attacker behavior drifts, so metrics, thresholds, and models all need standing owners and continuous re-evaluation.
The best spam and phishing detection systems are not the ones with the smartest single model — they are the ones where cheap, fast, and boring checks handle the overwhelming majority of traffic so that the expensive, clever, adaptive parts of the system can focus their attention on the small, dangerous minority where they actually earn their cost.
