Picture a newsroom from the 1990s. Editors would gather every morning, look at wire service reports, phone calls, and yesterday’s newspaper sales, and decide what was “important” for that day’s front page. The feedback loop between something happening in the world and the public being told it mattered was measured in hours, sometimes a full day.

The internet compressed that loop first to minutes, through live news tickers and search engine query spikes. Then social platforms compressed it further, to seconds, because the “signal” of what the world cares about right now is no longer something an editor has to notice and decide on — it is something that can be measured directly, in real time, from the aggregate behavior of millions of people simultaneously searching, posting, watching, and sharing.

A “Trending Now” page is the product surface built on top of that measurement. It answers one continuously changing question: out of everything happening on this platform right now, what is the world suddenly paying disproportionate attention to, compared to a normal moment? A trending page that updates once an hour is not really “trending” — it is a stale summary of history. The entire value proposition of this feature is speed: catching a breaking news event, a viral clip, a sudden earthquake, or a major sports upset within seconds of it starting to spread, not after it has already peaked.

It is also worth noting upfront why this particular feature has become such a common interview and system-design topic in its own right, separate from its product importance. It sits at a genuinely interesting intersection of several distinct hard problems that rarely all appear together in one system: high-throughput event ingestion, low-latency stream processing, applied statistics for anomaly detection, memory-efficient approximate algorithms, real-time push delivery to millions of clients, and adversarial abuse resistance. Very few systems require a strong, working understanding of all six of these areas simultaneously, which is exactly what makes designing one from scratch such a thorough test of end-to-end systems thinking.

Let us precisely define what we are building before drawing any boxes. The platform wants a “Trending Now” page that:

• Continuously ingests a firehose of user activity events — searches, posts, likes, shares, views, comments — across every topic, hashtag, and entity on the platform.
• Computes, in near real time, which topics are experiencing an unusual, statistically significant spike in attention relative to their own recent baseline.
• Ranks those topics and serves an updated, ordered trending list to hundreds of millions of viewers, refreshed within single-digit seconds of the underlying shift occurring.
• Segments trends by geography and, often, by category (news, sports, entertainment, gaming) so a global spike and a regional spike are both surfaced appropriately to the right audience.
• Filters out spam, bot-driven manipulation, and coordinated inauthentic activity, so trending reflects genuine organic attention.
• Remains stable enough that legitimate trends do not flicker in and out of the list every second due to noise, while still being fast enough to catch a real spike almost immediately.

2.1 Three Forces Pulling Against Each Other

Why is this hard? Three forces pull against each other simultaneously:

3.1 Functional Requirements

• Ingest all relevant engagement events platform-wide (posts, searches, shares, views, reactions) with topic/entity extraction attached.
• Maintain a continuously updated attention score per topic, computed over multiple sliding time windows (for example, the last 5 minutes compared against the trailing 24-hour baseline for that same topic).
• Detect statistically significant spikes and rank the top trending topics globally and per region/category.
• Serve the current trending list to clients with end-to-end latency, from event occurring to appearing on someone’s screen, of single-digit seconds at the P95 level.
• Push updates to actively viewing clients without requiring a manual page refresh.
• Detect and suppress manipulated or spam-driven trends before they reach real users.

3.2 Non-Functional Requirements

• Latency: event-to-visible-on-page latency under 5–10 seconds at P95; API read latency under 100 ms for a cached trending snapshot.
• Availability: 99.95%+, since this is a highly visible, high-traffic surface, though brief staleness is more tolerable than a hard outage.
• Consistency: eventual consistency is not just acceptable but expected — trending is inherently an approximate, continuously-refreshed signal, not a transactional record.
• Throughput: must absorb extreme, unpredictable spikes in the input event stream itself (the same real-world event that causes a topic to trend also causes overall platform activity to spike).
• Accuracy vs. cost trade-off: approximate counting techniques are explicitly acceptable in exchange for massive scalability.

3.3 Back-of-the-Envelope Estimation

3.4 Memory Footprint of Approximate Structures

It is worth walking through why approximate counting is not just a nice-to-have but close to mandatory at this scale. If the platform tracks meaningful statistics for even five million concurrent topics, and each topic needs a handful of counters (short window count, long window baseline mean, baseline variance, distinct-user estimate), exact tracking with typical data structures could easily consume many tens of gigabytes of actively-updated memory, replicated across every parallel processing node for redundancy. A Count-Min Sketch and a HyperLogLog structure, by contrast, can represent an entire high-cardinality stream in a few kilobytes to a few megabytes total, regardless of how many distinct topics exist, because their memory footprint is fixed by design rather than growing with cardinality. This is the difference between a system that scales roughly linearly with the number of topics being tracked, and one that scales with a small constant, which is exactly the property needed when the set of “topics being discussed right now” is effectively unbounded.

Let us look at the full picture before diving into individual pieces. Every box in the diagram below names the concrete technology or role it plays, exactly as you would want to present it on a whiteboard.

4.1 Reading the Diagram Left to Right

Every user action across the platform is emitted as an event into the ingestion layer’s Kafka firehose. A stream processing layer continuously extracts topics, maintains sliding-window counters using approximate data structures, and scores each topic for how anomalous its current activity level is compared to its own baseline. A ranking service turns those scores into an ordered top-K list per region and category, storing the live result in Redis. From there, two delivery paths exist simultaneously: a fast-cache, mostly-static path through the CDN for casual visitors, and a live push path through a WebSocket/SSE gateway for anyone actively watching the page, so it updates without a manual refresh.

Now let us walk through every box: what it is, why it exists, where it sits in this system, a simple analogy, and how it behaves in production.

5.1 GeoDNS

Translates a domain name into an IP address, choosing the nearest healthy regional cluster based on where the request originates, so a viewer in Tokyo is routed to infrastructure in or near Japan rather than across the ocean, shaving critical milliseconds off a latency budget that is already tight.

5.2 WAF & DDoS Protection

A Web Application Firewall filters malicious traffic patterns before they reach application servers. This matters especially here because trending pages are an attractive target for coordinated manipulation attempts trying to force a topic onto the page artificially, and the WAF is the first filtering layer against obviously automated abusive traffic.

5.3 Global Load Balancer

Distributes incoming HTTP and WebSocket connection requests across many backend instances so no single machine is overwhelmed, and so failed instances are automatically routed around. At this scale, load balancing happens in layers: a global Layer-7 balancer for geographic routing, and per-service internal load balancers spreading requests across each microservice’s instance pool.

5.4 API Gateway

The single entry point for standard request-response API calls (fetching the current trending snapshot, fetching trend details). Handles authentication token validation, rate limiting, and routing, so individual backend services do not each reimplement these cross-cutting concerns.

5.5 WebSocket / SSE Gateway

A specialized entry point that maintains long-lived, persistent connections with actively viewing clients, so that when the Ranking Service produces an updated trending list, it can be pushed to every open browser tab and app screen immediately, instead of clients having to repeatedly poll an API and waste both bandwidth and time discovering nothing has changed yet.

5.6 Producer Services

Every part of the platform that generates a user action — the posting service, the search service, the reactions service, the sharing service — emits a lightweight event describing that action onto the event bus. These producers do not know or care who consumes the events; they simply publish and move on, which decouples the trending pipeline entirely from the core product services generating the underlying activity.

5.7 Event Bus (Kafka)

The backbone of the entire ingestion layer. Kafka accepts an extremely high-throughput stream of events, partitioned (commonly by a hash of the extracted topic or entity) so that all events about the same topic land on the same partition, letting downstream consumers maintain per-topic state efficiently without needing to coordinate across the whole cluster for every single update.

5.8 Topic / Entity Extraction Service

Raw events (a search query, a post’s text, a hashtag) need to be mapped to a normalized topic or entity identifier before they are useful for aggregation — “Taylor Swift,” “#TaylorSwift,” and “taylor swift” all need to collapse into the same underlying entity. This service applies lightweight natural language processing and entity-linking to tag each incoming event with the canonical topic(s) it relates to.

5.9 Stream Processor (Flink)

The computational heart of the pipeline. A distributed stream processing engine like Apache Flink maintains continuously updated sliding-window aggregates — for example, a rolling five-minute count and a rolling one-hour count per topic — processing the Kafka firehose in real time rather than in periodic batch jobs, which is what makes single-digit-second latency achievable in the first place.

5.10 Approximate Counting Workers

At this event volume, keeping an exact count of every mention of every topic, updated in real time, is prohibitively expensive in both memory and compute. Probabilistic data structures like Count-Min Sketch (for approximate frequency counting) and HyperLogLog (for approximate distinct-user counting) trade a small, bounded amount of accuracy for dramatic reductions in memory footprint, making it feasible to track millions of concurrent topics with fixed, predictable resource usage.

5.11 Spike Scoring Service

Takes the continuously updated counters and asks, per topic, “is the current short-window rate unusually high compared to this topic’s own established baseline?” using statistical techniques like exponentially weighted moving averages and z-scores, described in detail later in this article. This is what actually distinguishes “trending” from “merely popular.”

5.12 Spam and Manipulation Filter

Runs candidate trending topics through checks for coordinated inauthentic behavior — a burst of activity from a suspiciously narrow set of accounts, newly created accounts acting in a synchronized pattern, or other bot-like signatures — before a topic is allowed to surface, protecting the integrity of the page from manufactured trends.

5.13 Ranking Service

Consumes scored, filtered topics and produces the final ordered top-K trending list per region and category, writing the result into Redis for fast reads and pushing it out through the WebSocket gateway for live updates.

5.14 Redis Cluster

Holds the live, continuously updated trending list using sorted sets, which are a natural fit since they maintain elements ordered by score with efficient insert, update, and top-K retrieval, letting the API Gateway or CDN origin fetch “top 20 trending topics right now” in a single fast operation.

5.15 Time-Series Database

Stores historical per-topic activity data (Druid, ClickHouse, or a similar system built for high-cardinality time-series aggregation) that the Spike Scoring Service reads from to establish each topic’s expected baseline behavior, and that product analytics and post-incident review can query later.

5.16 Object Storage (Raw Event Archive)

Every raw event is also durably archived to cheap, long-term object storage, both for compliance and so that the stream processing pipeline can be replayed from a specific point in time if a bug in the scoring logic is discovered and needs to be corrected retroactively.

5.17 Service Discovery & Config Service

Service discovery lets components locate each other’s current network addresses dynamically as instances scale up and down. The config service centralizes tunable parameters — decay rates, spike thresholds, region definitions — so they can be adjusted live without a full redeployment.

5.18 Observability Stack

Prometheus and Grafana track pipeline throughput, processing lag, and scoring latency. An ELK or Loki stack aggregates logs across every stage of the pipeline. Distributed tracing stitches together the path of a single event from ingestion through to its effect on the final ranked list, which is essential for debugging exactly where latency accumulates in a pipeline with this many stages.

5.19 Internal Pub/Sub Fan-Out Layer

Sitting between the Ranking Service and the fleet of WebSocket Gateway instances, this lightweight internal publish-subscribe layer (often built on Redis Pub/Sub or a dedicated broker) lets a single ranking update be broadcast once and received by every gateway instance in parallel, each of which then relays it to its own set of connected clients. Without this layer, the Ranking Service itself would need to know about and directly message every individual gateway instance, creating an unnecessary and fragile coupling between a compute service and the delivery infrastructure’s ever-changing instance count.

5.20 CDN Origin Refresh Mechanism

The CDN serves a cached snapshot of the trending list to the large majority of casual, non-live-viewing traffic. A small background process periodically pulls the freshest snapshot from Redis and pushes a cache invalidation or refresh to the CDN’s origin, on a cadence tuned to balance freshness against origin load — often every few seconds — which is looser than the sub-second live-push guarantee given to actively connected WebSocket clients, but still far fresher than a typical CDN cache policy for less time-sensitive content.

Let us trace the full journey of a single event, step by step, from the moment a user does something on the platform to the moment that action might contribute to a topic appearing on someone else’s trending page.

6.1 Why Sliding Windows Instead of Fixed Buckets?

A naive approach might bucket events into fixed one-minute intervals and compare bucket-to-bucket. The problem is that a fixed-bucket approach can miss or misjudge a spike that straddles a bucket boundary, and it updates in discrete jumps rather than smoothly. A sliding window continuously represents “the last 5 minutes as of right now,” recalculated incrementally as new events arrive and old events age out, giving a much smoother and more immediately responsive signal.

6.2 Why Run the Spam Filter After Scoring, Not Before?

Running expensive spam and manipulation detection on every single incoming event, before it is even known whether the associated topic is anywhere close to trending, would waste enormous compute on the overwhelming majority of events that will never come close to the trending threshold. Instead, the pipeline only invests spam-detection effort on the comparatively small set of topics that have already crossed the statistical spike threshold, keeping the expensive check proportional to what actually matters.

6.3 How Sliding Windows Are Actually Maintained Incrementally

It helps to be precise about what “sliding window” means computationally, since it is a common follow-up question. Rather than re-summing every event in the last five minutes from scratch every time a new event arrives, which would be wasteful and slow, the stream processor maintains a running total and incrementally adds each new event’s contribution while separately subtracting the contribution of events that have just aged out of the window, often using a structure like a time-bucketed ring buffer under the hood. This means the cost of updating a topic’s windowed count stays constant regardless of how much total activity has occurred, which is essential for keeping per-event processing overhead low even for the platform’s most active, highest-volume topics.

6.4 What Happens During a Momentary Processing Delay

If the stream processing layer falls slightly behind, perhaps due to a transient burst in event volume, the system does not simply produce incorrect output — it produces slightly stale output for a brief period, since Kafka retains the backlog of unprocessed events durably rather than dropping them. As soon as processing capacity catches up, the pipeline works through the backlog and the trending list converges back to a fresh, accurate state. This graceful-staleness-rather-than-data-loss property is one of the most valuable characteristics of building this system on a durable, replayable event log rather than a fire-and-forget messaging system.

A topic moves through a recognizable lifecycle as attention around it rises and falls. Modeling this explicitly as a state machine makes the ranking and suppression logic much easier to reason about.

7.1 Each Transition Is a Concrete Pipeline Decision

Each transition in this state machine corresponds to a concrete pipeline decision. The move from Baseline to Rising happens continuously inside the Flink stream processor as counters update. The move from Rising to Candidate is the exact moment the Spike Scoring Service’s z-score crosses the configured threshold. The UnderReview state is deliberately explicit rather than skipped, because it is the checkpoint where the spam and manipulation filter gets a chance to block a manufactured trend before it ever reaches a real viewer.

7.2 Debouncing to Prevent Flicker

Without care, a topic hovering right at the spike threshold could rapidly toggle between Trending and Fading as its score oscillates around the boundary, causing the visible list to flicker distractingly. A small hysteresis gap — requiring a topic to fall meaningfully below the threshold, not just barely below it, before transitioning out of Trending — smooths this out, trading a small amount of responsiveness for a much more stable, trustworthy-looking page.

Different data in this pipeline has very different volume, access pattern, and consistency needs, so, as with most large-scale systems, a single storage technology cannot serve everything well.

8.1 Why Redis Sorted Sets Specifically

A Redis sorted set stores members ordered by an associated numeric score, and supports operations like “give me the top 20 members by score” in logarithmic time. This maps almost perfectly onto “give me the top 20 trending topics right now” — the Ranking Service simply updates a topic’s score as new scoring output arrives, and reads are essentially free in terms of computation.

8.2 Sharding the Live Ranking Store

Rather than one giant global sorted set, the system maintains separate sorted sets per region and category combination (for example, trending:us:sports or trending:in:entertainment), which naturally shards the write and read load across many smaller keys instead of funneling every single update through one massive, contended structure.

8.3 Checkpointing in the Stream Processor

Flink periodically checkpoints its in-memory windowed state to durable storage, so that if a processing node crashes, it can resume from the last checkpoint rather than losing all accumulated counter state and needing to replay the entire event history from scratch. This checkpointing interval is itself a trade-off: more frequent checkpoints mean faster, less lossy recovery, at the cost of more overhead during normal operation.

8.4 Why a Time-Series Database Specifically for Baselines

Establishing a reliable baseline for a topic benefits from being able to efficiently query “what did this topic’s activity look like at this same time of day over the past several weeks,” since many topics have entirely normal, predictable daily and weekly rhythms — a sports topic naturally spikes during game hours, a finance topic naturally spikes during market open. General-purpose relational databases are not optimized for this kind of high-cardinality, time-bucketed aggregation at the volume this system produces, which is exactly the gap that purpose-built time-series and analytical databases like Druid or ClickHouse are designed to fill, offering fast rollups and range queries over billions of historical data points.

9.1 Example API Surface

GET  /v1/trending?region=IN&category=sports&limit=20
GET  /v1/trending/{topicId}/detail
GET  /v1/trending/{topicId}/history?window=24h
WS   /v1/trending/stream?region=IN&category=all
POST /v1/internal/events                 // producer-to-ingestion, internal only
GET  /v1/internal/pipeline/health

9.2 Synchronous vs. Asynchronous Boundaries

The read path (a client asking “what is trending right now”) is synchronous and must be fast, typically served straight from the Redis-backed cache with the API Gateway adding only authentication and rate limiting overhead. The write/compute path (raw events flowing through extraction, scoring, and ranking) is entirely asynchronous and event-driven, which is precisely what allows it to absorb massive, bursty throughput without ever blocking the fast, synchronous read path that users actually interact with.

9.3 A Simplified Java Example: The Spike Scoring Logic

Below is a simplified Java implementation showing how the Spike Scoring Service might compute a z-score-style anomaly signal for a single topic using an exponentially weighted moving average as the baseline. It is intentionally simplified for teaching purposes.

@Service
public class SpikeScoringService {

    // Smoothing factor for the exponentially weighted moving average.
    // Smaller alpha means the baseline adapts more slowly to recent activity.
    private static final double ALPHA = 0.05;
    private static final double SPIKE_THRESHOLD_Z_SCORE = 3.5;

    private final TopicBaselineStore baselineStore;

    public SpikeScoringService(TopicBaselineStore baselineStore) {
        this.baselineStore = baselineStore;
    }

    public ScoringResult scoreTopic(String topicId, double currentWindowRate) {
        TopicBaseline baseline = baselineStore.get(topicId);

        double previousMean = baseline.getMean();
        double previousVariance = baseline.getVariance();

        // Update the exponentially weighted moving average and variance.
        double delta = currentWindowRate - previousMean;
        double newMean = previousMean + ALPHA * delta;
        double newVariance = (1 - ALPHA) * (previousVariance + ALPHA * delta * delta);

        baselineStore.update(topicId, newMean, newVariance);

        double stdDev = Math.sqrt(Math.max(newVariance, 1e-6));
        double zScore = (currentWindowRate - newMean) / stdDev;

        boolean isSpike = zScore >= SPIKE_THRESHOLD_Z_SCORE
            && currentWindowRate >= baseline.getMinimumVolumeFloor();

        return new ScoringResult(topicId, zScore, isSpike);
    }
}

This section is worth its own dedicated treatment, because the scoring algorithm is the single most important intellectual piece of this entire system — everything else is infrastructure built to make this calculation possible at scale and speed.

10.1 Exponentially Weighted Moving Average (EWMA) as a Baseline

Rather than storing a topic’s entire history to compute a true average, an EWMA keeps a single running value that gets nudged toward each new observation, weighted so that recent observations matter more than old ones but nothing is ever fully forgotten. Formally, at each step:

$$mu_t = alpha cdot x_t + (1 – alpha) cdot mu_{t-1}$$

This gives an efficient, constant-memory way to track “what does normal currently look like for this topic,” while still letting that definition of normal drift slowly over time as genuine long-term popularity shifts (a topic that was once niche and has organically grown a bigger audience should have its baseline rise accordingly, without every historical data point needing to be reprocessed).

10.2 Z-Scores for Spike Detection

A z-score expresses how many standard deviations the current observation is from the baseline mean:

$$z = frac{x_t – mu_t}{sigma_t}$$

A high z-score means the current activity level is a statistical outlier relative to that specific topic’s own normal pattern, which is exactly the “sudden shift in attention” signal the whole system is trying to detect, independent of whether the topic is normally huge or normally tiny.

10.3 Count-Min Sketch for Approximate Frequency Counting

A Count-Min Sketch is a probabilistic data structure that can estimate how many times an item has been seen using a small, fixed amount of memory, at the cost of occasionally overestimating counts (it never underestimates). This trade-off is extremely favorable at this scale: tracking exact counts for tens of millions of distinct topics would require memory proportional to the number of distinct topics, while a Count-Min Sketch’s memory footprint stays fixed and small regardless of how many distinct topics exist.

10.4 HyperLogLog for Distinct-User Counting

A topic that gets a thousand mentions from a thousand different people is a much stronger organic trending signal than a topic getting a thousand mentions from twenty people repeatedly. HyperLogLog estimates the number of distinct elements (unique users engaging with a topic) in a stream using a remarkably small, fixed amount of memory, letting the scoring pipeline factor in engagement diversity, not just raw volume, without the prohibitive memory cost of tracking an exact set of every user who has touched every topic.

10.5 Decay Functions

Beyond the sliding window itself, individual event contributions are often weighted with a decay function so that an event from thirty seconds ago contributes more to the current score than an event from four minutes and thirty seconds ago, even though both fall inside a five-minute window. This produces a smoother, more immediately responsive signal than a hard cutoff window, where a topic’s score would otherwise drop abruptly the instant an old burst of activity ages out of the window entirely.

10.6 Choosing Window Lengths in Practice

Picking the actual duration for the short-term “current activity” window and the longer baseline window is as much a product decision as a technical one. A very short current window, such as 60 seconds, reacts almost instantly to a genuine spike but is also more susceptible to random short-term noise, especially for topics with naturally lower baseline volume. A longer current window, such as 10 minutes, produces a smoother, more statistically stable signal at the cost of a slower reaction time. Many production systems use several windows simultaneously — a very short window to catch the earliest signs of an emerging spike, and a medium window to confirm it is sustained rather than a single noisy burst — and only promote a topic to the visible trending list once both signals agree, trading a small amount of additional latency for meaningfully higher precision.

10.7 Category- and Region-Specific Threshold Tuning

A single global z-score threshold rarely works well across an entire platform, because different categories and regions have very different natural volatility. A small regional forum topic might naturally have huge relative swings in mention volume simply due to low absolute numbers, while a globally popular entertainment category has enormous absolute volume but proportionally much smaller relative swings. Production systems typically maintain separate baseline models and thresholds per category and per region, sometimes further refined with machine-learned threshold calibration that adapts over time as a category’s typical volatility pattern shifts, rather than relying on a single hand-tuned constant applied everywhere.

10.8 Combining Multiple Signals Into a Single Score

In practice, the final ranking score for a candidate trending topic is rarely just the raw z-score alone. It is typically a weighted combination of several signals: the statistical anomaly score itself, the distinct-user engagement diversity from HyperLogLog, a freshness or recency factor from the decay function, and sometimes a manually curated boost or suppression list maintained by an editorial or trust-and-safety team for known sensitive situations. Combining these into one final ranking score, rather than relying on any single signal in isolation, produces a noticeably more robust and harder-to-manipulate ordering than any one technique could achieve alone.

11.1 Key Trade-off: Freshness vs. Stability

This is the defining tension of the whole system. A very short sliding window and an aggressive, low spike threshold make the page maximally fast and sensitive, but also maximally prone to noisy, flickering, short-lived spikes that do not reflect anything genuinely significant. A longer window and stricter threshold make the page calmer and more trustworthy, but slower to catch real breaking moments. Production systems typically resolve this not with one single setting, but with multiple tiers — for instance, a fast, sensitive “just happening” section alongside a more conservative, stable main trending list — giving users both signals without forcing one universal trade-off.

11.2 Key Trade-off: Infrastructure Cost vs. Latency Guarantee

Every additional second shaved off the end-to-end latency budget tends to cost meaningfully more in infrastructure, since it typically means smaller batch sizes, more frequent checkpointing, tighter autoscaling triggers with more headroom held in reserve, and more expensive, lower-latency storage and networking choices throughout the pipeline. A platform needs to be deliberate about where on this curve it actually needs to sit: a five-second latency target is achievable with meaningfully less infrastructure spend than a one-second target, and the right choice depends on how much genuine product value the difference actually delivers, rather than defaulting to the fastest technically achievable number without weighing its cost.

12.1 Horizontal Scaling of the Stream Processing Layer

Flink jobs are parallelized across many task slots, with the event stream partitioned (typically by topic hash) so that each parallel instance owns a consistent subset of topics. Adding more processing capacity means adding more parallel task slots and, correspondingly, more Kafka partitions, allowing the pipeline’s throughput ceiling to scale roughly linearly with added compute.

12.2 Handling the “Same Event Causes the Spike” Problem

A defining characteristic of this system, different from many others, is that the exact real-world event that causes a topic to trend also tends to cause an overall platform-wide traffic and event-ingestion spike at the same moment. The system must be provisioned with meaningful headroom above steady-state average load specifically to absorb this correlated spike, and autoscaling policies need to react within seconds, not minutes, since a slow scale-out response directly undermines the feature’s entire value proposition during exactly the moments it matters most.

12.3 Fan-Out for the Push Layer

When the ranked list updates, that update needs to reach every actively connected WebSocket client simultaneously. Rather than the Ranking Service pushing directly to millions of individual connections, updates are published to a lightweight internal pub/sub layer that each WebSocket Gateway instance subscribes to, letting the fan-out to end clients happen in parallel across many gateway instances rather than funneling through a single bottlenecked broadcaster.

12.4 Regional Sharding of Computation

Since trending lists are inherently segmented by region, much of the pipeline’s computation can be regionally sharded as well — a spike in Brazil-specific topics does not need to be computed on infrastructure physically located in, or contending for resources with, a spike happening simultaneously in Southeast Asia, which both improves latency and provides natural computational isolation between regions.

13.1 Multi-Region Deployment

The system runs across multiple geographically distributed regions, both to serve read traffic with low latency worldwide and to provide resilience against a single region’s infrastructure failing entirely. Regional Kafka clusters and stream processing deployments typically operate semi-independently for regional trends, while a smaller, lighter-weight aggregation layer combines cross-region signal for genuinely global trending topics.

13.2 Circuit Breakers & Backpressure

If a downstream stage of the pipeline (say, the spam filter) slows down, the system needs explicit backpressure handling so that upstream stages do not simply pile up unbounded work and eventually run out of memory. Kafka’s own consumer-lag model naturally provides a buffer here, but stream processing frameworks also implement backpressure signaling internally, slowing ingestion at the source when downstream processing genuinely cannot keep up, trading a temporary increase in end-to-end latency for overall pipeline stability.

13.3 Graceful Degradation

Not every partial failure should take the whole feature down. If the spam and manipulation filter becomes temporarily unavailable, the system might choose to fall back to a more conservative default (holding candidate topics rather than either blocking everything or bypassing the check entirely) rather than causing a full outage of the trending page. If the live WebSocket push layer fails, clients should gracefully fall back to a slower polling or CDN-cached experience rather than the page breaking outright.

13.4 Disaster Recovery

Because the live trending state itself is inherently ephemeral and continuously recomputed from the incoming event stream, disaster recovery for this system looks different from a typical database-backed application: rather than restoring from a backup, a full regional failure is recovered from by resuming stream processing from the last durable Kafka offset and checkpoint, with the trending list naturally rebuilding itself within the pipeline’s normal end-to-end latency window once processing resumes.

13.5 Aggregating Regional Signals Into Global Trends

A topic that is trending simultaneously across many independent regions is a stronger signal of genuine global significance than a topic trending in just one place. A lightweight cross-region aggregation layer periodically compares regional trending outputs and promotes topics showing consistent, independent spikes across multiple regions into a separate global trending view. This layer runs on a slightly longer cadence than the regional pipelines themselves, since confirming a genuinely global pattern benefits from a small amount of additional data, and it deliberately does not sit on the critical path of regional trending, so a slow or temporarily unavailable global aggregation layer never degrades the core regional experience that most users actually see.

13.6 Failover Testing for Streaming Infrastructure

Just as with other critical infrastructure, the resilience mechanisms described here are only trustworthy if they are regularly exercised under controlled conditions rather than assumed to work. Deliberately failing over a stream processing node, or simulating a Kafka broker outage, in a staging or canary environment on a regular cadence verifies that checkpoint-based recovery genuinely restores processing within the expected time window, rather than discovering during a real incident that a recovery procedure everyone assumed would work actually has an undocumented gap.

14.1 Why This System Is a High-Value Manipulation Target

A “Trending Now” page carries implicit editorial authority — millions of people take its presence as a signal that something is genuinely, organically significant. This makes it a natural target for anyone wanting to artificially inflate the visibility of a topic, product, or narrative, whether through simple bot networks or more sophisticated coordinated campaigns using real but coordinated human accounts.

14.2 Bot and Coordinated Activity Detection

The Spam and Manipulation Filter looks for signatures like: an unusually narrow set of accounts driving a disproportionate share of a topic’s volume, accounts with suspiciously synchronized timing patterns, newly created or historically low-activity accounts suddenly acting in concert, and engagement patterns that do not match the organic diversity typically seen in genuine viral spread.

14.3 Authentication & Rate Limiting

Standard API authentication (JWT-based) protects both the read API and, especially, the internal event-ingestion endpoints, since an attacker able to directly inject fabricated events into the pipeline could otherwise manipulate trending output far more easily than through normal platform interactions. Rate limiting at the API Gateway also protects the read path from being used as a vector for scraping or denial-of-service attempts.

14.4 Human Review and Escalation Paths

Fully automated detection cannot catch every sophisticated manipulation attempt, so production systems typically maintain a human trust-and-safety escalation path: topics flagged as borderline by automated detection, or reported by users as suspicious, can be manually reviewed and, if necessary, suppressed even after initially passing automated checks.

14.5 Data Privacy in the Ingestion Pipeline

The raw event stream flowing through this pipeline is, at its core, a record of what large numbers of individual people are doing on the platform in near real time. Even though the trending feature only ever surfaces aggregate topic-level signals rather than any individual’s activity, the underlying pipeline still needs to treat that raw event stream with real care: stripping or hashing personally identifying details before they are needed for anything beyond the immediate scoring computation, enforcing strict access controls on the raw event archive in object storage, and ensuring that debugging tools built for engineers to inspect pipeline behavior surface aggregate statistics rather than an easy way to browse individual users’ raw activity.

14.6 Encryption and Network Isolation

As with any production system handling user activity data, all network traffic between pipeline stages uses TLS in transit, and the raw event archive is encrypted at rest. The internal Kafka cluster and stream processing infrastructure typically sit in a private network segment with no direct public internet exposure, reachable only through the narrow, well-defined internal service boundaries described earlier in this article, rather than being broadly accessible the way the public-facing API Gateway and CDN necessarily are.

15.1 Pipeline-Specific Metrics That Matter Most

15.2 The Three Pillars of Observability

Metrics (Prometheus/Grafana) provide the numeric, alertable time-series view of pipeline health. Logs (ELK/Loki) provide detailed, searchable event-level records for debugging specific incidents. Distributed tracing follows a single event’s path across every pipeline stage, which is invaluable for pinpointing exactly which stage is contributing unexpected latency when the end-to-end number degrades.

15.3 Alerting Philosophy

Alerts should center on symptoms that directly threaten the feature’s core promise: end-to-end latency exceeding target, consumer lag growing rather than shrinking, or a sudden unexplained drop in the number of topics being scored (which could indicate a silent upstream failure). Avoid alerting on every minor fluctuation in raw event volume, which is expected to vary enormously and naturally throughout a normal day.

16.1 Containerization & Orchestration

Stateless services (API Gateway consumers, WebSocket Gateway instances, the Ranking Service) run as containers on Kubernetes, benefiting from automatic scaling and self-healing. The stream processing layer typically runs on a dedicated, purpose-built cluster (a managed Flink or similar deployment) since stateful streaming workloads have different operational characteristics than simple stateless request-response services.

16.2 CI/CD for a Streaming Pipeline

Deploying changes to stream processing logic requires particular care, since a naive redeploy can lose in-flight state. Production pipelines typically use safe upgrade strategies like Flink’s savepoint mechanism, which captures a consistent snapshot of all in-flight state that the new version of the job can resume from, rather than an ordinary rolling deployment that would simply drop and recreate the processing state from scratch.

16.3 Infrastructure as Code

Kafka topic configurations, Flink job definitions, autoscaling policies, and Redis cluster topology are all defined declaratively and version-controlled, so the entire pipeline’s infrastructure is reproducible, auditable, and can be recreated in a new region or after a disaster without manual reconstruction.

17.1 Patterns Used in This System

17.2 Common Anti-Patterns to Avoid

18.1 Best Practices

• Treat the read path and the compute path as fully independent systems with independent scaling and independent failure modes.
• Use approximate, fixed-memory data structures deliberately and explicitly, understanding and documenting their accuracy trade-offs rather than treating them as a hidden implementation detail.
• Build spam and manipulation detection into the pipeline as a first-class stage from day one, not as an afterthought once abuse is discovered in production.
• Use synthetic canary events to continuously and objectively measure true end-to-end latency, rather than relying on internal per-stage metrics alone.
• Design explicit debouncing and hysteresis into the ranking logic to keep the visible page stable and trustworthy.

18.2 Common Mistakes

• Under-provisioning capacity for the exact correlated-spike scenario (a real event driving both the trending signal and overall platform load simultaneously) that the feature exists to handle.
• Tuning spike thresholds using only global defaults, ignoring that different categories and regions have wildly different natural activity baselines and noise characteristics.
• Treating checkpoint and recovery strategy as an afterthought for the stream processing layer, leading to painful, slow recovery after any node failure.
• Forgetting that the trending list is fundamentally a product surface with real-world consequences (amplifying misinformation, enabling manipulation) and under-investing in the trust-and-safety layer relative to the pure engineering.
• Coupling the visible ranking logic too tightly to a single hard-coded threshold, rather than exposing key parameters through the config service so they can be adjusted quickly in response to real-world calibration issues without a full redeployment.

18.3 A Checklist for Reviewing This Kind of System in an Interview

When presenting a design like this out loud, a short mental checklist helps ensure the conversation covers what interviewers are most likely to probe:

• Have I clearly separated the always-fast, cached read path from the asynchronous, event-driven compute path, and explained why that separation matters for latency guarantees?
• Have I explained specifically what makes a topic “trending” rather than just “popular,” using a concrete statistical mechanism rather than a vague notion of a spike?
• Have I addressed the memory and scalability implications of tracking millions of concurrent topics, and why approximate structures are the right tool here?
• Have I discussed how the system defends against manipulation, given that this feature carries implicit public trust?
• Have I addressed what happens during a correlated spike in both the input event stream and the trending computation itself, since that is exactly the scenario the feature exists to handle well?

19.1 A Common Thread Across These Systems

Looking across these examples, a consistent pattern emerges: every platform that has successfully built a genuinely real-time trending or anomaly-detection feature at scale eventually converged on the same core architectural shape described throughout this article — an event-driven ingestion layer decoupled from a streaming computation layer, approximate data structures to make high-cardinality tracking affordable, and a strict separation between the always-fast cached read path and the continuously running compute path. This convergence, arrived at somewhat independently across different companies and different problem domains, is a strong signal that these are not arbitrary implementation choices but genuinely load-bearing architectural decisions that any system with similar latency and scale requirements is likely to need in some form.

19.2 Where These Systems Tend to Differ

The differences between real-world implementations tend to show up less in the core architecture and more in the specific tuning and trust-and-safety layers: how aggressively a platform weighs freshness against stability, how sophisticated its manipulation-detection models are, and how much editorial or human review sits alongside the fully automated pipeline. These differences usually reflect each platform’s specific risk profile and audience expectations rather than a fundamentally different technical approach, which is a useful thing to point out when discussing this system in an interview setting, since it shows an understanding of what is architecturally fundamental versus what is a tunable business decision layered on top.

20.1 Frequently Asked Questions