Open Instagram and tap the little magnifying glass. Open TikTok and you land on “For You” without tapping anything at all. Open Pinterest and the entire home screen is one enormous river of pins you never asked for by name. All of these are the same idea wearing different clothes: a page that shows a user things from people and creators they do not follow yet, chosen automatically by software rather than by the user’s own choices.

This is different from a normal “Home” or “Following” feed. A following feed is easy to reason about — you show a person the posts from the accounts they already follow, roughly in time order, maybe with some ranking. The explore page throws that constraint away. There is no follow graph to lean on. The system has to guess, for a stranger it has never been explicitly told anything about, which of literally billions of pieces of content that stranger would enjoy enough to stop scrolling for.

That single sentence — guess what a person wants, from everything that exists, without being told — is one of the hardest, highest-value problems in modern software engineering. It sits at the intersection of distributed systems, machine learning, and product design, and it is asked about constantly in system design interviews because it forces a candidate to reason about scale, latency, personalization, and trade-offs all at once.

A short history of discovery systems

Recommendation and discovery systems did not start on social apps. Their intellectual ancestors are much older, and it’s worth walking through the arc briefly because every idea layered onto today’s explore feeds echoes something from this history.

The system this tutorial designs draws from all of that lineage: a candidate-generation stage inspired by collaborative filtering and embeddings, a ranking stage inspired by YouTube’s and Instagram’s neural ranking work, and a real-time signal pipeline inspired by TikTok’s approach to freshness.

What this tutorial will and won’t cover

This tutorial focuses specifically on the system design and infrastructure side of the problem: how requests flow through the system, how services are organized, how data is stored and moved, and how the whole thing stays fast and reliable at scale. It deliberately treats the internals of any specific machine learning model (the exact neural network architecture, loss function, or training hyperparameters) as a supporting detail rather than the main subject — the goal is to understand the shape of the system a ranking model lives inside, which is largely independent of which specific model you eventually choose to plug in. Where a modeling concept is essential to understanding an architectural decision — embeddings, the two-tower model, calibration — it’s explained at a conceptual level, with pointers to where a deeper dive would matter, rather than derived mathematically.

Before diving into architecture diagrams and service boundaries, it’s worth spending a moment on why a platform would take on all this complexity in the first place — because every engineering choice in the rest of this tutorial only makes sense once the underlying business pressure is clear.

Every social or content platform faces the same cold, uncomfortable fact: the average user follows a small, static set of accounts, and that set grows slowly. If the only thing a platform ever shows a user is content from people they already follow, the platform has a growth ceiling. New creators struggle to be discovered because nobody already follows them. Users eventually run out of fresh content from their existing follow list and leave the app. Advertisers cannot reach new audiences beyond a creator’s existing base.

The explore page exists to solve three business problems simultaneously:

The core engineering problem

Strip away the product framing and the engineering problem is this: given a user u, and a universe of candidate items I that can run into the billions, return an ordered list of k items (typically 20–50 for one page load) that maximizes some combination of engagement, relevance, and platform health — within a latency budget of roughly 100–200 milliseconds, for hundreds of millions of users, continuously, as both the user population and the content corpus change every second.

That definition immediately exposes four sub-problems that shape everything else in this tutorial:

1. The search space problem

You cannot score a billion items for every single request. Scoring one item with a neural network might take a millisecond; scoring a billion sequentially would take over 11 days per request. The system needs a funnel: cheaply narrow billions down to thousands, then thousands down to hundreds, then hundreds down to the final twenty — spending more computation only on candidates that survived the earlier, cheaper filters.

2. The cold-start problem

A brand-new user has no history. A brand-new piece of content has no engagement data. Both are “cold.” The system must have a fallback strategy — popularity-based, demographic-based, or content-similarity-based — for exactly these cases, or new users and new creators are permanently disadvantaged.

3. The freshness vs. accuracy trade-off

The most accurate ranking model is one retrained on the very latest behavior data. But retraining and redeploying large models constantly is expensive and slow. Real systems compromise: a large model retrained periodically (hours to a day), combined with lightweight real-time signals (what the user did in the last five minutes) blended in at serving time.

4. The filter bubble / diversity problem

An algorithm purely optimizing for “engagement” will happily show a user the same ten cat videos forever, because that maximizes short-term watch time. Left unchecked this produces filter bubbles, echo chambers, and eventually user fatigue. The system needs deliberate diversity and exploration mechanisms, not just exploitation of what already works.

“The goal of the explore page is not to show the best content in the world. It is to show the best content for this specific person, right now, that they haven’t already seen.”

— A useful mental model for the entire system

Formalizing “good”: what metric are we actually optimizing?

Before any architecture can be designed, engineers need a precise, measurable definition of success — vague goals like “show good content” don’t translate into a loss function a model can be trained against. In practice, teams define a composite objective, typically a weighted sum of several predicted probabilities: probability of a like, probability of a share or save, probability of watching to completion (for video), probability of following the creator, and a negative weight for probability of an explicit “not interested” signal or a report. The weights themselves are product decisions, not purely technical ones, and they get revisited regularly as the platform’s priorities shift — for instance, weighting “follow” more heavily when the business goal is creator-graph growth, versus weighting “watch time” more heavily when the goal is session length.

It’s worth being explicit that this weighted-sum objective is itself a trade-off, not a solved problem. Optimizing too heavily for short-term engagement signals (likes, watch time) can inadvertently reward sensational or addictive content patterns; optimizing too heavily for “meaningful” signals like shares or follows can under-serve content types that are enjoyable but less shareable. Most mature platforms pair their engagement-based objective with periodic human-rated surveys (“was this a good use of your time?”) specifically to catch cases where the automated metric and genuine user satisfaction start to diverge.

Before drawing any boxes and arrows, let’s build a shared vocabulary. Every term below will reappear constantly in the rest of this tutorial, so take the time to internalize each one with its analogy. Skimming this section and coming back to it later, once a term reappears in a later architecture discussion, is a perfectly reasonable way to read it too — the goal is that by the end of Section 4, none of the labels on the architecture diagram feel unfamiliar.

3.1 Candidate generation (a.k.a. retrieval)

What: The first stage of the pipeline. Its job is to shrink billions of possible items down to a few thousand plausible ones, quickly and cheaply.

Why: Because you cannot afford to run an expensive, precise ranking model on every item that exists — you need a coarse, fast filter first.

Where: Runs first in the pipeline, often across multiple parallel “candidate sources” whose results get merged.

Beginner example: “Show me items similar to the last 5 posts this user liked” is a simple candidate generator.

Production example: Pinterest’s discovery system uses multiple candidate generators in parallel — one based on the user’s recent “pin” interactions, one based on user-to-user similarity (people like you), one based on trending/popular content, and one based on the boards a user has saved to — then merges all of their outputs.

3.2 Ranking

What: The second stage. Takes the smaller candidate set (thousands) and scores each one precisely using a more expensive model, usually a neural network, to predict the probability of a positive outcome (like, share, watch-to-completion, follow, etc).

Why: Precision matters most for the items that will actually be shown, so it’s worth spending more compute here — but only here, because the set is now small.

3.3 Re-ranking / business-logic layer

What: A final pass over the ranked list that applies rules unrelated to raw predicted engagement: diversity (don’t show 5 posts from the same creator in a row), content policy filters (remove content the user has muted, blocked, or that violates guidelines for that user’s region/age), and business objectives (occasionally insert a promoted or sponsored item).

Why: A pure ML ranking score, applied blindly, can create a bad or even unsafe user experience. Business rules act as guardrails on top of the model.

3.4 Embeddings

What: A way of representing a user, or a piece of content, as a fixed-length vector of numbers (say, 128 or 256 floating-point numbers) such that “similar” things end up close together in that vector space.

Why: Computers can’t natively understand “this photo is about hiking” — but they can compute the mathematical distance between two vectors extremely fast. Embeddings translate messy, human concepts (taste, topic, style) into geometry a machine can search over efficiently.

Software example: A user embedding might be built from an average of the embeddings of the last 50 posts they engaged with. A content embedding might come from a neural network that looked at the image, the caption text, and the audio track together.

3.5 Approximate Nearest Neighbor (ANN) search

What: An algorithm/index structure (e.g., HNSW, IVF-PQ, ScaNN) that finds vectors close to a query vector in high-dimensional space, without checking every single vector — trading a small amount of accuracy for a massive amount of speed.

Why: An exact nearest-neighbor search over a billion 256-dimensional vectors is far too slow for a 150ms budget. ANN indexes make this a few milliseconds by cleverly organizing the vector space in advance.

3.6 Feature store

What: A specialized system that stores and serves pre-computed “features” (numeric or categorical signals) about users and content — e.g., “this user’s average session length,” “this post’s like-rate in the last hour” — with very low read latency.

Why: Ranking models need dozens to hundreds of input features per candidate, computed consistently between training and serving. Recomputing them all live, per request, from raw data would be far too slow.

3.7 Collaborative filtering vs. content-based filtering

3.8 The two-tower retrieval model

What: A neural network architecture split into two independent halves — a “user tower” that turns a user’s features into an embedding, and an “item tower” that turns a content item’s features into an embedding, in the same vector space — trained so that the dot product (or cosine similarity) between a user’s vector and an item’s vector approximates how likely that user is to engage with that item.

Why it matters: Because the two towers are independent, the item tower can be run once, offline, for every piece of content in the corpus, and the resulting vectors indexed in the Vector Database ahead of time. At request time, only the (cheap) user tower needs to run live, and the rest becomes a fast ANN lookup. This split is precisely what makes embedding-based candidate generation feasible within a tight latency budget — you are never running a full neural network per candidate at serving time, only once per user request.

3.9 CAP theorem, applied to this system

What: The CAP theorem states that a distributed data store can only guarantee two of three properties at once during a network partition: Consistency (every read sees the latest write), Availability (every request gets a response), and Partition tolerance (the system keeps working despite network splits between nodes). Because network partitions are a real, unavoidable possibility in any distributed system, the practical choice is really between consistency and availability when a partition occurs.

Where it applies here: Different stores in this architecture deliberately make different choices:

This is a recurring theme worth internalizing: a large system is never “CP” or “AP” as a whole — each component makes its own, deliberate choice based on what happens if it’s wrong versus what happens if it’s unavailable.

Now we assemble every concept from Section 3 into one coherent, end-to-end architecture. The diagram below shows every major component a production explore-page system needs — from the client device all the way down to offline model training — including the infrastructure layer (API Gateway, Load Balancer) that every real system needs but that tutorials often skip.

Component-by-component breakdown

Load Balancer

Role: The very first infrastructure component every client request hits. Distributes incoming traffic across many identical API Gateway instances so no single machine is overwhelmed. Typically an L7 (application-layer) load balancer capable of routing based on HTTP path, using algorithms like round-robin, least-connections, or weighted routing for canary releases.

Why it’s here: Without it, a single gateway instance becomes both a bottleneck and a single point of failure. In production this is usually a managed layer (AWS ALB/NLB, GCP Cloud Load Balancing, NGINX, Envoy, or HAProxy).

API Gateway

Role: The single, well-defined front door for all client requests. Handles authentication (verifying the user’s session token), authorization, per-user and per-IP rate limiting (so one client can’t hammer the explore endpoint thousands of times a second), request validation, and routing to the correct backend service. It also often handles response caching headers and request/response logging for observability.

Why it’s here: Centralizing these cross-cutting concerns means individual services (candidate generation, ranking, etc.) don’t each need to reimplement auth and rate limiting — they trust that anything reaching them has already been validated.

Explore Orchestrator Service

Role: The “conductor.” Receives the validated request from the API Gateway, calls the candidate generation service, waits for (or times out on) its results, passes them to the ranking service, then to the re-ranking/business-rules service, assembles the final response, and returns it. Often implements fan-out/fan-in patterns, calling multiple candidate sources in parallel.

Candidate Generation Service

Role: Runs several retrieval strategies in parallel — ANN vector similarity search, “people similar to you” collaborative filtering lookups, trending/popular content lookups, and graph-based signals (friends-of-friends) — then merges and de-duplicates results into a candidate pool of a few thousand items. Itself horizontally scaled behind an internal load balancer.

Ranking Service (model serving)

Role: Loads a trained ML model (often served via TensorFlow Serving, TorchServe, or a custom low-latency inference server) and scores each candidate using features pulled from the Feature Store. Runs as a pool of stateless, horizontally scaled instances, load balanced internally, often with GPU or specialized inference hardware for large models.

Re-ranking / business rules service

Role: Applies diversity constraints (no more than N consecutive posts from one creator), safety/policy filters, deduplication against recently-seen content, and business logic like inserting a sponsored post every M positions.

Feature Store

Role: A low-latency key-value store (Redis, DynamoDB, or a purpose-built system like Feast) holding pre-computed features about users and content, read by both candidate generation and ranking, and written to continuously by the stream processor.

Vector Database / ANN index

Role: Stores embeddings for users and content and serves approximate nearest-neighbor queries in milliseconds. Sharded across many nodes for a billion-scale content corpus, itself sitting behind internal load balancing so no single shard becomes a hotspot.

Explore result cache

Role: Caches the final ranked list per user for a short TTL (e.g., 2–10 minutes) so that repeated pull-to-refresh or pagination requests from the same user don’t recompute the entire pipeline every time.

Content Metadata DB & Social Graph DB

Role: Sharded, replicated relational or wide-column databases holding the canonical content records (captions, media URLs, timestamps) and the follow/block/mute graph used to exclude content the user should never see.

Message queue, stream processor, batch pipeline, model registry

Role: Every view, like, skip, and share is published as an event onto a Kafka-style message queue. A stream processor (Flink/Spark Streaming) consumes these in near real-time to update the Feature Store. A separate offline batch pipeline periodically retrains the ranking and embedding models on accumulated historical data, and pushes new model versions to a Model Registry that the Ranking Service loads from.

CDN

Role: Once the ranked list of content IDs is returned to the client, the actual images/video are served from a CDN’s edge caches, not from the origin servers, keeping media delivery fast and origin load low.

Let’s trace exactly what happens, internally, in the milliseconds after a user opens the explore tab. This is the point where the boxes in Figure 1 stop being static labels and start being a concrete sequence of calls, each with its own budget of time and its own opportunity to fail gracefully.

Stage 1: Candidate generation in detail

The Candidate Generation Service typically runs multiple independent strategies in parallel, not sequentially, and merges their results:

• Embedding similarity: take the user’s embedding vector, run an ANN query against the content vector index, get back the ~1,000 nearest content items.
• Collaborative signals: “users who share your recent likes also liked…” via a precomputed item-to-item similarity table or a two-tower retrieval model.
• Graph-based: content liked by accounts the user already follows, or by friends-of-friends, gives a social-proof signal even without a direct interest match.
• Trending/popular: a fallback pool of currently trending content, refreshed every few minutes, ensures the pipeline always has something to show even for a brand-new user with zero history (see cold-start handling in Section 15).

These parallel result sets are merged, deduplicated by content ID, and filtered against a simple exclusion list (already-seen content, blocked/muted accounts) before being passed to ranking.

Stage 2: Ranking in detail

The ranking model is usually a two-tower or gradient-boosted / deep neural network model trained to predict multiple objectives at once — probability of like, probability of share, probability of watching to completion, probability of following the creator — combined into one score via a weighted formula tuned by the product team. Each candidate’s features are fetched in a single batched call to the Feature Store (never one-by-one — batching keeps latency predictable), and the model runs one forward pass per candidate, or a single batched forward pass across all candidates for GPU efficiency.

Stage 3: Re-ranking in detail

The re-ranking pass walks the ranked list top to bottom and applies deterministic rules: skip an item if its creator already appears twice in the last five slots (diversity), skip an item that fails content-policy checks for this user’s age/region, and probabilistically insert exploratory or sponsored items at fixed intervals. This stage is intentionally cheap — simple rule evaluation, not further ML inference — because by now the candidate count is small (hundreds, not thousands).

Inside the ANN index: how HNSW actually works

It’s worth understanding, at a conceptual level, what data structure makes millisecond-scale search over a billion vectors possible, since it’s one of the most commonly probed implementation details in interviews for this kind of system. HNSW (Hierarchical Navigable Small World) builds a multi-layer graph over the vector space: the top layer has very few nodes and long-range connections, letting a search jump across large distances quickly; each lower layer has progressively more nodes and shorter-range connections, letting the search refine its position. A query starts at the sparse top layer, greedily walks toward the closest node, then drops down a layer and repeats, narrowing in on the true nearest neighbors layer by layer — similar to how a person navigating a new city might first use a highway map to get to the right neighborhood, then a street map to find the right block, then walk the final few meters by eye.

This graph structure is why HNSW search is logarithmic in the size of the corpus rather than linear — doubling the content corpus does not double search time, which is exactly the scaling property needed as a platform’s content library grows into the billions.

Concurrency inside a single Ranking Service instance

Within one Ranking Service instance, thousands of candidates need to be scored per request without blocking on I/O. Production implementations typically use an asynchronous, non-blocking execution model: feature fetches for different candidates are issued concurrently (as shown in the batching code in Section 8), and model inference itself is batched into a single tensor operation rather than looped candidate-by-candidate, letting the underlying hardware (GPU or vectorized CPU instructions) exploit data parallelism. Getting this concurrency model right is often the difference between a ranking service that meets its latency SLO and one that doesn’t, independent of how good the model itself is.

Section 5 traced a single read request. But the explore system is really two loops running at different speeds: a fast read path (what we just traced) and a slower write/feedback path that continuously improves what the read path serves. Understanding both is essential — a system that only has the read path would never learn anything new. It’s easy, when first sketching this kind of system, to draw only the read path and treat everything else as “the ML team’s problem” — but the two are inseparable in practice, since the read path’s quality is entirely a function of how well the write path has been feeding it.

Lifecycle of a single piece of content

1. Upload: Creator posts content. It’s written to the Content Metadata DB and queued for embedding generation (an offline or near-real-time job computes its content embedding from image/video/text).
2. Indexing: The new embedding is inserted into the Vector DB / ANN index, making it eligible for retrieval by candidate generation within minutes of upload.
3. Cold serving: With no engagement history yet, the content initially relies on content-based similarity and a small “exploration budget” — a small percentage of explore slots deliberately reserved for under-exposed content — to get its first impressions.
4. Warm serving: As real engagement events accumulate, the Feature Store’s item-level features (like-rate, share-rate, average watch time) update, and the ranking model starts using real signals instead of only content similarity.
5. Decay: Older content’s freshness-related features decay over time, naturally lowering its ranking score unless it continues to earn strong engagement (evergreen content).
6. Archival: Very old or consistently low-performing content may be pruned from the hot ANN index to keep the index smaller and faster, while remaining queryable through slower, colder storage if ever needed.

From raw events to training data

It’s worth tracing exactly how a single user interaction eventually becomes a labeled training example, since this connective tissue is often the least visible part of the system despite being essential to it. When a user views an explore item, the client emits an impression event (this item was shown, at this position, at this time) and, separately, an engagement event if the user liked, shared, or skipped it. Both events land on the message queue, get archived to a data lake in a columnar format optimized for analytical queries, and are later joined together in a batch job: every impression is paired with whether or not it was followed by a positive engagement within some time window, producing a labeled row of (user features, item features, context features, label) exactly the shape a supervised ranking model is trained on. This impression-to-label join is one of the most operationally important, and easiest to get subtly wrong, pieces of the entire pipeline — a bug in how impressions and engagements are matched up can silently corrupt every model trained afterward, which is why this join logic typically has its own dedicated data-quality monitoring and unit tests independent of the rest of the system.

No architectural decision in this tutorial is free. Every component that makes the explore page fast, personalized, and resilient also adds operational surface area, infrastructure spend, and engineering complexity that a simpler system wouldn’t carry. Before committing to this design for a real product, it’s worth being explicit about exactly what’s being traded away, so the decision is a deliberate one rather than a default.

Key trade-offs every design must confront

Where the latency budget goes

A realistic 150ms budget breaks down roughly like this: 10–15ms for load balancer + API gateway overhead (auth, rate limiting), 20–30ms for candidate generation (mostly the ANN vector search), 60–80ms for ranking (the most expensive stage — batched model inference), 10–15ms for re-ranking/business rules, and the remainder as network/serialization overhead. Every millisecond saved in candidate generation is a millisecond that can be spent on a more accurate ranking model, so profiling and optimizing the cheap stages pays for itself.

Horizontal scaling strategy

Every stateless service in the architecture — API Gateway, Orchestrator, Candidate Generation, Ranking, Re-ranking — is deployed as a horizontally scaled pool of identical instances behind an internal load balancer, auto-scaled based on CPU/GPU utilization and request queue depth. The stateful components (Vector DB, Feature Store, Content DB) scale differently:

Batching for throughput

The single highest-leverage performance technique in this architecture is batching at every layer: batch feature-store reads (one round trip for 3,000 candidates instead of 3,000 round trips), batch model inference (GPUs are dramatically more efficient scoring 3,000 candidates in one forward pass than 3,000 separate passes), and batch ANN queries where possible.

// Batches feature-store reads instead of issuing one call per candidate.
// This single change can cut ranking-stage latency by an order of magnitude.
public class BatchFeatureFetcher {

    private final FeatureStoreClient featureStore;
    private static final int MAX_BATCH_SIZE = 500;

    public BatchFeatureFetcher(FeatureStoreClient featureStore) {
        this.featureStore = featureStore;
    }

    // Splits a large candidate list into fixed-size batches and fetches
    // features concurrently, then merges results into one map.
    public Map<String, FeatureVector> fetchFeatures(List<String> candidateIds) {
        Map<String, FeatureVector> result = new ConcurrentHashMap<>();
        List<List<String>> batches = partition(candidateIds, MAX_BATCH_SIZE);

        List<CompletableFuture<Void>> futures = batches.stream()
            .map(batch -> CompletableFuture.runAsync(() -> {
                Map<String, FeatureVector> batchResult = featureStore.multiGet(batch);
                result.putAll(batchResult);
            }))
            .collect(Collectors.toList());

        CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
            .join();

        return result;
    }

    private List<List<String>> partition(List<String> ids, int size) {
        List<List<String>> batches = new ArrayList<>();
        for (int i = 0; i < ids.size(); i += size) {
            batches.add(ids.subList(i, Math.min(i + size, ids.size())));
        }
        return batches;
    }
}

This pattern — partition, fetch concurrently, merge — recurs throughout high-scale recommendation systems, not just for features but for any per-candidate lookup (metadata, safety flags, creator info).

Capacity planning: back-of-the-envelope math

A useful exercise, and a very common interview follow-up, is estimating the scale this system needs to handle. Take a platform with 500 million daily active users, where roughly 30% open the explore tab at least once a day, averaging 3 explore-page loads per visit. That’s 500M × 0.3 × 3 ≈ 450 million explore requests per day, or roughly 5,200 requests per second on average — but average is the wrong number to design for. Real traffic is highly diurnal, with peak-hour traffic commonly 3–5x the daily average, so the system needs to comfortably handle on the order of 20,000–25,000 requests per second at peak without breaching the latency SLO.

Each request fans out into a candidate generation call touching roughly 2,000–3,000 candidates, a batched ranking call scoring those same candidates, and a handful of feature-store lookups. Multiplying through, the Feature Store alone needs to sustain tens of millions of key lookups per second at peak, which is precisely why it’s implemented as a horizontally sharded, in-memory-first store rather than a traditional relational database — no disk-backed relational system comfortably sustains that read volume at single-digit-millisecond latency.

This kind of estimation exercise — daily active users, engagement rate, fan-out factor, peak multiplier — is the standard way to translate a product requirement into an infrastructure sizing decision, and it’s worth being able to walk through it from memory rather than needing exact numbers memorized.

A production explore page cannot simply fail when one component has an issue — showing a user a blank screen or an error is far worse for the business than showing slightly-less-perfect recommendations. The entire architecture is built around graceful degradation: every stage has a fallback that is cheaper and less accurate, but always available. Reliability, in this sense, is a design property that has to be planned into every stage from the start — it cannot be bolted on after the fact once the happy path is already built, because retrofitting a fallback into a service that was never designed to have one usually means rewriting it.

Reliability techniques used throughout

Availability targets in practice

Most large platforms target 99.9%–99.99% availability for read-heavy, user-facing endpoints like explore. At 99.9% (“three nines”), that’s roughly 8.7 hours of allowed downtime per year; at 99.99% (“four nines”), roughly 52 minutes per year. Because the explore page is not a payments or safety-critical system, teams often deliberately choose availability and low latency over perfect ranking accuracy — a slightly worse ranked list served instantly beats a perfect one served after a 2-second stall, or not served at all.

Replication and consensus underneath the data layer

Every stateful store described in Section 13 relies on replication to survive individual node failures, and replication in turn depends on a consensus protocol to agree on the true, current state across replicas when nodes disagree. The Content Metadata DB, which requires strong consistency for a creator’s post record, typically uses a leader-based replication scheme with a consensus protocol like Raft to elect a single writable leader and safely replicate writes to followers — if the leader crashes, the remaining replicas run a new leader-election round and continue serving, at the cost of a brief unavailability window measured in seconds. The Feature Store and Vector Database, by contrast, favor leaderless, eventually-consistent replication (closer to a Dynamo-style design), because their AP choice from Section 3 means a small window of staleness across replicas is an acceptable trade for never blocking on leader election.

Failure recovery in practice

When a node in any replicated store fails, the practical recovery sequence looks the same across the stack: health checks detect the failure within seconds, the load balancer or service mesh routes traffic away from the failed node, a replacement instance is provisioned (Kubernetes reschedules the pod; a managed database service spins up a replacement replica), and the new node catches up on missed state — either by streaming the replication log from a healthy replica or, for the Vector Database, by re-syncing its shard of the ANN index from the source-of-truth embedding store. None of this requires human intervention in a well-built system; it’s designed to be a routine, automatic event rather than an incident.

Security for a recommendation pipeline is not just about keeping attackers out — it’s about making sure the system’s own optimization pressure never gets turned against the platform or its users. That distinction shapes every mitigation described below: some defend the perimeter, but several exist specifically to keep the ranking model itself from being quietly gamed.

An explore/discovery pipeline is a tempting target: it decides what millions of people see, it exposes an API surface that touches user behavior data, and it’s a natural place for bad actors to try to manipulate visibility (fake engagement to boost content) or scrape data.

Threats specific to a recommendation pipeline

Defense in depth across the stack

• Edge: The Load Balancer and API Gateway terminate TLS, enforce authentication (short-lived OAuth2/JWT tokens), and apply rate limiting before any request reaches internal logic.
• Service-to-service: Internal calls between the Orchestrator, Candidate Generation, Ranking, and data stores use mutual TLS (mTLS) within a service mesh, so a compromised service can’t silently impersonate another.
• Least privilege: The Ranking Service, for example, only has read access to the Feature Store and Model Registry — it has no ability to write to the Content Metadata DB or Social Graph DB.
• Content safety layer: Independent of engagement-based ranking, a separate policy/safety classifier can veto any candidate before it’s ever shown, regardless of its predicted engagement score.

Compliance and regulatory considerations

Algorithmic feeds increasingly fall under direct regulatory scrutiny in multiple jurisdictions — rules requiring platforms to explain, in general terms, how recommendation systems work (transparency requirements), to offer users a non-personalized or chronological alternative, or to give users the ability to view and delete data used to personalize their experience. Architecturally, this means the system needs to support: exporting a user’s stored features and embeddings on request, deleting a user’s behavioral history and retraining/reindexing without it when requested, and maintaining audit logs (see Section 11) detailed enough to reconstruct why a specific recommendation was shown, in case of a regulatory inquiry. Building these capabilities in from the start is significantly cheaper than retrofitting them after the fact, since deletion and export need to reach into the Feature Store, Vector Database, and training data lake — every corner of the system that touches user data.

Recommendation systems fail silently more often than they fail loudly. The API can return a 200 OK with a perfectly well-formed but terrible list of recommendations, and nothing in a standard uptime dashboard will catch that. This means monitoring for this kind of system needs two distinct layers: systems health (is the infrastructure working) and model/product health (are the recommendations actually good).

Systems-level metrics

Model / product-level metrics

Logging & tracing

Every request carries a unique trace ID propagated through every service (Orchestrator, Candidate Gen, Ranking, Re-ranking), captured via distributed tracing tools (OpenTelemetry, Jaeger, Zipkin). This makes it possible to answer “for this exact user, at this exact timestamp, why did they see this exact list?” — essential both for debugging and for explaining decisions during policy or fairness reviews.

Defining SLOs and alerting without fatigue

A common early mistake is alerting on every metric that can be measured, which quickly produces so much noise that engineers start ignoring pages altogether — a failure mode sometimes called alert fatigue, and one that defeats the entire purpose of monitoring. The healthier approach is defining a small number of Service Level Objectives (SLOs) that genuinely matter to the user experience — for example, “P99 explore-page latency under 200ms for 99.9% of the trailing 28 days,” or “explore endpoint error rate under 0.1%” — and alerting primarily on those, using an error-budget model: as long as the system is within its allowed budget of bad minutes for the period, no page fires; once the budget is at risk of being exhausted, an alert fires with enough lead time for a human to intervene. Model-health metrics like calibration drift or diversity score are usually tracked on dashboards and reviewed on a regular cadence (daily or weekly) rather than paged on immediately, since they move more slowly and rarely require the same minute-by-minute urgency as an outright outage.

This distinction — page-worthy SLOs versus dashboard-worthy health metrics — keeps on-call engineers focused on what genuinely needs immediate attention, while still ensuring slower-moving model quality issues get regularly reviewed by the team that owns the ranking model rather than silently drifting unnoticed for weeks.

Everything described so far assumes the system is already running correctly. Getting it there — and, more importantly, getting changes into it safely once it’s live and serving real traffic — is its own discipline, and one where recommendation systems have specific failure modes that ordinary web services don’t.

Container orchestration

All stateless services (API Gateway, Orchestrator, Candidate Generation, Ranking, Re-ranking) are containerized and run on Kubernetes (or a managed equivalent like Amazon EKS, Google GKE), which handles horizontal pod autoscaling, rolling deployments, and self-healing. The Ranking Service, if GPU-backed, runs on GPU-enabled node pools with autoscaling tied to inference queue depth rather than plain CPU usage.

Deployment strategy for the ranking model

Model deployments are treated differently from ordinary code deployments because a bad model can silently harm engagement without throwing errors. The standard approach:

1. Offline evaluation: New model candidate is evaluated against held-out historical data first.
2. Shadow deployment: The new model runs in production alongside the current one, scoring real traffic, but its scores are logged and compared — never actually served to users.
3. Canary rollout: Once shadow metrics look healthy, the new model is served to a small percentage (e.g. 1–5%) of real traffic via the load balancer’s weighted routing, with engagement metrics compared to the control group.
4. Full rollout: If the canary A/B test shows a statistically significant improvement (or at least no regression), traffic is gradually shifted to 100%.
5. Instant rollback: The Model Registry keeps the previous model version hot and ready; if metrics regress post-rollout, traffic reverts instantly without a redeploy.

Blue-green vs. canary for infrastructure changes

Non-model infrastructure changes (a new version of the Orchestrator service, for instance) typically use canary deployments for the same reason: gradually shift a small percentage of load-balanced traffic to the new version, watch error rates and latency, and roll forward or back based on real signal — rather than an all-at-once blue-green cutover, which offers less granular risk control for a system this central to the product. Blue-green deployments still have a place for changes that are difficult to run partially — a breaking schema migration on the Content Metadata DB, for example, where running two incompatible versions of the schema side by side isn’t feasible — but for anything that can be run gradually, the finer-grained control of a canary rollout is almost always preferable for a system at this scale.

Infrastructure as code

The entire stack — load balancer configuration, Kubernetes manifests, Vector DB cluster topology, Feature Store provisioning, autoscaling policies — is defined declaratively (Terraform, Helm charts) and version controlled, enabling reproducible environments across staging and production, and making the multi-region failover setup described in Section 9 something that can be stood up consistently rather than manually.

Cost optimization

GPU-backed model serving and always-on streaming pipelines are the two largest cost centers in this architecture, so cost discipline matters as much as raw performance. Common techniques include model distillation (training a smaller, faster “student” model to approximate a larger “teacher” model’s predictions, cutting inference cost significantly with a small accuracy trade-off), quantization (reducing model weight precision from 32-bit to 8-bit or lower, shrinking both memory footprint and inference latency), and autoscaling GPU node pools aggressively down during off-peak hours rather than provisioning for peak capacity around the clock. On the storage side, tiering — keeping only recently active or high-traffic content in the hot Vector Database and Feature Store, while archiving older or low-traffic items to cheaper cold storage — keeps the expensive, low-latency tier smaller and therefore cheaper without materially affecting the recommendations most users actually see.

Database choices, and why

Caching strategy in depth

There are actually three distinct caching layers in this architecture, each solving a different problem:

Load balancing in depth

Load balancing appears at three distinct layers in this system, not just once at the edge:

1. Edge load balancer: Distributes client traffic across API Gateway instances, typically using round-robin or least-connections, with health checks removing unhealthy instances automatically.
2. Internal service load balancing: Within the service mesh, calls from the Orchestrator to Candidate Generation and Ranking are load balanced across their respective instance pools (often client-side load balancing via a service mesh sidecar, e.g. Envoy).
3. Data-tier load balancing: Reads to the Vector DB and Feature Store are distributed across replicas/shards, sometimes using consistent hashing so that repeated queries for the same key land on the same shard/cache-friendly node.

Sharding strategy for the Content Metadata DB

Sharding the Content Metadata DB by content ID (using a hash of the ID to pick a shard) rather than by creator ID is a deliberate choice worth explaining. Sharding by creator would cluster all of a single popular creator’s content on one shard, creating a hot spot the moment that creator goes viral — exactly the kind of imbalance a sharding scheme should avoid. Hashing by content ID spreads load evenly regardless of any one creator’s popularity, at the cost of making “fetch all posts by this creator” a scatter-gather query across shards rather than a single-shard lookup — an acceptable trade, since that particular query pattern is far less latency-sensitive (used for a creator’s own profile page, not the high-QPS explore pipeline) than the content-ID lookups the explore system performs constantly.

The explore endpoint

The client-facing API is deliberately simple — all the complexity described so far is hidden behind one endpoint. A mobile or web client should never need to know that a single response was assembled from a cache lookup, a five-service internal call chain, and a fallback decision; it should just receive a clean, well-formed list of items. A typical request/response contract:

GET /v1/explore?cursor=&limit=30
Authorization: Bearer <jwt>

// Response
{
  "items": [
    {
      "content_id": "c_9f21a",
      "creator_id": "u_7712b",
      "media_url": "https://cdn.example.com/media/9f21a.jpg",
      "score": 0.812,
      "reason": "similar_to_recent_activity"
    }
  ],
  "next_cursor": "eyJvZmZzZXQiOjMwfQ=="
}

Note the reason field — surfacing a lightweight, human-readable explanation for why an item was recommended (e.g., “similar to recent activity”, “popular in your area”) is both a UX best practice and increasingly a transparency/regulatory expectation for algorithmic feeds.

Internal service-to-service APIs

Internally, services communicate over gRPC rather than REST/JSON — the binary protocol buffer format is significantly faster to serialize/deserialize than JSON, which matters when the Orchestrator is making calls carrying thousands of candidate scores within a tight latency budget.

service RankingService {
  rpc RankCandidates (RankRequest) returns (RankResponse);
}

message RankRequest {
  string user_id = 1;
  repeated string candidate_ids = 2;
}

message RankResponse {
  repeated ScoredCandidate results = 1;
}

message ScoredCandidate {
  string content_id = 1;
  float score = 2;
}

Why microservices here, not a monolith

Each stage of the pipeline (Candidate Generation, Ranking, Re-ranking) has wildly different resource needs — Ranking is GPU-hungry and latency-sensitive, Candidate Generation is I/O-bound against the Vector DB, Re-ranking is pure CPU and nearly instant. Splitting them into independently deployable, independently scalable services means each can be scaled and optimized on its own hardware profile, and a bug or deployment issue in Re-ranking logic doesn’t require redeploying the expensive Ranking model-serving fleet.

Pagination without duplicates or gaps

Notice the next_cursor field in the response contract above rather than a simple page number. Cursor-based pagination is important here for a subtle reason: the underlying ranked list is not stable — it can change between requests as the user’s behavior updates their features in near-real-time, and as new content is continuously indexed. A naive offset-based approach (“give me items 30 through 60”) can silently skip or duplicate items if the underlying ranking shifts between page loads. A cursor instead encodes enough state (typically an opaque, server-generated token capturing the ranked list’s position and a snapshot identifier) to continue exactly where the previous page left off, even if the live ranking has since moved on for future requests.

Idempotency for client retries

Mobile clients on flaky networks routinely retry requests that appear to have failed but actually succeeded server-side. Because a GET request to the explore endpoint is naturally idempotent (repeating it doesn’t change server state), this is less of a concern for the read path than it would be for, say, a “like” or “follow” action — but it does mean the result cache described in Section 13 does double duty here: a retried request within the cache TTL window returns the identical list rather than triggering an entirely new, potentially different pipeline run, which keeps the user experience consistent across a retry.

The individual components covered so far — services, databases, caches — are only half the picture. How they’re arranged and how failures propagate between them follows a smaller set of recurring, well-known patterns, and avoiding an equally well-known set of anti-patterns matters just as much as picking the right technology for each box.

Patterns worth adopting

Anti-patterns to avoid

One more pattern: progressive enrichment

A subtler pattern worth naming explicitly is progressive enrichment across the funnel: each stage adds more expensive, more precise information to a candidate rather than starting from scratch. Candidate generation attaches a cheap similarity score and a source tag (“from embedding search”, “from trending pool”). Ranking attaches a full set of predicted engagement probabilities. Re-ranking attaches business-rule metadata (why an item was kept, boosted, or dropped). By the time a request finishes, every surfaced item carries a rich trail of exactly how it was chosen — which is precisely what powers the debugging workflow described in Section 16 and the “reason” field in the API contract in Section 14. Designing for this kind of progressive enrichment from day one is far cheaper than retrofitting explainability into a system that was never built to carry it.

Debugging a bad recommendation, step by step

When a user or a product manager reports “this recommendation makes no sense,” the debugging workflow this architecture enables is worth walking through explicitly, since it demonstrates why the logging and tracing investments from Section 11 pay for themselves. First, use the trace ID for that specific request to pull the full record of what happened: which candidate sources contributed that item, what raw features it had at scoring time, what score the ranking model assigned it, and whether it survived re-ranking on merit or was inserted by a business rule (an exploration slot, a promoted post). Second, compare that score against the model’s calibration curve — was 0.85 actually a reasonable prediction given the item’s features, or does this look like a model bug? Third, check whether the issue is isolated to one user (a personalization bug, perhaps a stale or corrupted user embedding) or widespread across many users (a systemic issue, likely a bad model deployment or a broken feature pipeline). This structured triage — trace, features, calibration, blast radius — turns a vague complaint into a concrete, assignable bug in minutes rather than hours.

Building a culture of experimentation

Because so many of the decisions in this system are inherently subjective trade-offs — how much exploration is enough, how heavily to weight shares versus watch time, how aggressive diversity rules should be — the healthiest long-term practice is treating the explore page not as a system you finish building, but as a continuous experimentation platform. That means investing early in reliable, statistically sound A/B testing infrastructure (proper randomization, sufficient sample sizes, guardrail metrics that block a launch even if the primary metric improves), and normalizing the expectation that most proposed changes, even ones that seem obviously beneficial, should be validated with real traffic before being trusted.

Designing an algorithmic explore page is fundamentally an exercise in taming an intractable problem — find the best content for a person out of billions of possibilities, in milliseconds — through disciplined architectural narrowing rather than brute force. Every design decision in this tutorial traces back to that one constraint.

“A great explore page isn’t the one with the fanciest model — it’s the one that stays fast, stays available, and stays fair to new creators, even on its worst day.”

— Closing principle for this design

Where to go from here

If you’re preparing this topic for a system design interview, the strongest preparation is not memorizing every component name in this tutorial, but being able to redraw Figure 1 from scratch, unprompted, and narrate why each box exists — what would break, specifically, if the Load Balancer, the Feature Store, or the exploration budget were removed. If you’re building something like this for real, the most valuable early investment is not the fanciest ranking model, but the observability and fallback infrastructure described in Sections 9 and 11 — a simple model running on a resilient, well-instrumented pipeline will outperform a brilliant model running on a fragile one, every time it actually matters.