Open a photo-sharing app anywhere in the world — on a fast office Wi-Fi connection or a shaky mobile signal in a rural area — and a photo appears almost instantly. That instant feeling is one of the most carefully engineered illusions in modern software. Behind it sits a genuinely difficult problem: billions of images, each needing to exist in several different sizes and formats, stored durably, and delivered to any person on Earth in well under the time it takes to blink.

Photo sharing at real scale is a relatively young engineering discipline, even though photography itself is old. Early web photo hosting services in the 2000s stored a single copy of each uploaded image and resized it on the fly using server-side scripts whenever a page requested it — a reasonable approach when traffic was modest, but one that falls apart completely once traffic reaches millions or billions of daily image views, because resizing an image is genuinely expensive computational work, and doing it repeatedly for the same image, for every single viewer, wastes enormous amounts of compute that could instead be spent once and reused millions of times.

The shift that made modern photo platforms possible was recognizing photo delivery as fundamentally a content distribution problem, not just a storage problem. Once platforms started pre-generating a small set of standard resolutions at upload time and distributing them through globally-distributed edge caching networks — content delivery networks, or CDNs — the physics of the problem changed. Instead of every image view requiring a round trip to a single, distant data center, most views could be served from a cache location physically close to the viewer, cutting latency from hundreds of milliseconds down to the tens of milliseconds needed to feel instant.

This tutorial works through exactly that architecture in depth: how to accept and store an enormous, ever-growing volume of images, how to generate the right set of derived resolutions and formats efficiently, and how to get any one of those billions of images in front of any viewer, anywhere, in well under 100 milliseconds.

A Short Timeline of Photo Delivery at Scale

Let’s break this into its genuinely hard sub-problems, because “store photos and show them to people” undersells just how many distinct engineering challenges are packed into this one sentence.

2.1 Sub-problem 1 — Images Are Big, and There Are a Lot of Them

A single high-resolution photo can easily be several megabytes. Multiply that by billions of images, and total storage requirements reach into the petabyte range before even accounting for the additional derived resolutions this system needs to generate and keep. Storage at this scale isn’t just “buy more disks” — it requires deliberate tiering, compression, and deduplication strategies, all covered later in this tutorial.

2.2 Sub-problem 2 — One Image Needs to Exist in Many Forms

The same photo needs to render well as a tiny thumbnail in a grid view, a medium-sized preview in a feed, and a full-resolution version when someone taps to view it in detail — and increasingly, in multiple modern compressed formats (like WebP or AVIF) alongside the traditional JPEG, since different devices and browsers support different formats with meaningfully different compression efficiency. Deciding when to generate these variants — eagerly at upload time, or lazily on first request — is one of the most consequential trade-offs in this entire design.

2.3 Sub-problem 3 — Physics Limits How Fast a Distant Server Can Respond

As introduced above, no clever server-side optimization changes the fact that data takes real, physical time to travel long distances. A user in Mumbai requesting an image stored only in a single data center in Virginia is going to experience real network latency no matter how fast that Virginia data center itself responds. Sub-100 ms global delivery is fundamentally a data placement problem before it’s anything else: the bytes need to already be close to the viewer before the request even happens.

2.4 Sub-problem 4 — Popularity Is Extremely Uneven

A small fraction of images — a viral post, a celebrity’s photo, a trending meme — receive a dramatically disproportionate share of total views, while the vast majority of uploaded images are viewed a handful of times by a small circle of people, and plenty are barely viewed at all after the first day. A design that treats every image identically, provisioning the same caching and replication effort for a rarely-viewed vacation photo as for a viral post, wastes enormous resources on the long tail while potentially under-serving the popular head.

Explicit functional and non-functional requirements — and back-of-the-envelope math that grounds every downstream architectural choice in concrete numbers rather than vibes.

3.1 Functional Requirements

• Users can upload photos, which are validated, stored, and made available for viewing.
• Each photo is automatically available in several standard resolutions (thumbnail, medium, large, original) and, ideally, multiple compressed formats.
• Users can view photos shared with them, subject to privacy settings.
• Photos can be deleted, and deletion should propagate across all stored copies and cached versions within a reasonable window.

3.2 Non-Functional Requirements

3.3 Back-of-the-Envelope Capacity Estimation

Let’s work through realistic numbers for a platform with 2 billion total stored images and 50 million new photo uploads per day — numbers broadly consistent with the scale of the largest real-world photo-sharing platforms.

Storage estimation

These numbers alone justify several design decisions covered later: this is clearly a scale where storage cost optimization (compression, tiering, deduplication) is a first-class engineering concern, not an afterthought, and where a single storage system or data center could never physically hold or serve all of this data with acceptable latency.

Read traffic estimation

Compare this read volume — north of two million requests per second at peak — against the roughly 580 uploads per second implied by 50 million daily uploads. The read-to-write ratio here is enormous, easily in the range of thousands to one. This confirms the intuition from the problem statement: this is overwhelmingly a read-serving and caching problem. The vast majority of engineering effort should go toward making reads (image views) as fast and cheap as possible, since they utterly dominate total system load, while the upload path — though it still needs to be reliable and reasonably efficient — can afford to do more work per operation, since it happens so much less frequently.

The overall shape has a clear center of gravity: the CDN. Nearly every architectural decision here exists either to feed the CDN correctly (upload and processing) or to fall back gracefully when the CDN doesn’t already have what’s needed (the origin serving path).

4.1 Component-by-Component

Even though uploads are far less frequent than views, the upload path still deserves careful design, because it’s the one place in this system where correctness mistakes (a lost photo, a corrupted upload) are the most visible and damaging kind of failure — a lost social media post is annoying, but a lost family photo can feel genuinely irreplaceable to the person who uploaded it.

5.1 Upload & Ingestion Pipeline

Why deduplication matters more here than in most systems

Photo-sharing platforms see a surprising amount of duplicate content — the same viral meme re-uploaded by many different users, the same stock or promotional image shared repeatedly, or a single user re-uploading their own photo after a failed upload retry. At the storage volumes calculated earlier, avoiding redundant storage for genuinely identical bytes is a meaningful cost saving, not a minor optimization.

public class UploadService {

    private final ContentHasher hasher;
    private final DedupIndex dedupIndex;
    private final BlobStore blobStore;
    private final MetadataStore metadataStore;
    private final EventPublisher eventPublisher;

    public UploadResult handleUpload(byte[] imageBytes, String ownerId) {
        String contentHash = hasher.sha256(imageBytes);

        Optional<String> existingObjectId = dedupIndex.lookup(contentHash);
        String objectId;
        boolean isNewContent;

        if (existingObjectId.isPresent()) {
            objectId = existingObjectId.get();
            isNewContent = false; // reuse existing bytes, save storage
        } else {
            objectId = generateObjectId();
            blobStore.putObject(objectId, imageBytes);
            dedupIndex.register(contentHash, objectId);
            isNewContent = true;
        }

        String photoId = generatePhotoId();
        metadataStore.createRecord(photoId, ownerId, objectId, contentHash, Instant.now());

        if (isNewContent) {
            // only kick off processing for genuinely new bytes --
            // a duplicate reuses variants generated previously
            eventPublisher.publish(new ProcessingEvent(photoId, objectId));
        }

        return new UploadResult(photoId, objectId);
    }
}

Why the metadata record is separate even when content is deduplicated

Notice that a duplicate upload still creates its own metadata record (photo_id), even though it points at the same underlying object bytes as an earlier upload. This matters because two different users’ uploads of the same visual content are still distinct social artifacts — different owners, different captions, different privacy settings, different upload timestamps — even though the underlying pixels are identical. Deduplicating at the byte level while keeping metadata fully independent per upload preserves correct product behavior while still capturing the storage savings.

Handling large uploads and unreliable client connections

Mobile uploads in particular happen over unreliable networks, so production upload paths typically support resumable, chunked uploads — the client uploads a large image in smaller pieces, and if the connection drops partway through, it can resume from the last successfully received chunk rather than restarting the entire upload from scratch. This meaningfully improves the experience for users on poor connectivity, which is a substantial fraction of a genuinely global user base.

Malware and content moderation scanning

Before an uploaded image is fully accepted, it typically passes through automated scanning — malware/exploit detection (since image files have historically been used as attack vectors through malformed file parsing) and content moderation classifiers (detecting policy-violating content). This scanning can run in parallel with metadata creation and initial storage, with the option to quarantine or remove content shortly after if it’s flagged, rather than blocking every single upload while waiting for a full moderation verdict.

5.2 Image Processing & Resolution Generation

This section covers how one uploaded original becomes the several variants a real product actually needs, and the eager-versus-lazy trade-off flagged earlier in more depth.

The standard variant set

Each of these is typically also generated in more than one file format — a traditional JPEG for maximum compatibility, and a modern format like WebP or AVIF for browsers and devices that support it, since these newer formats can achieve visually similar quality at 25–50% less file size, which directly reduces both storage cost and, more importantly given the read-heavy nature of this system, bandwidth cost and transfer time on every single view.

Eager versus lazy generation — the real trade-off

Most real, mature photo platforms land on a hybrid: eagerly generate the two or three most universally needed variants (typically thumbnail and medium, since nearly every uploaded photo gets viewed at least in a feed or grid context), and generate everything else — larger sizes, less common formats, unusual aspect-ratio crops — lazily, on first request, cached aggressively afterward so the cost is paid at most once per variant, ever.

public class ImageProcessingWorker {

    private static final List<VariantSpec> EAGER_VARIANTS = List.of(
        new VariantSpec("thumbnail", 150, 150, ImageFormat.WEBP),
        new VariantSpec("medium",    640,  -1, ImageFormat.WEBP)  // -1 = preserve aspect ratio
    );

    private final BlobStore blobStore;
    private final ImageTransformer transformer;

    public void handleProcessingEvent(ProcessingEvent event) {
        byte[] original = blobStore.getObject(event.getObjectId());

        for (VariantSpec spec : EAGER_VARIANTS) {
            byte[] resized = transformer.resize(original, spec.width(), spec.height(), spec.format());
            String variantKey = buildVariantKey(event.getPhotoId(), spec);
            blobStore.putObject(variantKey, resized);
        }
        // larger/rarer variants are intentionally NOT generated here --
        // ImageServingService produces them on demand, on first request
    }

    public byte[] generateVariantOnDemand(String photoId, String objectId, VariantSpec spec) {
        byte[] original = blobStore.getObject(objectId);
        byte[] resized  = transformer.resize(original, spec.width(), spec.height(), spec.format());
        blobStore.putObject(buildVariantKey(photoId, spec), resized);
        return resized;
    }

    private String buildVariantKey(String photoId, VariantSpec spec) {
        return photoId + ":" + spec.name() + ":" + spec.format();
    }
}

Format negotiation — serving the right format to the right device

Because not every browser or device supports every modern image format, the serving layer needs to determine, per request, which format the requesting client actually supports — typically read from a standard HTTP header the browser sends indicating which formats it accepts — and serve the best available match rather than a one-size-fits-all format. This means the CDN and Image Serving Service need to treat “photo X at medium size” as potentially several distinct cacheable objects (one per supported format), not a single object, which has real implications for cache key design covered in the caching section ahead.

Progressive and adaptive loading

Beyond generating discrete resolution variants, many image formats support progressive encoding — structuring the file so a lower-quality, blurry version of the whole image can render almost immediately while the remaining data continues loading and sharpens the image incrementally, rather than the image appearing abruptly all at once (or not at all) only once fully downloaded. This is a genuinely important technique for perceived performance, since it gives the viewer something meaningful to look at almost immediately even on a slow connection, well before the full sub-100 ms target is even relevant to the full-quality version.

This chapter addresses the headline requirement most directly. Let’s build up an understanding of exactly how a CDN makes sub-100 ms global delivery achievable, then walk a single photo end-to-end through the lifecycle stages that get it there.

6.1 Points of Presence (PoPs) & the Geography of Latency

A CDN operates a large number of geographically distributed edge locations, often called Points of Presence, each capable of caching and serving content independently. When a viewer requests an image, DNS-based or Anycast-based routing directs their request to the nearest (in network terms, not always strictly geographic distance) PoP, rather than all the way to a single origin data center. If that PoP already has the requested content cached — a cache hit — it serves the response directly from a location physically close to the viewer, which is what makes sub-100 ms delivery achievable: the request never has to travel further than the nearest PoP.

6.2 Why Cache Hit Ratio Is the Single Most Important Metric in This System

Given that a cache hit at a nearby PoP can plausibly land well under 100 ms, while a cache miss requiring a trip back to a distant origin frequently cannot, the fraction of requests served as cache hits — the cache hit ratio — is essentially the deciding factor in whether this system meets its core requirement at all. A design that achieves a 99.9% cache hit ratio will feel consistently fast; the same design with a 90% hit ratio will have a meaningfully worse tail latency experience, because 10% of requests are paying the much higher cross-region cost.

6.3 Maximizing Cache Hit Ratio in Practice

6.4 The Long Tail Problem, Revisited Through a CDN Lens

Recall the popularity skew from the problem chapter: most images are viewed rarely. This means most images will, realistically, experience occasional cache misses no matter how well-tuned the system is, simply because they don’t get requested often enough to reliably stay warm in every relevant cache. The honest, achievable goal isn’t “100% cache hit ratio” — it’s “a cache hit ratio high enough, and a cache-miss path fast enough, that the sub-100 ms target is met for the overwhelming majority of real-world requests,” accepting that a small fraction of first-time or rare requests may occasionally take somewhat longer.

6.5 Data Flow & Lifecycle of a Photo

Real systems engineering rarely gives you unambiguous wins. Being able to articulate what each major decision costs — not just what it buys — is one of the clearest signals of practical, production-grounded experience.

The overarching system-level trade-off, worth stating plainly: this design accepts meaningfully more upfront engineering complexity — multiple storage tiers, a hybrid processing strategy, careful CDN configuration — in exchange for meeting a genuinely demanding, physics-constrained latency requirement at a genuinely enormous scale. A simpler design (single data center, on-demand resizing, no CDN) would be far easier to build and reason about, but would have no realistic path to meeting a sub-100 ms global requirement no matter how much it was optimized within that simpler shape.

7.1 Why These Trade-offs Are Worth Stating Explicitly

It’s tempting, when presenting a finished design, to describe every decision as an obvious, unambiguous win. Real systems engineering rarely works that way, and being able to articulate genuinely what each major decision costs — not just what it buys — is one of the clearest signals of practical, production-grounded experience an interviewer can look for. A candidate who can only explain why a design choice is good, without being able to say what it gives up in exchange, likely hasn’t actually operated a system like this under real constraints, where every one of these trade-offs eventually shows up as a genuine cost someone on the team has to account for, whether in an infrastructure bill, an on-call incident, or a slower-than-ideal feature launch caused by unexpected complexity in a supposedly simple change.

Sub-100 ms is a hard latency budget. Understanding where the milliseconds actually go — and where compression, batching, and parallelism cut them — is the difference between hitting the target and merely hoping to.

8.1 Latency Budget Thinking

Meeting a hard latency target like sub-100 ms benefits from explicitly breaking the budget into pieces: DNS resolution and connection setup to the nearest PoP (a few milliseconds, often amortized across many requests through connection reuse), the PoP’s own cache lookup and response time (typically single-digit milliseconds for a well-provisioned PoP), and the actual network transfer time for the image bytes themselves (which scales with file size — reinforcing why compression and appropriately-sized variants matter directly for meeting the latency target, not just for storage cost). Adding these up should comfortably clear the 100 ms target for a cache hit; the entire architecture exists to make cache hits the overwhelming norm.

8.2 Horizontal Scaling Across Every Tier

Every stateful component here — metadata shards, blob storage, dedup index — scales horizontally by adding more machines, following the same consistent-hashing-based partitioning principles used throughout this tutorial series. The CDN layer scales by adding more edge PoPs and capacity within existing ones, typically managed by the CDN provider rather than the platform’s own engineering team directly, though the platform still influences hit ratio through the caching strategy decisions covered earlier.

8.3 Compression as a Direct Performance Lever, Not Just a Cost Lever

It’s worth being explicit that image compression (modern formats, appropriate quality settings per size) directly affects the latency budget discussed above, not just storage cost — fewer bytes to transfer means less time spent transferring them, which matters even for a cache hit at a nearby PoP, and matters considerably more for the comparatively rare but still latency-sensitive cache-miss path back to origin.

8.4 Parallelizing Image Processing

Generating multiple variants for a single upload is naturally parallelizable — thumbnail, medium, and format-converted versions can all be generated concurrently rather than sequentially, since they’re independent transformations of the same source bytes. Processing worker pools sized appropriately for this concurrency, and monitored for queue depth the same way discussed for other asynchronous pipelines in this tutorial series, keep processing latency (the delay before a freshly uploaded photo’s eager variants are ready) low even during upload volume spikes.

Durability is the standout requirement here, more so than in many other systems. A lost feed entry is annoying; a lost family photo is unforgivable.

9.1 Durability Is the Standout Requirement

While most systems in this tutorial series lean toward accepting some staleness in exchange for availability, this system has an additional, especially strict requirement worth calling out: photo durability. Users treat uploaded photos as effectively permanent, irreplaceable records, which means the blob storage layer needs extremely strong durability guarantees — commonly expressed as something like “eleven nines” of durability in commercial object storage offerings, achieved through aggressive replication (multiple copies, often across physically separate facilities) and continuous integrity checking (checksums verified regularly to detect and repair any silent data corruption before it can result in permanent loss).

9.2 Where This System Sits on the CAP Spectrum

For serving traffic, this system leans AP, favoring availability — a CDN serving a slightly outdated cached version of metadata-adjacent information, or a brief delay in a newly-uploaded photo appearing in absolutely every corner of the CDN, are both acceptable in exchange for the system staying up and responsive. For the underlying blob storage itself, the priority shifts slightly: durability (never losing data) matters more than either strict consistency or even availability in the narrow sense — a brief unavailability of a rarely-accessed cold-storage object is a far smaller problem than that object’s bytes being irrecoverably lost.

9.3 Replication Strategy

Blob objects are typically replicated across multiple availability zones within a region, and often across regions as well for the most critical, hard-to-replace content (originals especially, more so than derived variants, which can always be regenerated from the original if truly necessary). This replication serves two distinct purposes simultaneously: protecting against data loss (durability) and enabling continued availability if one replica location becomes temporarily unreachable.

9.4 Graceful Degradation Patterns Specific to This System

9.5 Disaster Recovery With Durability as the Top Priority

Cross-region replication of original images specifically (accepting that derived variants, being regenerable, don’t need quite the same level of redundancy) protects against the loss of an entire region’s storage infrastructure, while regular integrity verification catches silent corruption before it can compound into unrecoverable loss.

9.6 Backup vs. Replication — A Distinction Worth Being Precise About

Replication protects against infrastructure failure but faithfully copies any logical error (an accidental bulk deletion, a bug that corrupts data) to every replica just as quickly as it copies legitimate data. Point-in-time backups or object versioning (retaining a previous version of an object for some window even after it’s been “deleted” or overwritten) protect against exactly this different class of failure, and are worth maintaining even in a system that already has strong replication.

9.7 Cost Optimization — A First-Class Concern at This Scale

Signed URLs let private content flow through aggressive CDN caches without leaking access, while defense against hotlinking, malware, and content-policy violations sits alongside standard TLS and at-rest encryption.

10.1 Signed URLs for Private Content

Public photos can be served through simple, stable CDN URLs, but private or restricted-audience photos need access control enforced even though the content sits cached at edge locations far from any central authorization server. The standard solution is signed URLs: the metadata API generates a URL that includes a cryptographic signature and an expiry timestamp, and the CDN itself (most commercial CDNs support this natively) validates the signature and expiry before serving cached content, rejecting requests with an invalid or expired signature without ever needing to contact the origin for that check.

public class SignedUrlGenerator {

    private final String secretKey;
    private static final Duration DEFAULT_EXPIRY = Duration.ofMinutes(15);

    public String generateSignedUrl(String baseUrl, String photoPath) {
        long expiryEpochSeconds = Instant.now().plus(DEFAULT_EXPIRY).getEpochSecond();
        String payload = photoPath + ":" + expiryEpochSeconds;
        String signature = hmacSha256(payload, secretKey);

        return baseUrl + photoPath
            + "?expires="   + expiryEpochSeconds
            + "&signature=" + signature;
    }

    private String hmacSha256(String data, String key) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(key.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
            byte[] hash = mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
            return Base64.getUrlEncoder().withoutPadding().encodeToString(hash);
        } catch (Exception e) {
            throw new RuntimeException("Failed to sign URL", e);
        }
    }
}

This approach elegantly resolves the tension flagged in the caching section: cached bytes at the CDN edge can remain aggressively, indefinitely cached, while access control is still meaningfully enforced through the time-limited signature rather than needing an active permission check against origin on every single request — the CDN can validate the signature itself, entirely at the edge, without a round trip back to any central authorization service.

10.2 Preventing Hotlinking and Unauthorized Embedding

Beyond private-content access control, platforms often want to prevent third-party sites from directly embedding (hotlinking) their publicly-hosted image URLs, which consumes bandwidth without any corresponding benefit to the platform. Checking the HTTP referrer header at the CDN edge, and rejecting or redirecting requests from unrecognized origins, is a common mitigation — imperfect, since referrer headers can be spoofed or omitted, but effective against the more casual majority of unauthorized embedding.

10.3 Malware and Content Scanning, Revisited

As introduced in the upload section, every uploaded image passes through automated scanning before being considered fully accepted — both for malicious file content (exploiting parser vulnerabilities in image-processing libraries has historically been a real attack vector) and for policy-violating content. Running this scanning asynchronously, in parallel with normal processing, keeps upload latency low while still providing meaningful protection, with the ability to quarantine and remove flagged content shortly after acceptance if needed.

10.4 Encryption at Rest and in Transit

Blob storage typically encrypts data at rest by default in modern object storage offerings, protecting against a scenario where physical storage media is somehow compromised. All traffic between clients, the CDN, and origin infrastructure runs over encrypted connections (TLS), protecting image bytes and metadata from interception in transit — standard practice throughout the industry, but worth stating explicitly as a baseline requirement for a system handling personal user content at this scale.

Given how central cache hit ratio is to meeting the core latency requirement, it deserves monitoring not just as a single global number, but broken down by region, by content age, and by variant size and format.

11.1 The Metric That Matters Most: Cache Hit Ratio, Broken Down by Dimension

An aggregate hit ratio that looks healthy can hide a specific region or content category performing meaningfully worse, exactly the kind of gap worth catching before it accumulates into a broader pattern of user complaints from one part of the world.

11.2 Latency Percentiles, Tracked Separately for Hits and Misses

Blending cache-hit and cache-miss latency into one aggregate metric obscures what’s actually happening — a healthy system should show cache-hit latency comfortably under the 100 ms target with very tight variance, and cache-miss latency higher but still bounded and predictable. Tracking these separately (and tracking the ratio between them) makes it far easier to diagnose whether a latency regression stems from degraded cache performance, a shift in cache hit ratio, or a slowdown specifically in the origin miss-handling path.

11.3 Processing Pipeline Health

11.4 Storage Growth Tracking

Given the multi-petabyte-per-year growth calculated in the capacity estimation section, dashboards tracking actual storage growth against projections help the team anticipate capacity needs and catch anomalies early — a sudden, unexplained jump in storage growth rate might indicate a bug in the deduplication logic (failing to catch duplicates it should be catching) or an unexpected shift in upload patterns worth investigating.

11.5 Distributed Tracing Across the Upload-to-Serving Pipeline

Tracing a single photo’s journey — upload, dedup check, storage, processing, first CDN population, first serve — using a consistent trace or correlation ID propagated through each stage, makes it possible to answer concrete diagnostic questions like “why did this specific photo’s medium variant take unusually long to become available” by following its actual path through the system, rather than piecing together disconnected logs from several services after the fact.

Origin infrastructure follows the standard containerized, orchestrator-managed pattern. The CDN edge, by contrast, is typically a managed external layer whose value comes largely from configuration — TTLs, signed-URL validation, purge integration — rather than how it’s deployed.

12.1 Origin Infrastructure: Containerized and Orchestrated

The Upload Service, Image Processing Workers, Metadata Service, and Image Serving Service all follow the same containerized, orchestrator-managed deployment pattern used throughout this tutorial series, each scaling independently based on its own relevant load signal — upload rate, processing queue depth, metadata query volume, and cache-miss rate respectively.

12.2 The CDN as a Largely Managed, Externally-Operated Layer

Unlike most other components in this design, the CDN’s edge infrastructure itself is typically operated by a specialized third-party provider (or, for the largest platforms, a purpose-built internal system like Netflix’s Open Connect), rather than infrastructure the platform’s own engineering team deploys and manages directly. The platform’s deployment responsibility here is largely about correct configuration — cache TTL rules, signed URL validation setup, purge/invalidation API integration — rather than deploying and scaling edge servers itself.

12.3 CI/CD for the Processing Pipeline Specifically

Changes to image processing logic — a new compression setting, an adjusted quality parameter, a newly-added format — deserve particularly careful, gradual rollout, since a subtle bug could silently degrade image quality or dramatically increase file sizes across a huge volume of newly processed content before anyone notices through normal functional testing alone. Canary rollout to a small percentage of new uploads first, with automated comparison of output file size and a basic quality metric against the previous version, catches this kind of regression before it reaches full production traffic.

12.4 Multi-Region Origin for Resilience, Not Primarily for Latency

Unlike the CDN edge layer, where geographic spread exists specifically to minimize viewer-facing latency, having origin infrastructure in more than one region is motivated primarily by resilience — protecting against a full regional outage taking down the entire cache-miss-serving and upload-accepting path — rather than latency, since origin traffic volume (cache misses plus uploads) is a small fraction of total system load compared to the CDN-served majority.

One chapter for the entire data plane: the model of what a photo actually is, the storage systems chosen for each shape of data, the caching layers that sit above them, and the sharding scheme that keeps everything horizontally scalable.

13.1 Data Model & Storage Schema

Why variants are modeled as fully independent objects

It’s tempting to think of a “medium” variant as just “the original, resized” rather than a stored object in its own right. But materializing each variant as its own persisted blob object — generated once, stored, and reused for every subsequent request — is exactly what makes CDN caching effective. If variants were instead computed fresh on every single read, there would be nothing stable for a CDN to cache at all; the whole caching strategy this tutorial builds around depends on variants being stable, addressable, immutable objects.

Content-addressable object IDs

A common and effective pattern is deriving an object’s storage key directly from its content hash (or a hash of the combination of the original’s hash and the variant’s transformation parameters), rather than from an arbitrary sequential ID. This has two benefits worth calling out: it naturally supports the deduplication behavior covered earlier — identical bytes always produce the same key — and it makes CDN caching trivially safe to set extremely long TTLs on, since a given key’s content can never change; if the underlying image is edited, that edit produces new bytes and therefore an entirely new key, never a mutation of an existing one.

Why metadata and blob storage are strictly separate systems

The Metadata Store needs to support relatively rich, small, frequent queries — “get this user’s photos,” “check this photo’s privacy setting” — patterns well suited to a traditional database. Blob Storage needs to efficiently store and retrieve very large binary payloads by a simple key, with strong durability guarantees, a very different optimization target. Forcing both needs onto a single storage system — say, storing image bytes directly inside rows of a relational database — works poorly at this scale, bloating the database with data it’s not well-suited to handle efficiently and making routine database operations like backups and replication dramatically more expensive than they need to be.

13.2 Storage Systems Deep Dive

Why a general-purpose file system doesn’t work at this scale

It might seem natural to imagine “just store files on disk, organized in directories.” At billions of objects, this breaks down badly — traditional file systems have real limits on how many files a single directory can efficiently hold, metadata operations (like listing a directory) become slow, and there’s no natural built-in mechanism for replication, geographic distribution, or the kind of massive horizontal scaling this system needs. Purpose-built object storage systems solve exactly these problems, treating each object as an opaque blob addressed by a flat key rather than a hierarchical path, and handling replication and distribution internally.

Haystack — a genuinely relevant real-world case study

Facebook’s engineering team published a widely-referenced description of Haystack, a storage system they built specifically because general-purpose file systems and even general-purpose object storage of the era proved inefficient for photo storage at their scale — the specific problem being that a huge fraction of the overhead in serving a small image file came from file system metadata operations rather than the actual data transfer. Haystack’s core idea was to store many small photo objects together in larger physical files, with a compact in-memory index mapping each photo’s ID directly to its exact byte offset within one of those larger files — dramatically reducing the number of expensive metadata lookups needed to serve a single photo. This is a great example of a storage system deliberately co-designed around one specific, well-understood access pattern (many small, immutable, rarely-updated objects, read far more often than written) rather than relying on a generic storage solution.

Storage tiering: hot, warm, and cold

Not all stored bytes deserve the same storage cost. A tiered approach — placing frequently-accessed content on faster, more expensive storage, and rarely-accessed content on slower, cheaper storage, with automated policies migrating objects between tiers based on observed access patterns — meaningfully reduces total storage cost without materially affecting the experience for the content that actually matters for the sub-100 ms requirement, since cold-tier content is by definition rarely requested and therefore rarely on the critical path anyway.

13.3 Caching Strategy Beyond the CDN

Cache key design — more subtle than it looks

A naive cache key might just be the photo ID. But as the processing chapter established, a single photo can be requested at multiple sizes and in multiple formats, so the actual cache key needs to encode all of the dimensions that produce a genuinely different byte payload: photo ID, size label, and format at minimum. Getting this wrong in either direction causes real problems — too coarse a key (ignoring format) risks serving the wrong format to a client that can’t render it; too granular a key (encoding irrelevant details) fragments the cache unnecessarily, hurting hit ratio by treating what should be the same cached object as many different ones.

public class ImageCacheKeyBuilder {

    public static String buildKey(String photoId, String sizeLabel, ImageFormat format) {
        return String.join(":", photoId, sizeLabel, format.name().toLowerCase());
        // e.g. "photo_9182734:medium:webp"
    }
}

Origin-side caching, one layer beneath the CDN

The Image Serving Service itself benefits from a caching layer in front of blob storage — even though blob storage is fast for a purpose-built object store, an additional in-memory or local-SSD cache for the most commonly requested objects at the origin reduces load on blob storage and speeds up the (already comparatively rare, but still latency-sensitive) cache-miss path from the CDN.

Metadata caching

Photo metadata — privacy settings, owner information — is checked on essentially every serving request (to enforce access control before serving any bytes), making it a natural candidate for aggressive caching using the same cache-aside pattern discussed for metadata lookups in other systems, since metadata changes far less often than it’s read.

Client-side caching

Mobile and web clients themselves cache recently-viewed images locally, which is a meaningfully important layer for perceived performance even beyond server-side caching — a photo the user just scrolled past and scrolls back to should render instantly from local device storage without any network round trip at all, CDN or otherwise. Standard HTTP caching headers (long max-age values, given the immutability of variant objects) are what make this client-side caching behavior work correctly and safely.

13.4 Sharding & Load Balancing

Sharding the metadata store

Photo metadata is naturally partitioned by owner_id, following the same reasoning applied to other systems in this series — the dominant query pattern is “get this user’s photos,” so keeping one user’s metadata together on a single shard avoids expensive cross-shard queries for the most common access pattern. Consistent hashing with virtual nodes, as covered generally for distributed sharding, applies directly here to distribute metadata shards evenly and allow smooth rebalancing as the platform grows.

Distributing objects across blob storage

Object storage systems typically handle internal sharding and replication themselves, but the choice of object key still matters for how evenly load spreads across the underlying infrastructure. Using content-hash-derived keys has a helpful side effect here: hash-derived keys are naturally well-distributed across the keyspace, which tends to spread storage and request load evenly across the underlying storage cluster, avoiding the kind of hotspot that a sequential or predictable key scheme could create.

Load balancing the Image Serving Service

As the layer handling CDN cache misses, the Image Serving Service sits behind standard load balancers distributing requests across many stateless instances, scaled based on cache-miss volume — which, given how central cache hit ratio is to this whole design, should ideally represent a small and relatively stable fraction of total traffic, making this tier considerably smaller and cheaper to run than it would need to be if it were handling the full read volume calculated earlier.

Regional origin placement

While the CDN’s edge PoPs are deliberately spread very widely, the origin infrastructure itself (Image Serving Service, primary blob storage, metadata stores) is typically consolidated into a smaller number of well-provisioned regions, since it only needs to handle the comparatively modest cache-miss traffic rather than the full global read volume — an important cost and complexity distinction from the CDN’s edge layer, which needs far broader geographic spread specifically because it’s absorbing the overwhelming majority of total requests.

The metadata API tells the client where to find each variant. The CDN handles the actual byte delivery. Keeping these two responsibilities separate is why the metadata service can stay small even at millions of image views per second.

14.1 The Public API Surface

Authorization: Bearer <token>

Response 200 OK:
{
  "photoId":    "photo_9182734",
  "ownerId":    "u_5521",
  "uploadedAt": "2026-07-20T09:14:00Z",
  "privacy":    "public",
  "variants": {
    "thumbnail": "https://cdn.example.com/img/photo_9182734/thumbnail.webp",
    "medium":    "https://cdn.example.com/img/photo_9182734/medium.webp",
    "large":     "https://cdn.example.com/img/photo_9182734/large.webp",
    "original":  "https://cdn.example.com/img/photo_9182734/original.jpg"
  }
}

Notice the actual image bytes are fetched through CDN URLs returned by this metadata response, not through the API Gateway directly — the metadata API’s job is simply to tell the client where to find each variant; the CDN handles the actual, extremely high-volume byte delivery. This separation is important: the metadata API can be relatively lightweight and doesn’t need to scale anywhere near the roughly two million requests per second calculated for image views, since a client typically fetches metadata once and then reuses the returned URLs directly against the CDN for the actual image bytes, often across many subsequent views without ever calling the metadata API again.

14.2 Why the CDN URL, Not the API, Is the Actual Read-Path Bottleneck to Optimize

This is worth stating explicitly, since it’s easy to over-invest engineering effort in the wrong place: the metadata API’s performance matters, but the CDN’s performance is what the sub-100 ms requirement is actually about. A perfectly optimized metadata API sitting in front of a poorly-tuned CDN configuration would still fail to meet the stated requirement, while a mediocre metadata API in front of an excellent CDN setup would likely still succeed for the specific, headline latency target — the load times measured against the requirement are the direct image byte fetches, not the metadata round trip that precedes them.

The whole system is a small stack of well-understood patterns wrapped around one central idea — content-addressable immutability — plus a matching set of anti-patterns it deliberately refuses to use.

15.1 Patterns Worth Naming Explicitly

15.2 Anti-Patterns to Avoid

Six habits worth internalizing, and five failure modes worth actively avoiding — each of them derived from the specific shape of the photo-serving problem, not from generic distributed-systems folklore.

Real-world implementations that mirror the shape of this design, plus the specific algorithms — frequency-aware eviction, consistent hashing, perceptual hashing, rate limiting — that make each subsystem work at scale.

17.1 Facebook Haystack

Already introduced in the storage chapter, Haystack remains one of the most widely cited real-world examples of a storage system purpose-built around exactly this tutorial’s core photo-serving access pattern: many small, immutable, read-heavy objects, where reducing metadata overhead per object mattered enormously at scale.

17.2 Netflix Open Connect

Open Connect represents an extreme, deliberate version of the “move bytes physically close to the viewer” principle this entire tutorial builds around — placing caching infrastructure directly inside internet service provider networks, going even further than a typical third-party CDN arrangement in minimizing the physical distance content needs to travel to reach viewers.

17.3 On-Demand Image Transformation Services

Several commercial services specialize specifically in on-the-fly image resizing and format conversion, cached aggressively behind their own CDN layer — a productized version of the hybrid eager/lazy and CDN-first approach this tutorial develops from first principles, packaged as a service platforms can integrate rather than building fully in-house. Studying how these services structure their URL-based transformation APIs (encoding size, format, and crop parameters directly in the request URL, which then becomes the cache key) is a useful, concrete real-world illustration of the cache key design principles discussed in the caching section.

17.4 Large-Scale Photo & Visual Discovery Platforms

Visual discovery and photo-sharing platforms operating at massive scale have published engineering discussions describing very similar overall shapes to this tutorial’s design — heavy emphasis on image processing pipelines generating multiple derived sizes, aggressive CDN usage, and deliberate attention to the long-tail popularity distribution problem raised early in this tutorial, since visual content discovery in particular tends to produce exactly the kind of highly uneven popularity distribution this design accounts for throughout.

17.5 CDN Cache Eviction: Why Plain LRU Isn’t Always the Best Fit

A CDN edge node has finite cache capacity and needs a policy for deciding what to evict when it fills up. Plain Least Recently Used works reasonably well here too, but image-serving workloads have a specific characteristic worth considering: a piece of content that’s been requested many times over a longer period (frequency) might be a better bet to keep cached than something requested only once, very recently (recency alone). This is why many production CDN and cache systems use variants like LFU (Least Frequently Used) or hybrid approaches like LRU-K or ARC (Adaptive Replacement Cache), which weigh both recency and frequency rather than recency alone.

public class LFUCache<K, V> {

    private final int capacity;
    private final Map<K, V>       values         = new HashMap<>();
    private final Map<K, Integer> frequencies    = new HashMap<>();
    private final Map<K, Long>    lastAccessTime = new HashMap<>();

    public LFUCache(int capacity) {
        this.capacity = capacity;
    }

    public V get(K key) {
        if (!values.containsKey(key)) return null;
        frequencies.merge(key, 1, Integer::sum);
        lastAccessTime.put(key, System.nanoTime());
        return values.get(key);
    }

    public void put(K key, V value) {
        if (!values.containsKey(key) && values.size() >= capacity) {
            evictLeastFrequentlyUsed();
        }
        values.put(key, value);
        frequencies.merge(key, 1, Integer::sum);
        lastAccessTime.put(key, System.nanoTime());
    }

    private void evictLeastFrequentlyUsed() {
        K victim = null;
        int  minFreq       = Integer.MAX_VALUE;
        long oldestAccess  = Long.MAX_VALUE;

        for (K key : values.keySet()) {
            int  freq       = frequencies.get(key);
            long accessTime = lastAccessTime.get(key);
            if (freq < minFreq || (freq == minFreq && accessTime < oldestAccess)) {
                minFreq      = freq;
                oldestAccess = accessTime;
                victim       = key;
            }
        }
        values.remove(victim);
        frequencies.remove(victim);
        lastAccessTime.remove(victim);
    }
}

17.6 Consistent Hashing for CDN Request Routing

The same consistent hashing technique covered generally elsewhere in this tutorial series applies to how a CDN internally routes a given content key to a specific caching node within a PoP (many PoPs have multiple physical machines behind them), ensuring an even distribution of both storage and request load across the PoP’s internal capacity, and minimizing disruption if a caching node within a PoP is added or removed.

17.7 Perceptual Hashing — a More Advanced Form of Similarity Detection

The content-hash-based deduplication covered in the upload chapter catches byte-for-byte identical uploads, but it won’t catch two images that are visually near-identical but not byte-identical — a slightly re-compressed copy of the same photo, or the same image with a tiny watermark added, produce completely different content hashes despite looking essentially the same to a human viewer. Perceptual hashing addresses this by computing a hash based on visual characteristics (a coarse representation of the image’s overall structure and luminance pattern) rather than its exact bytes, so that visually similar images produce similar or identical perceptual hashes even when their underlying bytes differ.

public class SimplePerceptualHash {

    private static final int GRID_SIZE = 8; // 8x8 = 64-bit hash

    public long computeHash(BufferedImage image) {
        BufferedImage resized = resizeToGrayscale(image, GRID_SIZE, GRID_SIZE);

        long[] pixelValues = new long[GRID_SIZE * GRID_SIZE];
        long sum = 0;
        int index = 0;
        for (int y = 0; y < GRID_SIZE; y++) {
            for (int x = 0; x < GRID_SIZE; x++) {
                int gray = resized.getRGB(x, y) & 0xFF;
                pixelValues[index++] = gray;
                sum += gray;
            }
        }
        long average = sum / pixelValues.length;

        long hash = 0;
        for (int i = 0; i < pixelValues.length; i++) {
            if (pixelValues[i] >= average) {
                hash |= (1L << i); // bit set if pixel brighter than average
            }
        }
        return hash;
    }

    // hamming distance between two hashes indicates visual similarity --
    // a small distance means the images are likely near-duplicates
    public int hammingDistance(long hashA, long hashB) {
        return Long.bitCount(hashA ^ hashB);
    }

    private BufferedImage resizeToGrayscale(BufferedImage src, int w, int h) {
        BufferedImage resized = new BufferedImage(w, h, BufferedImage.TYPE_BYTE_GRAY);
        Graphics2D g = resized.createGraphics();
        g.drawImage(src, 0, 0, w, h, null);
        g.dispose();
        return resized;
    }
}

This kind of near-duplicate detection is used less for storage deduplication (exact content-hash dedup already handles the storage-saving case well) and more for other product and trust-and-safety purposes — detecting re-uploads of previously removed policy-violating content even after minor modifications, or identifying that a “new” viral image is actually a re-share of something that already went viral previously under a different upload.

17.8 Rate Limiting the Upload Path

The same token bucket and sliding window techniques covered generally elsewhere in this tutorial series apply directly to upload rate limiting here — protecting against a single account attempting to upload an unreasonable volume of content in a short period, whether due to a buggy client retry loop or deliberate abuse.

Common interviewer follow-ups — the answers to which show whether a candidate has genuinely internalized the shape of the design, or merely memorized its parts.

Here’s the narrative worth being able to walk through cleanly, start to finish, if asked to design this system live.

Placed alongside the other systems covered in this tutorial series — a content feed built around fan-out economics, and a profile-view tracker built around write-heavy event aggregation — this photo-sharing design rounds out a genuinely useful pattern for approaching any large-scale system design problem: identify the specific physical or traffic-shape constraint that dominates the problem, whether that’s an uneven fan-out distribution, an inverted read-write ratio, or in this case the hard physical limits of network latency across global distance, and let that dominant constraint drive the architecture, rather than reaching for a familiar template and hoping it happens to fit.