One librarian and one bookshelf works fine for a hundred books. Ten million books need a whole city of branch libraries. Sharding is how the internet solved that same problem for data.

Picture a school library with a single librarian and a single bookshelf. When there are a hundred books on that shelf, the librarian can find any one of them in seconds. But what happens when the library’s collection grows to ten million books? One shelf cannot hold them. One librarian cannot remember where every title lives. Every student ends up standing in a long line just to ask a simple question.

That is exactly the situation big websites and apps eventually run into with their data. In the early years of the internet, a single database sitting on a single computer was more than enough to power an entire application. As companies grew from thousands of users into hundreds of millions, one database was no longer close to enough. Engineers needed a way to split their data across many computers while still making it feel like “one library” to the people using it.

That solution is called sharding.

A short history

The word “shard” literally means a small broken piece of something larger — a shard of glass or pottery. The idea of splitting data across machines is older than the modern web; mainframe systems in the 1980s already used a technique called “horizontal partitioning” to spread large tables across storage devices.

The term “sharding” in its modern sense took off in the early 2000s, largely thanks to massively multiplayer online games (MMOs) like Ultima Online and EverQuest. Those games split their game world into multiple parallel copies called “shards” — each shard being a separate server that held a subset of players and game state, so the game as a whole could support far more players than any single server could.

Around the same era, web companies like Google, Amazon, and later Facebook, Twitter, and Uber were hitting the very same wall with their relational databases. Google’s Bigtable paper (2006) and Amazon’s Dynamo paper (2007) popularised the idea of automatically partitioning data across thousands of commodity machines. From there, sharding rapidly became a standard pattern — taught in every system design course and asked about in nearly every senior engineering interview.

To really understand sharding, you first have to understand the problem it fixes: a single database server running out of room to grow. Every machine has walls, and eventually one of them is what a system hits first.

To understand sharding, you first need to understand the problem it exists to fix: a single database server, no matter how expensive, running out of room to grow.

The limits of a single database server

Every computer, however powerful, has physical limits:

• Storage limit — a hard disk or SSD can only hold so much data.
• Memory (RAM) limit — only so much data can be cached in memory for fast access.
• CPU limit — only so many queries can be processed per second.
• Network limit — only so much data can move in and out over the network per second.

When a company is small, none of that matters. But as the number of users and the volume of data climbs into the millions or billions of rows, a single machine — no matter how expensive — eventually cannot keep up. Engineers call the moment you hit this a “scaling wall.”

Vertical scaling vs. horizontal scaling

There are two broad ways to deal with a database that is running out of capacity:

Sharding vs. replication — not the same thing

Beginners often confuse sharding with replication, but the two solve genuinely different problems:

In real production systems, sharding and replication are almost always used together: every shard is itself replicated for safety. We will return to this combination later in the High Availability chapter.

Eight short definitions to keep in your pocket. Master these, and every other section in this guide clicks into place.

3.1 What is sharding?

Sharding (also known as horizontal partitioning) is the process of splitting a large dataset into smaller, independent pieces called shards, and then storing each shard on a separate database server. Every shard holds a different subset of the rows in a table, but all shards together share the exact same table structure (columns).

3.2 Horizontal vs. vertical partitioning

It helps to clearly separate two often‑confused ideas:

• Horizontal partitioning (sharding) — splits a table by rows. Users 1–1,000,000 go to Shard A, users 1,000,001–2,000,000 go to Shard B. Every shard has the same columns.
• Vertical partitioning — splits a table by columns. For instance, keeping frequently‑used columns (name, email) in one table and rarely‑used columns (biography, preferences) in another, possibly on a different server.

3.3 Shard key (partition key)

The shard key is the piece of data used to decide which shard a particular row belongs to. Picking the shard key is the single most important design decision in a sharded system — a poor choice can quietly undo all the benefits of sharding.

3.4 Shard (partition, node)

A shard is one independent database instance that holds a slice of the total data. Each shard is a fully working, self‑contained database with its own storage, memory, and processing power. Shards do not share data with each other — they only know about their own slice.

3.5 Logical shard vs. physical shard

Modern systems often separate the concept of a “logical shard” (a bucket, e.g. “bucket 173 of 1024”) from a “physical shard” (an actual database server). Many logical shards can be mapped onto one physical server. That extra layer of indirection makes it much easier to move data around later without touching the shard key logic — we will unpack this under Consistent Hashing.

3.6 Shard map / routing layer

The shard map (or routing table) is the lookup table that tells the application which physical shard holds a given logical shard or key range. A router or proxy component uses this map to send each query to the correct shard.

3.7 Rebalancing

Rebalancing is the process of moving data between shards after they are already created — usually because one shard has grown much larger or busier than the others (a “hot shard”). This is one of the hardest operational challenges in any sharded system.

3.8 Cross‑shard query

A cross‑shard query is any query that needs data from more than one shard (for example, “find the total number of orders across all customers”). These queries are slower and more complex than single‑shard queries because the application has to ask multiple shards in parallel and combine the results itself.

A production sharded system typically has five building blocks working together. The application talks to a router, the router consults a shard map, and the router forwards the request to the right shard — not the other way around.

A production sharded system typically has five building blocks working in concert.

Where routing logic can live

Levels of sharding maturity

Teams tend to move through a predictable set of stages as their data grows, and it helps to recognise where you are on this ladder:

1. Single database — one server handles everything. Perfectly fine for most applications, even fairly large ones.
2. Vertical scaling + read replicas — a bigger primary server for writes, with read‑only copies to spread out read traffic. This alone solves the majority of early scaling problems.
3. Functional federation — splitting one big database into several databases by subject area (users, orders, payments), each still a single server.
4. Sharding — one or more of those functional databases is further split horizontally across multiple servers, because a single server can no longer hold that subject area’s data or traffic.
5. Globally distributed sharding — shards spread not just across servers but across regions and continents, often using a distributed SQL database, so a worldwide user base gets low latency everywhere.

Very few companies ever need to reach stage 5. Recognising which stage your system is actually at — rather than assuming you need the most advanced setup — is one of the most valuable skills an architect can develop.

At the heart of every sharded system is one small function: given a piece of data, tell me which shard it belongs to. Five common ways to answer that question — each with its own personality.

The heart of sharding is the function that decides, for any given piece of data, which shard it belongs to. This is called the partitioning function or sharding algorithm. Let’s walk through the five approaches you will see most often.

5.1 Hash‑based sharding

The shard key is passed through a hash function (a function that converts input into a fixed‑size number that looks random but is always the same for the same input). The hash result is then used, usually with a modulo operation, to pick a shard.

shard_number = hash(shard_key) % number_of_shards

Why hashing helps

Hash‑based sharding spreads data evenly and avoids “hot spots” — situations where one shard gets far more traffic than the others. That is its biggest advantage over simple range‑based splitting.

The big weakness: adding or removing shards

The formula hash(key) % N has a serious problem: if N (the number of shards) changes — say you grow from four shards to five — almost every key’s hash % N result changes as well. Which means almost all the data must be moved around at once, which is extremely expensive and disruptive in a live production system.

5.2 Consistent hashing

Consistent hashing is a smarter hashing technique, introduced in a 1997 paper by Karger et al. at MIT, and popularised in production by Amazon’s Dynamo paper. It dramatically reduces how much data needs to move when shards are added or removed.

Here is how it works, step by step:

1. Imagine a circle (a “ring”) representing all possible hash values, from 0 up to some very large number, wrapping back round to 0.
2. Each shard (server) is placed at one or more points on this ring, based on hashing the server’s name or ID.
3. Each piece of data is also hashed to find its own position on the ring.
4. To find which shard owns a piece of data, walk clockwise from the data’s position until you hit the first shard.

The magic of consistent hashing: when a new shard is added, it only takes over a small slice of the ring from its neighbour. All other shards are completely unaffected. Instead of moving nearly 100% of the data (as with plain modulo hashing), you typically only move about 1/N of the data, where N is the new number of shards.

Virtual nodes

In practice, each physical shard is given many points on the ring (called “virtual nodes” or “vnodes”) instead of just one. That spreads load more evenly and makes rebalancing smoother when a shard is added or removed. Cassandra, DynamoDB, and Riak all use virtual nodes internally.

5.3 Range‑based sharding

Data is split according to ranges of the shard key’s value. For example, customer IDs 1–1,000,000 go to Shard A, 1,000,001–2,000,000 go to Shard B, and so on.

5.4 Directory‑based (lookup table) sharding

Instead of computing the shard with a formula, a separate lookup service simply stores, for every key, exactly which shard it lives on. This is the most flexible approach because any key can be moved to any shard just by updating the lookup entry — no formula, no ring, just a map.

5.5 Geo‑based sharding

Data is split by geography — European users’ data stays on shards in Europe, US users’ data stays on shards in the US, and so on. This is increasingly common not just for performance (keeping data close to users reduces latency) but also for legal compliance, such as GDPR requirements that certain data must stay within specific countries or regions.

Follow one request from application to storage — a single‑shard write, a single‑shard read, and the scatter‑gather query that fans out to every shard at once.

Let’s trace exactly what happens when an application reads and writes sharded data, from request to response.

Write path

1. The application sends a write request, e.g. “create an order for customer 8842.”
2. The router extracts the shard key (customer_id = 8842).
3. The router computes or looks up the target shard (e.g. Shard 3).
4. The write is sent only to Shard 3 and its replicas.
5. Shard 3 acknowledges the write; the router returns success to the application.

Read path (single‑shard query)

1. The application asks “get all orders for customer 8842.”
2. The router computes the shard for customer_id = 8842 — again Shard 3.
3. The query goes directly to Shard 3 (or one of its read replicas).
4. Shard 3 returns the result; the router passes it back unchanged.

Read path (cross‑shard / scatter‑gather query)

Some queries do not include the shard key — for example, “find all orders placed today across the entire company.” The router cannot know which single shard holds the answer, because the data is spread across all of them.

This pattern is called scatter‑gather (or fan‑out). It works, but it is inherently slower and more resource‑intensive than a single‑shard query, because the overall response time is limited by the slowest shard, and the results must then be merged, sorted, or aggregated in application code.

If sharding has one make‑or‑break decision, it is the shard key. Four qualities separate a great key from one that quietly ruins the system a year after launch.

If sharding has one “make‑or‑break” decision, it is the shard key. A poorly chosen key can create imbalanced shards, excessive cross‑shard queries, or both. Here are the qualities to look for.

Qualities of a good shard key

• High cardinality — many distinct possible values, so data spreads across many shards (e.g. user_id is good; country with only five values is risky).
• Even distribution — no single value should be dramatically more common than others (avoids “hot” shards).
• Matches query patterns — most of your application’s queries should already include this key, so most reads and writes stay on a single shard.
• Rarely changes — changing a row’s shard key means physically moving the row to a different shard, which is expensive.

Common anti‑pattern: sharding by a low‑cardinality field

A beginner mistake is sharding by something like status (“active”/“inactive”) or country. If 90% of users are in one country, that shard becomes massively overloaded while the others sit nearly empty — defeating the entire purpose of sharding.

Java example: a simple hash‑based router

Below is a simplified Java example showing how a router might decide which shard to send a query to, using a basic hash‑based approach.

public class ShardRouter {

    private final int numberOfShards;

    public ShardRouter(int numberOfShards) {
        this.numberOfShards = numberOfShards;
    }

    // Decides which shard a given key belongs to
    public int getShardForKey(String shardKey) {
        int hash = Math.abs(shardKey.hashCode());
        return hash % numberOfShards;
    }

    public static void main(String[] args) {
        ShardRouter router = new ShardRouter(4);

        String[] userIds = {"user_101", "user_202", "user_303", "user_404"};
        for (String userId : userIds) {
            int shard = router.getShardForKey(userId);
            System.out.println(userId + " -> Shard " + shard);
        }
    }
}

This works, but as we discussed earlier, it suffers from the “almost everything moves” problem when numberOfShards changes. Here is a simplified consistent hashing implementation that avoids that issue.

import java.util.SortedMap;
import java.util.TreeMap;

public class ConsistentHashRouter {

    private final SortedMap<Long, String> ring = new TreeMap<>();
    private final int virtualNodesPerShard;

    public ConsistentHashRouter(int virtualNodesPerShard) {
        this.virtualNodesPerShard = virtualNodesPerShard;
    }

    public void addShard(String shardName) {
        for (int i = 0; i < virtualNodesPerShard; i++) {
            long hash = hash(shardName + "#VN" + i);
            ring.put(hash, shardName);
        }
    }

    public void removeShard(String shardName) {
        ring.values().removeIf(name -> name.equals(shardName));
    }

    public String getShardForKey(String key) {
        long hash = hash(key);
        // Find the first shard clockwise from this hash value
        SortedMap<Long, String> tail = ring.tailMap(hash);
        Long nodeHash = tail.isEmpty() ? ring.firstKey() : tail.firstKey();
        return ring.get(nodeHash);
    }

    private long hash(String input) {
        // A simple, deterministic 64-bit hash (production code should use
        // MD5, MurmurHash, or a similar well-tested hash function)
        long h = 1125899906842597L;
        for (char c : input.toCharArray()) {
            h = 31 * h + c;
        }
        return h;
    }

    public static void main(String[] args) {
        ConsistentHashRouter router = new ConsistentHashRouter(100);
        router.addShard("ShardA");
        router.addShard("ShardB");
        router.addShard("ShardC");

        System.out.println("user_101 -> " + router.getShardForKey("user_101"));
        System.out.println("user_202 -> " + router.getShardForKey("user_202"));

        // Adding a new shard only remaps a small fraction of keys
        router.addShard("ShardD");
        System.out.println("After adding ShardD:");
        System.out.println("user_101 -> " + router.getShardForKey("user_101"));
    }
}

Sharding is powerful, but it is not free. The same properties that make it scale also introduce operational complexity that every architect eventually has to reckon with.

Advantages of sharding

Disadvantages of sharding

Doubling shards roughly doubles capacity — in theory. In practice, hot shards, tail latency, and network hops all decide whether that potential is real.

The whole point of sharding is scalability, so it is worth understanding precisely how it improves performance — and where its limits are.

Linear (near‑linear) scalability

In an ideal sharded system, doubling the number of shards roughly doubles the total write capacity, because each shard independently handles its own slice of traffic with no shared bottleneck. In practice, scalability is “near‑linear” rather than perfectly linear, because of overhead from routing, cross‑shard queries, and coordination.

Hot shards and skew

A “hot shard” is one that receives disproportionately more traffic or data than the others, becoming a bottleneck even though the system as a whole has spare capacity elsewhere. This is the single biggest threat to sharding’s performance benefits, and it almost always traces back to a poor shard key choice, or to a naturally skewed real‑world distribution (for example, a small number of extremely popular users or products).

Query latency considerations

• Single‑shard queries stay fast, because each shard is small and the query never leaves one machine.
• Cross‑shard queries are bound by the slowest participating shard, plus the time it takes to merge results — sometimes called the “tail latency” problem.
• Network hops matter — every extra hop (application → router → shard) adds latency, so routers are usually kept close to the shards.

Sharding on its own does not make a system reliable — it multiplies your points of failure. Combining sharding with replication is what turns fragile into resilient.

Sharding alone does not make a system reliable — in fact, on its own, it makes a system more fragile: with ten shards, you now have ten independent points of failure instead of one. This is exactly why sharding is almost always combined with replication.

Sharding + replication together

In production, each shard is not a single server — it is a small cluster: one primary server that accepts writes, and one or more replica servers holding up‑to‑date copies of that shard’s data. If the primary fails, a replica can be promoted to take over.

Failover

Failover is the process of automatically detecting that a shard’s primary server has failed and promoting one of its replicas to become the new primary, so the system keeps working with minimal disruption. Modern systems (MongoDB replica sets, PostgreSQL with Patroni, MySQL Group Replication) automate this process using consensus protocols, typically completing failover in seconds.

Backup and disaster recovery

Because losing a shard means permanently losing that slice of data (unless replicas or backups exist), sharded systems need a disciplined backup strategy:

• Regular automated snapshots of every shard, stored separately from the shard itself.
• Point‑in‑time recovery, so a shard can be restored to its state just before a bad deployment or accidental deletion.
• Cross‑region replicas, so that an entire data centre failure does not take down every replica of a shard at once.

Distributed systems live under three unavoidable constraints. The CAP theorem names them; consensus algorithms are how real systems make peace with them.

The CAP theorem

The CAP theorem, formalised by computer scientist Eric Brewer, states that a distributed data system can only fully guarantee two out of these three properties at the same time, during a network failure:

• Consistency (C) — every read receives the most recent write, or an error.
• Availability (A) — every request receives a (non‑error) response, even if it might not be the latest data.
• Partition Tolerance (P) — the system keeps working even if network communication between servers is broken.

Because real networks do occasionally experience partitions (a switch fails, a cable is cut, a data centre loses connectivity), partition tolerance is not really optional for a distributed sharded system — so the real, practical choice is between consistency and availability when a partition happens.

Consensus algorithms

To keep replicas of a shard agreeing on who the current primary is (and what the latest committed data is), distributed systems use consensus algorithms — protocols that let a group of servers agree on a value even if some of them fail or messages get delayed.

• Raft — a consensus algorithm designed to be easy to understand; used by etcd, Consul, and CockroachDB.
• Paxos — an older, more complex consensus algorithm; used inside Google Chubby and Spanner.

Failure recovery workflow

1. Detection — a health‑check or heartbeat mechanism notices a shard’s primary is not responding.
2. Election — the remaining replicas run a consensus protocol to elect a new primary.
3. Promotion — the elected replica becomes the new primary and starts accepting writes.
4. Re‑routing — the shard map / config service is updated, so the router sends future traffic to the new primary.
5. Recovery — the failed server, once fixed, rejoins as a replica and catches up on missed data.

Sharding spreads data across many servers, which means security has to be enforced consistently everywhere. A single misconfigured shard becomes a weak link for the entire system.

Sharding spreads data across many servers, which means security has to be enforced consistently everywhere — a single misconfigured shard can become a weak link for the entire system.

Key security concerns

• Consistent access control — every shard must enforce the same authentication and authorisation rules. A permission fix applied to one shard but forgotten on another creates a security gap.
• Encryption in transit — traffic between the router and shards, and between shards and their replicas, should be encrypted (TLS), especially when shards live in different data centres or cloud regions.
• Encryption at rest — each shard’s underlying storage should be encrypted, so a stolen disk or backup does not expose data.
• Network isolation — shards typically sit in private subnets, reachable only through the router/proxy layer, not directly from the public internet.
• Data residency compliance — regulations like GDPR (Europe) or data localisation laws (in countries such as India and Russia) may require that certain users’ data physically stays within specific geographic boundaries — a strong motivation for geo‑based sharding.
• Audit logging — because data is spread out, security teams need centralised, aggregated audit logs across all shards to detect suspicious patterns that might only be visible when the whole system is looked at together.

With one database, monitoring is easy: watch one dashboard. With many shards, monitoring becomes an exercise in spotting the one shard out of a hundred that is quietly misbehaving.

With a single database, monitoring is simple — watch one dashboard. With many shards, monitoring becomes an exercise in spotting the one shard out of a hundred that is misbehaving.

Key metrics to track per shard

Aggregated dashboards

Beyond per‑shard metrics, teams build aggregated views — total QPS across all shards, the “top 5 hottest shards,” and heat maps showing load distribution — so a human operator can spot skew at a glance instead of checking a hundred separate dashboards.

Distributed tracing

For cross‑shard (scatter‑gather) queries, distributed tracing tools (such as Jaeger or Zipkin, following the OpenTelemetry standard) let engineers see exactly how long each shard took to respond within a single logical request, making it possible to pinpoint which shard is slowing everything down.

Modern cloud platforms have made sharding far more approachable than it used to be, by offering managed services that handle much of the routing, rebalancing, and failover for you.

Modern cloud platforms have made sharding far more approachable than it used to be, thanks to managed services that handle much of the routing, rebalancing, and failover automatically.

Managed sharding services

Containerisation & orchestration

Shards are commonly deployed as stateful containers managed by Kubernetes, using StatefulSets to give each shard’s pods a stable network identity and persistent storage — since (unlike stateless web servers) shards cannot simply be swapped for an identical replacement. Each one holds unique data.

Zero‑downtime shard migrations

Moving data to a new shard while the system stays live typically follows this pattern:

1. Dual writes — new writes are sent to both the old and the new shard.
2. Backfill — existing data is copied from the old shard to the new one in the background.
3. Verification — the new shard’s data is compared against the old shard’s to confirm correctness.
4. Cutover — the router’s shard map is updated to point reads (and eventually writes) to the new shard.
5. Cleanup — once confidence is high, the old copy of the data is removed.

Sharding lives alongside caching and load balancing, not instead of them. Understanding how the three interact is what turns a “sharded system” into an actually fast one.

Sharding and caching

Caching (using tools like Redis or Memcached) and sharding solve different but complementary problems. Caching keeps frequently‑accessed data in fast memory to avoid hitting the database at all; sharding spreads the database itself across multiple machines. Large systems typically use both: a cache in front of every shard, so even a single shard’s load is reduced further.

Sharding and load balancing

A load balancer spreads incoming network traffic across multiple identical servers. A shard router is a specialised form of this idea — except instead of sending traffic to any available server, it sends each request to one specific, correct server based on the data being requested. Within a single shard’s replica set, a traditional load balancer is often still used to spread read traffic across that shard’s replicas.

NoSQL databases built for sharding

Some databases were designed from the ground up to be sharded automatically, without engineers having to build custom routing logic:

• MongoDB — built‑in sharding using a “shard key,” with a routing component called mongos.
• Cassandra — uses consistent hashing internally to spread data (“partitions”) across a ring of nodes with no single point of failure.
• DynamoDB — automatically partitions tables based on a partition key, scaling storage and throughput transparently.
• Elasticsearch — splits indices into “shards” for both storage and parallel query execution.

Sharded systems have their own recurring patterns — and their own recurring bad habits. Learning to name both is what separates a candidate who has read about sharding from one who has actually run it.

Useful patterns

Anti‑patterns to avoid

• Sharding too early — adding the complexity of sharding before the system actually needs the scale, when a bigger single server or read replicas would have been simpler.
• Sharding by a low‑cardinality or skewed key — as discussed earlier, this creates hot shards and defeats the whole purpose.
• Ignoring cross‑shard transactions until it is too late — not planning for how multi‑shard operations (like “transfer money between two users on different shards”) will be handled.
• Manual, undocumented rebalancing — moving data between shards by hand, without tooling or verification, is a common source of data‑loss incidents.
• No shard‑level monitoring — only watching aggregate, system‑wide metrics, which can hide a single badly overloaded shard.

A tight checklist of habits that consistently produce healthy sharded systems — plus the mistakes that quietly sink otherwise good designs six months in.

Best practices

Common mistakes

• Picking a shard key that matches today’s data, but not tomorrow’s query patterns.
• Forgetting that unique constraints (like a unique email address) are hard to enforce across shards, since each shard only sees its own slice of data.
• Underestimating how expensive and risky live rebalancing will be, and not building tooling for it early.
• Treating each shard as equally important, when in reality some shards (e.g. large enterprise customers) may need dedicated capacity.
• Not testing failure scenarios (a shard going down, a replica lagging) until it happens for real in production.

Every major platform you use runs on a sharded system somewhere. Here is how six of them made the trade‑offs in real production.

Ten of the questions candidates and engineers ask most often about sharding — answered without hand‑waving.

Sharding is the technique of splitting a large dataset by rows across multiple independent database servers, so that no single machine has to store or process all the data alone. It is one of the foundational tools for scaling from thousands to billions of records.

Sharding is the technique of splitting a large dataset by rows across multiple independent database servers, so that no single machine has to store or process all the data alone. It is one of the foundational tools for building systems that can grow from thousands to billions of records without hitting a hard capacity wall.

Understanding sharding well means understanding both sides of it: the elegant idea of splitting data to scale, and the very real engineering costs — complexity, cross‑shard queries, and rebalancing — that come with it. Every experienced architect learns to ask not “should we shard?” but “do we actually need to shard yet, and if so, what shard key will serve us for years, not just months?”