Start with the big picture before we meet the details.

The simple definition

Bigtable is a NoSQL wide-column database. “NoSQL” means it does not follow the classic relational design with joined tables. “Wide-column” means each row can hold a very large and flexible set of columns, and different rows can have completely different columns. Bigtable is also fully managed, so Google handles the machines, software updates, storage, repairs, and scaling plumbing.

You can think of a Bigtable table as a huge, sorted map. You give it a row key, and it gives you back the data for that row, very quickly, no matter how many billions of rows exist.

Why it exists

Google needed a storage system for products such as web search indexing, maps, and email, where data is enormous and must be served in milliseconds. Its engineers described the design in a well-known research paper in the mid-2000s. That design inspired many other databases, such as HBase and Cassandra. Google later offered the same system to everyone as Cloud Bigtable.

Where it is used

Teams choose it for time series data such as sensor readings, for user profiles and personalisation, for financial market data, for advertising and analytics, for recommendation and machine learning features, and for any workload that writes data constantly at a very large scale.

Six words describe how every piece of Bigtable data is organised. Learn these first.

flowchart TD
  A["Table"] --> B["Row with key user123"]
  B --> C["Column family profile"]
  B --> D["Column family activity"]
  C --> E["Qualifier name"]
  C --> F["Qualifier city"]
  D --> G["Qualifier last_login"]
  G --> H["Cell version at time 1"]
  G --> I["Cell version at time 2"]
    

An example row

Notice that empty cells are not stored at all. This is why the table is called sparse. A table with ten thousand possible columns costs almost nothing for rows that use only three. Bigtable stores only what exists.

Sorted by row key

All rows are kept in alphabetical order of their row keys, byte by byte. This is the single most important fact about Bigtable. It means that rows with similar keys sit next to each other, and a request for “all rows starting with this prefix” is very fast. Row key design is therefore the heart of good Bigtable work, which we study in Chapter 7.

Here are the building blocks that sit between your application and the stored data.

flowchart LR
  A["Application"] --> B["Bigtable Client Library"]
  B --> C["Front End Servers"]
  C --> D["Node 1"]
  C --> E["Node 2"]
  C --> F["Node 3"]
  D --> G["Colossus Distributed Storage"]
  E --> G
  F --> G
  H["Cluster Manager"] -.-> D
  H -.-> E
  H -.-> F
    

The main parts

Compute and storage are separate

This is a clever part of the design. The nodes do not own the data. The data sits in shared storage, and a node only holds the instructions that say which tablets it is responsible for. If a node fails, or the load changes, Bigtable simply gives its tablets to other nodes. No big data copying is needed, so rebalancing and recovery are fast.

Follow one write and one read to see why Bigtable is so quick.

sequenceDiagram
  participant App as Application
  participant N as Node
  participant L as Write Ahead Log
  participant M as Memory Table
  participant S as Sorted Files in Storage
  App->>N: Write a row
  N->>L: Record the change safely
  L-->>N: Log saved
  N->>M: Add change to memory
  N-->>App: Write confirmed
  Note over M,S: Later the memory table is flushed into a sorted file
  App->>N: Read a row
  N->>M: Check memory first
  N->>S: Check sorted files
  N-->>App: Return merged result
    

The write path

When you write, the node first records the change in a log so it can never be lost. Then it places the change in an in-memory structure that is already sorted. Because nothing has to be searched or rearranged on disk at that moment, writes are very fast. Once the memory area grows large enough, its contents are flushed into an immutable sorted file in storage. Immutable means the file is never edited, only replaced later.

The read path

When you read, the node looks in memory and in the sorted files, merges what it finds, and returns the newest version. Frequently used blocks are kept in a cache to make repeated reads even faster.

Compaction and deletes

Over time many small files pile up. Bigtable runs compaction in the background, which merges small files into bigger sorted ones and throws away old versions and deleted data. A delete is first written as a small marker, often called a tombstone, and the real clean-up happens later during compaction. This is why storage usage may not drop the instant you delete data.

Strong consistency within a cluster

Inside a single cluster, a row is handled by one node at a time. So a read right after a write sees that write. Changes to one row are atomic, meaning all of the changes to that row succeed together or none do. However, Bigtable does not give you transactions that span many rows or many tables. Designs must keep related data in one row when they need all-or-nothing updates.

Every Bigtable project passes through the same stages. Knowing them helps you plan.

Ways to get data in and out

• Client libraries. Official libraries exist for popular languages such as Java, Go, Python, Node.js, and C++.
• HBase-compatible API. Applications written for Apache HBase can often move over with little change.
• Dataflow. Google’s stream and batch processing service reads from and writes to Bigtable, and has ready-made templates.
• Dataproc and Spark. Big data jobs can scan Bigtable for analysis.
• BigQuery. Analysts can query Bigtable data from the BigQuery warehouse without copying it first.
• Import and export. Bulk files can be exchanged with Cloud Storage for loading or archiving.
• Command line and emulator. A simple command line tool and a local emulator help with development and testing.

Garbage collection of old versions

Each cell can hold many versions, which could grow without limit. So every column family has a garbage collection policy. For example: keep only the newest three versions, or delete anything older than thirty days. Bigtable removes expired versions automatically during compaction, which keeps storage tidy and costs down.

Bigtable rewards careful planning. A good design makes it fly. A poor design makes it crawl.

Think in queries, not in tables

With relational databases you often design the tables first and write the queries later. With Bigtable you do the opposite. Write down every important query first, then shape the row key so that each query becomes either a single-row lookup or a scan over a neat range of rows.

Column families

• Keep the number small. A handful of families is typical. Dozens can hurt performance.
• Group by access pattern. Put columns that are read together in the same family.
• Group by lifetime. Put short-lived data and long-lived data into different families so each can have its own cleaning rule.
• Use short names. Family and column names are stored with the data in every row, so shorter names save space at huge scale.

Denormalise

Because there are no joins, you usually store the data the way you will read it, even if the same fact appears in more than one place. This is called denormalisation. It uses more storage but gives fast reads, and storage is cheap compared to the cost of slow queries at scale.

Size habits

Keep row keys short, since they are repeated throughout the system. Keep individual cells reasonably small, and avoid giant rows. Bigtable documents recommended limits for the size of a row key, a cell, and a whole row, and very large values belong in object storage with only a pointer stored in Bigtable.

This is the most important chapter. The row key decides speed, balance, and cost.

The hot spot problem

Remember that rows are sorted by key and tablets hold neighbouring rows. If all new writes have keys that sort next to each other, they all land in the same tablet, which one node handles. That one node becomes overloaded while the others sit idle. This is called a hot spot.

flowchart TD
  A["Choose a row key"] --> B{"Does it start with a timestamp or a rising number"}
  B -->|"Yes"| C["Hot spot risk on one node"]
  B -->|"No"| D{"Is the first part well spread such as a user id or device id"}
  D -->|"Yes"| E["Writes spread across nodes"]
  D -->|"No"| F["Add a salt or hash prefix"]
  C --> G["Move the timestamp later in the key"]
  G --> D
  F --> E
    

Good and bad keys

Useful techniques

• Field promotion. Move a well-spread field, such as a device id, to the front of the key, and put the timestamp after it.
• Salting. Add a small computed prefix, such as the remainder of a hash divided by the number of buckets, to spread writes. The trade-off is that a range read must query every bucket.
• Reverse timestamps. Store a number that gets smaller as time moves forward, so the newest rows sort first.
• Delimiters. Join the parts of a key with a clear separator so prefixes make sense.
• Avoid sensitive data in keys. Keys appear in logs and tools, and they cannot be encrypted separately.

What if a zone fails, or users are spread across the world?

flowchart LR
  A["App in Asia"] --> B["App Profile Asia"]
  C["App in Europe"] --> D["App Profile Europe"]
  B --> E["Cluster Mumbai"]
  D --> F["Cluster Belgium"]
  E |"Automatic replication"| F
  E --> G["Colossus Storage Mumbai"]
  F --> H["Colossus Storage Belgium"]
    

An instance can contain several clusters in different zones or regions. Bigtable copies changes between them automatically, in both directions. Usually the delay is small, but replication is eventually consistent, which means a change made in one cluster may take a moment to appear in the other.

App profiles and routing

An app profile tells Bigtable how a particular application should be routed. With single-cluster routing, all of that app’s requests go to one chosen cluster, which keeps reads consistent with writes and allows multi-row style patterns that rely on a single place. With multi-cluster routing, requests go to the nearest healthy cluster, and if one fails, traffic moves to another automatically.

Why use replication

• High availability. A zone or region problem does not stop your service.
• Lower delay for users. Place clusters close to where users live.
• Workload isolation. Send live traffic to one cluster and heavy analytics jobs to another, so reports never slow down customers.
• Disaster protection. A second region protects against a regional outage.

Cost and capacity note

Each cluster has its own nodes and storage, so replication multiplies cost. Also, when you rely on automatic failover, each cluster must have spare capacity to take over the other’s traffic, so you should keep average load lower than in a single cluster set-up.

Bigtable scales by adding nodes. Here is how to do it wisely.

Scaling out

To get more speed, add nodes. Because the data lives in shared storage, new nodes take over some tablets quickly, without a long data copy. Throughput grows roughly in line with the number of nodes, provided your row keys spread the load well. You can set the node count by hand or let autoscaling follow a target such as average processor use. Autoscaling suits traffic that rises and falls during the day.

Storage choices

What affects speed

• Row key design. The biggest factor, as Chapter 7 explained.
• Node count. More nodes can handle more requests.
• Access pattern. Single-row lookups and short range scans are fast. Scanning an entire huge table is slow and heavy.
• Row and cell size. Smaller rows give more operations per second.
• Network distance. Keep applications in the same region as the cluster.
• Warm-up. A brand-new or freshly scaled cluster needs a little time under steady load to spread tablets and fill caches.
• Client behaviour. Reuse connections and send batches, instead of creating new connections for every request.

Keep headroom

A commonly recommended practice is to keep average processor load well below the maximum, for example around two-thirds on a single cluster and lower when using failover between clusters. This leaves room for traffic spikes and for background work such as compaction. Always check the current guidance in the documentation.

An honest view of what you gain and what you give up.

The big trade-offs

Speed versus flexibility. You gain extreme speed for the queries you planned for, and you lose the freedom to ask any question you like. Teams often add BigQuery beside Bigtable for flexible analysis.

Consistency versus availability. A single cluster is strongly consistent for rows. Multi-cluster routing is more available but only eventually consistent.

Cost floor versus scale. Bigtable is a poor bargain for tiny data, and generally recommended for data in the range of a terabyte or more, or for very high request rates. Below that, other databases are simpler and cheaper.

Denormalised copies versus storage. Storing the same data in several shapes speeds reading but uses more space and needs your code to keep copies in step.

Databases hold valuable data, so protect them in layers.

Who may do what: IAM

Access is controlled by IAM, Google’s permission system. Roles can be given at the project, instance, or table level. Some roles can only read data, some can read and write, and some can manage the instance. Give each person and each program the smallest role that works. An application that only shows data should have read access only.

Protecting data

• Encryption at rest. Always on by default. You can use your own keys through customer-managed encryption keys if your rules require it.
• Encryption in transit. Connections are encrypted.
• Private access. Use private connectivity so traffic stays off the public internet where possible.
• VPC Service Controls. Build a protective boundary around your projects to reduce the risk of data leaving by mistake.
• Service accounts. Use a separate service account for each application, with narrow roles, and avoid long-lived downloaded keys.
• Audit logs. Record who administered the instance and, if you enable it, who read or wrote data.
• Keep secrets out of keys. Row keys appear in logs and tools, so do not place personal data or passwords in them.
• Separate environments. Use different projects for testing and production.

Replication protects against failure. Backups protect against mistakes.

If a bug writes bad data or someone deletes the wrong rows, replication faithfully copies that damage to every cluster. Only a separate, earlier copy can save you. Bigtable offers managed backups of a table, which are stored with the cluster and kept for a time you choose. You can restore a backup into a new table, and you can often restore it into a different instance if your rules allow.

Plan with two numbers

RPO is how much recent data you can afford to lose. RTO is how long you can afford to be unavailable. Replication with failover gives a very short RTO for zone failures. Frequent backups give a small RPO against human error. Practise a restore at least once, because an untested backup is only a hope.

You cannot fix what you cannot see. Watch the numbers that matter.

What to watch

• Average processor load. Rising load means it is time to add nodes or to improve the design.
• Hottest node load. If one node is far busier than the average, you probably have a hot spot, and adding nodes will not help.
• Latency. Watch the slowest few percent of requests, not only the average.
• Storage use. Make sure growth is expected and old versions are being cleaned.
• Error rates and throttling. A rising count means overload or client problems.
• Replication delay. Important when you route traffic across clusters.
• Backup success. Verify that scheduled backups finish.

The pair of “average load” and “hottest node load” is especially helpful. When the average is low but one node is overloaded, you have a design problem. When all nodes are busy, you have a capacity problem. The cures are different, so reading the right chart saves both money and time.

Helpful habits and harmful habits seen again and again in real projects.

Helpful patterns

Anti-patterns to avoid

More things to steer clear of

• Full table scans in live traffic. They load every node and slow all users.
• Hundreds of column families. They hurt performance and are hard to manage.
• Huge values in cells. Put large files in Cloud Storage and store only a link.
• Running near full capacity. No room for spikes or compaction.
• Relying on replication as a backup. It copies mistakes too.
• Many tiny writes with new connections each time. Reuse connections and batch your requests.

A checklist of habits that experienced teams rely on.

Habits that work

• Write down the queries first. Key design follows from them.
• Test with realistic data. Use true volume, true key patterns, and true request rates before launch.
• Look at Key Visualizer after every major change.
• Set garbage collection policies on purpose. Never leave unlimited versions by accident.
• Use autoscaling for changing traffic, with sensible minimum and maximum limits.
• Keep clients in the same region as the cluster, and reuse connections.
• Use app profiles to separate serving and analytics.
• Schedule backups and practise restores.
• Use least-privilege IAM and separate production from testing.
• Plan capacity headroom so failover and spikes do not cause an outage.
• Label resources for cost tracking and ownership.

Mistakes beginners often make

Common ways teams use Bigtable. These are typical scenarios, not claims about any particular company.

Google Cloud has many databases. Here is how Bigtable compares.

A simple rule of thumb: if you have enormous data or request rates and you know your access pattern by key, choose Bigtable. If you need joins and transactions, choose a relational option. If you need flexible analysis of everything, choose BigQuery. If you have small data, choose something simpler. Many large systems combine them, using Bigtable to serve and BigQuery to analyse.