Compression is the art of representing the same information with fewer bytes.

The plain definition

Compression is a technique that takes some data, such as a text file, a log line, a message, or a database page, and rewrites it in a smaller form. Later, a matching step called decompression turns the small form back into the original. The program that does the shrinking and un-shrinking is called a codec, which is short for coder-decoder. gzip, Snappy, LZ4 and zstd are all codecs.

Compression works because real data is rarely random. Human language repeats words. Log files repeat the same timestamps, field names, and error messages thousands of times. JSON repeats the same keys for every record. Whenever data repeats or follows a pattern, a codec can say the pattern once and then point back to it, instead of writing it out every time.

Why anyone bothers

There are three big reasons. First, storage costs money. Cutting a data set to one third of its size can cut the disk bill by the same amount. Second, networks have limits. Sending fewer bytes means messages arrive sooner and a link can carry more traffic. Third, many systems are limited by how fast they can read from disk or the network, not by how fast the CPU can think. If reading is the slow part, reading fewer bytes and then spending a little CPU time to expand them can actually make the whole job faster.

A handful of words show up in every compression conversation, so let us define them once.

Compress speed and decompress speed are different things

Beginners often talk about one speed, but there are two. Compression is usually the heavier job because the codec has to search for patterns. Decompression is usually much lighter because it just follows instructions. This matters a lot in real life. Data is often written once and read many times, so a codec that is slow to compress but very fast to decompress can be a great deal.

Almost every codec here is built from the same two ideas: remove repeats, then write what is left as compactly as possible.

Stage one: finding repeats with LZ77

The family of ideas behind all four codecs comes from a 1977 method called LZ77, named after Abraham Lempel and Jacob Ziv. The codec walks through the data and keeps a window of the recent bytes. When it sees a sequence it has already seen, it does not write it again. Instead it writes a tiny note that says: go back this many bytes, and copy this many bytes. That note is called a match or a back-reference. Anything that has no earlier copy is written out as it is, and these bytes are called literals.

Stage two: entropy coding

After the repeats are removed, you are left with a stream of literals and match notes. Some of these appear far more often than others. Entropy coding gives short bit patterns to the frequent ones and long patterns to the rare ones, like Morse code giving a single dot to the letter E because it is so common. Huffman coding is the classic way to do this. A newer family called ANS, which zstd uses through a design named Finite State Entropy, reaches similar or better savings while running much faster.

flowchart LR
  A["Raw data"] --> B["Find repeats in sliding window"]
  B --> C["Literals and back-references"]
  C --> D{"Entropy coding used?"}
  D -->|"gzip and zstd"| E["Huffman or FSE codes"]
  D -->|"Snappy and LZ4"| F["Write directly"]
  E --> G["Framed compressed output"]
  F --> G
        

You can rarely have small size, fast compression, and fast decompression all at once.

Picture a triangle with three corners: smallest output, fastest compression, and fastest decompression. A codec sits somewhere inside it. Moving toward one corner usually pulls you away from another. A very small output needs a long search for patterns, which makes compression slow. A very fast compressor cannot afford to search long, so it leaves patterns unfound and the output is larger.

The extra cost that people forget: memory

There is a fourth corner that is easy to miss. Codecs need memory for their window, hash tables, and buffers. If your service runs thousands of compression streams at the same time, such as thousands of connections each compressing their own traffic, the memory per stream adds up quickly. A codec that looks great in a single-file benchmark can become a problem at scale for exactly this reason.

gzip is the old reliable of compression. It is everywhere, and almost every tool on earth can read it.

What it is

gzip was created in the early 1990s as a free replacement for older compression tools. The file format is described in RFC 1952, and the compression method inside it is called DEFLATE, described in RFC 1951. DEFLATE combines an LZ77 search with Huffman coding. Because it was standardized so early and so openly, gzip became built into web servers, browsers, operating systems, programming languages, and countless tools.

How it works

gzip looks back over a window of 32 kilobytes to find repeats. It then encodes the literals and matches using Huffman codes, and it can build fresh Huffman tables for each block of data to fit that block well. It adds a small header and a checksum at the end, which lets the reader detect damaged data. Levels run from 1 to 9. Level 1 is the fastest, level 9 is the smallest, and level 6 is the usual default that balances the two.

Snappy trades some size for a lot of speed, and it made that trade on purpose.

What it is

Snappy came out of Google, where it was used internally for years under an earlier name before being released as open source in 2011. Its goal is very different from gzip. It does not try to produce the smallest output. It tries to keep the CPU cost so low that compression becomes almost invisible in busy systems like databases and distributed processing engines.

How it works

Snappy uses an LZ77-style search but keeps it very simple. It looks at small groups of bytes, uses a quick hash to guess where an earlier copy might be, and moves on quickly when it does not find one. It works on small blocks of around 64 kilobytes. It has no entropy stage at all, so literals are stored as they are and matches are stored in a compact fixed format. There are no levels to tune. Snappy has one speed setting, which is fast.

LZ4 is designed so that decompression runs at speeds close to how fast memory can deliver bytes.

What it is

LZ4 was created by Yann Collet and released around 2011. Its core promise is extremely fast compression and even faster decompression, with a modest ratio. Like Snappy, it is a pure LZ77-style codec with no entropy stage. Its format is designed so that a decoder is little more than a tight loop of copying bytes, which modern CPUs execute very quickly.

How it works

LZ4 encodes data as a series of sequences. Each sequence holds a run of literals followed by a match reference, with small headers that store lengths in a very compact way. The matching window is 64 kilobytes, and the minimum match length is short. The default mode uses a fast hash search. There is also a high-compression variant called LZ4 HC, which searches more thoroughly to get a smaller output while keeping decompression just as fast. This is a very useful property: you can spend extra time once when writing and still enjoy the same lightning-fast reads.

zstd was designed to beat gzip on ratio and speed at the same time, and to cover a huge range of use cases with a single tool.

What it is

Zstandard, usually written zstd, was created by Yann Collet at Facebook and open sourced in 2016. It is standardized in RFC 8878. The goal was ambitious: match or beat gzip on size while being much faster, and also scale down to speeds close to LZ4 or up to very high ratios, all within one codec family. Today it is used in Linux kernels, package managers, databases, log pipelines, and messaging systems.

How it works

zstd starts with a familiar LZ77-style search, but it supports much larger windows than gzip, so it can find repeats that are far apart. It then applies two entropy coders: Huffman for the literals, and Finite State Entropy for the match information. FSE is based on a modern method called Asymmetric Numeral Systems, which gets close to the theoretical best size while running fast. The result is a codec that is both small and quick.

Seeing the four codecs together makes their personalities obvious.

A typical picture in numbers

On a well-known mixed benchmark data set, published results for the zstd project show a familiar pattern. gzip at its default level lands around a 2.7 ratio at roughly 100 megabytes per second compression and around 400 megabytes per second decompression. zstd at its fastest standard level lands around a 2.9 ratio at several hundred megabytes per second compression and well over a thousand megabytes per second decompression. Snappy and LZ4 land around a 2.1 ratio, with LZ4 decompressing fastest of all. These are approximate and vary by machine, but the shape is consistent: zstd tends to beat gzip on both size and speed, and LZ4 wins on raw decompression speed.

quadrantChart
    title Speed versus compression ratio
    x-axis "Smaller output" --> "Faster speed"
    y-axis "Lower efficiency" --> "Higher efficiency"
    quadrant-1 "Fast and small"
    quadrant-2 "Small but slow"
    quadrant-3 "Slow and larger"
    quadrant-4 "Fast but larger"
    gzip: [0.25, 0.55]
    Snappy: [0.75, 0.35]
    LZ4: [0.9, 0.3]
    zstd: [0.6, 0.8]
        

You already use compression every day, even when you never turn it on yourself.

Following one message from the producer to the reader shows exactly where the cost is paid.

In a typical pipeline, data is compressed once by the writer, stored or transmitted in its small form, and decompressed each time a reader needs it. That means the writer pays the compression cost once, while every reader pays the decompression cost. If you have one writer and fifty readers, decompression speed matters fifty times more often than compression speed.

sequenceDiagram
  participant P as Producer
  participant C as Codec
  participant S as Storage or Network
  participant R as Reader
  P->>C: Send raw batch
  C->>C: Find repeats and encode
  C->>S: Write small compressed batch
  S->>R: Deliver compressed batch
  R->>C: Ask to decompress
  C->>R: Return original batch
        

Batch size changes everything

Compression works best on larger chunks because there are more chances to find repeats. Compressing each tiny message alone finds very little to reuse, and the header overhead can even make a tiny message larger. That is why messaging systems compress whole batches, and why zstd dictionaries exist for cases where you must compress small independent records.

A simple set of questions gets you to a good answer for most projects.

flowchart TD
  A["Start: what matters most?"] --> B{"Must every client and tool read it?"}
  B -->|"Yes"| G["Use gzip"]
  B -->|"No"| C{"Is latency or CPU the top concern?"}
  C -->|"Yes"| D{"Need slightly smaller output?"}
  D -->|"No"| L["Use LZ4"]
  D -->|"Yes"| Z1["Use zstd at a low level"]
  C -->|"No"| E{"Is storage or bandwidth cost the top concern?"}
  E -->|"Yes"| Z2["Use zstd at a medium or high level"]
  E -->|"No"| F["Use zstd at its default level"]
        

Rules of thumb

Compression is a tool, and like any tool it can be misused against you.

Side-channel leaks

When secrets and attacker-controlled text are compressed together and then encrypted, the size of the result can leak information. If an attacker can inject guesses into a page that also contains a secret, a correct guess will compress a little smaller than a wrong one. By watching the sizes, they can slowly reveal the secret. Attacks with names like CRIME and BREACH used this idea against encrypted web traffic. The safest habit is to never compress secrets together with untrusted input before encrypting, or to add protections such as randomizing sizes.

Decompression bombs

A tiny, carefully crafted file can expand into gigabytes or more, exhausting memory or disk on the system that opens it. This is called a decompression bomb or zip bomb. Any service that accepts compressed data from outsiders should enforce limits on the expanded size, the expansion ratio, and the time spent decoding.

Integrity and trust

Checksums inside compressed formats help detect accidental damage, but they do not prove the data is authentic. If tampering is a concern, use proper signatures or authenticated encryption in addition to compression.

You cannot tune what you do not measure, and compression needs measurements from both the size side and the speed side.

What to log

Record which codec and level produced each block, in the block header or metadata. Future readers will need this to decode correctly, and future you will need it to understand why one data set is bigger than another. When you change a codec, keep the ability to read the old one until all old data has aged out.

A few habits keep showing up in well-run systems, and a few mistakes keep showing up in troubled ones.

Helpful patterns

Anti-patterns to avoid

These habits will save you from most of the pain teams run into.

The entropy stage is where gzip and zstd earn their extra savings, so it is worth understanding how it thinks.

The idea behind Huffman coding

Normally every character in a text file takes the same amount of space, eight bits. But in English text, the letter E shows up all the time, while the letter Z is rare. Huffman coding notices this and gives the common symbols short codes, maybe only two or three bits, and gives rare symbols longer codes. Since the common symbols appear so often, the total size drops. The codec builds a small table that lists which code belongs to which symbol, and it stores that table right next to the data so the reader can decode it.

Why Huffman has a limit

Huffman codes must use a whole number of bits for every symbol. A symbol that ideally deserves only 1.2 bits still has to take at least 1 bit, or more often 2. That small rounding loss adds up across millions of symbols. It is a bit like a shop that can only sell whole loaves of bread even when you only want a little more than one.

What FSE does differently

Finite State Entropy, the method zstd uses for match information, is built on ideas from Asymmetric Numeral Systems. Instead of giving each symbol its own separate code, it keeps a small running state that carries fractional information forward from one symbol to the next. This lets it get very close to the ideal size, without the rounding loss of Huffman, and it does so using a table lookup that runs quickly on modern CPUs. In practice this means better compression at high speed.

A level is a dial that tells the codec how much effort to spend hunting for patterns.

What a level really changes

At a low level, the codec checks only a few places in its window for earlier copies of the current data, and it takes the first decent match it finds. At a high level, it checks many more places, compares several candidates, and sometimes even looks ahead to see whether skipping a match now gives a better result later. More checking means a smaller output, but each extra bit of savings costs more and more time. The curve is steep at the beginning and flat at the end.

Diminishing returns

With gzip, the jump from level 1 to level 6 gives a real improvement in size, while the jump from level 6 to level 9 gives a small extra gain at a much higher time cost. zstd behaves the same way, only across a wider range. The middle levels are usually the sweet spot. The very top levels of zstd are meant for special jobs, like packaging a software release that will be downloaded millions of times, where spending minutes of compression once is worth it.

Knowing how a codec packages its output helps you avoid confusing errors when systems disagree.

Block compression versus streaming

Some uses compress a fixed block of data in one go, such as a database page or a batch of messages. Others compress a continuous stream, such as a network connection that keeps sending data for hours. Block compression is simple and lets you decompress any block on its own. Streaming carries the codec memory from one piece to the next, which usually improves compression because later data can refer back to earlier data. The price is that you often must read from the start of the stream to decode a later part.

Framing matters

The raw compressed bytes are not enough by themselves. A reader also needs to know where each block starts, how long it is, what codec made it, and whether it arrived intact. This packaging is called framing. Different libraries and tools use different framing for the same codec. Snappy is a well-known case: a raw Snappy block, and the framed format meant for streams, are not the same bytes. LZ4 also has a raw block form and a frame form. If one system writes the raw form and another expects the frame form, decoding fails, even though both say they use the same codec.

flowchart LR
  A["Writer picks codec and form"] --> B["Header describes format"]
  B --> C["Compressed blocks"]
  C --> D["Checksum"]
  D --> E["Reader parses header"]
  E --> F{"Format understood?"}
  F -->|"Yes"| G["Decode and verify"]
  F -->|"No"| H["Decode error"]
        

Zstd adds three special tools that solve problems the older codecs cannot.

Dictionaries for tiny data

A codec learns about your data as it reads. When a record is only a few hundred bytes, the codec finishes before it has learned anything useful. A dictionary solves this by giving the codec a head start. You train it on a few thousand sample records, and it captures the common field names, common values, and common phrases. Both the writer and the reader load the same dictionary. Now even a tiny record can point into the dictionary for its repeated parts. For small JSON events, the size reduction from a dictionary can be large compared with compressing without one.

Long-range matching for huge files

Ordinary windows look back a limited distance. For a large backup or a virtual machine disk image, the same content may repeat hundreds of megabytes apart. Long-range mode lets zstd remember far more history so it can find those distant repeats. It uses more memory, and the reader must also be able to allocate that much, so both sides need to agree on the window size.

Using many cores

Compression of a large input can be split into chunks that run on different CPU cores at the same time. zstd supports this directly, so on a multi-core machine a big compression job can finish several times faster. The chunks are independent enough that the output is still a valid stream any zstd reader can decode. There is a small cost in ratio at chunk boundaries, usually minor.

The right codec depends on the machine it runs on and the money it saves or spends.

CPU and cache behavior

Speed is not only about clever algorithms. Modern CPUs are very fast at simple loops that copy bytes in a predictable pattern and slow when they must jump around memory. That is why LZ4 decompression, which is little more than a tight copy loop, reaches such high speeds. Larger windows and bigger tables, like those used at high zstd levels, do not fit in the fast CPU caches, so speed falls off. Keeping the working memory small can make a codec faster in practice.

Memory per stream

Imagine a server that handles ten thousand connections, and each one keeps its own compression state. If each state needs one megabyte, that is about ten gigabytes of memory just for compression. Lowering the window size or the level can shrink the state dramatically. This is a frequent surprise when a team turns on a stronger setting and then sees memory usage jump.

A simple way to think about cost

Compression saves money in one place and spends it in another. It saves on storage, network transfer, and sometimes reading time. It spends on CPU, and sometimes memory. To decide well, put both sides into the same unit, such as dollars per month, using your real cloud prices. Often the storage savings are large and the CPU cost is small, so a stronger codec wins. In other cases, especially on already busy CPUs, the extra compute is what tips the balance toward a lighter codec.

A careful test takes an hour and can save months of regret.

flowchart TD
  A["Collect real sample"] --> B["Run each codec at chosen levels"]
  B --> C["Record size, speeds, memory"]
  C --> D["Compare against your priorities"]
  D --> E{"Clear winner?"}
  E -->|"Yes"| F["Pilot in staging under load"]
  E -->|"No"| G["Refine sample and levels"]
  G --> B
  F --> H["Roll out readers first, then writers"]
        

Each codec was born to solve a problem its creators were facing, and knowing that story explains its personality.

What the history teaches

Notice the pattern. Each new codec appeared when the surrounding hardware and workloads changed. gzip came when disks and modems were slow and CPUs were modest. Snappy and LZ4 came when servers had many fast cores and the goal was to keep those cores busy with real work. zstd came when engineers realized they could use modern entropy coding tricks to get close to the best of both worlds. Expect the story to continue, because hardware keeps changing and new needs keep appearing.

Two places where most engineers meet these codecs first are message brokers and analytics files, and each has its own habits.

Message brokers

In a broker such as Kafka, a producer collects messages into a batch and compresses the whole batch before sending it. The broker usually stores the batch in its compressed form and hands it to consumers as it is, which means the compression work is done once by the producer and undone once per consumer. This design is efficient, but it has consequences. The larger the batch, the better the compression, but the longer messages may wait for the batch to fill. Settings that control batch size and waiting time therefore affect both compression quality and delivery delay.

Choosing the codec here is a balance. A very light codec keeps producers fast and latency low. A stronger codec reduces network traffic and broker disk usage, which matters when there are many consumers or long retention. Since every consumer must decompress, decompression speed is especially valuable. Also remember that brokers and clients must both support the codec you pick, and older client versions may not know newer codecs.

flowchart LR
  P["Producer collects batch"] --> C["Compress batch once"]
  C --> B["Broker stores compressed batch"]
  B --> K1["Consumer one decompresses"]
  B --> K2["Consumer two decompresses"]
  B --> K3["Consumer three decompresses"]
        

Columnar analytics files

Columnar file formats store all the values of one column together. A column of country names, for example, contains the same few dozen values repeated over and over, which is ideal for compression. These formats also apply lightweight tricks before the general codec runs, such as replacing repeated strings with small numbers or storing only the difference between neighboring values. Then a general codec like Snappy or zstd squeezes the result again. Because of this two-layer approach, the ratio you see is often much higher than you would get by compressing a plain text file with the same codec.

Analytics queries usually read the same file many times, so decompression speed is a big deal. This is why fast codecs have been popular defaults. As zstd matured, many teams moved to it because it produces noticeably smaller files while still decompressing quickly, which reduces both storage cost and the time to read from remote object storage.

The web adds a special rule: the sender and receiver must agree on a codec while they are talking.

How the agreement works

When a browser asks for a page, it includes a header that lists the compression methods it understands. The server looks at this list, picks one it also supports, compresses the response, and adds a header that names the method it used. The browser then decompresses the response before showing the page. Because the browser announces its abilities on every request, servers can safely offer newer methods to modern clients while falling back to gzip for older ones.

Static versus dynamic content

Files that do not change, such as scripts and style sheets, can be compressed once ahead of time at a very high level and then served as they are. The expensive work is paid a single time and every visitor benefits. Content that is generated on the fly, such as personalized pages or API answers, must be compressed for every request, so a faster, lighter setting is usually wiser. Mixing these two strategies is common and works well.

When something goes wrong, the symptom usually points quickly to the cause.

A calm way to investigate

Start by asking three questions. First, did something change recently, such as a codec, a level, a library version, or the shape of the data? Second, is the problem in size, in speed, or in correctness? Third, can you reproduce it with a small sample outside production? Answering these usually narrows the search quickly. Most compression trouble turns out to be a mismatch between two parts of a system rather than a fault in the codec itself.

Walk through these questions in order and the right answer will usually appear on its own.

Walking through realistic situations shows how the ideas from earlier chapters combine into decisions.

Scenario one: an event pipeline with tiny JSON messages

A company collects click events from millions of phones. Each event is a small JSON document of about three hundred bytes, and many share the same field names and similar values. Compressing each event alone would gain little, since there is almost nothing to reuse inside a single tiny document. The team first groups events into batches before sending them, which lets the codec find repeated field names across many events. Then they train a zstd dictionary from a sample of real events, so even small batches compress well. They pick a low zstd level to keep phone battery and server CPU costs modest. The result is far less network traffic, and the change is safe because each batch carries the identifier of the dictionary that produced it.

Scenario two: a cache that must answer in microseconds

A service keeps large serialized objects in memory to avoid slow database calls. Memory is expensive, so the team wants to shrink these objects, but the whole point of the cache is speed. Any codec that slows reads would defeat the purpose. They choose LZ4, because its decompression is so quick that the added delay is tiny compared with the time saved by avoiding the database. They measure the hit latency before and after, and confirm that it barely moves, while the memory footprint drops enough to hold many more objects in the same machines.

Scenario three: a huge archive of old logs

A team must keep several years of logs for auditing, but nobody reads them except during rare investigations. The volume is large and the storage bill grows every month. Here speed of reading barely matters, and compression happens only once in a nightly job with plenty of spare CPU time. The team recompresses day-old logs with a high zstd level using multiple threads. Storage shrinks noticeably, and the extra CPU cost for the nightly job is small compared with the monthly saving. During an audit, they can still decompress a day of logs in a short time, because zstd decompression stays fast even for heavily compressed data.

Scenario four: a public website with heavy scripts

An online shop notices that mobile visitors on slow connections leave before the page finishes loading. The scripts and style sheets are large text files that change only when the team releases a new version. The team compresses these files once during every release at the strongest setting that both gzip and newer browser-supported codecs allow, and the web server hands the small precompressed file to each visitor. Product pages, which are generated for each request, use a lighter setting so the server keeps up during busy sales. Images are left untouched since they are already compressed. Pages load noticeably faster, servers stay calm under load, and older browsers still receive gzip, so nobody is left with a broken page.

Looking at how large systems evolved shows where the industry is heading.

In most measurements zstd gives a better size, better speed, or both, so it is often the better technical choice. But gzip is understood by nearly every system on earth. If the other side cannot read zstd, gzip is the right answer no matter how good zstd looks on paper.

Snappy skips the entropy coding stage and searches for repeats quickly and lightly. It gives up some size to gain a lot of speed. That was a deliberate design choice, not a flaw.

No. Once data has been compressed, the patterns are gone, so a second pass finds almost nothing and may even make the result slightly larger. Choose one good pass instead.

Usually not by much. Higher levels mostly make the compressing side slower and hungrier for memory. Decompression speed stays similar across levels in modern codecs like zstd.

Your data may already be compressed, encrypted, or close to random. Images, videos, archives, and ciphertext have very little pattern left for a codec to find.

A dictionary is a shared block of typical content that both writer and reader know ahead of time. It helps most when compressing many tiny records, where each record alone has too little material to find repeats. It adds the work of training and versioning it, so use it when the savings justify that work.

It depends on the bottleneck. If your system waits on disk or network, spending a little CPU to move fewer bytes usually speeds things up. If CPU is already the limit, a heavy codec can slow you down, and a lighter one like LZ4 is a better fit.

Store the codec name with each block, update all readers to understand the new codec first, then switch the writers. Keep support for the old codec until all old data has expired or been rewritten.

It can be, but order and content matter. Compress first and then encrypt, since encrypted data does not compress. Be careful not to mix secrets with attacker-controlled content in the same compressed block, because size differences can leak information.

If both sides support it, start with zstd at its default level. It is fast, small, and flexible. If you need maximum compatibility, choose gzip. If speed is the only thing that matters, choose LZ4.

Compression trades a little computing effort for a lot of savings in storage and bandwidth. All four codecs in this guide are lossless and built on the same core idea of replacing repeated data with short references. They differ in how much effort they spend. gzip is the universal classic. Snappy and LZ4 are built for raw speed and skip the entropy stage. zstd combines both stages in a modern design and offers one adjustable family that covers almost every need. The best choice always comes from understanding your bottleneck, your data, and how often it is read, and then measuring rather than guessing.