Why Use Feature Flags?
How a simple on/off switch in your software lets teams ship faster, release safer, run experiments, and recover from disasters in seconds instead of hours.
Imagine you run a busy restaurant. You have just invented a brand new dish, and you are very proud of it. But you are also nervous. What if customers hate it? What if the kitchen cannot cook it fast enough on a Saturday night? What if it makes the whole kitchen slow and every other order arrives late?
A careful chef would not put the new dish on the menu for everyone at once. The chef would let a few friendly regulars try it first. If they love it, the chef would offer it to more tables. If something goes wrong, the chef would quietly take it off the menu without closing the restaurant or rebuilding the kitchen.
Software teams face exactly this problem every day, and a feature flag is the tool that gives them the chef’s superpower. A feature flag is a switch inside your software that decides, while the program is running, whether a particular feature is visible and active. In this guide you will learn what feature flags are, why companies like Google, Facebook, Netflix, and Uber rely on them, how they work inside a real production system, where they go wrong, and how to use them well.
No earlier knowledge is needed. Every new term is explained the first time it appears, and every big idea comes with an everyday example.
1What Is a Feature Flag?
Before asking why, we need a clear picture of what. A feature flag is simpler than it sounds.
A feature is any piece of behavior that a user can notice. A new payment button, a redesigned home page, a faster search algorithm, a recommendation list, and a dark mode setting are all features.
A flag is a named value that can be switched between states, most often “on” and “off”. Programmers also call it a toggle, a switch, a feature toggle, or a feature switch. These names all mean nearly the same thing.
A feature flag joins the two ideas. The code for the new feature is already written and already running on your servers, but it sits behind a check that asks a question: “Is this feature turned on for this person right now?” If the answer is yes, the user sees the new behavior. If the answer is no, the user sees the old behavior.
Think of the light switch on your wall. The wiring, the bulb, and the electricity are all installed in advance. Nothing about the room changes until you press the switch. A feature flag is that wall switch for software. The feature is fully built and waiting. The flag decides when the light comes on, and for whom.
The Key Idea: The Decision Moves Out of the Code
Without a flag, the decision to show a feature is made when the programmer writes the code and when the team deploys it. Once it is deployed, everybody gets it. With a flag, the decision is made later, while the software is running, and it can be changed at any time without writing new code or deploying again.
That single shift, moving the decision from “build time” to “run time”, is the reason feature flags are so powerful. Every benefit you will read about in this guide comes from it.
What a Flag Can Know
A flag does not have to be a plain on/off value for the whole world. A good flag system can look at information about the request and decide differently for different people. This information is called the context. Common pieces of context include the user identifier, the country, the type of device, the subscription plan, the app version, and the percentage bucket that the user falls into.
So a flag can say: “Turn this on for all employees, for ten percent of users in Canada, and for nobody else.” That level of control is what separates a feature flag from a simple setting hard-coded in a file.
A flag is not the feature. The flag is only the switch. The feature is the code behind the switch. Keeping those two ideas separate will save you from many confusing conversations later.
2The Problem: Deploying and Releasing Are Welded Together
To understand why flags matter, you first need to feel the pain they remove.
Two words get mixed up all the time in software, and separating them is the heart of this whole topic.
Deployment means putting new code onto the servers where it runs. It is a technical act done by engineers and automated tools.
Release means letting real users actually see and use a new behavior. It is a business act, decided by product managers, designers, marketers, and leaders.
In the traditional world, these two acts are welded into one. The moment the code is deployed, it is released. There is no gap between them. That sounds convenient, but it creates a long list of problems.
Problem One: Big, Scary Releases
Because deploying equals releasing, teams become afraid of deploying. Fear makes them wait. While they wait, more and more changes pile up. When they finally deploy, they are not shipping one small change but hundreds of changes at once. If something breaks, nobody knows which change caused it. Engineers call this a “big bang release”, and it is exactly as stressful as it sounds.
Problem Two: Long-Lived Branches
A branch is a separate copy of the code where a developer works privately. If a feature is not ready to be released, developers often keep it on a private branch for weeks or months so it does not leak out to users. The longer a branch lives, the further it drifts from the main code. When the day comes to merge it back, there are many merge conflicts, which are places where two people changed the same lines in different ways. Sorting them out can take days and often introduces new bugs.
Problem Three: Rollback Means Another Deployment
If a new release has a serious bug, the only remedy in the old world is to deploy the previous version of the code again. This is called a rollback. A rollback has to go through the build system, the tests, the approval steps, and the actual rollout across hundreds or thousands of servers. Even a fast team needs fifteen minutes to an hour. For a large website, an hour of broken checkout can mean millions of dollars lost.
Problem Four: Everyone or No One
With deploy equal to release, a feature reaches all users at the same moment. You cannot test it on a small group first. You cannot show it only to employees. You cannot let a big customer preview it before a contract signing. It is all or nothing.
flowchart LR
A["Write Code"] --> B["Merge to Main"]
B --> C["Deploy to Servers"]
C --> D["All Users See Feature Instantly"]
D --> E{"Is there a bug?"}
E -->|"Yes"| F["Emergency Rollback Deployment"]
F --> G["Wait 15 to 60 Minutes"]
E -->|"No"| H["Celebrate"]
Notice how the diagram has only one path to safety, and that path is slow. Feature flags add a whole second road, which you will see in the next chapters.
Picture a theater that must open the curtain the instant the set is delivered. Stagehands could not finish building, actors could not rehearse on the real stage, and the audience would watch the set being assembled. Feature flags are the curtain. The set can be installed days in advance, tested quietly, and revealed only when the director gives the signal.
3The Core Idea: Separate Deploy from Release
Feature flags cut the weld. Deployment becomes a boring technical event, and release becomes a controlled business decision.
With a flag in place, the new feature code is deployed in the “off” state. Real users do not see it, so deploying is no longer risky. The team can deploy ten times a day, even on a Friday afternoon, because deploying only moves hidden code onto servers.
Later, when the business is ready, someone opens a dashboard and turns the flag on for a small slice of users. They watch the numbers. If everything looks healthy, they widen the slice. If something looks wrong, they turn the flag off, and the old behavior returns immediately. No new deployment is needed.
flowchart LR
A["Write Code Behind a Flag"] --> B["Merge to Main Daily"]
B --> C["Deploy with Flag OFF"]
C --> D["Internal Testing in Production"]
D --> E["Release to 1 Percent"]
E --> F["Release to 25 Percent"]
F --> G["Release to 100 Percent"]
E -->|"Problem found"| H["Flip Flag OFF in Seconds"]
F -->|"Problem found"| H
Who Benefits
Engineers benefit because they can merge small pieces of work often and sleep better after a deploy. Product managers benefit because they pick the launch moment instead of waiting for the next release train. Support teams benefit because they can switch off a troublesome feature during an incident. Executives benefit because the company moves faster with less risk. Customers benefit because they get new things sooner and see fewer outages.
This separation is the single most important sentence in the whole article. If you remember nothing else, remember that feature flags let you ship code without shipping behavior.
4The Big Reasons to Use Feature Flags
Here are the eight benefits that make teams adopt flags and never go back.
Instant Kill Switch
If a feature misbehaves, turn it off in seconds. No emergency deployment, no waiting for a build, no midnight panic.
Small, Frequent Merges
Unfinished work can live safely on the main line of code, hidden behind a flag. Developers stop fighting giant merge conflicts.
Gradual Rollouts
Show a feature to one percent of users, then five, then fifty. Problems are caught while they only affect a handful of people.
Experiments and A/B Tests
Show version A to half the users and version B to the other half, then measure which one performs better with real data.
Targeted Access
Give a feature only to employees, beta testers, a single country, or one important customer, and nobody else.
Launch on Your Schedule
Code can be ready weeks early. Marketing picks the exact minute of the announcement and flips the switch.
Graceful Degradation
Under heavy load, switch off expensive features such as recommendations to keep the core of the product alive.
Safe Big Changes
Moving to a new database or service? Flags let you move traffic slowly, compare results, and go back at any moment.
Each of these deserves a deeper look, and the next chapters give each one its own story. For now, notice a pattern. Every benefit is about one of three things: reducing risk, increasing speed, or increasing learning. Few single techniques in software improve all three at once. That is why feature flags are treated as a foundational practice by high-performing engineering organizations.
5Types of Feature Flags
Not all flags are the same. They live for different lengths of time and serve different people. Mixing them up is a common source of trouble.
Engineers often sort flags into four families. This grouping became popular through writing by Pete Hodgson and by Martin Fowler, and it is a very handy way to think. The two big questions that separate the families are: how long will this flag live? and how often will its value change?
| Flag Type | Main Purpose | Typical Lifespan | Who Controls It |
|---|---|---|---|
| Release flag | Hide unfinished or newly shipped features until ready | Days to a few weeks | Engineers and product managers |
| Experiment flag | Run A/B or multivariate tests to compare versions | Weeks, until the result is clear | Product and data teams |
| Operational flag | Control system behavior, such as kill switches and load shedding | Months to forever | Operations and on-call engineers |
| Permission flag | Give features to certain groups, such as paid plans or beta testers | Months to forever | Product, sales, and support |
Release Flags
These are the most common. A team is building a new checkout page. Instead of waiting until it is perfect, they merge it behind a release flag, deploy it hidden, test it, and then release it. After the feature is fully rolled out and stable, the flag has done its job and should be deleted. A release flag that lives for two years is almost always a mistake.
Experiment Flags
An A/B test is a fair contest between two versions. Half the users see version A and the other half see version B. The team measures something that matters, such as how many people finish buying. An experiment flag handles the splitting. Each user must see the same version every time they come back, otherwise the results would be a mess. This is called sticky assignment, and good flag systems do it by turning the user identifier into a number using a fixed mathematical recipe, so the same user always lands in the same group.
Operational Flags
These control how the system behaves under stress. A kill switch turns a risky feature off in an emergency. A load shedding flag turns off optional features, such as personalized recommendations, when traffic spikes so the vital features survive. Operational flags often live a long time because the need for an emergency brake never goes away.
Permission Flags
These decide who is allowed to use something. A premium analytics screen might be on for customers on the “Enterprise” plan and off for everyone else. A beta feature might be on for a list of volunteer testers. Permission flags can be long-lived, and they start to blur into the idea of entitlements, which are the rights that come with a customer’s plan.
Different flag types need different handling. A short-lived release flag should have an expiry date and an owner. A long-lived operational flag needs documentation and regular testing. Treating them all the same is how teams end up with thousands of mystery switches nobody dares to touch.
6How a Feature Flag System Is Built
A flag is only a name in code, but a production flag system has several moving parts working together.
You can write a tiny flag system in an afternoon by reading a value from a configuration file. But companies running millions of requests per minute need something much stronger. Let us walk through the pieces of a real design.
flowchart LR
A["Engineer or Product Manager"] --> B["Flag Management Dashboard"]
B --> C["Flag Configuration Store"]
B --> J["Audit Log"]
C --> D["Delivery Layer: Streaming and Polling"]
D --> E["Server-side SDK in Backend Services"]
D --> F["Client-side SDK in Web and Mobile Apps"]
E --> G["In-Memory Rule Cache"]
G --> H["Application Code Makes Decision"]
F --> H
H --> I["Analytics and Metrics Pipeline"]
I --> B
Management Dashboard
A website where humans create flags, write targeting rules, and flip switches. It is the control room. It should have login, roles, and approval steps for risky changes.
Configuration Store
A reliable database that holds every flag, its rules, and its history. This is the single source of truth. It must be backed up and highly available.
Delivery Layer
The pipe that carries flag rules from the store to your applications. Some systems use polling, where apps ask for updates every few seconds. Others use streaming, where the platform pushes changes the moment they happen, often through a long-lived connection. Many global platforms put a content delivery network in front, which is a worldwide set of cache servers, so rules reach users quickly everywhere.
SDK (Software Development Kit)
A small library added to your application. It downloads the rules, keeps them in memory, and answers the question “is this flag on for this user?” The SDK does the evaluation locally, which makes it very fast and keeps working even if the flag platform has a temporary outage.
Analytics Pipeline
A stream of records noting which user saw which variation. This feeds experiments and also helps answer “who is still using the old behavior?” before a flag is removed.
Audit Log
A permanent diary of who changed what, when, and why. When an incident happens, the first question is always “what changed?” and the audit log answers it.
Server-Side versus Client-Side Flags
A server-side flag is evaluated inside your own backend servers. The user never sees the rules. This is safer for sensitive logic, because secrets such as pricing rules never leave your data center.
A client-side flag is evaluated on the user’s device, in a web browser or a phone app. The device receives only the final answers for that one user, not the whole rule book, because anything sent to a browser can be read by a curious person. Mobile apps add one more twist: once an app version is installed on a phone, you cannot easily change its code, but you can change its flags. That makes flags especially valuable for mobile, where app store review can take days.
The dashboard is the head office writing the daily menu. The configuration store is the master recipe book. The delivery layer is the delivery truck. The SDK is the copy of the menu pinned up in every branch kitchen, so the cooks never need to phone head office for every order. Even if the phone line breaks, the branch keeps serving from its pinned copy.
7How a Flag Decision Works Inside
Let us follow one request through the system, step by step, to see where the decision is made and why it is fast.
sequenceDiagram
participant U as User
participant LB as Load Balancer
participant API as App Server
participant SDK as Flag SDK
participant C as In-Memory Cache
participant FS as Flag Service
U->>LB: Open checkout page
LB->>API: Forward request
API->>SDK: Is new checkout on for user 42?
SDK->>C: Read cached rules
C-->>SDK: Rules for new checkout
SDK-->>API: Yes, show new checkout
API-->>U: New checkout page
FS-->>SDK: Background stream of rule updates
Step by Step
The App Starts Up
When the server boots, the SDK connects to the flag service and downloads all the rules. It keeps them in memory. If the flag service is unreachable at that moment, a well-built SDK falls back to a saved copy or to safe default values written in the code.
A Request Arrives
A user opens a page. The application builds a small context: the user identifier, country, plan, and device.
Rules Are Checked in Order
The SDK looks at the flag’s rules from top to bottom. Is the flag switched off entirely? Is this user on a “always on” list? Does the user match a targeting rule, such as “country equals Canada”? If none match, the flag falls through to a default percentage rollout.
The Percentage Bucket
For a percentage rollout, the SDK combines the flag name and the user identifier and runs them through a hash function, which is a mathematical recipe that turns any input into a seemingly random number, always the same for the same input. The number is mapped to a bucket between 0 and 99. If the rollout is at 10 percent, buckets 0 to 9 get the feature. Because the recipe never changes, user 42 gets the same answer on every visit, and raising the rollout from 10 to 20 percent keeps the original 10 percent inside the new group.
The Answer Returns in Microseconds
Because all of this happens in memory inside your own process, it usually takes a tiny fraction of a millisecond. There is no network call in the hot path, which is why a flag check adds almost no delay even at massive scale.
Updates Flow in the Background
When someone changes a rule on the dashboard, the platform pushes or serves the new rules. SDKs update their memory copy within seconds. The next request after that sees the new behavior.
Why Local Evaluation Matters
Imagine if every flag check required a network trip to a central server. A single page might check twenty flags, and a busy site serves hundreds of thousands of pages per minute. The central server would become a bottleneck, and its failure would break everything. By downloading rules and evaluating locally, the flag system removes itself from the critical path of user requests. This is a classic design pattern called cache the rules, not the answers.
Always give every flag a safe default value in the code. If the flag system is ever unreachable, the application should fall back to the old, proven behavior, not crash and not accidentally turn on a half-finished feature.
8Reason One: Gradual Rollouts and Canary Releases
The safest way to launch is to start small, watch carefully, and grow slowly.
Early coal miners carried canary birds into mines. If dangerous gas was present, the small bird would show distress first, giving the miners time to escape. A canary release uses the same idea. You expose the new feature to a very small group first. If something is wrong, only that small group is affected, and the team learns about it before the rest of the world does.
Feature flags make this easy because the rollout percentage is just a number on a dashboard. A typical plan has several rings.
flowchart LR
A["Ring 0: Developers"] --> B["Ring 1: All Employees"]
B --> C["Ring 2: 1 Percent of Users"]
C --> D["Ring 3: 10 Percent"]
D --> E["Ring 4: 50 Percent"]
E --> F["Ring 5: 100 Percent"]
C -->|"Error rate rises"| G["Pause or Roll Back"]
D -->|"Latency rises"| G
E -->|"Business metric drops"| G
What to Watch at Each Ring
While the rollout grows, the team compares the group that has the new feature with the group that does not. They look at three kinds of signals. Technical health includes error rates, response time, and server load. User behavior includes clicks, completed purchases, and time spent. Business outcomes include revenue and cancellations. If any signal moves in the wrong direction, they stop or turn the flag off.
Real-World Pattern: Dark Launches at Facebook
Facebook has long described using an internal system for gating features, which lets engineers release to employees first and then expand outward to wider groups of users. When Facebook introduced its chat feature around 2008, the team used a technique called a dark launch, where the heavy backend work ran for real users behind the scenes before any chat window was visible, so the engineers could measure the load on real traffic and fix problems before the public launch.
Real-World Pattern: Early Continuous Deployment at Flickr
Flickr was one of the early well-known teams to talk publicly about deploying many times per day. Part of how they made that safe was by hiding unfinished work behind flags so that new code could be pushed daily without changing what visitors saw until the team chose to reveal it.
Targeting Beyond Percentages
Percentages are only one tool. A team might release to a certain country first because that market is smaller and less risky, or because it has a legal requirement. They might release to users on a slow network to learn how the feature performs under bad conditions. They might release to a loyal group of beta volunteers who have agreed to give feedback. Every one of these is a rule in the flag, not a change in the code.
Wait long enough at each ring to collect meaningful data. One percent of a small user base may produce too few events to prove anything. The right waiting time depends on traffic, and a rollout is not a race.
9Reason Two: The Kill Switch and Faster Recovery
When something breaks at three in the morning, the speed of your fix is measured in how few steps stand between you and “off”.
Every system fails sometimes. The skill is not in preventing every failure but in recovering quickly. Engineers measure this with MTTR, which stands for Mean Time To Recovery: the average time from the moment a problem starts until normal service returns. Lower is better, and feature flags are one of the most effective ways to lower it.
Compare two recovery paths for the same bug.
flowchart TD
A["Alert: Errors Spike After Release"] --> B{"Is the new feature behind a flag?"}
B -->|"No"| C["Find the bad change"]
C --> D["Revert the code"]
D --> E["Run tests and build"]
E --> F["Redeploy to all servers"]
F --> G["Recovered after 30 to 90 minutes"]
B -->|"Yes"| H["Open flag dashboard"]
H --> I["Switch flag OFF"]
I --> J["Rules reach all servers in seconds"]
J --> K["Recovered after 1 to 3 minutes"]
A Famous Lesson: Knight Capital
In August 2012, a trading firm called Knight Capital deployed new software to its servers. The rollout was done by hand to eight servers, and one of them did not receive the new code. That server still contained an old, long-retired feature that had been switched by a flag, and the new release reused that same flag for a different purpose. When the flag was turned on, the one stale server woke up the old trading logic and began sending huge numbers of unwanted orders into the market. In roughly forty-five minutes the firm lost about 440 million dollars, and the company was soon forced to seek rescue financing.
There are several lessons in this story, and they point in both directions. First, a fast, well-rehearsed kill switch and clear monitoring could have limited the damage. Second, the disaster itself was caused partly by poor flag hygiene: an old flag left in the system and then reused for a new meaning. Feature flags are powerful, and power needs discipline. You will see this theme again in the chapters on anti-patterns and best practices.
Never reuse an old flag name for a new meaning, and remove dead code and dead flags together. A forgotten flag is a loaded trap.
Operational Kill Switches in Everyday Use
Turn Off Recommendations
A streaming service sees its recommendation engine slowing down. Operators switch it to a simple popular-titles list so users can still browse and play.
Disable a Failing Partner
An outside payment provider has an outage. A flag reroutes payments to a backup provider while the first one recovers.
Pause Heavy Background Jobs
During a traffic spike on a sale day, a flag pauses report generation so servers focus on customer purchases.
Read-Only Mode
During a risky database repair, a flag turns the site read-only, so users can browse while writes are paused safely.
A kill switch is like the circuit breaker box in your home. If one appliance starts to spark, you do not rebuild the house. You flip one breaker, the sparking stops, and the rest of the house keeps its lights on. You fix the appliance calmly afterward.
10Reason Three: Trunk-Based Development and Continuous Delivery
Feature flags are the quiet partner that makes fast, frequent shipping possible.
Earlier you met long-lived branches and their merge pain. The modern alternative is called trunk-based development. The trunk is the main line of the code. In this style, every developer merges small changes into the trunk at least once a day, and there are very few private branches. Google, for example, has publicly described keeping the vast majority of its code in one shared repository with developers working close to the main line.
The obvious worry is: “What if my feature is only half-finished? I cannot let users see half a feature.” Feature flags are the answer. The half-finished code is merged, but it sits behind a flag that is off, so it is invisible and harmless. The next day, the developer merges the next small slice. After ten days, the feature is complete, and the flag is turned on.
flowchart LR
subgraph OLD["Long-Lived Branch Style"]
A1["Main Line"] --> A2["Branch for 6 Weeks"]
A2 --> A3["Painful Merge"]
A3 --> A4["Big Release"]
end
subgraph NEW["Trunk and Flag Style"]
B1["Small Merge Day 1"] --> B2["Small Merge Day 2"]
B2 --> B3["Small Merge Day 3"]
B3 --> B4["Flag ON When Ready"]
end
Connecting to CI and CD
Continuous Integration, often shortened to CI, means that every change is automatically built and tested as soon as it is merged. Continuous Delivery, shortened to CD, means the software is always in a state that could be released at any time. These practices only work well if the trunk is always healthy and always shippable. Feature flags protect that health, because unfinished work never reaches users.
Researchers who study software teams, in the long-running reports often called the DORA research programme, have repeatedly found that teams who deploy often and recover quickly tend to do better on both technical and business results. Practices such as trunk-based development and feature flags are commonly listed among the habits that support those outcomes. The exact numbers change from year to year, so treat the general direction as the lesson.
Slice the Feature Small
Break the work into pieces that can each be merged on their own, even if each piece does nothing visible yet.
Wrap the Entry Point in a Flag
Place one flag check at the place where users would first reach the new behavior, such as a menu item or a route.
Merge Daily
Each slice goes into the trunk and passes through automated tests and deployment, still hidden.
Test in Production Safely
Turn the flag on for the team only. They can try the real feature on real infrastructure, which often reveals problems that a test environment never shows.
Release and Clean Up
Roll out gradually, then delete the flag and the old code path.
When merges are small, reviews are quick, bugs are easy to trace, and developers rarely get blocked waiting for each other. Flags are the safety net that lets a team work this way without fear.
11Reason Four: Experiments and Learning from Real Users
Opinions are cheap. Data from real users is better. Flags are the engine behind most online experiments.
Inside every product team there are debates. Should the button be green or blue? Should the price be shown with the tax included? Should the home page show a big picture or a list? Without evidence, the loudest person in the meeting usually wins. With experiments, the users decide.
An experiment flag has more than two states. It can have variations, such as “control” (the current design) and “treatment” (the new design), and sometimes more. The flag system splits users fairly, remembers which variation each person got, and records the exposure so analysts can later compare results.
How a Good Experiment Runs
State a Hypothesis
Write a clear guess, such as “A shorter sign-up form will raise the number of people who finish signing up.”
Choose the Measurement
Pick one main metric before starting, plus a few guardrail metrics, such as page speed and refund rate, that must not get worse.
Split and Run
The flag assigns users randomly and consistently. The test runs long enough to collect enough data and to cover normal weekly ups and downs.
Analyze with Care
Check whether the difference is large enough to be real and not just luck. Statisticians call this statistical significance. Peeking at results too early and stopping at a lucky moment is a classic mistake.
Decide and Clean Up
Roll the winner out to everyone, delete the losing code, and remove the flag.
Real-World Pattern: Experimentation at Netflix and Booking.com
Netflix has publicly written about running large numbers of controlled experiments to decide how its product looks and behaves, including how artwork and rows are presented to members. Booking.com has also spoken widely about running a very large number of experiments at the same time, and about the cultural habit of letting data settle product debates. These companies depend on the same basic ingredients: stable user assignment, careful measurement, and the ability to turn variations on and off without a new deployment.
Running two experiments that touch the same screen can make both results unreliable, because the effects mix together. Keep a calendar of experiments, and plan which tests can safely overlap.
A Gentle Form of Experiment: Dark Launch and Shadow Traffic
Sometimes the goal is not to learn what users like but to learn whether a new technical component can handle the load. In a dark launch, the new code runs for real requests, but its results are quietly thrown away and the user still gets the old answer. In shadow traffic testing, a copy of real requests is sent to the new system so engineers can compare its answers with the old system’s answers. Flags control how much traffic is copied and let the team stop the copying instantly if it causes trouble.
12Reason Five: Safe Migrations and Big Changes
The scariest work in software is not adding new features. It is replacing the parts that already work.
Sooner or later every company must move something big: a database, a payment provider, a search engine, a programming language, or an entire service. A mistake here can lose data or stop the business. Feature flags turn a frightening one-night cutover into a calm, reversible journey.
The Classic Migration Dance
flowchart LR
A["Phase 1: Old System Only"] --> B["Phase 2: Write to Old and New"]
B --> C["Phase 3: Compare Results Quietly"]
C --> D["Phase 4: Read from New for 5 Percent"]
D --> E["Phase 5: Read from New for 100 Percent"]
E --> F["Phase 6: Stop Writing to Old"]
F --> G["Phase 7: Retire Old System"]
D -->|"Mismatch found"| C
E -->|"Problem found"| D
Dual Writes
A flag turns on writing every new piece of data to both the old and the new system. The old system remains the source of truth.
Shadow Reads and Comparison
The app reads from both systems and compares the answers. Users still receive the old answer. Any difference is logged and studied. This pattern is sometimes called a shadow read or an experiment-style comparison, and GitHub has written about a tool of this kind, named Scientist, used when rewriting critical code paths.
Gradual Read Cutover
A percentage flag slowly moves real reads to the new system, from a tiny slice to everyone.
Retire the Old Path
Only after weeks of calm do the teams stop the dual writes, remove the flags, and decommission the old system.
Real-World Pattern: Moving Traffic Between Services
Large companies that moved from a single big application to many smaller services, a style known as microservices, have often described doing it one slice at a time. A flag decides whether a request is handled by the old big application or the new small service. If the new service misbehaves, the flag sends traffic back. This approach is often called the strangler pattern, because the new system slowly grows around the old one until the old one can be removed.
Think of rebuilding a bridge while cars keep crossing. You do not close the old bridge and hope the new one is ready. You build the new lanes beside it, let a few cars try them, then more, and only when everything is smooth do you take down the old structure. Flags are the traffic cones that guide cars from the old lanes to the new ones.
13The Life of a Flag: From Birth to Deletion
A flag is a temporary tool unless there is a clear reason for it to stay. Good teams manage the whole life of every flag.
Most beginners think about creating flags and forget about removing them. But the removal step is where the discipline lies. A flag is like a scaffold on a building. It is essential during construction, and it must come down afterward, otherwise it blocks the view and becomes dangerous.
flowchart TD
A["Plan: Name, Owner, Type, Expiry Date"] --> B["Create Flag in Platform"]
B --> C["Add Flag Check in Code"]
C --> D["Deploy with Flag OFF"]
D --> E["Test with Internal Users"]
E --> F["Gradual Rollout"]
F --> G{"Stable at 100 Percent?"}
G -->|"No"| H["Fix or Roll Back"]
H --> F
G -->|"Yes"| I["Make ON Permanent"]
I --> J["Delete Old Code Path"]
J --> K["Delete Flag from Platform"]
What to Write Down When a Flag Is Born
A Clear Name
Describe the feature, not the mood. Use a consistent pattern so anyone can guess what the flag does by reading its name.
An Owner
One person or one team is responsible. When there is no owner, there is no cleanup.
A Type and Purpose
Release, experiment, operational, or permission. This tells everyone how long to expect it to live.
An Expiry or Review Date
Set a reminder to remove the flag. Many teams automatically open a ticket when a release flag passes its date.
Create the cleanup ticket on the same day you create the flag. If removing the flag is a planned task from the very beginning, it will not be forgotten when the exciting launch is over.
14Advantages, Disadvantages, and Trade-offs
No tool is free. A fair look at feature flags shows both the gifts and the costs.
Advantages
- Deploying becomes safe and boring, so teams ship more often.
- Problems can be undone in seconds with a kill switch.
- Rollouts can be gradual, targeted, and measured.
- Small merges reduce conflicts and long-lived branches.
- Product and business teams control launch timing.
- Experiments turn debates into data.
- Big migrations become reversible, step-by-step journeys.
- Beta programs and plan-based access are easy to offer.
Disadvantages
- Code gets more complicated, with extra branches to read.
- Testing grows harder because there are more combinations.
- Old flags pile up as technical debt if nobody cleans them.
- A flag system is one more piece of infrastructure to run and secure.
- A wrong flag change can cause an outage as easily as it can prevent one.
- Behavior can differ between users, which makes bugs harder to reproduce.
- Teams may use flags as a crutch to ship half-baked work.
The Testing Explosion
This is the cost people feel first. If your application has one flag, there are two possible worlds to test: flag on and flag off. With two flags there are four worlds. With ten flags there are 1,024 worlds. With twenty flags there are over one million. You cannot test every combination, and you do not need to, but you must be thoughtful.
Practical teams manage this in several ways. They keep flags independent of each other wherever possible, so a combination never matters. They test the two most important states for every flag, which are the old default and the new target. They run a small number of full tests in the exact combination that will be live in production. They keep the number of live flags low by deleting old ones promptly. And they avoid flags that depend on other flags, since that creates chains that grow quickly.
Flags trade a little code complexity today for a lot of safety and speed. The trade is excellent when flags are few, short-lived, and well-owned. It turns bad when flags multiply and never go away.
The Failure Risk Inside the Flag System Itself
Because a flag can change the behavior of every server in seconds, the flag platform is a very powerful lever. A mistaken change, such as turning on a half-built feature for everyone or switching off a vital safety check, can cause an outage quickly. This is why mature teams treat flag changes like code changes: they require reviews and approvals for risky flags, they record every change in the audit log, and they roll changes out gradually even for emergency switches where time allows.
15Anti-Patterns: How Feature Flags Go Wrong
An anti-pattern is a habit that looks helpful but causes harm over time. These are the ones that hurt feature flag users most often.
What It Looks Like
The company has hundreds or thousands of flags. Nobody knows which are still used. Some have been on at one hundred percent for years.
Why It Hurts
Every dead flag leaves a dead code path behind. New engineers read both paths and wonder which one matters. The risk of an accident, like the Knight Capital story, rises with every forgotten switch.
The Fix
Give each release flag an owner and an expiry date. Run a regular cleanup. Add automated reports that list flags unchanged for a long time.
What It Looks Like
One flag check sits inside another, which sits inside a third. The behavior depends on a chain of switches.
Why It Hurts
The number of possible states multiplies. Nobody can say with confidence what a user will see. Bugs appear only for odd combinations.
The Fix
Keep each flag independent. If two features truly depend on each other, put them under one flag, or document the dependency in the dashboard and test it explicitly.
What It Looks Like
The same flag is checked in thirty different places across the code base.
Why It Hurts
It is easy to miss one place and leave the feature half-switched. Removing the flag later becomes a big, risky hunt.
The Fix
Check the flag at one decision point, such as the entry to a feature, and let the code below that point choose the right implementation. Good teams hide the choice behind a clear interface, so the rest of the code does not know a flag exists.
What It Looks Like
An old, retired flag name is recycled for a new feature to save time.
Why It Hurts
Any server or client that still holds the old code reads the new setting and behaves in a dangerous, unexpected way. This is the core of the Knight Capital disaster.
The Fix
Every flag name is unique forever. Never reuse a name, even after deletion.
What It Looks Like
Release flags are left in place and slowly become customer-specific configuration, so the code is full of special cases.
Why It Hurts
Release flags are designed for short lives. Used as permanent configuration, they hide real product decisions and make the system hard to reason about.
The Fix
If a behavior should really differ by customer for a long time, turn it into a proper, documented product setting or plan entitlement with its own tests.
What It Looks Like
The code assumes the flag service will always answer. When it does not, the page crashes or the feature turns on by accident.
Why It Hurts
The flag platform becomes a single point of failure for the whole business.
The Fix
Always code a default that matches the old, proven behavior, and make the SDK keep a local copy of the last known rules.
16Best Practices and Common Mistakes
These habits separate teams who love their flags from teams who dread them.
Best Practices
Name Flags Clearly and Consistently
Use a pattern that includes the team, the feature, and the type. A good name tells a stranger what happens when the flag is on.
Give Every Flag an Owner and an End Date
Release flags and experiment flags should be treated as loans that must be repaid by a certain date.
Keep the Flag Check Small and Central
Put the decision at one place, and keep the on and off paths easy to compare and easy to delete.
Choose Safe Defaults
When anything fails, the system should land on the old, proven behavior.
Test Both Paths
Automated tests should cover the flag on and the flag off. When the old path is later removed, remove its tests too.
Protect Risky Flags with Approvals and Roles
Not everyone should be able to flip every switch in production. Use roles, approvals for sensitive flags, and an audit trail.
Roll Out Gradually and Watch the Dashboards
Tie each rollout ring to specific health measurements, and agree beforehand what result means “stop”.
Practice the Kill Switch
Rehearse emergency flips during calm times, like fire drills. A switch that has never been tested may fail when it matters most.
Clean Up as Part of Done
A feature is not finished when it launches. It is finished when its flag and old code path are gone.
Common Mistakes
Flipping on a Friday Evening with No Plan
Just because a flag is easy to change does not mean every change is safe. Plan rollouts for times when people are available to watch.
Testing Only the New Path
When the flag is turned off in an emergency, the old path must still work. Teams that stop testing it find it broken at the worst moment.
Changing Data Shapes Under a Flag
If the new feature writes data in a new format, switching the flag off may not undo that. Plan how old code will read new data.
Using User Lists Instead of Rules
Long hand-typed lists of user identifiers are hard to maintain. Prefer rules based on attributes such as plan, country, or group.
Forgetting Mobile Versions
Old app versions stay in people’s pockets for months. A flag removed too early may break users who never updated.
Sending Secrets to the Browser
A client-side flag payload can be read by anyone. Never put private rules or confidential names in it.
17Security, Performance, and Monitoring
A flag platform controls the behavior of your whole product, so it deserves the same care as any critical system.
Security
Access control decides who may change flags. Production flags that affect payments, login, or data access should need approval from a second person. Sign-in to the dashboard should use strong authentication, such as multi-factor sign-in.
Data privacy matters because the context sent to a flag system can include user details. Send the smallest amount of information needed. Often a random, non-identifying user key is enough. If you use an outside flag vendor, be clear about what personal data leaves your systems, and consider privacy laws such as the European GDPR.
Separation of environments keeps mistakes contained. Development, testing, and production should each have their own flag settings, so that a change made while experimenting never touches real customers.
Client exposure needs attention. As noted earlier, anything delivered to a browser can be inspected. A competitor could read the names of your unreleased features if you send all flags to the client. Send only the flags and answers that the client truly needs.
Performance
Because evaluation happens locally in memory, the cost of a single flag check is very small. The real performance concerns lie elsewhere. The first is the startup download, where a large rule set can slow a cold start, so keep the payload lean. The second is too many flag checks in a tight loop, such as inside code that runs a million times per second; evaluate once and reuse the result. The third is update fan-out: when a rule changes, thousands of servers fetch the update at once, so systems use streaming, caching, or a content delivery network to absorb the burst. Mobile apps should also avoid blocking the first screen while waiting for flags, and should show a sensible default instead.
Monitoring and Observability
Observability means being able to understand what a system is doing by looking at the signals it emits: logs, metrics, and traces. With flags, observability has an extra job: connecting every behavior change to the flag state that caused it.
flowchart LR
A["Flag Change Event"] --> B["Audit Log"]
A --> C["Annotation on Monitoring Dashboards"]
D["Application Metrics: Errors and Latency"] --> E["Monitoring System"]
C --> E
E --> F{"Metrics Worsen After Change?"}
F -->|"Yes"| G["Alert On-Call Engineer"]
G --> H["Flip Flag OFF"]
F -->|"No"| I["Continue Rollout"]
Annotate Dashboards
Draw a vertical line on every graph at the moment a flag changed. Engineers can see cause and effect at a glance.
Tag Logs with Flag State
Include the active variation in log lines and traces, so a bug report can be matched to the exact state the user saw.
Alert on Automated Guardrails
Some platforms can automatically pause or reverse a rollout when error rates cross a limit, so humans do not have to be the only safety net.
Track Flag Inventory
Keep a live count of total flags, flags past expiry, and flags without owners. A simple chart of this count shows whether debt is growing.
18Build, Buy, or Use Open Source?
Once a team decides to use flags, the next question is where the flag system will come from.
There are three broad routes, and each suits a different stage of a company.
| Route | Good For | Strengths | Watch Out For |
|---|---|---|---|
| Build your own | Very small needs, or very large companies with special demands | Full control, no vendor cost, fits internal tools perfectly | You must build dashboards, audit logs, targeting, experiments, and scale. The work is much larger than it first appears. |
| Open source | Teams that want control and lower license cost | Transparent code, self-hosting, active communities; examples include Unleash, Flagsmith, and the OpenFeature standard for common code interfaces | You run and secure the servers yourself, and advanced features may need extra work |
| Commercial service | Teams that want speed and polish | Managed reliability, rich targeting, experimentation, approvals, and support; examples include LaunchDarkly and similar vendors | Ongoing cost, data sharing with a vendor, and the risk of tight dependence on one vendor |
| Cloud provider feature | Teams already deep in one cloud | Fits existing accounts, permissions, and billing; AWS offers feature flags through AWS AppConfig | Features may be simpler than specialist products, and portability to other clouds can be harder |
The Value of a Standard Interface
Because flag vendors each have their own library, switching later can be painful. An open standard called OpenFeature defines a common way for application code to ask for flag values, regardless of which backend is used. Adopting such a layer means your application code talks to a neutral interface, and the real provider can be swapped with limited changes. This is similar in spirit to how a standard electric plug lets you change appliances without rewiring the house.
A Gentle Starting Path
Start Simple
Begin with a small set of release flags and one kill switch for your riskiest feature. A plain configuration service may be enough for the first few weeks.
Write Your Rules of the Road
Decide naming, ownership, expiry, and approval rules before the number of flags grows.
Adopt a Real Platform When You Feel the Pain
When you need percentage rollouts, targeting, audit logs, or experiments, choose a mature tool instead of stretching a home-made one.
Measure and Improve
Track how often you deploy, how quickly you recover, and how many stale flags you have. Improve the practice every quarter.
19Frequently Asked Questions
Short answers to the questions beginners ask most often.
20Summary and Key Takeaways
Key Takeaways
- Feature flags separate deployment from release. Code can reach production hidden, and be revealed later by a deliberate decision.
- They shrink risk. Gradual rollouts, canary groups, and instant kill switches limit how many people any bug can hurt and how long it lasts.
- They speed up delivery. Unfinished work lives safely on the main line of code, which means smaller merges and fewer painful branches.
- They power learning. Experiments, dark launches, and shadow traffic turn opinions into evidence from real users.
- They make big migrations reversible. Moving databases, services, and providers becomes a slow, safe walk instead of a leap.
- They need discipline. Give every flag an owner, a type, and an end date, and never reuse names. Flag debt and nested flags are the main dangers, and Knight Capital remains the warning story.
- Design the system for failure. Evaluate locally, keep safe defaults, protect access with roles and audit logs, and watch your dashboards whenever a flag changes.