Three small ideas come first: the feature flag, the value it carries, and the word “variate”.

The feature flag

A feature flag (also called a feature toggle) is a switch inside software that decides whether a behaviour is active. The code is already deployed. The flag decides who gets to use it. Changing the flag changes behaviour without a new release.

The value

Many flags carry more than on or off. They can carry a value, such as a piece of text, a number or a small bundle of settings. Flagsmith calls this a remote configuration value. For example, a flag called “home_banner_text” might carry the sentence “Welcome back!”.

The multivariate idea

The word variate simply means “a version that can vary”. A multivariate feature is a feature that has several possible values at once, each called a variation, together with a rule that says what share of people receives each one. One feature, many versions, one fair way of dividing people between them.

A tiny worked example

Suppose an online shop has a button on its checkout page. Today the button says “Buy now”. The marketing team suspects other wordings might persuade more visitors to finish. They create a multivariate feature named checkout_button_label. The control value is “Buy now”. They add two variations, “Add to basket” and “Get it today”. They give each variation twenty-five percent. Half of the visitors keep seeing “Buy now”, a quarter see “Add to basket” and a quarter see “Get it today”.

Notice what the developers did not have to do. They did not build a lottery. They did not write code that remembers which visitor received which wording. They did not create three separate switches. They wrote one line that asks for the value of one feature, and Flagsmith handled the sharing, the remembering and the fairness.

Why the word “feature” and not “flag”

Flagsmith uses the word feature for the thing you manage, because one feature can have an on or off state and also a value. A multivariate feature is simply a feature that has been given extra versions of its value. In everyday speech people still say “flag”, and this article uses both words for the same idea.

An on or off switch answers “should this exist?”. Multivariate answers “which version should this be?”.

Questions that need several versions

What happens without it

A team without multivariate support has to invent its own sharing logic. Engineers create three boolean flags, write code that picks one, and then argue about how to keep each person in the same group. It is slow, hard to read, and easy to get wrong. Multivariate moves that logic into Flagsmith, where it is visible, adjustable from a dashboard, and consistent across web, mobile and servers.

What large companies teach us

Companies known for large online experiments, such as Netflix, Google, Booking.com and Microsoft, have written publicly about running many experiments at once and about needing each person to stay in the same group throughout. Their lesson is simple: measurements are only trustworthy if assignment is steady. Flagsmith is not claimed to be used by any of these companies. They are mentioned only to show why steady assignment matters.

The cost of guessing

Without testing, teams decide by opinion. The loudest voice in the meeting, or the most senior person, picks the wording or the layout. Opinions are often wrong in surprising ways. Teams that measure frequently discover that the version they disliked performs better with customers. Multivariate features turn the argument into a measurement: show each version to similar people and see what they do.

The cost of all-at-once changes

Releasing a change to everyone at once is risky. If the change contains a bug or confuses people, everyone suffers at the same moment. A multivariate feature lets the team release the change to a small share, watch what happens and widen gradually. The same tool that tests wording also protects the whole customer base from a bad release.

The cost of private tooling

Some companies build their own experiment systems. This can work, but the system needs steady care: storing assignments, handling many devices, offering a screen for non-engineers and keeping an audit trail. A shared product such as Flagsmith moves those chores to a place that the whole team can see and use, so product managers can change weights without waiting for an engineer to release code.

Five small words appear everywhere on the multivariate screen. Learn them once and the dashboard becomes easy to read.

The control value

The control value is what the feature normally says. In experiments, it is the baseline you compare against. In Flagsmith it is the value held by the feature itself, and anyone who is not allotted a variation receives it. Thinking of it as “what everyone would see if no experiment were running” keeps the idea clear.

Variations and weights

Each variation carries its own value and its own weight, a percentage. If one variation has weight thirty and another has weight thirty, then sixty percent of people receive a variation and the remaining forty percent receive the control value. Weights add up to at most one hundred percent. Check the current documentation for how your version displays and validates them.

pie title Sharing 100 percent of people
  "Control: Buy now" : 40
  "Variation A: Add to basket" : 30
  "Variation B: Get it today" : 30
    

Reading the weights like a recipe

A weight plan can be read aloud as a sentence: “Out of every hundred people, thirty see the first variation, thirty see the second, and forty see the control.” If you cannot say the plan as a clear sentence, the plan is probably too complicated. Teams find it useful to write that sentence in the feature description so that anyone who opens the feature understands the experiment in ten seconds.

Rounding and small audiences

Weights are shares of a large population. With a hundred thousand visitors, a thirty percent weight gives very close to thirty thousand people. With only forty visitors, the real split may be eleven, fourteen and fifteen instead of twelve, twelve and sixteen. This is ordinary chance, not a fault. It is another reason experiments need enough people before conclusions are drawn.

Flagsmith is organised as a tree. Seeing where multivariate settings live explains why they behave the way they do.

At the top is the Organisation. Below are Projects, one per product. Beneath each Project are Environments, such as Development, Staging and Production. A feature is defined once in the Project, and its state lives in each Environment. For multivariate features, a very useful split follows: the list of variations and their values belongs to the feature in the Project, while the weights are chosen separately for each Environment.

graph TD
  ORG["Organisation"] --> PRJ["Project: Store"]
  PRJ --> FEAT["Feature: checkout_button_label, control = Buy now"]
  FEAT --> V1["Variation A: Add to basket"]
  FEAT --> V2["Variation B: Get it today"]
  PRJ --> DEV["Environment: Development"]
  PRJ --> STG["Environment: Staging"]
  PRJ --> PRD["Environment: Production"]
  DEV --> W1["Weights: A 50, B 50"]
  STG --> W2["Weights: A 20, B 20"]
  PRD --> W3["Weights: A 0, B 0, all control"]
    

Why not define variations per Environment?

If every Environment had its own private list of variations, Staging could drift away from Production, and a test that passed in Staging might not match what Production would serve. By keeping the list of variations with the feature, Flagsmith makes sure that every Environment knows the same set of versions. Only the sharing plan, which is naturally different in Development, Staging and Production, is chosen locally.

Promotion across Environments

A healthy path is to create the feature in Development, try every variation with test identities, copy the weights to Staging with a modest audience, and finally set Production. Because the variations already exist everywhere, promotion is only a matter of typing different weights. Some teams record each step in a checklist so that nobody skips Staging in a hurry.

Flagsmith offers several ways to give different people different things. Knowing which tool fits which question avoids over-engineering.

The key difference

A segment is about who you are. You are in the premium group because of a fact about you. Multivariate is about chance with memory. You receive variation A because your steady bucket number happened to fall in its slice, and it will fall in the same slice tomorrow. These two ideas can be combined: a segment can restrict the experiment to one country, and the multivariate weights then share people inside that country.

Combining the tools in one product

Real products use all of these tools together. A plain flag acts as the kill switch for the whole checkout page. A segment gives staff the new design first. An identity override pins one troubled customer to the old design. A multivariate feature shares everyone else between three button wordings. Each tool answers a different question, and none of them replaces the others. Learning which one fits which question is the main skill of a good flag designer.

The dashboard makes the process friendly. Here is the whole journey in plain steps, with the reason for each one.

Naming that scales

A worked walk-through

Imagine you are the developer at PageTurn, the fictional bookstore. In Development you create checkout_button_label and type “Buy now” as the control. You add two variations and give them readable names. You set both weights to fifty percent just in Development, so with a handful of test identities you are almost certain to see both versions quickly. You write the small piece of code that maps each label to a displayed text, and you check that an unknown label shows “Buy now”.

Next, in Staging, you reduce the weights to twenty-five percent each and you create three test identities in different segments. You confirm that the staff segment shows the new wording and that a customer from the strategic segment always sees control. You also switch off network access for a moment in a test device and watch the page fall back to control without any error.

Only after those checks do you open Production, where you begin with five percent for each variation. You write the owner, the ticket number and a planned end date in the feature description. You add a note in the team channel with the start time. Then you watch the exposure counts and your guardrail numbers for the first hour.

A variation is just a value. What kinds of values make useful variations?

Typical value kinds

Prefer names over raw details

A very useful habit is to make the variation value a name that the code understands, such as “layout_grid”, rather than a long block of settings. The code then decides what “layout_grid” means. This keeps changes in code review, keeps flags small, and lets you rename the effect without touching the live configuration.

Keep the shape consistent

Every variation of a feature should have the same kind of value. If the control is a short label, each variation should be a short label too. Mixing a number in one variation and a sentence in another forces the code to guess what it received, which creates bugs on the exact path where you are measuring.

Reviewing values with designers

If a variation holds text that customers will read, ask a writer or designer to review it before it goes live. A typing error in a variation is seen by a real share of customers, and it can also spoil an experiment, because the wording problem is mixed with the idea being tested. A short review step costs minutes and protects the whole test.

This is the heart of the feature. Behind the friendly dashboard is a simple, clever trick that makes the choice both random-looking and perfectly repeatable.

The goal

Flagsmith must pick a variation for each person so that the shares match the weights, and so that the same person always gets the same variation, on every device and every visit, unless the weights change. Pure random picking would break the second promise, because a coin flip gives a different answer each time.

The trick: a stable number from a hash

A hash function is a recipe that turns any text into a long scrambled number. The same text always gives the same number, but similar-looking texts give very different numbers. Flagsmith combines the person’s identifier with information about the feature, runs the recipe, and reduces the result to a number between zero and one hundred. This number is the bucket. Flagsmith then walks along the variations, each owning a range of buckets according to its weight, and the person receives the variation whose range contains their bucket. Check the current documentation for the exact recipe, because internal details can change between versions.

flowchart LR
  A["Identifier: user_48213"] --> B["Mix with the feature"]
  B --> C["Hash recipe"]
  C --> D["Bucket number, e.g. 37"]
  D --> E{"Which range holds 37?"}
  E -- "0 to 29: Variation A" --> F["Variation A"]
  E -- "30 to 59: Variation B" --> G["Variation B"]
  E -- "60 to 99: Control" --> H["Control value"]
    

Why mixing in the feature matters

If the bucket came only from the person, then the same people would land in the “first thirty percent” of every experiment, and some customers would always be the guinea pigs. By mixing in the feature, each feature reshuffles people differently. A person in the first slice of one experiment may be in the last slice of the next. This spreads the experience fairly across the customer base.

What the hash does not do

The hash does not look at who the person is, what they bought or where they live. It treats the identifier as an anonymous piece of text. This is a useful property: the sharing is fair with respect to everything else about the person, so the groups tend to look alike on average. That is exactly what an experiment needs, because differences in outcome can then be linked to the version rather than to the kind of people in each group.

Two people with similar identifiers

Even identifiers that look almost the same, such as user_1001 and user_1002, produce very different bucket numbers. The hash scrambles them thoroughly. So consecutive customers do not cluster in the same group, and early sign-ups do not all end up in one variation.

Same person, two features

Because the feature is mixed into the hash, one person lands in different places for different features. This avoids a hidden problem: if a person were always in the “first thirty percent” of every experiment, they would be the first to see every untested idea. Mixing the feature spreads that burden.

The hash needs something steady to work from. That something is the identity.

An identity in one paragraph

An identity is Flagsmith’s name for one end user, account or device that your application tells it about, named by an identifier, usually an internal user ID. When the application asks for flags for an identity, Flagsmith can personalise the answer. When it asks without one, Flagsmith has nothing steady to hash.

What happens without an identity

If the application asks for flags without supplying an identifier, Flagsmith cannot place the person in a bucket. In that case the person typically receives the control value of the feature, not a variation. The multivariate mechanism needs a steady name to work. Check the current documentation for the exact behaviour of each SDK.

Choosing the identifier for experiments

Account-level experiments

In business products, everyone in one company often needs to see the same version, otherwise colleagues sitting side by side would see different screens and call support. In that case use the account ID as the identifier for the experiment. All people in a company then share one bucket and one variation. The trade-off is that you have fewer units to compare, so the test needs more companies or more time.

Steadiness is what turns a random split into a trustworthy experiment. Understanding exactly when people stay put, and when they move, protects your results.

Staying put

As long as the identifier, the feature and the weights stay the same, a person’s bucket and variation never change. They keep seeing the same version on their phone, laptop and tablet, if all three use the same identifier.

When people move

People can move between variations when weights change, because the ranges of buckets are redrawn. Imagine ranges 0 to 29 for A, 30 to 59 for B and the rest control. If you raise A to forty percent, the range for A becomes 0 to 39, and B’s range may shift. People who were at the edges can change variation. The shift is small, but it exists.

stateDiagram-v2
  [*] --> Planned: Variations and weights chosen
  Planned --> Running: Feature on, people bucketed
  Running --> Running: Same people, same versions
  Running --> Adjusted: Weights changed
  Adjusted --> Running: Some people moved, measure from here
  Running --> Stopped: Weights set to zero or feature off
  Stopped --> [*]
    

A story of a weight change

A team starts with variation A at ten percent, variation B at ten percent and control at eighty percent. After a week the numbers look safe, so they raise A to twenty-five percent and B to twenty-five percent. Everyone whose bucket lay in the lowest ten percent for A still sees A. New people with buckets in the next fifteen percent now also see A. Some people who used to see control now see a variation, because they moved into a variation’s widened range. Some people who saw B may now see A, because the ranges are laid end to end and shifted when the first range grew.

This is why teams prefer to grow an experiment in one clear step, record the date, and later analyse the period before and after the change separately. People who stayed in the same version across both periods are the cleanest group to study.

Stopping an experiment cleanly

The cleanest stop is to set all variation weights to zero. Everybody then receives the control value at once, and no code needs to be released. If the winner is a variation, the best next step is to make its value the new control value in the same moment, so that people never see a flicker of the old version.

Several rules can apply to one person at once. Flagsmith resolves the conflict with a fixed order, and multivariate sits inside it.

Three layers

The identity override is a special answer for one person. The segment override is a special answer for a named group. The Environment state is the general answer, and for a multivariate feature this is where the weighted split lives. The most specific layer that applies wins.

flowchart TD
  A["Request: feature X for identity I"] --> B{"Identity override for X?"}
  B -- Yes --> C["Use that exact value"]
  B -- No --> D{"In a segment that overrides X?"}
  D -- Yes --> E["Use the segment's value"]
  D -- No --> F["Compute bucket and pick variation by weight"]
  F --> G["Variation or control value"]
  C --> H["Return to application"]
  E --> H
  G --> H
    

Why this ordering is useful

Reading the effective value

When you open an identity in the dashboard, it shows the effective value of each feature and the reason: pinned by an identity override, set by a segment, or chosen by the weighted split. Make this the first stop whenever someone asks, “why does this person see that version?”. The answer is usually visible in under a minute and removes guesswork from the conversation.

Follow a single request from the app to Flagsmith and back, and the whole mechanism becomes a short story.

The cast

The application is your product. The SDK is a small library inside it that knows how to talk to Flagsmith. The Flagsmith API is the service that answers. The rules store holds features, variations, weights, segments and identities.

sequenceDiagram
  participant U as Person
  participant A as Application
  participant S as SDK
  participant F as Flagsmith API
  participant R as Rules Store
  U->>A: Opens the checkout page
  A->>S: Get value of checkout_button_label for user_48213
  S->>F: Request with Environment key and identifier
  F->>R: Load feature, variations, weights, overrides
  R-->>F: Rules for this Environment
  F->>F: Check overrides, then compute bucket and pick variation
  F-->>S: Value: Add to basket
  S-->>A: Value ready
  A-->>U: Shows the button labelled Add to basket
    

Reading the diagram

The person opens a page. The application asks the SDK for the feature value, passing the identifier. The SDK adds the Environment key, which tells Flagsmith which Environment the question concerns. Flagsmith loads the rules, checks overrides, computes the bucket and chooses the variation. The value travels back and the application draws the correct button.

What the application actually does

The code does not contain three separate branches wired to three flags. It reads one value and acts on it, usually by choosing among a small number of known names. The control value is the safe fallback when a value is unknown or missing, so a typo in a variation cannot crash the page.

What can slow the journey

Three things commonly slow the trip: a slow network, a request made too often, and a request that waits for a response before drawing anything. The remedies are simple. Cache results, ask once per screen rather than once per element, and draw the page with the control version while waiting if a short timeout expires. Customers prefer a page that appears quickly in a safe version over a perfect page that appears late.

Getting the value is the easy part. Using it safely is where good habits pay off.

The mapping habit

Treat the incoming value as a label and map it to behaviour inside your own code. The code holds a small table: “label_add_to_basket means show this text”, “label_get_it_today means show that text”. If the incoming label is not in the table, the code uses the control behaviour. This protects against typos, unexpected values and the case where a variation is deleted while some devices still hold an older cached answer.

Record what the person saw

An experiment is only useful if you can later tell who saw what. When the application receives a variation, it should send an event to its analytics tool saying, for example, “person user_48213 saw variation label_add_to_basket for feature checkout_button_label”. Without this, you cannot link behaviour to variation. Flagsmith offers integrations with some analytics tools, and many teams record this themselves. Check the current documentation for which integrations are available.

Do it once, at the moment of exposure

Record the exposure when the person actually sees the variation, not when the app merely fetches the value. If you record exposure for everyone who loaded the app but only some reached the checkout page, your comparison will include people who never saw the button.

A tidy way to read the value in code

A good habit is to wrap the Flagsmith call in a small function of your own, such as “get checkout label”, which hides the details. Inside, it asks Flagsmith, checks the answer against the list of known labels and returns control for anything unexpected. The rest of the application only calls the small function. If you later change your flag tool or the way names are stored, only that one function changes.

The wrapper is also the best place to record exposure, to apply safe defaults and to add timing measurements. One place for all the careful work means the careful work actually happens.

Flagsmith splits people fairly. Whether the result teaches you anything depends on how you design the experiment around it.

Start with a hypothesis

A hypothesis is a clear sentence that can turn out wrong. For example: “Changing the button text to ‘Add to basket’ will increase the share of visitors who start checkout.” Writing it down before the test prevents the habit of looking at many numbers afterwards and picking whichever one looks good.

Choose one main measure

Pick one primary metric, such as the share of visitors who finish a purchase. Other numbers can be watched as guardrails, such as page speed or complaint rate, to make sure a win in one area does not cause harm in another.

Decide how long and how many

Small differences need many people to detect. A test run for one afternoon with a hundred visitors may show a difference that is only luck. Statisticians use a calculation called a power analysis to estimate how many people are needed. Many analytics tools provide a sample size calculator. Fix the duration in advance, and cover at least a full weekly cycle so weekday and weekend behaviour are both included.

An example hypothesis, written well

“For visitors who reach the checkout page, changing the button label from ‘Buy now’ to ‘Add to basket’ will raise the share who complete a purchase within the same visit by at least two percentage points, without increasing the rate of customer support contacts.” This sentence names the audience, the change, the primary metric, the size of improvement worth caring about and the guardrail. Anyone can later check whether it came true.

Minimum meaningful change

Before the test, decide the smallest improvement that would justify the effort. A change that raises purchases by a hundredth of a percent is real but may not matter to the business, and detecting it would need millions of visitors. Fixing the smallest interesting change shapes the sample size calculation and prevents a long test chasing something not worth having.

Run the test through full cycles

Customer behaviour follows rhythms. Weekends differ from weekdays. Pay day differs from the end of the month. A test that runs from Tuesday to Thursday sees only part of the picture. Running at least one or two full weeks, and avoiding unusual events such as big sales unless they are the target, gives a fairer view.

Decide before, not after

The numbers you type into the weight boxes decide how fast you learn and how much risk you take.

Common weight plans

More variations, less certainty

Every extra variation divides the audience into smaller groups. With the same number of visitors, four variations give each group a quarter of the evidence. Small groups produce noisy results. Teams that want to test many ideas should either wait longer or narrow the list before the live test, using earlier research and common sense.

flowchart TD
  A["All eligible people"] --> B["Weights split them"]
  B --> C["Control: 50 percent"]
  B --> D["Variation A: 25 percent"]
  B --> E["Variation B: 25 percent"]
  C --> F["Compare metric"]
  D --> F
  E --> F
  F --> G["Decide: ship, drop or iterate"]
    

Writing the plan in the description

Put the weight plan and the reasons for it into the feature description, for instance “Variation A 25, B 25, control 50, chosen to get an answer in two weeks with ordinary traffic.” When another person opens the feature, they see the intent as well as the numbers. If the numbers later change, add a dated line to the description. This tiny journal makes the history readable to everyone, even without opening the audit log.

Not every experiment should include everybody. Combining segments with multivariate lets you test on exactly the right audience.

Why limit the audience

A change meant for new customers should not be tested on veterans. A feature for one country may not make sense elsewhere. Including the wrong people dilutes results and may annoy people for no reason.

Two ways to combine

Keeping the audience description honest

When you limit an experiment to a segment, say so in every report: “results apply to new customers in India on Android”. It is tempting to speak as if the result applies to everyone. A finding for one audience may not hold for another, and a clear description of who was in the test prevents costly over-generalising.

Multivariate is not only for A/B tests. It is also a safe way to change behaviour slowly and to find a good setting.

Engine migrations

A team replacing its search engine can create a feature with variations “engine_old”, “engine_new_a” and “engine_new_b”. Starting with a tiny share on the new engines, they watch speed and error rates, then grow the weights gradually. If something goes wrong, they set the new weights to zero and everyone returns to the old engine within moments.

Tuning a number

When nobody knows the best cache lifetime, a feature with variations 30, 60 and 120 seconds lets live traffic reveal the answer, guided by measurements of speed and cost. Once the answer is clear, the winner becomes the control value and the experiment ends.

Fair load testing in production

Sending a small percentage of real traffic to a new service, sometimes called a canary release, is easy with weights. The name comes from the canary birds miners once carried to warn of danger: if the small group has trouble, the team learns before everyone is affected.

flowchart LR
  A["Day 1: new 1 percent"] --> B["Day 3: new 5 percent"]
  B --> C["Day 5: new 25 percent"]
  C --> D["Day 7: new 50 percent"]
  D --> E["Day 9: new 100 percent"]
  B -. "problem seen" .-> F["Set weights to zero, old engine again"]
  C -. "problem seen" .-> F
    

A migration story in detail

RoamEasy, the fictional travel site, wants to replace its search engine. The feature search_engine has three values: “engine_old”, “engine_new_a” and “engine_new_b”. At first both new engines receive one percent each. The team compares response time, error rate and the share of searches that lead to a click. They also run a “shadow” check offline, comparing the results of the engines for sample searches, which does not involve customers at all.

When the first week shows no problems, they raise each new engine to five percent and then to fifteen. At fifteen percent, engine B shows slower answers on mobile networks. They set B to zero and keep A. A week later A reaches fifty percent, then one hundred, and the old engine is finally retired. At no moment did the team need to deploy new code to change exposure. Every step was a change on the dashboard that anyone with permission could review in the audit log.

Why the old path must keep working

A gradual migration is only safe if the old behaviour keeps working until the end. If the team deletes the old engine early, setting weights to zero no longer rescues anyone. Keep the old path alive and tested until the new one has served everyone for a comfortable period.

Where the choice of variation is made affects speed, freshness and what you need to protect.

Remote evaluation

In remote evaluation, the SDK sends the identifier to Flagsmith and Flagsmith does the thinking. The rules never leave the server. It is the default and is simple and safe.

Local evaluation

In local evaluation, a server-side SDK downloads the Environment document, including variations and weights, and works out the answer in its own memory. It needs the secret server-side key. It avoids a network call for each decision, but its copy of the rules refreshes on a schedule, so a weight change takes a short time to arrive. Check the current documentation for which SDKs support this mode and how bucketing is kept consistent between modes.

A word on SDK differences

Different SDKs, for example for web, mobile and various server languages, can differ in the features they support and in the details of caching and evaluation. When your experiment runs across several platforms, read the documentation for each SDK you use, and test each one in Staging with the same identifier to confirm that the same variation appears. Differences found in Staging cost nothing. Differences found in Production cost data.

The dashboard shows the whole picture on one screen. Knowing where to look saves time and prevents expensive slips.

The feature list

Inside an Environment, the features page lists every feature with its on or off state, its control value and, for multivariate features, an indicator that variations exist. Searching, filtering and tagging help when the list grows. Tags such as “experiment”, “kill-switch” or the owning team make clean-up much easier later.

The feature panel

Opening a multivariate feature shows the control value, the list of variations and the weight boxes for this Environment. A summary of what percentage of people receive each version is usually visible. Here you edit weights, update values, add segment overrides and read the change history.

Permissions

Not everyone should be able to change Production weights. Flagsmith lets administrators grant permissions per Environment and per Project. A sensible pattern is wide freedom in Development, some freedom in Staging and a small trusted group in Production. Some plans also offer change requests, where a second person approves a Production change before it takes effect. Check the current documentation for what your plan includes.

Automation

The Flagsmith API lets scripts create features, edit variations and adjust weights. This is useful for staged ramps that follow a timetable, but automation should be limited, logged and reviewed. A script that raises weights automatically must also have a way to lower them when a guardrail metric worsens.

Tags and naming for large teams

A big organisation may have hundreds of features. Tags such as “experiment”, “migration”, “ops” and the owning team name keep the list navigable. A short naming convention helps even more: start names with the area of the product, such as checkout_ or search_, so related features sort together in the list.

A multivariate feature is meant to be temporary. Knowing the stages makes sure it leaves cleanly.

stateDiagram-v2
  [*] --> Proposed: Idea and hypothesis
  Proposed --> Built: Feature, variations, code mapping
  Built --> Testing: Weights in Development and Staging
  Testing --> Live: Production weights set
  Live --> Decided: Result analysed
  Decided --> Promoted: Winner becomes control
  Decided --> Abandoned: No improvement
  Promoted --> CleanedUp: Variations and code branches removed
  Abandoned --> CleanedUp
  CleanedUp --> [*]
    

The clean-up habit

When the winner is chosen, three things must follow. First, set the winning value as the control value. Second, remove the code branches that handled the losing variations. Third, delete the unused variations and, if the feature is no longer needed, the feature itself. Teams that skip this build up a pile of stale experiments that nobody dares to touch, which makes the code harder to read and increases the chance of accidents.

Making clean-up a team habit

Cleaning up is dull, so teams forget it. A few small rules help. Put the end date and owner in every feature description. Tag every experiment with the word “experiment”. Once a week, filter the feature list by that tag and read the end dates. Anything past its date gets a short message to its owner. When the owner confirms the decision, delete the variations, then the code branch, then the feature. Treat deletion as a normal, welcome task and celebrate the lighter codebase.

What can go wrong if you delete too early

If you delete a variation while some phones still hold an older cached answer, those phones may receive a label that your new code no longer knows. This is why the label-mapping habit with a control fallback matters: even stale answers cannot break the screen. Remove the code branch first in a release, wait until old app versions have mostly updated, then delete the feature in Flagsmith.

Flags sit on the path of every page. They must be fast, and they must never be the reason a page fails.

Cache what you can

Fetching variations on every screen wastes time. SDKs generally keep results in memory, and many can store them on the device so the next launch starts instantly. Teams refresh in the background every so often and accept a few minutes of staleness in return for speed.

Always have a default

If Flagsmith cannot be reached, the SDK needs something to fall back on. Good applications supply a default value in code for each flag. For a multivariate feature the safe default is nearly always the control behaviour, which is the version already proven in production.

flowchart TD
  A["App needs the value for identity I"] --> B{"Fresh cached value?"}
  B -- Yes --> C["Use cached value"]
  B -- No --> D["Ask Flagsmith with short timeout"]
  D --> E{"Answer in time?"}
  E -- Yes --> F["Use and cache it"]
  E -- No --> G{"Older cached value?"}
  G -- Yes --> H["Use older cached value"]
  G -- No --> I["Use control behaviour from code"]
    

Consistency during failures

A subtle risk: if the network fails and the app falls back to control while another device still shows a variation, the same person may see different versions on a single day. This is acceptable for a short outage but worth noting when you analyse results. Exposure logs should record what was actually shown, not what was meant to be shown.

Rules of thumb

• Never block the first paint of a page on a flag request for more than a moment.
• Read the value once and reuse it, rather than asking inside loops.
• Test what happens when Flagsmith is unreachable, in Staging, before an outage teaches the lesson.
• Make sure the control behaviour is always a fully working experience.

Choosing a timeout

A good timeout is short enough that a slow answer never freezes the page and long enough that normal answers arrive. Many teams start with a few hundred milliseconds for the first paint and a longer allowance for background refresh. Measure real response times before choosing, and revisit the number when networks or traffic change.

Variations are visible to the devices that receive them. Treat everything in a variation as readable by a curious stranger.

Public keys, visible values

Client-side Environment keys are designed to be public, which means anyone who finds one can request values for any identifier they invent. They could therefore learn the names and values of your variations. Never place secrets, private addresses or confidential pricing rules inside a variation value or a feature name.

Flags are not permissions

A multivariate feature decides what to show, not what is allowed. If one variation unlocks a more powerful screen, the server must still check that the person is entitled to use it. Hiding a button is a convenience, not a lock.

Privacy of identifiers

Experiments depend on a steady identifier, and an identifier can be personal data if it can be linked to a person. Use opaque internal IDs rather than email addresses, record only what analysis needs, and remove old data when it is no longer required. Rules such as the European GDPR may apply, so involve your privacy team. This article is technical guidance, not legal advice.

Fairness and care

Experiments touch real people. Avoid testing changes that could harm, mislead or disadvantage a group, such as hiding important fees in one variation. A good question before any test is, “Would I be comfortable if every customer knew exactly what we were trying?”

A short privacy checklist

• Does the identifier avoid personal details such as an email address?
• Are exposure logs limited to what analysis needs?
• Is there a retention rule for old identities and logs?
• Can a person’s data be found and deleted on request?
• Has the privacy team reviewed the experiment if it touches sensitive areas?

When someone says “I see something different from my colleague”, good tools turn the puzzle into a short checklist.

The audit log

Flagsmith records changes: who edited a weight, added a variation or created an override, and when. A sudden change in results often matches an entry in the log, so checking it is the first move when numbers behave strangely.

The debugging checklist

Signals worth watching

The sample ratio check

Analysts use a simple test called a sample ratio check. If the weights are fifty-fifty and the logged exposures are sixty-forty over thousands of people, something is wrong, for example a variation crashing for some devices so exposure is never logged. Catching this early protects the whole experiment from silent bias.

A debugging story

An analyst notices that the exposure logs show forty percent of people on control instead of the expected fifty. The sample ratio check has failed. The engineer opens the feature and finds the weights are correct. Then she looks at the logs by platform and sees that Android exposures for variation A are far below expectation. The cause: the Android build crashes on the screen for variation A for a small group of older phones, so no exposure is ever logged for them. Fixing the crash restores the expected ratio. Had nobody checked the ratio, the experiment would have wrongly concluded that variation A performed worse, because the struggling phones were quietly missing from its numbers.

Test identities

Keep a short list of named test identities for each variation, such as “test_variation_a” with an override pinning it to that version. Engineers and testers can sign in as these identities and see exactly what each version looks like on every device. Remember to leave these identities out of any analysis.

Multivariate features are powerful. An honest look at both sides helps teams use them wisely.

When not to use multivariate

If a change is obviously good, such as fixing a typo, simply release it. If a change is obviously needed for law or safety, release it without a test. If the audience is too small to measure any difference, a test only adds complexity. Use the tool where uncertainty is real and where the answer will change a decision.

A pattern is a solution many teams have found useful again and again.

Pattern 1: Control plus one challenger

The simplest and often best design. One control, one variation, a fifty-fifty split. It gives the fastest and clearest answer, because each group is as large as it can be.

Pattern 2: Label-based variations

Make every variation a short name, and let the code map names to behaviour. Unknown names fall back to control. This keeps flags small, safe against typos and easy to review.

Pattern 3: Staff-first, then percentage

Pin a staff segment to the new variation so the team sees it first. Then give a small weight to everyone else, and grow it as confidence grows.

flowchart LR
  A["Ring 1: staff pinned to new version"] --> B["Ring 2: 1 to 5 percent of customers"]
  B --> C["Ring 3: 25 percent"]
  C --> D["Ring 4: 50 percent, measure result"]
  D --> E["Winner becomes control"]
    

Pattern 4: Holdback

After shipping a change to almost everyone, keep a small group, perhaps five percent, on the old behaviour for a few weeks. Comparing them later reveals long-term effects that a short test cannot see, such as customers returning less often after a novelty wears off.

Pattern 5: Guardrails with automatic stops

Link important guardrail metrics, such as error rate, to an alert that tells a human, or a carefully tested script, to set the variation weight to zero. A good test can end itself safely.

Pattern 6: One decision owner

Every multivariate feature has a named person who decides when the experiment stops, which variation wins and who cleans up. Without one, experiments drift on forever.

Pattern 7: The decision record

When an experiment ends, write a short note in a shared place: the hypothesis, the dates, the weights, the number of people in each group, the main result, the guardrail result and the decision. Keep it brief, but keep it. Months later, when someone proposes the same idea again, the note saves weeks of repeated work and prevents repeated mistakes.

Pattern 8: Shared vocabulary of labels

Agree on naming rules for labels, for example “layout_grid” or “label_add_to_basket”, using lowercase words and underscores. When every team uses the same style, reports are easier to read and mistakes are easier to spot. A short naming guide pinned in the team wiki is enough.

An anti-pattern looks like a good idea and causes pain later.

The tempting shortcut

The most tempting anti-pattern is to run an experiment and quietly keep adjusting it until the result looks good. Each adjustment feels small, but together they make the result unreliable. If you want to learn something different, start a new experiment with a new plan. Honest records are worth more than flattering numbers.

Habits that keep multivariate work healthy, and the mistakes teams make without them.

Best practices

Common mistakes

Teach the habits

Good habits spread when they are written down and practised. Hold a short walkthrough for every new engineer and product manager using a harmless sample feature in Development. Let them create variations, set weights, open a test identity and read the effective value. Twenty minutes of practice removes most of the fear and prevents the common mistakes described above.

Imaginary but realistic stories show how the pieces fit. The companies are invented for illustration.

Scenario 1: The online bookstore

PageTurn, a fictional bookstore, wants more people to finish checkout. The team creates a multivariate feature called checkout_button_label with control “Buy now” and variations “label_add_to_basket” and “label_get_it_today”. Staff are pinned to the new labels by a segment. In Production the two variations each get twenty-five percent, leaving fifty percent on control. The code maps the labels to text and falls back to “Buy now” for anything unknown. Exposure is logged when the button appears. After three weeks, “Get it today” shows a modest improvement without more complaints. The team sets it as the control value, deletes the other variation and removes the old code branch.

Scenario 2: The streaming service

StreamNest, a fictional video service, tests three home screen layouts: list, grid and cards. Because traffic is large, a three-way split is possible. The identifier is the internal user ID, so each viewer keeps the same layout on phone and television. Guardrails include time to first play and error rate. The cards layout slows older televisions, so its weight is reduced to zero within hours, and the other two continue.

Scenario 3: The travel booking site

RoamEasy, a fictional travel site, is moving its search to a new engine. A feature with variations “engine_old”, “engine_new” begins at one percent for new engine, grows to five, twenty-five, then fifty. At twenty-five a spike in slow queries appears. The team sets the weight to zero, fixes an index, and resumes the ramp the next week. Customers barely notice, because the old engine was always one weight change away.

Scenario 4: The business software vendor

LedgerLeaf, a fictional accounting product, wants to test a shorter onboarding flow. The experiment is limited to new accounts by a segment, and the identifier is the account ID so that everyone in one company sees the same flow. Strategic customers are pinned to control. After analysis, the shorter flow wins and is promoted.

Lessons across all four

• A stable identifier was chosen first.
• Control was always a working, safe experience.
• Weights moved gradually, and a quick path back existed.
• Exposure and guardrails were measured.
• The winner was promoted and the rest was removed.

Short answers to common questions about multivariate features.

Is multivariate the same as A/B testing?

A/B testing compares two versions. Multivariate features in Flagsmith can serve two or more versions, so they support A/B and A/B/n tests. Flagsmith provides the splitting. Your analytics tool provides the measurement.

Do I need identities to use multivariate?

For a steady split, yes. Without an identifier, the person generally receives the control value.

Will the same person always see the same variation?

Yes, while the identifier, feature and weights stay unchanged. Changing weights can move some people.

What is the control value?

The ordinary value of the feature, used as the baseline and as the share left over after the variations take theirs.

Can weights add up to less than one hundred?

Yes. The remaining people receive the control value.

Can different Environments have different weights?

Yes. Variations are defined with the feature, but weights are chosen for each Environment.

Can I pin a person to one variation?

Yes, with an identity override, or a whole group with a segment override. Pinned people are no longer randomly assigned, so consider excluding them from analysis.

What if Flagsmith is unreachable?

The SDK uses a cached value if it has one, otherwise the default in your code. Make that default the control behaviour.

How many variations should I use?

As few as answer your question. Each extra variation divides the audience and slows learning.

Can I change a variation’s text during a test?

It is better not to. Doing so mixes two different treatments in one group. Create a new variation or a new experiment.

Does Flagsmith analyse the results?

Flagsmith decides who sees what. Analysis normally happens in your analytics tools. Check the current documentation for available integrations and any built-in reporting.

Is it safe to put values in variations?

Yes for presentation and tuning values. No for secrets, because client applications can read them.

Should I remove the feature after the test?

Usually yes. Promote the winner, remove unused variations and code branches, and delete the feature if it is no longer needed.

Where can I find exact, current details?

The official Flagsmith documentation. SDK methods, limits, permission names and bucketing details can change between releases.

A short glossary to keep nearby

A final walk through the essentials, in plain words.

A multivariate feature in Flagsmith is one feature that carries several possible values, called variations, with a control value as the baseline. Each variation has a weight, the percentage of people who receive it, and weights are set separately for each Environment. For each person, Flagsmith turns the identifier and the feature into a steady bucket number, finds which variation owns that bucket and returns its value, so the same person sees the same version every time. Identity overrides and segment overrides come first, and only people with no override are shared out by weight.

The craft lies around the mechanism. Write a hypothesis, choose one main metric and a few guardrails, keep weights steady while measuring, record exposure at the moment of display, and keep the control behaviour safe and working. Keep values short and free of secrets, enforce real permissions on the server, and clean up once a decision is made.