A plain on-or-off switch cannot answer the question “which version is better?” Weighted variations can.

The limit of on and off

A feature flag is a switch that controls whether a feature is available. Sometimes that is all you need. But many questions are not about on versus off. They are about choosing among several good options: three headlines, two layouts, four discount sizes. A simple switch can show only one version to everybody.

Guessing is expensive

Without a way to compare, teams argue. One person likes the green button, another likes the blue one, and the loudest voice wins. Real customers might disagree with both. Showing different versions to different people at the same time, and measuring what happens, replaces opinion with evidence.

Doing it without new releases

You could build the split yourself, with a random number in your code and a new release for every change. That is slow and hard to adjust. With weighted variations, the percentages live in Flagsmith. You change them from a dashboard in seconds, and the code only asks, “Which version should this person see?”

Weighted variations sit on top of a few simple ideas. Let us meet each one.

Project and environment

A project holds one product’s settings. Inside it are environments, which are separate copies of those settings, such as Development, Staging, and Production. Changing a setting in one environment does not touch the others.

Feature and feature state

A feature is a named switch, such as “welcome-message”. Its state in an environment is the current answer: on or off, and perhaps a value, such as a short piece of text or a number.

Identity and traits

An identity is one specific user, represented by an identifier your app sends. A trait is a small fact about that user, such as country or plan.

Segment and override

A segment is a named group defined by rules about traits. A segment override gives that group its own answer for a feature. An identity override gives one named person their own answer.

Variation

A variation is one of several possible values a feature can have. Weighted variations are variations that each receive a percentage share of the users.

A weighted variation is one possible answer for a feature, together with the share of users who should receive it.

Flagsmith calls the idea multivariate

In Flagsmith, a feature with several weighted options is commonly called a multivariate feature, and the options are called multivariate values or variations. “Multi” means many, and “variate” means version. The name may vary slightly by version of the dashboard, but the idea is the same everywhere.

Two ingredients

Each variation has a value and a weight. The value is the content, such as the text “Welcome back” or the colour name “green”. The weight is a percentage that says what share of users should receive it. A feature may have two variations, or several more, up to whatever limits your plan sets.

One person, one variation at a time

For a given feature, each person receives exactly one value. They never see a blend. The weights simply decide how the whole population is shared out, so that in a large group, about the right proportion lands on each variation.

flowchart TD
    F["Feature: welcome-message"] --> V1["Control value: Welcome back (50 percent)"]
    F --> V2["Variation: Hello again (30 percent)"]
    F --> V3["Variation: Great to see you (20 percent)"]
    V1 --> U["Each user receives exactly one"]
    V2 --> U
    V3 --> U
    

Every comparison needs a baseline. In Flagsmith, the feature’s own value usually plays that role.

What a control is

In an experiment, the control is the original, unchanged version. You compare every new variation against it. Without a control, you cannot tell whether a new version is better, or whether everything simply improved that week for another reason.

How Flagsmith handles it

As far as is publicly described, the feature’s regular value acts as the control, and the weights you give to the extra variations are carved out of one hundred percent. Whatever share is left over goes to the control value. If you give variation B twenty percent and variation C ten percent, the control receives the remaining seventy percent. The total of the extra variations cannot go above one hundred. Confirm this behaviour in the documentation for your version.

Why the remainder idea is helpful

It means you never have to make the numbers add up by hand. You say how much each new idea should get, and the rest stays with what you already have. Increasing a new variation automatically reduces the control.

Weights are simple numbers, but a few details deserve attention.

Shares of the whole

A weight is a percentage of all users who reach this feature. A weight of twenty means about one person in five. It does not mean exactly one in five at every moment, which a later chapter explains.

Whole numbers and small shares

Weights are typically entered as whole percentages, though the precise rules depend on your version. A very small share, such as one percent, is possible but gives very few people, so you may wait a long time before learning anything.

Zero is a valid weight

A variation with a weight of zero exists but reaches nobody, unless an override sends people to it on purpose. This is useful for preparing a variation in advance, or for keeping a version available to staff while hiding it from the public.

The sum cannot exceed one hundred

The shares of the extra variations together cannot be more than the whole. The dashboard usually prevents this. If you ever see numbers that seem impossible, treat the dashboard as the truth and re-check your arithmetic.

The assignment must be fair across the crowd and steady for each individual. Flagsmith achieves both with a clever trick.

Not a dice roll on every visit

If the system rolled a die each time someone opened the app, the same person could see green on Monday, blue on Tuesday, and green again on Wednesday. That would spoil both the experience and the experiment. So Flagsmith does not roll fresh dice.

A steady number from the identity

As far as is publicly described, Flagsmith combines the identity and the feature into a number between zero and one hundred, using a calculation that always gives the same result for the same inputs. The number is then compared with the weights, which are laid end to end along the line from zero to one hundred. Whichever band the number lands in is the person’s variation.

Why this works

The calculation spreads people very evenly across the line, so in a large crowd each band receives about its fair share. And because the inputs never change, the number never changes, so the same person lands in the same band every time. Including the feature in the calculation means that different features shuffle people differently, so the people in the early group for one test are not automatically the early group for all tests.

flowchart LR
    ID["Identity plus feature"] --> N["Steady number from 0 to 100"]
    N --> B1{"0 to 20: Variation B"}
    N --> B2{"20 to 30: Variation C"}
    N --> B3{"30 to 100: Control"}
    B1 --> R["The person's variation"]
    B2 --> R
    B3 --> R
    

Stickiness means a person keeps seeing the same variation. It is a precious property that is easy to break.

Why stickiness matters

A person who sees a new layout today and the old layout tomorrow will be confused, and any measurement of their behaviour becomes meaningless. Stickiness keeps the experience and the experiment consistent.

What keeps people sticky

The same identifier on every visit is the main requirement. If your app sends the same user identifier whether the person visits from a phone or a laptop, they stay in the same group on both. If it sends a different identifier on each device, they may be placed differently on each.

What breaks stickiness

Several things shake people loose. Changing the weights moves the band edges, so people near an edge may switch. Using temporary identifiers for logged-out visitors means a cleared browser creates a new person. And anonymous visits with no identifier at all may be handled differently, as a later chapter describes.

Changing weights with care

Increasing one variation’s share and decreasing another’s moves some people. If you are mid-experiment, such moves make the results harder to read. Changing weights is perfectly safe for the app, but think about the effect on your measurements.

A fifty-fifty split does not give exactly fifty-fifty at every moment. This surprises many beginners.

Chance behaves in crowds

If you flip a coin ten times, you rarely get exactly five heads. You might get three or seven. If you flip it ten thousand times, you will be very close to half. Weighted assignment works the same way. With a few visitors the shares look uneven. With many visitors they settle near the weights.

The danger of reading early results

Teams sometimes look at an experiment after one hour, see variation B ahead, and declare victory. With so few people, the lead may simply be luck. Decide in advance how many people you need and how long to wait, and avoid stopping the moment the numbers look good.

Small audiences make it worse

When you combine weights with a narrow group, such as customers in one small country, the number of people shrinks and the unevenness grows. Check that the group is large enough to teach you anything.

The exact screens vary between versions, but the sequence of ideas is the same.

Weights per environment

As far as is publicly described, the variations belong to the feature, but the weights are set separately in each environment. That means Development can run an even split for easy testing while Production holds everyone on the control. Confirm this in your version.

The code stays simple

Your application does not need to know about the weights. It asks for the feature’s value for the current person and displays whatever comes back. This is what makes later changes so easy: the code is written once, and the percentages are a dashboard matter.

The values you offer decide how much you can learn and how safely you can learn it.

Keep the values small and simple

A value can be a word, a number, or a short piece of structured text. Simple values such as “green”, “compact”, or “7” are easy to read in code and easy to explain. If each variation needs a long block of settings, consider whether the variation is really one idea or several.

Test one idea at a time

If variation B changes the button colour, the headline, and the price at once, and it wins, you will not know which change helped. Prefer variations that differ in one clear way, so the result teaches you something you can reuse.

Make every variation acceptable

Each version will be shown to real people, so each must be safe, accurate, and respectful. Do not include a variation you would be embarrassed to see in production. An experiment is not an excuse for a broken experience.

Name them well

Names like “control”, “shorter-text”, and “bigger-button” tell everyone what each is. Names like “B” and “test2” mean nothing in three months.

A weighted feature still has an on-or-off state. The switch and the split do different jobs, and mixing them up causes confusion.

The switch decides whether

The on-or-off state decides whether the feature is available at all. If the feature is off for a person, the whole idea is hidden from them, and your code should behave as if the feature does not exist.

The split decides which

The weights decide which value a person receives when the feature is available. They matter only for people who are included. A feature that is off for everyone has no audience for its split.

How your code should read both

A careful app first checks whether the feature is enabled. If it is, it reads the value and applies it. If it is not enabled, it uses its own safe default. Skipping the first check and reading the value alone can show a variation to people who were never meant to see the feature.

flowchart TD
    A["Ask for the feature"] --> B{"Is the feature enabled for this person?"}
    B -- No --> C["Use the safe default in code"]
    B -- Yes --> D["Read the value assigned by the weights"]
    D --> E["Show that variation"]
    

Segments let you choose who takes part and who is pinned to a particular version.

Pinning a group

A segment override can give a group a fixed variation, so those people skip the random split. Staff may always see variation B so that they can check it. Key customers may always see the control so that they are never surprised.

Overrides come first

In the usual order of authority, identity overrides outrank segment overrides, and segment overrides outrank the default state and its split. So a pinned person receives their pinned value, and the weights never get a chance to decide for them. Confirm the order for your version.

Pinned people are not experiment subjects

Someone who was pinned did not have a fair chance of each variation. If you include them in your results, you will measure the effect of the pinning, not the effect of the variation. Record who was pinned and leave them out of your analysis.

A split applies to everyone who reaches the feature. Often you want it to apply only to a certain audience.

Limit the audience with segments

An experiment on the sign-up page should involve only people who are signing up. An experiment on a premium tool should involve only premium customers. You can keep the feature off, or on the control, for everyone else, and use segment overrides to bring the intended audience into the split. Exactly how you arrange this depends on your version and design, so test it with sample identities.

Why a narrow audience matters

If you test a sign-up change on everyone, most people were never affected by it, and their behaviour dilutes the result. A focused audience makes the signal stronger and the experiment shorter.

The price of narrowness

A narrow audience also has fewer people. If the group is small, you may need a long time to gather enough observations. Balance sharpness against size.

Follow one request from the moment a person opens the app to the moment a variation appears.

sequenceDiagram
    participant App
    participant FS as Flagsmith
    App->>FS: Identifier plus traits
    FS->>FS: Identity override?
    FS->>FS: Segment override, first match?
    FS->>FS: Otherwise steady number against the weights
    FS-->>App: State and one value
    App->>App: Show the variation
    

The value of knowing the order

When someone asks why a particular person saw a particular variation, you walk the steps. Was there a pin? Was there a segment? If not, which band did their number fall in? The answer is always explainable.

In remote evaluation, Flagsmith’s servers choose the variation and the app simply receives it.

How it works

The app sends the identifier and traits to Flagsmith’s API. The server applies the overrides, runs the weighted split, and returns the chosen value. The app never sees the weights or the other variations.

Benefits

The weights stay private. A change to the percentages applies to the next request. The app needs no splitting logic of its own.

Costs

Every question needs a network trip. If Flagsmith cannot be reached, the app needs a fallback, and a fallback means the person sees the control, which should be safe. And remember that cached answers may keep a person on an older variation for a short while after a change.

In local evaluation, a server-side SDK downloads the plan and runs the split itself.

How it works

The SDK downloads an environment document that includes the features, their variations, weights, segments, and overrides, and refreshes it on a timer. When your code asks about a user, the SDK runs the same steps inside your own process, with no network trip per question.

Consistency matters

The SDK must calculate the steady number in the same way that Flagsmith’s own servers do, so that a person receives the same variation whether the question is answered remotely or locally. Flagsmith’s SDKs are designed for this, but if you mix evaluation styles in one product, test that a handful of identities get identical answers in both.

Responsibilities

The SDK needs a secret server-side key, because the document contains the whole plan. Changes to weights reach your service only at the next refresh. And your code must supply the identifier and any traits with each question.

The kind of app you build changes where the variation is chosen and how much you can trust the inputs.

Client-side apps

Web pages and mobile apps normally use remote evaluation with a client-side environment key. The key is public, which is fine because it can only read. The weights stay on the server, so a curious user cannot see the plan.

Server-side apps

Back-end services use a secret key and can evaluate locally. They often decide what to render on a page and send only the finished page to the browser, which prevents flicker, as the next chapter describes.

Where the user can interfere

A user can often change what a client-side app sends, such as traits. If a variation matters for money or access, do not let self-reported traits steer it. Compute those facts on the server.

A visitor should never see one version flash and then change into another.

The flicker problem

If a web page loads with the control and then swaps to variation B a moment later, the visitor sees the page jump. That feels broken, and it can ruin the very behaviour you wanted to measure.

Ways to avoid it

One approach is to fetch the variation on the server and render the correct version from the start. Another is to hold back the part of the page that depends on the variation until the answer arrives, showing a calm placeholder. A third is to keep the last answer from the previous visit and use it immediately, then refresh quietly. Each has trade-offs between simplicity and speed, so choose the one that fits your product.

Mobile apps

For mobile apps, a common pattern is to fetch the answer when the app starts and apply it at the next screen, rather than changing the interface while someone is looking at it. This also gives a more stable experience in poor network conditions.

The steady number comes from an identity. What happens when a visitor does not have one?

No identity, no steady number

A request without an identifier cannot be placed on the line from zero to one hundred in a repeatable way. Depending on your SDK and setup, such a request may simply receive the feature’s default value, or the weights may be applied in a way that is not steady between requests. Check what your SDK does and test it, because experiments with unsteady anonymous traffic are unreliable.

Give visitors a temporary identifier

The usual answer is to generate a random identifier, store it in the browser or app, and send it with every request. The visitor then has a steady place in the split, for as long as the identifier lasts.

The seam when someone signs in

When the visitor logs in, you may switch from the temporary identifier to the real user identifier. At that moment their number changes, and they may land in a different variation. For experiments on the sign-in journey, this is a real concern. Plan for it, perhaps by measuring only after sign-in, or by using the same identifier before and after.

A split is only useful if you can connect each person’s behaviour to the version they saw.

Flagsmith decides; your analytics measures

Flagsmith chooses the variation, but the measuring of clicks, sign-ups, purchases, and speed happens in your own analytics tools. Flagsmith can pass information to some analytics systems through integrations, depending on your version and plan, but you should not assume that every system is connected.

Record the exposure

The most important habit is to record, in your analytics, an event saying “this person saw variation B of this feature at this time”. Everything else is then simple: group the later behaviour by that record and compare. Without the exposure record, you are comparing groups you cannot identify.

Record it once the person really saw it

Be careful about when you log the event. If you log it as soon as the answer arrives but the person never reaches that screen, you count people who were never exposed, and the difference between versions shrinks. Log it at the moment the variation is displayed.

flowchart LR
    A["Flagsmith picks the variation"] --> B["App displays it"]
    B --> C["App records exposure in analytics"]
    C --> D["Person behaves: click, sign up, buy"]
    D --> E["Analytics groups behaviour by variation"]
    E --> F["Team compares results"]
    

Weights make an experiment possible. A few habits make it trustworthy.

Start with a question

Write down what you want to learn and what you will measure, before you start. “Does the shorter headline increase sign-ups?” is a question. “Let’s see what happens” is not, and it leads to finding patterns in noise.

Decide the size and length in advance

Estimate how many people you need and how long you must wait, taking into account weekday and weekend differences. Do not stop the moment one version leads. Stopping on a lucky streak is the most common way to fool yourself.

Keep the conditions steady

Do not change the weights, the audience, or the variations mid-way. If you must, record it and treat the experiment as starting again.

Watch the guardrails

Besides the measure you hope to improve, watch a few measures that must not get worse: errors, speed, cancellations, and complaints. A variation that raises sign-ups but doubles errors is not a winner.

Decide what counts as a win

Agree before the test how big a difference matters. A tiny improvement may not be worth the extra complexity of keeping the winning version. Write the decision rule down, so that nobody argues afterwards.

Weights are not only for experiments. They are also a careful way to move everyone from an old version to a new one.

The ramp

Start with a small share on the new version, such as five percent, and watch. If the health measures look fine, raise it to twenty-five, then fifty, then one hundred. At each step, the worst-case damage is limited to the share who currently receive the new version.

How it differs from an experiment

In an experiment you are trying to learn which version is better, and you hold the split steady while you measure. In a rollout, you already believe the new version is good, and you are checking that it is safe. The same tool serves both, but the habits differ. A rollout moves quickly toward one hundred, while an experiment waits.

Being ready to go back

The biggest benefit of a weighted rollout is the way back. If something goes wrong at twenty-five percent, you lower the share and the affected people return to the old version within moments. Decide in advance who may do this and what signals trigger it.

flowchart LR
    A["5 percent on new"] --> B{"Healthy?"}
    B -- Yes --> C["25 percent"]
    C --> D{"Healthy?"}
    D -- Yes --> E["50 percent"]
    E --> F{"Healthy?"}
    F -- Yes --> G["100 percent"]
    B -- No --> R["Lower the share and investigate"]
    D -- No --> R
    F -- No --> R
    

Weighted variations are not limited to two choices, but each extra version has a price.

Why add more versions

Sometimes you have three good ideas and want to compare them at once. Testing them together can be faster than testing them one after another, and it avoids the trap of changing the world between tests.

The price of each extra version

Every version needs enough people to produce a clear result. If you split the same crowd three ways instead of two, each group has fewer people and the experiment takes longer. And the more comparisons you make, the greater the chance that one version looks better by luck alone. Keep the number of versions modest.

A sensible approach

Many teams test two or three versions, pick a winner, and then test refinements of the winner in a second round. This keeps each experiment clear and each result useful.

An experiment is not finished when the numbers look good. It is finished when the code and the dashboard are tidy.

Make the decision

Compare the results against the rule you wrote beforehand. Include the guardrail measures. If nothing clearly won, that is also a result: it tells you the change does not matter much, and you can keep the simpler version.

Move everyone to the winner

Set the winning variation to one hundred percent, or change the control value to the winner’s value, depending on how you prefer to keep things. Watch the health measures during the final step as you would for any rollout.

Clean up

Remove the losing variations from the code and from the feature. Delete any overrides that existed for the experiment. Remove the feature if the choice is now permanent. Write a short note describing what you tested, what you saw, and what you decided, so the next team does not repeat it.

A busy product has many questions at the same time. Experiments can interfere with each other.

The problem of overlap

If one experiment changes the headline and another changes the button on the same page, a person may be in both. Their behaviour then reflects both changes at once, and each result is muddied by the other. Sometimes the effects are small enough to ignore. Sometimes they interact strongly, such as a new price test alongside a new discount banner.

Different features, different shuffles

Because the feature is part of the steady-number calculation, a person’s place in one experiment is generally unrelated to their place in another. Across a large crowd, each group in one experiment contains a mixture of the groups in the other. This spreads the interference evenly, which helps, but does not remove it.

Keeping risky pairs apart

For experiments that clearly touch the same screen or the same decision, avoid running them together. Run them one after another, or use a segment so that each person is eligible for only one. A shared calendar of running experiments helps everyone see what is already in progress.

People change over days and weeks. Experiments must allow for that.

Weekdays and weekends

Shoppers behave differently on a Tuesday than on a Saturday. An experiment that runs for two days may catch only one kind of behaviour. Running for at least one full week, or several, smooths this out.

Special days

Holidays, sales, and news events change behaviour for everyone. An experiment that spans such a day may show unusual results. Avoid starting experiments just before major events, or at least note them when reading results.

The novelty effect

A new version often attracts curiosity simply because it is new. Clicks rise at first, then fall as the novelty fades. Conversely, change aversion can make a new design look worse at first, until people get used to it. Waiting long enough, and looking at the trend rather than the first days, protects you from both mistakes.

Returning users

Long-time users, who know the old version well, react differently from newcomers. If your audience mixes both, consider looking at them separately.

Weighted variations are not only for colours and words. They can carry numbers and settings that change how the product behaves.

Behaviour variations

A variation may be a number such as a timeout, a page size, a retry limit, or a discount percentage. Splitting users among different numbers lets you learn which setting works best in real conditions, which is difficult to find with tests in a laboratory.

The extra caution these need

A variation that changes behaviour can hurt, not merely annoy. A shorter timeout may cause more failures. A larger page size may slow the screen. Keep the range of tested values inside what you know to be safe, validate the value in your code, and start with a small share. Treat each such variation like a mini release.

Validation in the code

Your application should check every received value: right type, inside the allowed range, not empty. If the check fails, fall back to the control. This protects you from a typing slip in the dashboard.

Because variations are shown to real people, a responsible team asks whether each version is fair.

Do not harm

No version should deceive, pressure, or disadvantage the people who receive it. A test that hides important information from one group, or makes cancelling harder for one group, is not an experiment. It is a trick.

Price and access need extra care

Showing different prices to different people for the same thing can upset customers when they find out, and in some places it may raise legal questions. Involve the right people in your organisation before testing prices or access to essential services.

Watch who is affected

A variation may help most users and hurt a small group, such as people using assistive tools or older devices. Look at results for such groups, not only the overall average.

Be ready to stop

If a variation shows signs of harm, stop it at once. The ability to change weights instantly is one of the great safety benefits of the tool; use it.

Choosing a variation is cheap. The costs hide in how often you ask and how much you carry.

The cost of one decision

Calculating a steady number and comparing it with a handful of weights is very quick. It is rarely the slowest part of a request.

Where the time goes

In remote evaluation, the network trip dominates. In local evaluation, downloading and holding the environment document is the main cost. Asking again and again for the same answer at every line of code adds up in both.

Ask once, reuse the answer

Request the features you need once per page or session and keep the answer in memory. Reusing it also guarantees consistency: the person sees one variation throughout the visit even if you change the weights in the middle.

Keep the document lean

With local evaluation, every variation and segment lands in the document. Remove finished experiments so that the document stays small and quick to fetch.

A good design assumes that any network service will sometimes fail.

The control as the safety net

If your app cannot get an answer, it should fall back to the control value, which is the version you have always shipped and the one you trust. Define this fallback in your code, not only in the dashboard.

Caching helps

Many SDKs remember the last good answer. Local evaluation keeps the last downloaded document. Client-side apps can store the last response. These measures reduce the harm from short outages and keep a person on the variation they were already seeing, which also protects the experiment.

Outages and your results

During an outage, people who should have seen variations see the control instead, and your results for that period are polluted. Record outages, and exclude affected periods from the analysis if needed.

flowchart TD
    A["Need a variation"] --> B{"Can we reach Flagsmith or a fresh copy?"}
    B -- Yes --> C["Use the assigned variation"]
    B -- No --> D{"Do we have a cached answer?"}
    D -- Yes --> E["Use the cached variation and log a warning"]
    D -- No --> F["Use the control from code and alert the team"]
    

A change to the weights does not reach every user in the same instant.

Layers of delay

A server using local evaluation holds the old document until its next refresh. A browser or phone may have cached the last answer. A delivery layer between you and Flagsmith may hold its own copy briefly. Together these decide how fast a change spreads.

A mix of old and new

During the delay, some servers use the old weights and some use the new. That is normal and harmless for most experiments, but it means the exact moment of a change is blurry. When you stop a harmful variation, a few people may still see it for a short time.

Plan for it

Know the longest delay in your setup. For emergencies, test how quickly a change really takes effect before you need it. Record the time of each change so that analysts can allow for the blur.

A variation shown on a screen is visible to the person looking at it. Assume they can look closely.

Values are not secret

Any value delivered to a browser or phone can be read by that user. Never place passwords, private keys, internal addresses, or confidential details inside a variation value.

Traits can be faked

If a client-side app sends traits that influence which segment someone falls into, the user can edit them. Do not let a self-reported trait unlock something valuable. Compute important facts on the server.

The user can look for the other versions

A curious person may change their identifier and watch how the page changes, in order to see the other variations. This is rarely harmful for a headline test, but it matters if a variation reveals unreleased plans or special prices. Keep unreleased material out of any variation that real users can reach.

Protect the keys

Client-side keys are public by design. Server-side keys can read the whole plan, including all weights, so keep them in a secrets manager. Administration credentials, which can change weights, deserve the strongest protection.

Experiments follow people’s behaviour, so they carry privacy responsibilities.

Collect the minimum

Send Flagsmith only the identifier and traits you need. Record in analytics only the events that answer your question. More data is not automatically better; it is more to protect.

Prefer non-personal identifiers

An internal user number, or a random identifier, is better than an email address. If data leaks or is exported, it reveals much less.

Know what is stored and for how long

Flagsmith may store identities and traits, and your analytics tool stores exposure and behaviour records. Treat both as personal data where the law says so. Know how long each is kept and how a person’s data can be removed on request.

Respect consent

In some regions, measuring behaviour requires the person’s consent. If a visitor has declined tracking, do not record their exposure, and think about whether they should be excluded from the experiment altogether.

A weight change can alter the experience of thousands of people in seconds. That deserves clear responsibility.

Roles and permissions

Flagsmith lets you control who may view features and who may edit them in each environment. The details, and which plans offer them, can differ, so check the current documentation. A sound principle is to give the widest access in Development and the narrowest in Production.

Approvals

For sensitive features in Production, a second pair of eyes catches mistakes. Where your plan offers change requests or scheduled changes, use them for experiments that affect money, access, or core flows.

The audit log

Flagsmith records who changed what and when. When results look odd, check whether the weights were changed during the experiment. The log is also how you reconstruct what happened when writing up the result.

Ownership

Every experiment should have an owner who can explain it, defend it, and shut it down. Unowned experiments tend to run forever.

You cannot manage what you cannot see. Make the effect of each variation visible.

Split your dashboards by variation

For each running experiment, make the key health graphs available by variation: errors, response time, and the success measure. A problem that affects only one variation is invisible in the overall average.

Check that the split is what you set

Count how many people were exposed to each variation. If you set fifty-fifty and see seventy-thirty with thousands of people, something is wrong: perhaps exposure events are logged inconsistently, perhaps one variation fails to load, or perhaps an override is pinning many people. This check is sometimes called a sample ratio check.

Mark changes on graphs

Use the audit log and webhooks to mark weight changes on your monitoring graphs, so a bend in a line can be matched to a change.

Everything you can do in the dashboard can usually be done by a program too.

Why automate

For a few experiments, clicking is fine. For many environments, or when teams want changes reviewed like other configuration, automation is better. Flagsmith provides an administration interface that can create features, set variations, and change weights. It uses a more powerful credential than the one used for evaluating features.

Automating a ramp

Some teams write a small tool that raises the share of a new version in steps, waiting and checking health measures between steps, and lowering it if a measure turns bad. This gives repeatable, careful rollouts. Keep a human in the loop for important features, and make sure the tool cannot do something that a human would not be allowed to do.

Separate evaluation from administration

Asking “which variation does this person get?” and “change the weights” use different interfaces and different keys. A service that only evaluates should never hold an administration credential.

These are tried and tested ways to use weights well.

These habits look harmless at first and become painful later.

If you remember only a handful of habits, make them these.

Common mistakes behind these habits

Almost every mistake with weighted variations comes from skipping one of the habits above. Teams peek and stop too early, so they ship a version that only looked better by luck. They change weights mid-way, so nobody knows who saw what. They forget the clean-up, so dead branches pile up. The habits are not complicated; they only need to be done every time.

You do not need advanced mathematics to avoid the biggest mistakes. A few pictures in your head will do.

Small groups are noisy

If ten people see version A and ten see version B, and three sign up from A and four from B, nobody should celebrate. A single extra sign-up changes the story. With ten thousand people in each group, a difference of a few percent starts to mean something.

Small differences need big crowds

The smaller the improvement you hope to detect, the more people you need. A huge improvement shows up quickly. A tiny one may need weeks of traffic. If your product has little traffic, test bolder changes, since subtle ones will be lost in the noise.

Rare events need even more

If only one visitor in a hundred buys, you need many visitors to see enough purchases in each group. Measuring a rare event directly is slow; sometimes it is better to measure an earlier step that happens more often, such as adding to a basket, as long as that step truly relates to the final goal.

Use tools, not gut feeling

Your analytics tool or your data team can calculate the number of people needed and tell you how confident you can be in a result. Flagsmith assigns the variations; the judging of results belongs to your measurement tools and your analysts.

Here are some stories that show weighted variations at work in ordinary companies.

Let us follow one team through a complete weighted experiment.

A language-learning app wants to know which welcome message makes new users complete their first lesson more often.

Step 1: Ask the question

The team writes: “Does a friendly question, instead of our current plain greeting, increase first lesson completion among new users?” The measure is the share of new users who finish the first lesson within a day. Guardrails are app crashes and the time to load the home screen.

Step 2: Plan the size and the length

The analyst says that, with the app’s traffic, they need about four thousand new users per group to detect a two percent improvement. At current sign-up rates, that takes about two weeks. They decide to run for three weeks to cover whole weeks and wait for novelty to fade.

Step 3: Build the feature

They create a feature called “welcome-message”. The control value is the current plain greeting. They add one variation, the friendly question. In Development they test with sample users. In Production the weights start at five percent for the new message.

Step 4: Pin and exclude

Staff are pinned to the new message through a segment override, so they can check it. The same override group is excluded from the analysis. The code validates that the value is not empty and falls back to the control otherwise.

Step 5: Ramp to the test split

At five percent, the team watches crashes and load time for a day. All is well, so they move to fifty-fifty for the measured part of the experiment. They log exposure when the welcome screen is displayed, and verify that the exposure counts come out near fifty-fifty after the first thousand users.

Step 6: Hold still

For three weeks, nobody touches the weights. The team resists looking at the result daily to declare a victory; they glance only at the guardrails and the split check.

Step 7: Decide

At the end, completion is 3.1 percent higher in the friendly group, above the two percent they cared about, and guardrails are unchanged. They apply the rule they wrote at the start and choose the friendly message.

Step 8: Clean up

They raise the friendly message to one hundred percent, watch for a week, then make it the new control value, delete the old variation, remove the branch from the code, and delete the staff override. They write a short note about the result in the team’s experiment log.

flowchart LR
    A["Ask the question"] --> B["Plan size and length"]
    B --> C["Build feature and variation"]
    C --> D["Pin staff, validate values"]
    D --> E["5 percent, check health"]
    E --> F["50/50 split, log exposure"]
    F --> G["Hold still for three weeks"]
    G --> H{"Rule met?"}
    H -- Yes --> I["Roll out the winner"]
    H -- No --> J["Keep the control"]
    I --> K["Clean up"]
    J --> K
    

Weighted variations are powerful, and like every power tool they have costs.

The central trade-off

Every variation buys knowledge at the price of complexity. The art is to test the ideas that matter, finish each test, and clean up after it. A small number of well-run experiments teaches more than a pile of half-finished ones.

When a result surprises you, work through the evaluation steps in order. Most puzzles have simple causes.

A calm routine

First, view the identity and its identifier. Second, check for an identity override. Third, check segment overrides in order. Fourth, look at the feature state and the weights in this environment. Fifth, read the audit log for recent edits. Write down what you find, because the same puzzle often returns.

Many teams begin with a random split written directly into their code. Moving it into Flagsmith is a calm, step-by-step job.

Find the hidden splits

Search your code for places where behaviour depends on a random number, a user number remainder, or a hard-coded percentage. List what each does, who owns it, and how often it changes.

Mirror the current behaviour

Create a feature in Flagsmith whose control matches the current default and whose variations match the old branches. Set the weights to the same percentages. A home-made split will probably place users differently from Flagsmith’s calculation, so some people will switch variations at the moment of the move. For an ongoing experiment, that is a reason to finish or restart it rather than carry it across.

Compare, then switch

For a short time, let the old code decide but log what Flagsmith would have chosen. Check that the overall shares match. Then switch the code to rely on Flagsmith.

Remove the old split

Delete the hard-coded logic so only one source of truth remains. Leaving both behind makes it impossible to say which one decided.

flowchart LR
    A["Find hidden splits in code"] --> B["Create matching feature and weights"]
    B --> C["Run both and log differences"]
    C --> D["Switch code to Flagsmith"]
    D --> E["Delete the old split"]
    

Tools help, but habits keep experimentation healthy over years. A light process is enough.

Before starting

Fill in a short plan: the question, the audience, the variations, the measure, the guardrails, the size, the duration, and the decision rule. Check the shared calendar for overlapping experiments. Name an owner.

During the run

Announce the start in the team chat. Check the guardrails and the split ratio only. Do not change the weights, and record any outage or incident that might affect the results.

At the decision

Compare results to the written rule. Note surprises. If the outcome is unclear, say so honestly and decide whether to run longer, run a bolder test, or decide on judgement.

After the decision

Roll out the winner, clean up the feature and the code, and file a short summary in the experiment log. Over time that log becomes a valuable memory of what your customers like and what they do not.

Walk through these questions before relying on a split for something important.

How to use the list

Hold a short review before your first important experiment goes live, and again when your use grows. Ask for real answers, not hopes. Any row where the team hesitates is a place where something will eventually go wrong. Write down the gap, give it an owner, and fix the most dangerous one first.

Flagsmith offers several ways to choose who sees what. Beginners often mix them up, so it helps to put them side by side.

Weighted variations choose which version

The split among values answers the question, “Of the people who reach this feature, which version should each receive?” It is about sharing a crowd among options.

Percentage rollouts choose who gets it at all

A percentage rule inside a segment answers a different question: “Which slice of people should receive the feature, or a special answer?” It is about widening an audience step by step. The mechanics are similar, since both turn an identity into a steady number, but the purpose differs.

Segments choose by facts

A segment selects people because of what is true about them: their plan, their country, their app version. It is about relevance, not about chance.

How they combine

These tools work well together. A segment can select the relevant audience, a percentage rule can choose a safe slice of that audience, and a weighted split can share that slice among versions. When you combine them, say the combination aloud in plain words, and test it with sample identities, to be sure the order of decisions is what you intended.

Getting numbers is easy. Reading them honestly is the harder skill.

Look at the size, not only the direction

It is not enough that version B is ahead. Ask by how much, and whether that amount matters to the business. A difference that is real but tiny may not justify the cost of keeping a second version.

Beware of the many questions trap

If you look at twenty different measures, one of them will probably look impressive by luck alone. Choose your main measure before the experiment starts, and treat other findings as hints for the next experiment rather than conclusions.

Check the groups behind the average

An average can hide a story. A new design may help desktop users and hurt phone users, and the two effects may cancel out. Look at the main groups separately, but remember that smaller groups mean noisier results.

Ask whether it can be explained

A result that comes with a believable reason is easier to trust than one that appears from nowhere. If the friendly welcome message raised completion, you can imagine why. If a random layout change raised revenue by a large amount, double-check the data before you celebrate.

Be willing to report a draw

Sometimes the honest answer is that the versions are about the same. That is useful: it means you can choose the cheaper or simpler one, and spend your attention elsewhere.

A quick reference for every important word in this guide.

Short answers to the questions beginners ask most often.

Weighted variations let you share users among several versions of a feature, by percentage, and learn which works best.

A weighted variation is one possible value of a feature together with the share of users who receive it. The feature’s own value usually acts as the control and takes whatever share is left over. A steady calculation from the identity and the feature decides each person’s variation, so the same person sees the same version on every visit, while crowds divide close to the planned weights.

Overrides sit above the split, so staff and key accounts can be pinned, and then excluded from the analysis. Whether you evaluate remotely or locally, give your code a safe fallback to the control, validate every value, log exposure at the moment of display, and verify that the split matches your plan. Plan the question, the size, and the decision rule before you begin, hold the weights steady, ramp up carefully for rollouts, protect privacy, and always clean up when you are done.