Before Flagsmith enters the story, the idea itself needs to be clear.

The simple definition

An A/B test (also called a split test or a controlled experiment) compares two versions of something, called A and B, by showing each to a different group of similar people at the same time and measuring which group does better on a chosen goal. Version A is usually the current version, called the control. Version B is the new idea, called the treatment or variant.

Why “similar people at the same time” matters

Suppose you showed version A in January and version B in February. If sales rose in February, was it version B, or the winter sales, or a new advert, or the weather? You cannot tell. By running both versions together on groups that are alike, anything outside the product, such as season, news or advertising, affects both groups equally. The only difference left is the version itself. That is what makes the comparison fair.

What makes it a “test” and not just a change

A test has a question written in advance, a goal measured in numbers, groups chosen by chance, and a rule for deciding the winner. Without these, you have only made a change and hoped for the best.

A tiny worked example

An online shop has a sign-up form with five fields: name, email, password, phone and postcode. The team suspects the form is too long. They build a shorter form with three fields: name, email and password. For two weeks, half of the new visitors see the long form and half see the short form, chosen by a fair, steady split. At the end they count how many people in each group completed sign-up. If the short-form group completes sign-up clearly more often, and nothing else got worse, the team has evidence for the change.

Notice what makes this a fair comparison. Both forms were shown on the same days to similar visitors. The visitors did not choose their form. The goal was chosen before the test. The teams agreed beforehand what “clearly more” would mean. Remove any of these ingredients and the comparison becomes weaker.

The numbers above are invented for teaching. Real results are rarely so tidy, which is why later chapters explain how to judge whether a gap is real or luck.

An A/B test needs two jobs done well: splitting people and measuring them. Flagsmith does the first job. Understanding that boundary prevents confusion.

The feature flag in one paragraph

A feature flag is a switch in software that decides which behaviour a person receives, without releasing new code. Flagsmith is a feature flag and remote configuration service. It has a dashboard where people manage flags and small libraries called SDKs that applications use to ask, “what should this person see?”.

The two halves of an experiment

Flagsmith supports the assignment half through multivariate features, which carry several values and share people between them by weight. For the measurement half, Flagsmith connects with analytics and data tools through integrations, and your own code can also record which version each person saw. Check the current documentation for the exact list of integrations and for any reporting features your plan includes.

What Flagsmith brings to an experiment

The cost of guessing

Without testing, teams decide by opinion. The loudest voice in the meeting, or the most senior person, picks the wording or the layout. Opinions are often wrong in surprising ways. Teams that measure frequently discover that the version they disliked performs better with customers. An A/B test turns an argument into a measurement.

The cost of all-at-once changes

Releasing a change to everyone at once is risky. If the change contains a bug or confuses people, everyone suffers at the same moment. A test that begins with a small share protects the whole customer base while the team learns.

The cost of private tooling

Some companies build their own experiment systems. This can work, but the system needs steady care: storing assignments, handling many devices, providing a screen for non-engineers and keeping an audit trail. A shared product such as Flagsmith moves those chores to a place the whole team can see, so a product manager can change a weight without waiting for an engineer to release code.

What large companies teach us

Companies known for large online experiments, such as Netflix, Google, Booking.com and Microsoft, have written publicly about running many experiments and about assigning each person consistently. Their shared lesson is that measurements are only trustworthy when assignment is steady and plans are fixed in advance. These companies are mentioned only to illustrate the practice, and none is claimed to use Flagsmith.

A dozen small words appear in every experiment conversation. Learn them once and every report becomes readable.

Control and variant

The control is what people see today. It anchors the test, because every claim is of the form “the variant did better or worse than the control”. The variant is the idea you want to try. If you have two ideas, you may have two variants, which makes an A/B/C test.

Primary metric and guardrails

The primary metric is the single number that decides the winner. A guardrail is a number you watch to make sure the win does not hide harm. For example, a pop-up that doubles newsletter sign-ups is not a win if it also makes people leave the site twice as often.

Reading a result sentence

A result might be written like this: “Variant B raised the sign-up rate from 10.0 to 12.1 percent, a gain of 2.1 points, with a 95 percent confidence interval from 0.9 to 3.3 points, across 20,000 exposed visitors per group, with no change in fake accounts.” Every word in that sentence comes from the table above. If you can read the sentence and explain it to a colleague, you understand the vocabulary of experiments.

Relative and absolute change

An increase from 10 percent to 12 percent is two percentage points in absolute terms, and twenty percent in relative terms. Both are true, but they sound very different. Always say which one you mean. Reports that mix them confuse readers, and headlines that use only the larger relative number can exaggerate small gains.

An experiment is a small pipeline with a handful of pieces. Seeing the whole pipeline once makes every later chapter easier.

The cast

The application is your product. The SDK is the small library inside it that talks to Flagsmith. The Flagsmith service holds the feature, the variations and the weights. The analytics tool receives events about what people do. The analyst compares the groups and reports the answer.

flowchart LR
  P["Person"] --> A["Application"]
  A -->|"identifier"| S["Flagsmith SDK"]
  S --> F["Flagsmith service: feature, variations, weights"]
  F -->|"variation for this person"| S
  S --> A
  A -->|"shows version A or B"| P
  A -->|"exposure event: person, feature, variation"| T["Analytics tool"]
  P -->|"actions: clicks, purchases"| A
  A -->|"outcome events"| T
  T --> R["Analyst compares groups, reports result"]
    

Reading the diagram

The person uses the application. The application passes the person’s identifier to the SDK, which asks Flagsmith which version applies. Flagsmith answers with the variation, and the application shows that version. At that moment the application sends an exposure event to the analytics tool, naming the person, the experiment and the version. As the person continues to use the product, outcome events, such as clicks and purchases, flow to the same analytics tool. The analyst later joins exposure with outcomes and compares groups.

What can go wrong in the pipeline

Each stage is simple alone. The craft of experimentation lies in checking that the stages agree with each other, which is why later chapters return to checks such as the A/A test and the sample ratio.

Flagsmith is organised as a tree. Knowing where each experiment ingredient lives prevents surprises.

The Organisation is the top level, holding people and billing. Below it are Projects, one for each product. Beneath each Project are Environments, such as Development, Staging and Production. A feature is defined once in the Project, with its variations. The weights, segment overrides and identity overrides are set per Environment.

graph TD
  ORG["Organisation"] --> PRJ["Project: Web Shop"]
  PRJ --> FEAT["Feature: signup_form_version, control = five_fields"]
  FEAT --> V1["Variation: three_fields"]
  PRJ --> DEV["Environment: Development, weights 50 / 50"]
  PRJ --> STG["Environment: Staging, weights 20 / 80"]
  PRJ --> PRD["Environment: Production, weights 10 / 90 at start"]
  PRD --> SEG["Segments: staff, new customers, strategic accounts"]
  PRD --> ID["Identities: user_48213 and others"]
    

One experiment, many Environments

A healthy habit is to rehearse the same experiment in each Environment before Production. In Development, set large weights so that you see both versions within a few test identities. In Staging, use production-like conditions, and confirm that exposure events reach the analytics tool’s test project. Only then set the cautious Production weights. Because the variants are defined once per feature, nothing needs to be rebuilt between stages.

The dashboard makes the setup friendly. Here is the whole journey in plain steps, with the reason for each one.

The feature description is your lab notebook

Write the hypothesis, owner, ticket, start date and planned end date in the feature description. A year later, anyone who opens the feature will understand why it exists and whether it should still be there.

A worked walk-through

Imagine you are the developer at PageTurn, the fictional bookstore. In Development you create signup_form_version with control “five_fields” and one variant “three_fields”. You set the weight to fifty percent so that a few test identities quickly show both forms. You write a small function that asks Flagsmith for the value, checks it against the two known labels, falls back to control for anything else and returns the label. The screen then draws the right form and, at that moment, sends the exposure event.

In Staging you reduce the variant to twenty percent and create three test identities: a staff member, a new visitor and a strategic customer. You check that staff see the variant, the strategic customer sees control and the new visitor sees whichever version their bucket gives. You switch off the network on a test phone and watch the page fall back to control without any error.

Only after these checks do you open Production, starting at five percent. You write the owner, ticket and planned end date in the description, announce the start in the team channel and watch guardrails during the first hour. Calm, repeatable launches are good launches.

Before splitting anyone, decide what exactly is being split. This single choice shapes the validity of the whole test.

What an identity is

An identity is Flagsmith’s name for one end user, account or device that your application tells it about. It is named by an identifier, usually an internal user ID. Flagsmith turns the identifier into a steady number, which decides which version the identity receives. Without an identifier, there is nothing steady to work from, and the person generally receives the control value.

The unit of randomisation

The unit of randomisation is the thing that is assigned to a version as a whole. Common choices follow.

Match the unit to the metric

The unit you assign by should match the unit you measure. If you assign by user but measure per page view, a few heavy users can dominate the numbers. If you assign by account, measure per account. Analysts call a mismatch here a source of misleading precision.

Keeping the unit consistent across tools

The same unit must be used in Flagsmith, in the exposure event and in the analysis. If Flagsmith assigns by user ID while the analyst counts by device, one person with three devices becomes three observations, and the groups no longer match the assignment. Write the unit in the experiment plan in one clear sentence, for example “Unit: signed-in user ID, counted once per user”.

Interference between units

Sometimes the version one person sees affects another person. In a marketplace, a variant that gives buyers a discount may change what sellers experience. In a social product, a change for one friend alters what their friends see. When interference is likely, assign by larger clusters, such as an account, a region or a community, so that people who influence each other share a version.

Behind the friendly dashboard is a clever, simple trick. It makes the split look random yet perfectly repeatable.

The two promises

A good split keeps two promises. First, over many people, the shares match the weights. Second, the same person always receives the same version, on every visit and every device. A coin flip keeps the first promise but breaks the second, because the same person would get a new answer each time.

The trick: a hash and a bucket

A hash function is a recipe that turns any text into a long scrambled number. The same text always gives the same number, but similar-looking texts give very different numbers. Flagsmith combines the identifier with information about the feature, runs the recipe and reduces the result to a number between zero and one hundred, called the bucket. Each version owns a range of buckets according to its weight. The person receives the version whose range holds their bucket. Check the current documentation for the exact recipe, because internal details can change between versions.

flowchart LR
  A["Identifier: user_48213"] --> B["Mix with the feature"]
  B --> C["Hash recipe"]
  C --> D["Bucket, e.g. 37"]
  D --> E{"Which range holds 37?"}
  E -- "0 to 49: Variant" --> F["Variant: three_fields"]
  E -- "50 to 99: Control" --> G["Control: five_fields"]
    

Why the feature is mixed in

If the bucket came only from the person, the same people would always be in the first slice of every experiment and would meet every untested idea first. Mixing in the feature gives each experiment a different shuffle, which spreads the experience around and keeps experiments independent of each other.

Does the split look at who the person is?

No. The hash treats the identifier as anonymous text. The groups therefore tend to look alike on average, in age, country, device and habits. That similarity is exactly what an experiment needs: when the groups behave differently, the difference can be traced to the version and not to the kind of people in each group.

Same answer on every device

Because the bucket depends only on the identifier and the feature, a browser, a phone app and a server all reach the same answer for the same person, provided each sends the same identifier. This is why teams agree on one identifier format before the first experiment. A person who signs in on a laptop in the morning and a phone in the evening then sees the same version both times, which keeps the experience calm and the data clean.

Assigning a version is only half the job. To learn anything, you must record who saw what.

What an exposure event is

An exposure event is a small record saying, “this person saw this version of this experiment at this time”. It is the bridge between Flagsmith and the analytics tool. Later, the analyst uses it to sort every outcome into the right group.

Two ways to record exposure

When exactly to record it

Record exposure when the person actually sees the version, not when the app merely fetches the value. Suppose half of all visitors never reach the sign-up form. If you record exposure for everyone who loaded the home page, your groups will include many people who never saw anything different, and the real effect will be washed out.

sequenceDiagram
  participant U as Person
  participant A as Application
  participant F as Flagsmith
  participant T as Analytics tool
  U->>A: Opens the sign-up page
  A->>F: Which version for user_48213?
  F-->>A: three_fields
  A->>U: Draws the three-field form
  A->>T: Exposure: user_48213, signup_form_version, three_fields
  U->>A: Completes sign-up
  A->>T: Outcome: user_48213, signup_completed
    

A tidy way to read and record in code

A good habit is to wrap the Flagsmith call in a small function of your own, such as “get signup form version”. Inside, it asks Flagsmith, checks the answer against the known labels and returns control for anything unexpected. The rest of the application calls only the small function. The wrapper is also the best single place to add timing, safe defaults and, with the display code, exposure logging. If you later change your flag tool or analytics tool, only the wrapper changes.

A test is only as good as what it measures. Choosing metrics is the most important design decision after the hypothesis.

The primary metric

The primary metric should be directly tied to the goal of the change, easy to count and hard to cheat. For a sign-up form, “completed sign-ups per visitor who saw the form” is better than “clicks on the sign-up button”, because a click is not the real goal.

Guardrail metrics

Guardrails catch harm. Typical ones are page load time, error rate, refund rate, support contacts, unsubscribe rate and, for sign-ups, the share of accounts that are later found to be fake. If a variant improves the primary metric but worsens a guardrail beyond an agreed limit, the team stops and thinks.

Secondary metrics

Secondary metrics explain why the primary metric moved. For example, if sign-ups rose, did people also finish the form faster? They help the team learn, but they should not be used to rescue a failed test by picking whichever number looks good.

Good metric habits

• Define each metric precisely in writing, including who is counted and over what time window.
• Prefer rates (share of people) to raw counts, because group sizes may differ slightly.
• Choose metrics that respond within the test period. A goal that takes six months to appear cannot decide a two-week test.
• Use one primary metric. Several primaries multiply the chance of a lucky false win.

Time windows

Decide how long after exposure an outcome counts. For a sign-up form, completing sign-up within the same visit may be right. For a recommendation feature, purchases within seven days may be right. Use the same window for both groups and fix it before the test. A window chosen after seeing the data is another way to fish for a win.

Ratio metrics and averages

Some metrics are averages, such as revenue per visitor. A few very large purchases can dominate an average and make it jump around. Analysts often cap extreme values or use methods that are less sensitive to outliers. Ask your analyst how outliers will be handled before the test begins, so the decision is not made while looking at results.

Planning decides, before a single person is assigned, whether the test can possibly answer your question.

A hypothesis that can fail

A good hypothesis names the audience, the change, the metric and the expected direction. For example: “For new visitors who reach the sign-up form, replacing the five-field form with a three-field form will raise the completed sign-up rate by at least two percentage points, without raising the rate of fake accounts.” Anyone can later check whether it came true.

Minimum detectable effect

The minimum detectable effect (MDE) is the smallest improvement that would be worth acting on. A gain of two percentage points may justify the work. A gain of two hundredths of a point may not. Smaller effects need far more people to detect, so choosing the MDE honestly shapes the length of the test.

Sample size

The number of people needed in each group depends on the baseline rate, the MDE and how confident you want to be. Statisticians call the calculation a power analysis. Most analytics tools and many websites offer a sample size calculator. As a rule, rarer outcomes and smaller effects require larger groups.

If you cannot reach the sample size

If traffic is too small, you have options: test a bigger change, pick a more common metric, accept a longer duration, or skip the test and decide by other evidence such as user research. A test that cannot reach its sample size produces noise that looks like an answer, which is worse than no test.

A worked planning example

Suppose about 4,000 new visitors reach the sign-up form each day, and ten percent currently sign up. The team wants to detect a rise to twelve percent. Using a standard sample size calculator, they find they need roughly four thousand people in each group. With a fifty-fifty split and 4,000 visitors a day, that takes about two days of traffic. Because they also need full weekly cycles, they plan for two weeks anyway. If the team had wanted to detect a rise from ten to ten and a half percent, the calculator would have shown tens of thousands per group, a much longer test. The numbers here are illustrative only, and your own calculator will give exact values.

The lesson is the order of work: choose the effect worth finding, compute the sample, divide by daily traffic, and only then decide whether the test is practical.

How long to run a test is not only a matter of numbers. People have rhythms, and new things attract extra attention for a while.

Run full weekly cycles

Behaviour changes with the day. Weekends differ from weekdays, and pay day differs from the end of the month. A test that runs from Tuesday to Thursday sees only part of the picture. Running at least one or two complete weeks, and ending on the same weekday on which you began, includes each part of the rhythm equally.

The novelty effect

Anything new tends to attract curiosity. A bright new button may get extra clicks in the first days simply because it is different, and the effect fades as people get used to it. The opposite can also happen: regular users dislike change at first, then settle. This is called the novelty effect (or change aversion). To guard against it, run the test long enough for early reactions to settle, and compare the early days with the later days inside the test.

flowchart LR
  A["Day 1 to 3: curiosity spike or change aversion"] --> B["Day 4 to 10: behaviour settles"]
  B --> C["Day 11 to 14: steady reading"]
  C --> D["Decide using the settled period and the whole period"]
    

Avoid unusual periods

Big sales, holidays, outages and big marketing campaigns can distort behaviour. Either avoid them or accept that the result describes that unusual time. If an outage strikes during the test, note it and consider excluding the affected hours from the analysis for both groups.

Reading the days inside a test

Most analytics tools can show the difference between versions day by day. A healthy test shows a gap that wobbles around a steady value. A gap that starts large and shrinks every day suggests a novelty effect. A gap that grows steadily may suggest that people need time to learn the new design. Looking at the shape, not only the final number, teaches you what is really happening.

You do not need a degree to read an experiment result. A few ideas, explained simply, are enough.

Luck is always in the room

Even if two versions are exactly the same, two groups of people will not behave identically, by pure chance. Flip a fair coin a hundred times, and you will rarely see exactly fifty heads. Statistics is the tool for asking: is the difference I see bigger than what chance alone would plausibly produce?

The p-value

The p-value answers a narrow question: “If the two versions were truly identical, how often would I see a difference at least this large by luck?” A small p-value, such as below five percent, means such a difference would be rare under pure luck, so the data gives evidence of a real effect. It does not say how likely the variant is to be better, nor how big the effect is.

The confidence interval

A confidence interval is a range around the measured difference in which the true effect plausibly lies. A result such as “+1.8 points, with an interval from +0.6 to +3.0” is far more useful than a lone “significant”. It shows both the likely size and how uncertain you still are. If the interval includes zero, the data cannot rule out “no effect”.

Power

Power is the chance that your test will detect a real effect of the size you care about. Low power means that even a real improvement may be missed. Planning the sample size is how you buy power.

An example in words

Imagine a coin-flipping contest between two groups of ten thousand people, each flipping fair coins once. Group A gets 5,030 heads and group B gets 4,970 heads. Is group A luckier at heart? No, the coins are identical. A sixty-head gap in ten thousand flips is the sort of difference chance produces regularly. Now imagine 5,300 against 4,700. That gap would almost never come from chance. The p-value expresses this feeling in a number: the larger and cleaner the gap relative to the noise, the smaller the p-value.

Why wider intervals mean less certainty

Small groups give wide intervals, and large groups give narrow ones. A wide interval, such as from minus two to plus eight points, says “the effect could be harmful or very helpful, we cannot tell”. A narrow interval, such as from plus one to plus two, says “the effect is small but clearly positive”. Teams make better decisions by reading the width as well as the centre.

Most bad experiment results come from a few well-known traps. Knowing them is the best protection.

Trap 1: Peeking and stopping early

Looking at the results every day and stopping the moment the variant looks better greatly increases the chance of a false win. Early numbers swing wildly, and if you stop at a lucky swing, you will declare a winner that is not real. The cure is to decide the sample size or the end date in advance and to wait. Some teams use special sequential testing methods that are designed for continuous monitoring. If you use such a method, follow its rules exactly.

Trap 2: Too many metrics

If you check twenty metrics, one of them will probably look “significant” by luck alone. Use one primary metric to decide. Treat the others as clues for learning, not as ways to rescue a failed test. When many comparisons are unavoidable, ask the analyst to adjust for them.

Trap 3: Slicing until something shines

Cutting the results by country, device, day and age until one slice looks good is another form of the same trap. If a surprising slice looks interesting, treat it as a new hypothesis and test it again properly.

Trap 4: Changing the plan mid-test

Changing weights, changing the audience or editing the version while the test runs mixes different treatments in the same numbers. If something must change, record the date and analyse the periods separately, or restart.

An illustration of peeking

Imagine two identical versions and a daily check. Even though there is no real difference, the measured gap wanders up and down, and on some days it will cross the usual threshold for “significant” by luck. If you stop on the first such day, you will announce a winner far more often than the stated five percent of the time. A test that is checked daily for two weeks can produce a false win far more often than one that is read once at the end. This is the reason why fixed end dates, or proper sequential methods, matter.

There is a simple check that catches many hidden problems before they ruin a test. It looks only at group sizes.

What it is

If you set weights of fifty and fifty, the number of people logged in each group should be close to equal. If one group is noticeably larger, and the gap is bigger than chance explains, you have a sample ratio mismatch (SRM). It means that something is dropping or adding people unevenly, and any result from the test is suspect.

Common causes

• A version crashes on some devices, so exposure is never logged for those people.
• A version loads slowly, so impatient people leave before the exposure event fires.
• Bots or internal staff are treated differently in different groups.
• Exposure is logged in two places in one version and one place in the other.
• People change identifiers partway, so they are counted twice.
• Overrides pin some people to one version outside the random split.

flowchart TD
  A["Weights say 50 and 50"] --> B["Count exposed people per version"]
  B --> C{"Are the counts close to 50 and 50?"}
  C -- Yes --> D["Proceed to compare metrics"]
  C -- No --> E["Sample ratio mismatch"]
  E --> F["Investigate: crashes, slow load, bots, double logging, overrides"]
  F --> G["Fix, then restart or exclude affected data"]
    

How much mismatch is too much?

Some difference in group sizes is normal, because a random split is not perfectly even. Analysts use a simple statistical check on the counts to decide whether the gap is bigger than chance explains. With thousands of people, even a gap of one percentage point from the expected split can be suspicious. Many analytics tools provide the check, or an analyst can compute it in minutes. Treat a failed check as a stop sign, not a footnote.

What to do when the check fails

• Pause decisions based on the result.
• Compare group sizes by platform, country and day to find where the gap appears.
• Check the exposure logging code in both versions.
• Look for overrides, bots and staff accounts in the data.
• Fix the cause, then restart the test with a clean record.

Not everyone belongs in every test. Flagsmith’s segments and overrides decide who is in, who is out and who is pinned.

Segments

A segment is a named group defined by rules over traits, which are facts about an identity such as country or plan. Segments let you test only on the audience your hypothesis is about. A change aimed at new customers should not be tested on veterans.

Overrides and the order of decisions

For each feature, Flagsmith checks, in order, an identity override (a special answer for one person), then a segment override (a special answer for a group), then the general Environment state, where the weighted split happens. The most specific layer that applies wins. So pinned people never enter the random split.

flowchart TD
  A["Request for experiment feature, person I"] --> B{"Identity override?"}
  B -- Yes --> C["Use pinned version"]
  B -- No --> D{"In a segment with an override?"}
  D -- Yes --> E["Use the segment version"]
  D -- No --> F["Compute bucket and pick by weight"]
  C --> G["Return to application"]
  E --> G
  F --> G
    

Documenting the audience honestly

When you limit a test to a segment, say so in every report: “results apply to new customers in India on Android”. A finding for one audience may not hold for another, and a clear description prevents costly over-generalising. Segments need traits, so make sure the application sends the facts the segment depends on, such as country, in a consistent format. A missing trait means a person silently falls outside the segment and receives control.

The weights you choose decide how fast you learn and how much risk you take. A careful ramp gives both.

Common weight plans

The ramp-then-measure pattern

A very sound approach separates safety from measurement. First, ramp slowly, for example one, five and twenty percent, watching only guardrails and errors. If nothing goes wrong, move to the even split that you will actually analyse. Because changing weights moves some people between versions, the clean measurement window begins at the final weights.

flowchart LR
  A["Phase 1: 1 percent, safety check"] --> B["Phase 2: 5 percent, guardrails"]
  B --> C["Phase 3: 20 percent, guardrails"]
  C --> D["Phase 4: 50 / 50, measurement window starts"]
  D --> E["Decision at planned end"]
  A -. "problem" .-> X["Weights to zero"]
  B -. "problem" .-> X
  C -. "problem" .-> X
    

Why changing weights moves people

Each version owns a range of bucket numbers, laid end to end. When a weight grows, the ranges are redrawn, so people near the edges may move between versions. A person who was in control may now receive the variant, and rarely a person may swap between two variants. For a clean measurement, choose the measurement weights once, keep them, and start the analysis window at the moment they are set.

An experiment touches real people. Good teams decide in advance when they will stop early because of harm.

Reasons to stop early

A test should stop before its planned end if a guardrail metric crosses a limit written beforehand. Examples include a spike in errors, a sharp rise in support contacts, a serious drop in sales, a security or legal concern, or a bug that affects the experience of one version. These are not failures of the plan. They are the plan working as designed.

How to stop

With Flagsmith, the fastest stop is to set the variant weight to zero in Production. Everyone then receives the control value at once, and no code release is needed. The change is recorded in the audit log. If the problem is serious, switching the feature off entirely is also possible, provided the code treats “off” as the safe control behaviour.

Writing stopping rules in the plan

Put the rules into the plan as plain sentences: “If the error rate in the variant is more than double control for one hour, set the variant weight to zero and notify the owner.” Name who may press the button, which Environment it applies to and how the team will be told. When a rule exists on paper, a calm person can act at three in the morning without a meeting.

When the planned end arrives, the analyst compares the groups. The team then has to decide what to do with the answer.

The checklist before reading the result

Four possible outcomes

An inconclusive result is a result

A test that finds no clear difference has still taught you something: this change does not matter much for this goal at this size. That saves the team from spending months on a change that does little. Resist the urge to hunt for a win in slices of the data.

Write the decision record

Write a short note: hypothesis, dates, weights, group sizes, primary result with interval, guardrail results, decision and next step. Keep it where the team can find it. When someone proposes the same idea two years later, the note saves weeks of repeated work.

Explaining results to non-experts

Report in plain sentences. “The shorter form led to about three more sign-ups per hundred visitors, and we are fairly confident the true gain is between one and five. We saw no rise in fake accounts. We recommend shipping it.” This sentence includes size, uncertainty, guardrail and recommendation. Pictures help too: a simple bar chart of the two rates with interval bars is clearer than a table of statistics for most audiences.

An experiment is scaffolding. When the building stands, the scaffolding must come down.

stateDiagram-v2
  [*] --> Planned: Hypothesis and plan written
  Planned --> Built: Feature, variants, code mapping
  Built --> Rehearsed: Tested in Development and Staging
  Rehearsed --> Running: Production weights set
  Running --> Analysed: Planned end reached
  Analysed --> Promoted: Winner becomes control
  Analysed --> Dropped: No benefit
  Promoted --> CleanedUp: Code branches and variants removed
  Dropped --> CleanedUp
  CleanedUp --> [*]
    

The safe order of clean-up

Keeping the learning, not the clutter

When the feature is deleted, the knowledge must survive. Copy the decision record to a shared place: what was tried, what was found and what was decided. Link it from the ticket. Future teams searching for “sign-up form test” should find the note rather than repeat the experiment. Deleting configuration while keeping knowledge is the healthy end of an experiment.

Several techniques look alike. Knowing the differences helps you pick the right tool.

Same tool, different purposes

The Flagsmith machinery is similar in each case, but the questions differ. A percentage rollout asks whether a release is safe and moves to full exposure if it is. An A/B test asks which version is better and holds the weights steady while it measures. Mixing the two purposes causes confusion: a rollout that keeps growing cannot give a clean comparison, and an experiment held at fifty percent is not a rollout.

Choosing the right tool in three questions

1. Do I need evidence about which version is better? If yes, run an A/B test.
2. Do I only need to know that a release is safe? If yes, use a staged rollout with guardrails.
3. Do I need to decide who is allowed to have something? If yes, use segments and entitlements.

Many teams confuse these questions and use an A/B test where a staged rollout would be enough. The result is a long, expensive test that answers a question nobody asked.

Flags sit on the path of every page. They must be fast, and they must never be the reason a page fails.

Cache what you can

Asking Flagsmith on every screen wastes time. SDKs generally keep results in memory, and many can store them on the device so that the next launch starts instantly. Teams refresh in the background and accept a short period of staleness in return for speed.

Always have a default

If Flagsmith cannot be reached, the SDK needs a fallback. Good applications supply a default value in code for each feature. For an A/B test, the safe default is the control behaviour, because it is already proven in production.

flowchart TD
  A["App needs the version for identity I"] --> B{"Fresh cached value?"}
  B -- Yes --> C["Use cached value"]
  B -- No --> D["Ask Flagsmith with a short timeout"]
  D --> E{"Answer in time?"}
  E -- Yes --> F["Use and cache it"]
  E -- No --> G{"Older cached value?"}
  G -- Yes --> H["Use older cached value"]
  G -- No --> I["Use control behaviour from code"]
    

Failures and fair measurement

If the network fails and one device falls back to control while another still shows the variant, the same person may see two versions in a day. A short outage matters little, but exposure logs must record what was actually shown, not what was intended. Analysts can then handle such cases honestly.

Page speed is part of the experiment

If asking Flagsmith slows the page for everyone, the experiment itself may harm results. If the variant loads extra content and is slower than control, speed rather than design may explain a loss. Measure load time per version as a guardrail.

Choosing a timeout

A good timeout is short enough that a slow answer never freezes the page and long enough that normal answers arrive. Many teams start with a few hundred milliseconds for the first paint and a longer allowance for background refresh. Measure real response times before choosing, and revisit the number as networks and traffic change. If the timeout fires, use the cache or control and try again later.

Experiments involve real people and their data. Treat both with care.

Public keys, visible values

Client-side Environment keys are designed to be public. Anyone who finds one can ask Flagsmith for the values of any identifier they invent. So never place secrets, private addresses or confidential rules inside feature names or values. Use short labels, and keep sensitive logic on the server.

Flags are not permissions

A flag decides what to show, not what is allowed. If one version reveals a powerful screen, the server must still check that the person may use it.

Privacy of identifiers and events

Experiments depend on a steady identifier and on recorded behaviour. These can be personal data. Use opaque internal IDs, collect only the events the analysis needs, keep a retention rule, support deletion requests and involve your privacy team. Laws such as the European GDPR may apply. This article is technical guidance, not legal advice.

Ethics

A short privacy checklist

• Does the identifier avoid personal details such as an email address?
• Are exposure and outcome events limited to what the analysis needs?
• Is there a retention rule for old identities and event data?
• Can one person’s data be found and deleted on request?
• Has the privacy team reviewed any experiment touching sensitive areas?

When someone says “I see something different from my colleague”, good tools turn the puzzle into a short checklist.

The audit log

Flagsmith records who changed what and when: weights, variants, overrides and feature state. When results behave strangely, check the log for edits near the time the behaviour changed.

The debugging checklist

Test identities

Keep a few named test identities pinned to each version, such as one for the control and one for the variant. Engineers and testers can use them to see each version on every device. Remember to exclude these identities from analysis.

A debugging story

An analyst notices that exposure logs show fifty-seven percent in control instead of fifty. The engineer opens the feature and finds the weights correct. She then breaks the logs down by platform and sees that Android exposures for the variant are far below expectation. The cause: the variant screen crashes on some older phones, so exposure is never logged for them. After the fix, the ratio recovers. Had nobody checked, the test would have wrongly flattered the variant, because its worst experiences were silently missing from its data.

A/B testing with Flagsmith is powerful, but it is not free. An honest look at both sides helps teams use it wisely.

When not to run an A/B test

If a change is obviously a fix, such as correcting a typo, release it. If a change is required for law or safety, release it without a test. If traffic is too small to detect anything, use research and judgement instead. Use A/B tests where uncertainty is real and where the answer will change a decision.

Cost in people and time

The main cost of experimentation is attention rather than money. Each test needs someone to plan it, someone to build it, someone to analyse it and someone to clean up. Teams that run a few well-chosen tests learn more than teams that run many careless ones. Choose experiments where the possible gain is large and the question is truly open.

A pattern is a solution that many teams have found useful again and again.

Pattern 1: Control plus one challenger

The simplest and often best design: one control, one variant, an even split. It gives the clearest answer fastest.

Pattern 2: Labels, not details

Make each version a short label such as “three_fields”, and let the code map labels to behaviour. Unknown labels fall back to control. Flags stay small and typos cannot break pages.

Pattern 3: Staff first

Pin a staff segment to the variant so the team sees the new version before customers do. Exclude staff from the analysis.

Pattern 4: Ramp, then measure

Release slowly while watching only guardrails, then move to an even split and start the clean measurement window.

Pattern 5: Holdback

After shipping a winner, keep a small group, perhaps five percent, on the old behaviour for several weeks. Comparing them later reveals long-term effects that a short test cannot see.

Pattern 6: One decision owner and a written record

Every experiment has a named person who decides when it stops and who writes the decision record. Without an owner, experiments drift on forever.

Pattern 7: A/A test

Before trusting a new setup, run a test where both groups receive the same version. If the setup is healthy, the result should show no difference, and the group sizes should match the weights. An A/A test reveals logging mistakes, identifier mismatches and sample ratio problems at no risk to customers.

flowchart LR
  A["A/A test: both groups see control"] --> B{"Sizes match, no difference?"}
  B -- Yes --> C["Setup is healthy, run the real A/B test"]
  B -- No --> D["Fix logging, identifiers or bucketing, repeat"]
    

Pattern 8: Shared vocabulary of labels

Agree on naming rules for versions and features, for example lowercase words and underscores, with the product area first, such as signup_ or search_. When every team uses the same style, reports are easier to read and mistakes are easier to spot. A short naming guide pinned in the team wiki is enough.

An anti-pattern looks like a good idea and causes pain later.

The tempting shortcut

The most tempting anti-pattern is to run an experiment and quietly keep adjusting it until the result looks good. Each adjustment feels small, but together they make the result unreliable. If you want to learn something different, start a new experiment with a new plan. Honest records are worth more than flattering numbers.

Habits that keep experimentation healthy, and the mistakes teams make without them.

Best practices

Common mistakes

Teach the habits

Good habits spread when they are practised. Hold a short walkthrough for every new engineer, product manager and analyst using a harmless sample feature in Development. Let them create a variant, set weights, open a test identity, read the effective value and check that an exposure event arrives in the analytics test project. Twenty minutes of practice removes most of the fear and prevents the common mistakes described above.

Imaginary but realistic stories show how the pieces fit. The companies are invented for illustration.

Scenario 1: The shorter sign-up form

PageTurn, a fictional online bookstore, believes its five-field sign-up form scares people away. The team creates a multivariate feature called signup_form_version with control “five_fields” and variant “three_fields”. The hypothesis, written in the feature description, expects a rise of at least two points in completed sign-ups, with fake accounts and page speed as guardrails. They run an A/A check for two days and find the logging healthy. They ramp the variant at five percent for a day, then move to fifty-fifty for two full weeks. The sample ratio holds. The variant raises sign-ups by about three points, with an interval comfortably above zero, and fake accounts do not rise. The team promotes “three_fields” to control, releases code without the old form and deletes the feature.

Scenario 2: The inconclusive search page

StreamNest, a fictional video service, tests twenty versus fifty results per page. After three weeks, the confidence interval for watch time runs from slightly negative to slightly positive. The result is inconclusive. The team keeps twenty results, which loads faster, and records that page size does not matter much for watch time. This saves them from months of tuning with no benefit.

Scenario 3: The sample ratio alarm

RoamEasy, a fictional travel site, sets a fifty-fifty split but sees fifty-seven percent in control after two days. The analyst finds that the variant page crashes on older phones, so exposure is never logged for them. Left unnoticed, the test would have made the variant look better because its worst experiences were missing from the data. The team fixes the crash, restarts the test and gains a trustworthy result.

Scenario 4: The business onboarding flow

LedgerLeaf, a fictional accounting product for companies, tests a shorter onboarding flow. Because colleagues sit together and compare screens, the team uses the account ID as the unit of randomisation, so everyone in one company sees the same flow. Strategic customers are pinned to control. With fewer accounts than individual users, the test runs six weeks. The result is a modest win, and the flow is promoted.

Lessons across all four

• A plan was written before the test began.
• A stable identifier was shared between Flagsmith and analytics.
• Control was always a safe, working experience.
• The sample ratio was checked, and an unclear result was accepted honestly.
• The decision was recorded and the scaffolding removed.

Short answers to common questions about A/B testing with Flagsmith.

Does Flagsmith run the statistics for me?

Flagsmith decides who sees which version. Analysis normally happens in your analytics tool or data warehouse. Check the current documentation for any built-in reporting available on your plan.

How do I set up an A/B test in Flagsmith?

Create a multivariate feature with a control value and one variant, set weights per Environment, write code that maps the value to behaviour and logs exposure, and verify with test identities.

Do I need identities?

For a steady split, yes. Without an identifier, a person generally receives the control value.

Will the same person always see the same version?

Yes, while the identifier, feature and weights stay unchanged. Changing weights can move some people.

What split should I choose?

Fifty-fifty gives the fastest answer for one change. Use a small share first when the change is risky.

How long should a test run?

Until the planned sample size is reached, and at least one or two full weekly cycles.

Can I change the weights during a test?

You can, but people may move between versions. Record the date and analyse the periods separately, or restart.

What if the result is inconclusive?

That is a valid result: no evidence of a difference of the size you care about. Keep the simpler or safer option, or run longer if the idea is worth it.

Can I test more than two versions?

Yes, as an A/B/n test, but each group becomes smaller, so you need more traffic or more time.

How do I keep staff out of the results?

Pin staff to a version with a segment override and exclude them in analysis, or record them separately.

What happens if Flagsmith is unreachable?

The SDK uses a cached value if it has one, otherwise the default in your code. Make that default the control behaviour.

Is A/B testing the same as a percentage rollout?

No. A rollout asks whether a release is safe and grows to everyone. An A/B test asks which version is better and holds weights steady while it measures.

Where can I find exact, current details?

The official Flagsmith documentation. Integrations, SDK methods, limits and bucketing details can change between releases.

A short glossary to keep nearby

A final walk through the essentials, in plain words.

An A/B test compares a control and a variant on similar people at the same time, so that differences can be traced to the version. In Flagsmith, a multivariate feature carries the versions and weights, and a steady bucket derived from the person’s identifier gives every person the same version every time. Your analytics tool records exposure and outcomes, and an analyst compares the groups.

The craft lies around the mechanism. Write a hypothesis, choose one primary metric and a few guardrails, calculate the sample size, run full weekly cycles, check the sample ratio and keep the plan steady. Read results with sizes and intervals, accept inconclusive answers honestly, protect people with guardrails and stopping rules, and always clean up after the test.