What Is Multi-Tenant Telemetry Isolation in OpenTelemetry?

What Is Multi-Tenant Telemetry Isolation in OpenTelemetry?

Multi-tenant telemetry isolation is the practice of letting many customers, teams or business units share one observability pipeline while keeping their data, performance, costs and access separate. This guide explains the ideas, the OpenTelemetry building blocks, and the mistakes to avoid.

Imagine a large apartment building with a single mailroom. Hundreds of residents receive letters and parcels every day, and one team of mail clerks sorts everything. A good mailroom guarantees three things. Every resident gets only their own mail. One resident who receives a thousand parcels at once cannot block everyone else’s deliveries. And the building manager can tell how much work each resident causes, so shared costs can be split fairly. Multi-tenant telemetry isolation asks the same three questions about monitoring data: who may see it, who may slow others down, and who pays for what.

1Why Multi-Tenancy Exists

Before talking about isolation, we need to understand why anyone shares a telemetry system in the first place.

What a tenant is

A tenant is a group that owns a set of data and expects it to be kept apart from other groups. In a company that sells software to other companies, a tenant is usually a customer. In a large organisation, a tenant might be a product team, a department or a business unit. In a platform team’s view, a tenant can even be an environment such as production or testing. The word is flexible, and the first job in any design is to decide exactly what it means in your setting.

Telemetry is the stream of traces, metrics and logs that systems produce about themselves. When many tenants produce telemetry, there are two simple ways to handle it. You can give every tenant a completely separate monitoring system, or you can let them share one. Separate systems are easy to understand and strongly separated, but they are expensive and slow to manage. A shared system is cheaper and easier to operate as one platform, but it demands careful design so that tenants do not harm or spy on each other.

Simple Analogy

Think of the difference between everyone owning a private swimming pool and a community swimming pool with marked lanes. Private pools give perfect privacy at a huge cost. A shared pool is far cheaper, but it needs lane ropes, rules and lifeguards so that one swimmer cannot splash everyone else out of the water.

Why organisations choose to share

Reason 1

Cost

Running one platform is far cheaper than running hundreds, because hardware, people and licences are shared.

Reason 2

Consistency

One platform means one set of standards, one upgrade routine and one place to fix problems.

Reason 3

Speed of Onboarding

Adding a tenant to a shared platform takes minutes, not weeks of building new infrastructure.

Reason 4

Platform Insight

A central team can see patterns across tenants, such as which kinds of workloads cause trouble, and improve the system for all.

!
The Catch

Sharing is only safe when isolation is designed in from the start. Retrofitting isolation into a platform that was built as one big pool is possible, but it is painful, risky and often incomplete.

Who the stakeholders are

A multi-tenant platform serves several groups of people, and each of them cares about different things. Tenants want their data kept private, their dashboards fast and their bills fair. Platform engineers want a system that is simple to operate and does not wake them up at night. Security teams want clear evidence that data cannot leak. Finance teams want costs they can explain. Legal and compliance teams want proof that obligations about location and retention are met. A good design keeps all five groups in mind from the first day, because each can veto the platform if its needs are ignored.

StakeholderMain questionWhat they need to see
TenantIs my data private and my service reliable?Clear promises, usage reports, and fast, correct dashboards
Platform engineerCan I run this safely and simply?Small configuration, good monitoring and tested changes
Security teamCan one tenant reach another’s data?Verified identity, server-side access checks and audit records
Finance teamWho caused which cost?Per-tenant metering that can be explained and defended
Compliance teamAre legal obligations being met?Evidence of location, retention, redaction and deletion

2Core Concepts and Vocabulary

A shared vocabulary makes the rest of the guide much easier to follow.

Term 1

Tenant

A customer, team or unit that owns data and expects it to be separate from others.

Term 2

Isolation

The set of guarantees that keep one tenant’s data, performance and costs separate from another’s.

Term 3

Noisy Neighbour

A tenant that uses so much of a shared resource that others suffer. The classic risk of sharing.

Term 4

Tenant Identity

The label that says which tenant a piece of telemetry belongs to, such as an identifier carried as an attribute or a header.

Term 5

Resource Attribute

A label attached to the source of telemetry, describing things such as the service, host or, in our case, the tenant.

Term 6

Gateway Collector

A central Collector that receives telemetry from many sources, applies common rules and sends it onwards.

Term 7

Routing

Choosing the destination or processing path for data according to a property such as its tenant.

Term 8

Quota

A limit on how much a tenant may send or store, in volume, rate or number of distinct series.

An important fact about OpenTelemetry

OpenTelemetry is a set of standards and tools for producing and moving telemetry. It does not, on its own, define a built-in concept of tenancy. There is no single standard switch that says “this data belongs to tenant A and must never mix with tenant B”. Instead, multi-tenancy is something you build from the ordinary parts: attributes that carry identity, authentication that proves identity, processors and connectors that route and shape data, and backends that store data separately. Conventions for naming a tenant attribute also vary, so check the current guidance and choose a name your whole platform will use consistently.

i
Remember

Because tenancy is assembled rather than switched on, the quality of isolation depends entirely on how carefully the parts are assembled. Every gap between two parts is a place where data from one tenant could leak to another.

Telemetry has three signals, and tenancy touches each differently

Traces, metrics and logs behave differently, and so do their tenancy problems. Traces are made of spans linked together, so keeping a trace whole matters, and a single trace should never contain spans from two tenants. Metrics are numbers over time grouped into series, so the main danger is the sheer number of series a tenant can create. Logs are text with attributes, so the main danger is sensitive content hiding inside the message. A design that handles one signal well may handle another poorly, so review each signal separately.

Signal 1

Traces

Keep every span of a trace within one tenant and one processing group. Sampling decisions must never mix tenants.

Signal 2

Metrics

Limit the number of distinct series per tenant, and keep tenant labels on the resource rather than on every point.

Signal 3

Logs

Scan and redact message text and attributes, since logs are the signal most likely to contain personal data.

Linking signals without leaking across tenants

Users like to jump from a metric spike to a related trace to the logs of one request. This works because the signals share identifiers. In a multi-tenant system, the same tenant label must be present and trusted on all three signals, or a jump from one to another could cross a boundary. Treat the tenant label as part of the identity of every item, and make sure that links and lookups always include it.

i
Keep Labels Consistent

Use exactly the same name and format for the tenant label across traces, metrics and logs, and across every environment. Inconsistent naming is a quiet cause of data that falls through routing rules or appears under the wrong tenant.

3What Isolation Actually Means

Isolation is not one thing. It is a bundle of separate promises, and a design can keep some and break others.

DimensionThe promiseWhat failure looks like
Data isolationA tenant can see only its own telemetryA customer sees another customer’s traces or logs
Performance isolationOne tenant’s load cannot degrade othersA burst from one tenant causes delays or drops for everyone
Configuration isolationRules for one tenant do not alter another’sA sampling change for one customer silently changes another’s data
Access isolationOnly authorised people reach a tenant’s data and settingsAn engineer from one team edits another team’s dashboards
Cost isolationEach tenant’s usage is measured and charged correctlyOne tenant’s spike appears on another’s bill
Compliance isolationEach tenant’s legal needs, such as location and retention, are respectedEuropean data stored in the wrong region, or kept longer than agreed

Why the dimensions must be considered separately

It is easy to believe that adding a tenant label to everything solves the whole problem. A label gives you a way to separate data when you query it, which helps with data isolation, but it does nothing for performance. A single noisy tenant can still overwhelm a shared queue even if every item carries a perfect label. Similarly, strong performance limits do nothing to stop someone from reading the wrong data. A serious design looks at each dimension and decides how it will be protected.

Simple Analogy

A hotel needs locks on doors, soundproof walls, separate bills and fire exits. A hotel that has excellent locks but thin walls keeps your belongings safe and your sleep poor. Each kind of protection solves a different problem, and guests expect all of them.

Data
Who can see what
Load
Who can slow others down
Cost
Who pays for which usage

Threats that isolation must withstand

It helps to think like an attacker, or like a careless engineer, and list what could go wrong. A tenant might try to read another tenant’s data by guessing identifiers. A tenant might try to impersonate another by copying its label. A tenant might flood the platform on purpose to cause harm, or by accident through a bug. A platform operator might look at data they should not see. A configuration change might accidentally expose data. A shared cache or shared file might hold data from several tenants and return the wrong one. Each of these threats deserves a specific defence.

ThreatLikely causeMain defence
ImpersonationTrusting sender-supplied labelsVerify identity at the edge and overwrite labels
Cross-tenant readingMissing server-side access checksEnforce tenant access in the backend or trusted proxy
FloodingBug or attack from one tenantPer-tenant limits and separate queues
Operator misuseWide access without oversightLeast privilege, approval and audit records
Accidental exposureUnscoped rules or mixed batchesReviewed, tested, tenant-scoped configuration
Shared cache leakCache keys that omit the tenantInclude the tenant in every cache key and test for it
!
Caches Are Easy to Forget

Query caches, lookup tables and result stores are shared by design. If any of them is keyed without the tenant, one tenant’s answer can be served to another. Review every cache and every shared store for a tenant component in its key.

4Tenancy Models: Silo, Pool and Bridge

Every multi-tenant design sits somewhere on a spectrum between full separation and full sharing.

Model 1

Silo

Each tenant has its own dedicated collectors and storage. Separation is strongest, and cost is highest.

Model 2

Pool

All tenants share the same collectors and storage, and data is separated by labels and rules. Cost is lowest, and the design must be careful.

Model 3

Bridge

A mixture: some parts are shared, such as collectors, while others are separate, such as storage. Many real systems end up here.

flowchart LR
  subgraph Silo
    A1[Tenant A data] --> CA[Collector A] --> SA[Storage A]
    B1[Tenant B data] --> CB[Collector B] --> SB[Storage B]
  end
  subgraph Pool
    A2[Tenant A data] --> CP[Shared collector]
    B2[Tenant B data] --> CP
    CP --> SP[Shared storage with tenant labels]
  end
        
FIG 1 — Silo model with separate stacks versus pool model with shared stacks

Advantages of Sharing (Pool)

  • Much lower cost, because hardware and effort are shared.
  • Quick onboarding of new tenants through configuration alone.
  • One platform to upgrade, secure and monitor.
  • Better use of resources, since quiet tenants make room for busy ones.
  • A single place to apply common rules such as redaction and limits.

Disadvantages / Trade-offs of Sharing

  • A single mistake can expose one tenant’s data to another.
  • Noisy neighbours can degrade the whole platform.
  • Per-tenant legal needs are harder to satisfy in shared storage.
  • Deleting one tenant’s data cleanly can be difficult.
  • Large tenants may demand stronger guarantees than a pool can give.

Choosing a model

The right choice depends on the strength of the guarantees you owe and the money you can spend. A platform serving internal teams that trust each other may happily use a pool. A service holding regulated data for large customers may need silos for the largest of them. Many organisations adopt tiers: a shared pool for small tenants, with dedicated pipelines or storage for the few tenants who need or pay for stronger separation.

QuestionSiloPoolBridge
Strength of separationStrongestDepends on configurationStrong where it matters
Cost per tenantHighLowMedium
Effort to add a tenantHighLowMedium
Noisy neighbour riskNone between tenantsReal and must be managedReduced for separated parts
Operational complexityMany stacks to runOne stack, complex rulesBoth kinds of complexity

Tiering in practice

Most successful platforms define a small number of tiers, rather than negotiating a unique design with every tenant. A basic tier might use the shared pool with modest limits and standard retention. A standard tier might use the shared pool with higher limits and longer retention. A premium tier might use a dedicated pipeline, or even dedicated storage, with strict guarantees. Defining tiers in advance makes the platform easier to explain, price and operate, and it gives tenants a clear path to upgrade as their needs grow.

1

Basic

Shared pool, modest limits, standard retention, shared support.

2

Standard

Shared pool with higher limits, longer retention and tenant-specific rules.

3

Premium

Dedicated pipeline or storage, strong guarantees and priority handling.

4

Regulated

Regional, separated deployment with strict redaction, audit records and deletion on exit.

Whichever tiers you choose, write them down in plain language, including what is promised about data separation, limits, retention, response times and deletion. Tenants will compare tiers carefully, and unclear promises lead to disputes later.

5Where Tenant Identity Comes From

Every isolation mechanism depends on knowing, reliably, which tenant a piece of telemetry belongs to.

The golden rule

“Never believe a tenant’s claim about who they are. Believe only what the platform has verified.”

If the label that names a tenant is supplied by the sender and trusted without checking, any sender can pretend to be any tenant. A careless or malicious client could stamp its data with another tenant’s identifier, filling that tenant’s storage with junk or causing its quota to run out. The safe design is for the platform to decide the tenant identity, based on something it has verified, such as the credentials presented when the data arrived.

Common sources of identity

Source 1

Authentication Credentials

A token or certificate presented by the sender is checked at the edge, and the tenant is looked up from it. This is the most trustworthy source.

Source 2

Network Location

Telemetry arriving from a particular network or cluster is assigned to a tenant. Useful inside a company, weaker across the internet.

Source 3

Platform Metadata

Information from the orchestration platform, such as the namespace a workload runs in, mapped to a tenant by a trusted table.

Source 4

Sender Claims

A label or header set by the sender. Acceptable only as a hint that is verified against a stronger source, never as the final answer.

SourceTrust levelTypical use
Verified credentials at the edgeHighCustomers sending data from outside
Platform metadata from a trusted systemMedium to highTeams inside one orchestrated platform
Network locationMediumPrivate networks with one tenant each
Claim made by the senderLowOnly as a hint to be checked or overwritten
!
Overwrite, Do Not Append

If data arrives with a tenant label already attached by the sender, the gateway should replace it with the verified value, not add a second one. Leaving the sender’s label in place invites spoofing and confusing duplicates.

Baggage is a convenience, not a security tool

OpenTelemetry supports a mechanism called baggage, which carries small key and value pairs along with a request as it travels between services. It is tempting to put a tenant identifier there. Baggage is useful for passing context inside your own system, but it travels in ordinary request headers, may be forwarded to third parties, and can be forged by anyone who can send a request. Use it for convenience, such as letting downstream services label their telemetry, but never as proof of identity.

Identity for telemetry sent from inside a tenant’s own systems

Customers who send data to your platform usually run their own agents or collectors. Giving each customer a credential that is unique to them lets you identify the tenant reliably. Credentials should be easy to rotate and revoke, and should be scoped narrowly: the credential allows sending telemetry as that tenant and nothing else. Short-lived tokens, issued by an identity service and refreshed automatically, reduce the damage if one is stolen.

Within a single organisation, workloads often run in shared clusters, and identity can come from the platform: the namespace, the service account or a signed workload identity. Here the gateway reads trusted platform information and maps it to a tenant through a table. The table itself is critical, since an error in it sends data to the wrong tenant, so keep it under strict change control with review and testing.

Practice 1

Unique Credentials per Tenant

Never share a credential between tenants. Sharing makes it impossible to tell them apart or revoke one without the other.

Practice 2

Narrow Scope

A credential should permit sending as one tenant only, not reading, not administering and not sending as others.

Practice 3

Short Lifetimes

Tokens that expire quickly and renew automatically limit the usefulness of anything stolen.

Practice 4

Controlled Mapping Table

The table that maps identity to tenant is security-critical and needs review and version control.

6Architecture: Layers of a Multi-Tenant Pipeline

A well-formed design has clear layers, and each layer has a job in keeping tenants apart.

flowchart TD
  A[Applications and agents of many tenants] --> G[Edge gateway: authenticate and stamp tenant]
  G --> R[Routing layer: choose path by tenant]
  R --> P1[Pipeline for tenant group 1]
  R --> P2[Pipeline for tenant group 2]
  R --> P3[Default or quarantine pipeline]
  P1 --> E1[Exporter with tenant credentials]
  P2 --> E2[Exporter with tenant credentials]
  E1 --> S[Multi-tenant storage]
  E2 --> S
  P3 --> Q[Quarantine for investigation]
        
FIG 2 — Layers of a multi-tenant telemetry pipeline
Layer 1

Sources

Applications and agents produce telemetry. They may add their own labels, but these are treated as hints.

Layer 2

Edge Gateway

The first platform-controlled stop. It checks credentials, decides the tenant, and stamps data with a trusted label.

Layer 3

Routing and Processing

Data is sent down the path suited to its tenant, where redaction, sampling and limits are applied.

Layer 4

Export

Data leaves for storage with the tenant identity expressed in the way the storage expects, such as a header or a separate endpoint.

Layer 5

Storage and Query

The backend keeps tenants apart and checks who may read what when people query.

The most important principle in this layout is that trust is established once, early and by the platform. Everything after the edge gateway relies on the trusted label rather than on anything the sender said. Each later layer can then make decisions confidently, and an auditor can examine the gateway to see how identity is determined.

Simple Analogy

The edge gateway is like the reception desk of an office building. Visitors show identification once at the entrance and receive a badge. After that, the lifts and doors inside trust the badge instead of asking every visitor for papers again.

Agents near the tenant and gateways in the middle

A common arrangement uses two kinds of collector. Agents run close to the applications, perhaps inside the tenant’s own environment, and do light work such as collecting and forwarding. Gateways run centrally and do the platform work of authentication, stamping, routing and limits. Keeping the security-critical work in the gateway means that tenants cannot change or bypass it by altering their own agents.

It is tempting to let agents add the tenant label, since they know which application they serve. This can be allowed as a convenience, but the gateway must still verify and overwrite. If the label is accepted from the agent without verification, a compromised or misconfigured agent can impersonate another tenant.

TaskAgent near the tenantCentral gateway
Collect data from applicationsYesNo
Add hints such as environment nameYes, as hintsMay overwrite
Verify identityNo, it cannot be trusted toYes
Stamp the trusted tenant labelNoYes
Enforce limits and redactionOptional first passYes, as the authority
Route to storageNoYes
i
Design Hint

Treat everything that runs on the tenant’s side of the boundary as untrusted, even if your own team supplied it. The boundary is where trust begins, and the gateway is the checkpoint.

7The OpenTelemetry Building Blocks for Tenancy

The Collector offers a toolkit of components that, combined carefully, give you tenant isolation.

Names, options and availability change between versions, and some components live in the contributed set that must be included in your Collector build. Treat the list below as a map of ideas, and confirm details in the documentation for the version you run.

Block 1

Receiver Metadata Passing

Receivers can be told to keep incoming request headers and make them available inside the Collector, so later steps can use values such as a tenant header or a token.

Block 2

Authentication Extensions

Extensions that check tokens, certificates or passwords on incoming requests, rejecting senders who cannot prove who they are.

Block 3

Attribute and Transform Processors

Components that add, replace or remove labels, including setting a trusted tenant label and deleting any label the sender supplied.

Block 4

Routing Connector

A connector that sends data to different pipelines depending on a label or a header, with a default route for anything unmatched.

Block 5

Batch Processor with Metadata Keys

Batching can be done separately for each value of chosen metadata, such as a tenant header, so batches never mix tenants.

Block 6

Headers Setter Extension

An extension that sets headers on outgoing requests from information kept in the incoming context, so the tenant identity travels onwards to the backend.

Block 7

Memory Limiter

A guard that refuses data when memory use is high, protecting the Collector from overload by any source.

Block 8

Filter, Sampling and Redaction Processors

Components that drop unwanted data, keep a share of traces, and mask sensitive values, which can be applied differently for each tenant.

How the pieces fit together

1

The receiver accepts data and keeps metadata

It checks the sender through an authentication extension and retains the request headers.

2

A processor stamps the tenant label

The verified identity becomes a label on the data, replacing anything the sender claimed.

3

The routing connector picks a path

Data flows to the pipeline responsible for that tenant or group of tenants.

4

Per-tenant processing happens

Redaction, filtering, sampling and limits are applied according to that tenant’s rules.

5

The exporter sends data with tenant identity

The backend receives the data with the correct tenant header or credentials.

Avoiding common pitfalls with these components

Each building block has traps that catch people the first time. Keeping request headers in the context is useful but increases memory use, because the headers travel with the data through the pipeline, so retain only the headers you need. Batching per tenant metadata creates a separate batch for each distinct value, and with a very large number of tenants this can use a lot of memory, which is why the option has a limit on the number of distinct values it will track. Set that limit with care, since data for values beyond the limit may be handled differently from what you expect.

Routing connectors depend on matching rules, and a mistake in a rule can send data to the wrong place. Keep rules simple, test them with sample data, and always include a default route. Authentication extensions must be applied to every receiver that accepts outside data, including less obvious ones such as health or debugging endpoints, or an unauthenticated back door will exist.

Trap 1

Headers Kept Too Long

Keeping every header in the context wastes memory and may expose secrets. Keep only what is needed.

Trap 2

Unlimited Tenant Batches

Per-tenant batching uses memory for each tenant. Know the limit and what happens beyond it.

Trap 3

Unauthenticated Side Doors

Every receiver that accepts outside data needs authentication, not only the main one.

Trap 4

Untested Routing Rules

A small mistake in a rule misroutes data. Test each rule and keep a default.

8Stamping Tenant Identity at the Edge

The edge gateway is the most security-sensitive part of the design, so it deserves close attention.

sequenceDiagram
  participant C as Tenant client
  participant G as Edge gateway
  participant A as Authentication check
  participant T as Tenant table
  participant P as Next pipeline stage
  C->>G: Send telemetry with credentials
  G->>A: Verify credentials
  A-->>G: Verified identity
  G->>T: Look up tenant for identity
  T-->>G: Tenant identifier
  G->>G: Remove any sender-supplied tenant label
  G->>G: Add trusted tenant label
  G->>P: Forward stamped telemetry
        
FIG 3 — How the gateway turns credentials into a trusted tenant label

Step by step

When a request arrives, the receiver first checks the credentials through an authentication extension. Requests that fail the check are refused immediately, before any data is read. For requests that pass, the verified identity is used to look up the tenant, perhaps through a table kept in configuration or fetched from a trusted service. The gateway then removes any tenant label supplied by the sender and adds the verified one.

Where should the label go? Labels that describe the whole source, such as the tenant, belong on the resource, which is the top layer of the telemetry data model. Placing the tenant on the resource means that every item beneath it inherits the label, and it keeps the label out of the individual data points, where it would multiply the number of distinct series.

What to do with unknown or missing identity

Not every request will map cleanly. A credential may be valid but unknown to the tenant table. A trusted internal sender may omit its identity. The design must decide in advance what to do. The safest choices are to refuse data with no identity, or to send it to a clearly separate quarantine path where it is stored apart from any tenant and reviewed by the platform team. Silently assigning such data to a default tenant is dangerous, because it pollutes that tenant’s data and hides the problem.

i
Make Quarantine Visible

Whatever you do with unidentified data, count it and alert on it. A steady trickle of unidentified telemetry is a sign of a broken sender, a misconfiguration or an attempt to probe the system.

Keeping Secrets Out of Logs

Tokens and keys used for authentication must never appear in logs, error messages or telemetry. Make sure that any debugging output, and any component that prints headers, is configured to hide sensitive values.

9Routing Data by Tenant

Once data carries a trusted label, the platform can choose what happens to it.

A routing connector sits between pipelines. It looks at a property of incoming data, such as a resource attribute or a request header kept in the context, and sends the data to a matching pipeline. A default route catches data that matches nothing, which is exactly where unidentified or unexpected data should go. Routing can be done for traces, metrics and logs in the same way, with separate pipelines for each signal.

One pipeline per tenant, or per group?

The most obvious design is a separate pipeline for every tenant. It is easy to understand and gives each tenant its own processing rules and exporters. But the number of pipelines grows with the number of tenants, and each pipeline consumes memory and adds start-up time. With hundreds or thousands of tenants this does not scale, and configuration files become enormous.

A more scalable design groups tenants by what they need. All small tenants with standard rules share one pipeline, and the tenant label travels with the data to the exporter. Medium tenants with special rules share a second pipeline. Only the few large or demanding tenants get a pipeline of their own. This keeps configuration small while preserving the ability to treat important tenants differently.

DesignScales to many tenants?Per-tenant rules?Configuration size
Pipeline per tenantPoorlyFull freedomGrows with every tenant
Single shared pipeline, label travels with dataVery wellLimited, same rules for allSmall and constant
Pipelines per tier or groupWellPer groupGrows with the number of groups, not tenants
flowchart TD
  IN[Stamped telemetry] --> RC{Routing connector}
  RC -->|Large tenants| PL[Dedicated pipeline each]
  RC -->|Standard tenants| PS[Shared standard pipeline]
  RC -->|Regulated tenants| PR[Regional pipeline]
  RC -->|No match| PD[Default quarantine pipeline]
        
FIG 4 — Routing by tenant group instead of one pipeline per tenant
!
Do Not Forget the Default

A routing setup with no default route silently discards or misplaces data that fits no rule. Always define a default path, and make that path safe, visible and monitored.

Handling tenants that move between groups

Tenants change over time. A small tenant grows into a standard one, and a standard tenant signs a contract that requires regional storage. Routing based on a table that maps tenants to groups makes such moves a matter of editing the table, rather than rewriting pipelines. The table should be versioned, reviewed and applied in a controlled way, since an incorrect edit can send a tenant’s data to the wrong group.

When a tenant moves, think about what happens to data in flight and to history. New data starts flowing to the new group as soon as the table changes. Data already stored in the old group remains there, and queries must be able to find it, either by searching both groups for a period or by copying the history across. Decide in advance which approach you use, and tell the tenant what to expect.

1

Prepare the new group

Make sure the new pipeline and storage are ready and tested with sample data.

2

Change the mapping

Update the table so new data flows to the new group, using reviewed, version-controlled changes.

3

Keep history readable

Search both groups for a transition period, or copy history across, according to the agreement.

4

Retire the old path

Once history has aged out or been moved, remove the old route and data, and record the change.

10Per-Tenant Processing

Tenants differ in what they send, what they may keep and what they want removed.

Rule 1

Redaction

Some tenants require that certain fields, such as personal identifiers, are masked or removed before storage.

Rule 2

Filtering

Drop noisy or unwanted data, such as health checks, to save cost and clutter.

Rule 3

Sampling

Keep a controlled share of traces. Different tenants may buy or need different shares.

Rule 4

Enrichment

Add helpful labels such as region or environment, drawn from trusted tables rather than the sender.

Sampling with tenants in mind

Sampling decisions interact with tenancy in interesting ways. A fixed percentage is fair in proportion, but it gives a small tenant very few traces and a large one a flood. A limit on traces per second gives each tenant a predictable ceiling, which protects the platform but may leave a very large tenant with thin coverage. Many platforms combine approaches, with a base rate for all tenants and higher or lower rates according to tier. Whatever the policy, make it explicit and visible to tenants, so that nobody is surprised by what they can and cannot see.

Tail-based sampling, which decides after a trace is complete, needs all spans of a trace in one place. In a multi-tenant system it must also keep tenants apart: the decision memory for one tenant’s traces must never be shared with another’s. Routing by tenant before tail sampling, and then routing by trace identifier inside each group, keeps both needs satisfied.

i
Connects to Earlier Topics

Head samplers such as the rate-limiting sampler can run inside tenants’ own applications to cut volume at the source, while the platform applies tail-based policies in a stateful tier. A load balancing exporter in front of that tier keeps whole traces together. Tenancy adds one more dimension to the same design.

Rules must be scoped

A rule meant for one tenant must apply to that tenant only. Conditions such as “drop spans with this name” need an extra condition that checks the tenant label, or they will delete data belonging to everyone who uses the same name. This is a classic source of cross-tenant damage, and it is easy to make when configurations are edited by hand. Prefer structures where each tenant’s rules live in their own clearly named section, and use automated checks to catch rules that lack a tenant condition.

Letting tenants control some of their own rules

Tenants often want a say in what is kept: which services are sampled more heavily, which fields are masked, which noisy endpoints are dropped. A safe way to allow this is a self-service layer that produces rules in a restricted form. The platform checks each request against the tenant’s tier and limits, wraps the rule in a condition that restricts it to that tenant, and applies it through the same reviewed process as any other change. The tenant never edits the shared configuration directly.

Tenant requestPlatform checkResult
Drop a noisy endpointIs the rule scoped to this tenant only?Accepted, with a tenant condition added
Raise trace sampling rateIs it within the tier’s limits?Accepted or refused according to tier
Mask an additional fieldDoes it only remove data?Accepted, since stricter redaction is always safe
Add a label from another tenant’s dataDoes it reach outside the tenant’s own data?Refused
!
Never Let Tenants Edit Shared Files

Giving tenants direct write access to shared configuration, even with good intentions, makes one tenant’s mistake into everyone’s incident. Provide a controlled interface that generates safe, scoped rules instead.

11Exporting to Multi-Tenant Backends

The last step is telling the storage system whose data it is receiving.

Many observability backends have their own idea of tenancy. A common pattern is that the backend expects a header on each request that names the tenant, and it stores and queries data separately for each header value. Several popular open-source systems for traces, metrics and logs follow this approach, so your platform can often rely on the backend to keep tenants apart in storage, provided it tells the backend the right tenant every time.

Three ways to carry the identity to the backend

Way 1

Header from Context

The tenant is kept in the incoming context, and a headers extension copies it into a header on the outgoing request. Batches must be formed per tenant so that each request carries one tenant only.

Way 2

Exporter per Tenant Group

Each routed pipeline has its own exporter configured with a fixed header or credential. Simple and explicit, but it multiplies exporters as groups grow.

Way 3

Separate Endpoints

Different tenants are sent to different backend instances or regions. Strongest separation, used for silos or data residency.

!
Mixed Batches Are Dangerous

If a single outgoing request contains data from two tenants but carries only one tenant header, one tenant’s data lands in the other’s storage. Make sure batching is done separately for each tenant, using the batching option that groups by chosen metadata, before any export that depends on a per-request header.

Credentials for storage

The platform needs credentials to write to the backend. Keep these separate from anything tenants can influence, store them securely, and give the exporter only the permission to write, never to read. If different tenants require different credentials, keep them in per-group exporters, and rotate them on a schedule.

ApproachIsolation strengthOperational effortBest for
Shared storage with tenant headerMedium to high, relies on backendLowLarge numbers of small and medium tenants
Per-group exporters with fixed credentialsHighMediumTiers with different requirements
Separate backend instancesHighestHighRegulated or very large tenants

When the backend is unavailable for only one tenant

Backends sometimes fail for a single tenant, perhaps because that tenant has reached a storage limit or has been misconfigured on the backend side. If the exporter retries without limit, data for that tenant piles up, and if it shares a queue with others, their data waits too. The remedy is to separate queues by group, set sensible limits on retries and queue sizes, and make refusals visible both to operators and to the tenant concerned. Tenants should be told clearly when their data is being refused and why, so they can act.

Decide in advance what happens to data that cannot be delivered. Options include dropping it after a retry period, holding it in a limited disk queue, or sending it to a holding area. Dropping is simple but loses data. Holding is safer but costs resources and must be bounded. Whichever you choose, make the behaviour part of the tier’s promise, so that tenants know what to expect.

Option 1

Drop After Retries

Simple and cheap, but data is lost. Suitable for lower tiers where occasional loss is acceptable.

Option 2

Bounded Disk Queue

Holds data through short outages with a firm size limit, at the cost of storage and handling.

Option 3

Holding Area

Sends undeliverable data to a separate store for later replay, useful for higher tiers with strong promises.

12Resource Isolation and Noisy Neighbours

Keeping data apart is useless if one tenant can still bring the whole platform to its knees.

Where tenants compete

Shared 1

Network and Ingest Capacity

A tenant sending a very large volume can saturate the gateway’s ability to accept data.

Shared 2

Memory and Queues

Shared queues fill up when one tenant’s data is slow to export, and then everyone’s data waits behind it.

Shared 3

Storage and Series Count

A tenant that creates millions of distinct metric series strains storage and slows queries for all.

Shared 4

Query Capacity

Heavy queries from one tenant can slow dashboards and alerts for others.

Tools for fairness

The first tool is a limit on how much each tenant may send, expressed as a rate, as a volume per period, or as a number of distinct series. The Collector’s core components include a memory limiter that protects the process as a whole, but a limit per tenant often needs additional help. Some contributed components and many API gateways can enforce per-tenant limits, and some backends enforce limits themselves for each tenant header. Check what is available for your versions, and place the limit as early as possible, so that excess data is refused before it consumes resources.

The second tool is separation of queues. If every tenant shares one exporter queue, a slow backend for one tenant can block all. Batching and queuing per tenant group means that a problem in one group fills only that group’s queue. The third tool is separate pipelines for tenants of different sizes, so that a very large tenant cannot crowd out small ones within a shared pipeline.

1

Measure each tenant’s usage

Know the rate, volume and series count each tenant produces, so limits can be set sensibly.

2

Set limits per tenant or tier

Agree on a rate, volume and series ceiling for each tier, and write them into the service agreement.

3

Enforce early

Refuse excess data at the edge, with a clear signal so the sender can slow down or retry.

4

Separate queues and batches

Keep tenants or tenant groups from sharing a single queue that one slow destination could block.

5

Alert before the limit is reached

Warn tenants and operators as usage approaches a limit, so surprises are rare.

!
Cardinality Is a Weapon and an Accident

A single careless label with unbounded values, such as a user identifier on a metric, can create millions of series. Limit the number of series per tenant, and tell tenants clearly when a limit has been hit and which labels caused it.

13Data Protection: Redaction, Residency, Retention and Deletion

Different tenants owe different obligations to their own customers, and the platform must honour them.

Redaction and minimisation

Telemetry often accidentally contains personal or secret data: names in messages, tokens in headers, addresses in queries. The platform should provide redaction rules that remove or mask such values before storage, and these rules should be applied at the gateway so that sensitive data never reaches shared storage. Tenants with stricter needs can have stricter rules. Where possible, encourage tenants to avoid recording sensitive data in the first place, because removal after the fact is never perfect.

Data residency

Some tenants must keep their data in a certain country or region. A pooled design in one location cannot satisfy this. The usual answer is routing by tenant to regional pipelines and regional storage, so that data never leaves the permitted area. This also affects the gateway: if raw data crosses a border on its way to the region where it is processed, the rule may already be broken. Place gateways in each region, or route at the very edge.

Retention

Tenants may buy, or be required to have, different retention periods. In a pooled store this requires the backend to support separate retention per tenant, or requires separate storage for tenants with different periods. Be clear about what retention means: how long data stays searchable, and how long it stays on disk, which may differ.

Deletion and the right to be forgotten

When a tenant leaves, or when a person asks for their data to be erased, the platform must be able to delete it. In shared storage where many tenants’ data is mixed in the same files, deletion can be hard or even impossible at fine detail. Design for deletion at the tenant level from the start, choosing storage that can remove a tenant’s data entirely, and document what can and cannot be removed at finer levels.

ObligationTypical techniqueHard part
Mask sensitive fieldsRedaction rules at the gatewayFinding every place a sensitive value can hide
Keep data in a regionRegional gateways, pipelines and storageEnsuring nothing crosses a border in transit
Different retention periodsPer-tenant retention or separate storageBackends that cannot vary retention
Delete a tenant’s dataTenant-level storage separationMixed files in pooled storage
Encrypt dataEncryption in transit and at rest, possibly with per-tenant keysKey management at scale

Encryption and key management

Data should be encrypted while it travels, between tenants and the gateway and between every internal stage, and while it rests in storage. Encryption in transit relies on certificates, which must be issued correctly, renewed before they expire and checked on both sides. Encryption at rest is often handled by the storage system. Some tenants ask for their own keys, so that they can revoke access to their data by withdrawing a key. This is a powerful form of isolation, but it adds the burden of managing many keys safely, and it must be designed carefully so that a lost key does not mean lost data.

Keep keys separate from the data they protect, rotate them on a schedule, and limit who can use them. Record every use of a sensitive key. If the platform serves regulated tenants, expect auditors to ask for evidence of each of these practices.

1

Encrypt every hop

Use encrypted connections from tenants to the gateway and between all internal stages.

2

Encrypt storage

Protect stored data with encryption, using per-tenant keys for tenants who require them.

3

Separate and rotate keys

Keep keys apart from the data, rotate them regularly and restrict who may use them.

4

Audit key use

Record who used which key and when, for later review and for auditors.

14Access Control on the Query Side

Isolation must hold when people read data, not only when machines write it.

Writing data under the correct tenant is only half of the story. The other half is making sure that people can read only what they are allowed to read. When someone opens a dashboard, runs a search or sets an alert, the platform must attach their tenant, or the set of tenants they are allowed to see, to every query, and the backend must enforce it. If this check is done only in the user interface, anyone who calls the backend directly can bypass it.

Principles for safe querying

Principle 1

Enforce on the Server

The backend or a trusted proxy in front of it must enforce tenant access. The browser can never be trusted to do so.

Principle 2

Least Privilege

People get access only to the tenants and actions they need, such as read-only access to one team’s data.

Principle 3

Separate Roles

Distinguish viewers, editors and administrators, and keep platform operators separate from tenant users.

Principle 4

Audit Access

Keep records of who read which tenant’s data and when, especially for platform staff with wide access.

The platform operator’s special position

The people who run the platform can usually see everything, which is both necessary and risky. Reduce the risk by limiting how many people have this power, requiring approval and recording for its use, and giving most operators tools that show health and volume without revealing content. Tenants, especially regulated ones, will ask how operator access is controlled, and a clear answer builds trust.

Dashboards, alerts and shared settings

Dashboards and alert rules are also tenant data. A dashboard that queries across tenants, or an alert rule that reads another tenant’s series, is a leak waiting to happen. Keep dashboards and alert rules organised by tenant, give each tenant its own space, and review any shared or template dashboards to ensure they do not pull data from outside the viewer’s allowed set.

i
Test From the Outside

Verify query isolation by logging in as a user of one tenant and trying to read another’s data through every route: the interface, the programming interface, saved links and exports. The test must fail to find anything, in every route.

15Cost Attribution and Usage Metering

Sharing saves money only if everyone can see, and pay, their fair part.

Cost isolation begins with measurement. For each tenant, the platform should record how much data it sent, how much was stored and for how long, and how much was queried. Without this, shared costs are split by guesswork, and tenants have no reason to be careful. With it, you can charge tenants, show them their own usage, and set limits based on facts.

What to measure

MeasureWhy it mattersWhere to capture it
Items received per signalReflects ingest effortAt the gateway, after identity is stamped
Bytes receivedReflects network and storage cost better than countsAt the gateway or the export step
Items refused or droppedShows limit enforcement and billing disputesAt the point of refusal
Series count for metricsMain driver of metric storage costIn the backend or a counting step
Storage used and retentionDrives long-term costIn the backend
Query loadDrives query cost and fairnessAt the query layer

Taking care with the meter itself

Counting per tenant means adding a tenant label to the platform’s own metrics, and if the number of tenants is large, this itself creates many series. Keep the number of labels small, count at coarse granularity, and consider summarising per tier for dashboards while keeping detailed figures in a separate store used for billing. Counting must be done after identity is verified, so that spoofed labels cannot shift costs to another tenant.

Showback Before Chargeback

Many platforms first show tenants their usage and cost without charging, a practice sometimes called showback. This builds understanding and trust, surfaces mistakes in measurement, and lets tenants reduce waste before money is involved. Once the numbers are trusted, charging can follow.

!
Billing Disputes

If a tenant disputes a bill, you need evidence. Keep usage records for longer than the billing period, protect them from alteration, and be able to explain how each figure was measured.

Preventing cost surprises for tenants

The best bill is one that holds no surprises. Send tenants usage reports on a regular schedule and warnings when they approach limits. Show which services, signals and labels contribute most to their volume, so they can reduce waste. Many surprise bills come from a single change, such as a new label with many distinct values or a debug setting left on, and an alert on sudden growth catches these early, often before the tenant notices.

Report
Regular usage summaries per tenant
Warn
Alerts before limits are reached
Explain
Show which labels and services drive cost

Fair attribution also requires deciding what to do about shared costs that no tenant causes directly, such as the platform’s own monitoring or the cost of idle capacity kept in reserve. Some platforms spread these costs evenly, some in proportion to usage, and some absorb them as the cost of providing the service. Whatever is chosen, explain it clearly in advance.

16Monitoring the Platform Itself, per Tenant

A platform that promises isolation must be able to prove it is working.

The telemetry platform needs its own monitoring. Beyond the usual health figures such as memory, processor use and queue length, a multi-tenant platform needs views by tenant: how much each is sending, how much is being refused, how long their data takes to become searchable, and whether any limits are close. These views allow operators to spot a noisy neighbour early, answer questions from tenants, and verify that limits and routing behave as designed.

Signal 1

Ingest per Tenant

Volume received from each tenant over time, to spot sudden growth or silence.

Signal 2

Refusals per Tenant

Items rejected for authentication, limits or invalid data, which shows who is hitting limits.

Signal 3

Delay per Tenant

Time from arrival to being searchable, which shows whether one group is queuing behind another.

Signal 4

Unidentified Data

Amount of data landing in the default or quarantine path, a sign of broken senders or probing.

Service levels for tenants

Define what you promise: how quickly data is accepted, how long before it is searchable, how much availability to expect, and what happens when limits are exceeded. Different tiers may have different promises. Measure each promise from the tenant’s point of view, so that your reports match their experience. A promise that is not measured is only a hope.

1

Define the promise

State clearly what each tier gets, in terms a tenant can check.

2

Measure it per tenant

Record the relevant figures separately for each tenant or tier.

3

Alert on drift

Warn when a figure moves toward the promised limit, before it is broken.

4

Report openly

Share the results with tenants, especially after incidents, to build trust.

17Failure Modes and Anti-patterns

Most tenancy failures come from a small number of repeated mistakes.

ANTI-PATTERN-01 Avoid
Problem

Trusting a tenant label that the sender supplied, without verification.

Why It’s Harmful

Any sender can pose as any tenant, polluting data, consuming another tenant’s quota, or hiding its own activity.

Correct Approach

Verify identity at the edge, delete any sender-supplied tenant label, and stamp the trusted one.

ANTI-PATTERN-02 Avoid
Problem

Batching data from several tenants together before an export that carries a single tenant header.

Why It’s Harmful

Some tenants’ data is stored under another tenant’s name, which is a direct data leak.

Correct Approach

Batch separately by tenant, using the batching option that groups by chosen metadata.

ANTI-PATTERN-03 Avoid
Problem

Routing with no default path, or with a default that assigns unknown data to a real tenant.

Why It’s Harmful

Unmatched data is lost silently or pollutes a tenant’s records, and the problem stays hidden.

Correct Approach

Send unmatched data to a quarantine path, count it, alert on it and investigate.

ANTI-PATTERN-04 Avoid
Problem

Writing a processing rule for one tenant without a condition that restricts it to that tenant.

Why It’s Harmful

The rule affects everyone who matches it, silently deleting or changing other tenants’ data.

Correct Approach

Scope every rule by tenant, review changes, and test rules against sample data from several tenants.

ANTI-PATTERN-05 Avoid
Problem

Relying on the user interface alone to hide other tenants’ data.

Why It’s Harmful

Anyone who calls the backend directly bypasses the check and reads everything.

Correct Approach

Enforce tenant access on the server, in the backend or a trusted proxy, for every query.

ANTI-PATTERN-06 Avoid
Problem

Putting the tenant label on every metric data point instead of on the resource.

Why It’s Harmful

It adds to the number of distinct series, wastes memory and makes data harder to group.

Correct Approach

Place the tenant on the resource and keep data-point labels few and bounded.

18Scaling a Multi-Tenant Design

A design that works for ten tenants may break at ten thousand.

Scale in layers

The edge gateway scales by adding more identical collectors behind an ordinary load balancer, because authentication and stamping are stateless. The routing and processing layer can scale the same way where its work is stateless. The stateful parts, such as tail sampling, need data for each trace to arrive together, which is the job of a load balancing exporter that routes by trace identifier. In a multi-tenant system, combine this with tenant routing so that each stateful group serves a set of tenants and each trace stays whole inside its group.

Sharding tenants across fleets

Beyond a certain size, one fleet of collectors becomes too large or too risky. A single bad configuration affects everyone, and a single failure domain covers all tenants. Splitting tenants across several independent fleets, sometimes called cells, limits the blast radius of any one problem. Each cell serves a subset of tenants, and a thin routing layer in front of the cells sends each tenant to its home cell. A problem in one cell leaves the others untouched.

flowchart TD
  T[All tenants] --> FR[Front routing layer]
  FR --> C1[Cell 1: gateway, processing, storage]
  FR --> C2[Cell 2: gateway, processing, storage]
  FR --> C3[Cell 3: gateway, processing, storage]
        
FIG 5 — Splitting tenants into independent cells to limit the blast radius

Moving a tenant

Tenants grow, and a tenant that began small may need a bigger tier or its own cell. Plan the move in advance. Moving data while keeping history searchable and keeping live traffic flowing needs a routine: start sending new data to the new home, keep the old home readable for a period, and then retire it. Practising this on a small tenant before you need it for a large one is wise.

Stateless
Scale by adding identical collectors
Stateful
Scale by routing related data together
Cells
Limit the blast radius of any failure

Capacity planning with tenants in mind

Capacity for a shared platform is planned differently from capacity for a single application. Total volume matters, but so does the shape of the distribution: a platform where a few tenants produce most of the data behaves very differently from one where volume is spread evenly. Study the distribution, and plan for the largest tenants separately, since their growth can consume capacity faster than the average suggests. Keep headroom for sudden growth from any tenant, and review the figures regularly as tenants join, grow and leave.

Plan also for the busiest moments. Many tenants share the same patterns, such as a morning rush or an end-of-month report, so their peaks may coincide. A platform sized only for the average will struggle at these moments, so size for the combined peak and test with realistic bursts.

Planning questionWhy it matters
How is volume distributed across tenants?A few giants need separate planning from many small tenants
When do tenant peaks coincide?Combined peaks, not averages, decide required capacity
How fast is each tier growing?Growth rate decides how often to add capacity
What headroom is kept for surprises?Sudden growth from one tenant should not cause loss for others

19Testing and Verifying Isolation

Isolation that has never been tested is a belief, not a fact.

Test 1

Spoofing Test

Send data claiming to belong to another tenant and confirm that the gateway overwrites the claim or rejects the request.

Test 2

Cross-read Test

As a user of one tenant, try to read another’s data through every route, and confirm that nothing is returned.

Test 3

Noisy Neighbour Test

Flood the platform from one test tenant and confirm that others see no loss or delay beyond the agreed limits.

Test 4

Routing Test

Send sample data for each tenant group and verify that it arrives in the correct pipeline, with a default for unmatched data.

Test 5

Redaction Test

Send data containing known sensitive values and confirm that none appears in storage.

Test 6

Deletion Test

Remove a test tenant and confirm that all of its data, in all stores, is gone.

Canary tenants

Create a few synthetic tenants that exist only for testing. They continuously send known data, and automated checks read it back, verify that it arrived complete under the right tenant, and confirm that other tenants’ canary data is not visible. A failing canary check is an early warning that something in the platform has changed for the worse, even before any real tenant is affected.

Test configuration changes before they go live

Most isolation failures begin with a configuration change. Run each change in a test environment with sample data from several tenants, and compare the output with expectations. Automated checks can scan configurations for known dangers: rules without tenant conditions, pipelines without a default route, exporters with fixed credentials in the wrong group, and logging of sensitive headers.

i
Make Tests Routine

Run isolation tests on every release and on a regular schedule in production using canary tenants. Isolation tends to erode through small changes, and routine tests catch the erosion early.

20Operating the Platform: Onboarding, Change and Governance

Technology alone does not deliver isolation. Routines and responsibilities do.

Onboarding a tenant

1

Agree the tier and limits

Decide the tenant’s rate, volume, series and retention limits, and any special obligations such as location.

2

Create identity and credentials

Issue credentials and add the tenant to the trusted table that maps identity to tenant.

3

Assign routing and rules

Place the tenant in the right group, with its redaction, sampling and filtering rules.

4

Verify with test data

Send sample data, confirm it arrives correctly and cannot be seen by others.

5

Go live and watch

Enable real traffic, and watch usage and refusals closely during the first days.

Offboarding

When a tenant leaves, disable credentials first, so no new data arrives. Then remove routing and rules, delete or return stored data according to the agreement, remove the tenant from billing and dashboards, and record that the work was done. Forgotten tenants, with live credentials and stored data, are a quiet source of risk.

Managing configuration safely

Keep tenant configuration in version control, generate it from a single source of truth where possible, and require review for changes. Automated generation reduces the risk of the copy-and-paste mistakes that cause cross-tenant effects. Give tenants self-service where safe, such as choosing their own sampling within limits, while keeping identity, routing and limits under platform control.

AreaOwned by the platformMay be delegated to tenants
Identity and credentialsYesRotation of their own credentials
Routing and limitsYesRequests to change tier
Redaction rulesBaseline rulesAdditional stricter rules
SamplingAllowed rangeChoice within the range
Dashboards and alertsTemplates and safety checksTheir own content

21Real-World Scenarios

A few typical situations show how the choices fit together.

A Software-as-a-Service Company Observing Its Customers

The company runs one product for thousands of customers and wants per-customer insight for support. It stamps each request with a customer label derived from authentication, places it on the resource, and uses a shared pool with a tenant header to the backend. Large customers with strict contracts get dedicated pipelines. Usage is metered per customer and shown in a dashboard for the support team.

An Internal Platform Team Serving Product Teams

A central team offers observability as a service to dozens of product teams. Tenants are derived from the platform namespace through a trusted table. Each team gets a limit on volume and series, a showback report, and its own dashboards. The platform uses a routing connector with a few groups instead of a pipeline per team.

A Managed Service Provider with Regulated Clients

A provider monitors systems for banks and hospitals. Each client must keep data in its own country and have it deleted on exit. The provider runs regional gateways and regional storage, routes by client, applies strict redaction, and keeps full audit logs of operator access. Because separation matters more than cost, the largest clients get silos.

A Company Separating Environments

A company treats production, staging and development as tenants so that test data never mixes with production data and test floods cannot affect production monitoring. A simple routing rule and separate limits achieve this with little extra cost.

graph LR
  A[Small tenants] --> P[Shared pool with limits]
  B[Growing tenants] --> M[Shared pool with higher tier]
  C[Large or regulated tenants] --> S[Dedicated pipeline or silo]
  P --> U[Usage metering and reports]
  M --> U
  S --> U
        
FIG 6 — Tiered tenancy: shared for most, dedicated for a few, metered for all

What the scenarios share

In every case, identity was established by the platform, routing used groups rather than one pipeline per tenant, limits protected everyone, and usage was measured per tenant. Where problems occurred in other organisations, one of these four foundations was missing.

A Team That Learned About Mixed Batches

A platform team exported to a backend using a single tenant header, but their batching combined data from several customers. For weeks, a small share of one customer’s data appeared in another’s searches. The problem was found only when a customer reported unfamiliar service names in their dashboard. Switching to per-tenant batching fixed it, and a cross-read test now runs on every release to prevent a repeat.

A Team That Tested With Canary Tenants

Another team created three synthetic tenants that send known data every minute, and an automated check reads it back and confirms that no tenant sees another’s data. When a configuration change accidentally removed a tenant condition from a filtering rule, the canary check failed within minutes, and the change was reverted before any real customer was affected.

22Best Practices and Common Mistakes

A checklist in plain language, drawn from everything above.

1

Define what a tenant is and what you promise

Write down the isolation dimensions you guarantee, and for which tiers.

2

Derive identity from verified credentials

Overwrite sender labels, and treat baggage and claims as hints only.

3

Place the tenant on the resource

Keep data-point labels few and bounded to protect storage.

4

Route by group and always define a default

Use tiers instead of a pipeline per tenant, and send the unknown to quarantine.

5

Batch and queue per tenant or group

Never mix tenants in a request that carries one tenant identity.

6

Limit early, measure always

Apply rate, volume and series limits at the edge, and meter usage per tenant.

7

Test isolation continuously

Use spoofing, cross-read, noisy-neighbour and deletion tests, with canary tenants in production.

!
Frequent Mistakes

The usual errors are trusting sender labels, mixing tenants in batches, forgetting the default route, writing unscoped rules, enforcing access only in the interface, and never testing isolation after the first launch. Each has caused real data exposure in real organisations.

A sensible order of work

Begin by writing the promises and choosing the model. Build the edge gateway and identity stamping first, since everything depends on it. Add routing with a default path, then export with tenant identity. Add limits and metering. Add redaction and per-tenant rules. Build monitoring and tests, and only then onboard real tenants, starting with a friendly few. Following this order gives you a trustworthy foundation before the stakes become high.

23Frequently Asked Questions

Short, plain answers to the questions people ask most often.

Q1Does OpenTelemetry have a built-in tenant feature?

No single built-in switch exists. Multi-tenancy is assembled from attributes, authentication, processors, connectors, exporters and tenant-aware backends. Check the current documentation for any new conventions.

Q2Where should the tenant identifier be stored in the data?

On the resource, the top layer of the telemetry model, so it applies to everything beneath it without multiplying labels on individual data points.

Q3Can I trust a tenant header sent by the client?

Not as proof. Use it at most as a hint, verify identity from credentials at the edge, and overwrite the header with the verified value.

Q4Should I create one pipeline per tenant?

Only for a few large or special tenants. For many tenants, route by group and let the tenant label travel with the data, to keep configuration and memory use manageable.

Q5How do I stop one tenant from overwhelming others?

Apply limits on rate, volume and series count per tenant at the edge, separate queues and batches by group, and use pipelines of their own for the largest tenants.

Q6How can I keep tenant data from mixing in batches?

Use the batching option that groups data by chosen metadata such as the tenant header, so each batch and each outgoing request contains one tenant only.

Q7What should happen to data with no tenant?

Refuse it or send it to a clearly separate quarantine path. Count it, alert on it and investigate, rather than assigning it to a default tenant.

Q8Is baggage safe for carrying tenant identity?

It is convenient inside your own system but can be forged and may leak to third parties, so it must not be used as proof of identity or for security decisions.

Q9How do I handle data residency requirements?

Route tenants to regional pipelines and regional storage, and place gateways in each region so data does not cross borders before it is routed.

Q10How do I delete a tenant’s data?

Choose storage that can remove a tenant’s data entirely, document what can be removed at finer detail, and test deletion with a synthetic tenant before it is needed.

Q11How should I charge tenants fairly?

Meter volume, series and storage per tenant after identity is verified, begin with showback reports, and keep the records long enough to settle disputes.

Q12Do I need a separate backend for each tenant?

Not usually. Many backends support a tenant header and separate storage logically. Dedicated backends are for regulated or very large tenants who need the strongest separation.

Q13How do I know isolation is really working?

Test it routinely with spoofing, cross-read, noisy-neighbour and deletion tests, and run canary tenants in production whose data is read back and checked automatically.

Q14How does this relate to sampling and load balancing?

Tenant routing decides which group handles the data, sampling controls how much is kept, and a load balancing exporter keeps whole traces together inside each stateful group. They work together as stages of one design.

24Summary and Key Takeaways

Multi-tenant telemetry isolation lets many tenants share one observability platform without sharing their data, their performance or their bills. OpenTelemetry provides no single tenant switch, so isolation is built from ordinary parts: verified identity stamped at an edge gateway, a tenant label on the resource, routing by group with a safe default, per-tenant processing, exporters that carry tenant identity to the storage, limits that protect the platform, and metering that makes costs visible.

The strength of the whole design is the strength of its weakest gap. A single trusted-but-forged label, a single mixed batch, a single missing default route or a single query without a server-side check can undo everything else. Define what you promise, build trust once and early at the edge, scope every rule by tenant, and test isolation again and again, and a shared platform can be both economical and safe.

Key Takeaways

  • Isolation has many dimensions — data, performance, configuration, access, cost and compliance each need their own protection.
  • Tenancy is assembled, not switched on — OpenTelemetry offers building blocks, and the design determines how well tenants are kept apart.
  • Verify identity at the edge — derive the tenant from authentication, overwrite sender labels, and treat claims and baggage as hints.
  • Route by group with a safe default — avoid a pipeline per tenant, and send unmatched data to a monitored quarantine path.
  • Never mix tenants in a batch — group by tenant before any export that carries a single tenant identity.
  • Protect the platform from noisy neighbours — limit rate, volume and series per tenant early, and separate queues by group.
  • Measure, test and govern — meter usage per tenant, test isolation continuously with canary tenants, and manage configuration through reviewed, scoped changes.