What Is Multi-Tenant Telemetry Isolation in OpenTelemetry?
Multi-tenant telemetry isolation is the practice of letting many customers, teams or business units share one observability pipeline while keeping their data, performance, costs and access separate. This guide explains the ideas, the OpenTelemetry building blocks, and the mistakes to avoid.
Imagine a large apartment building with a single mailroom. Hundreds of residents receive letters and parcels every day, and one team of mail clerks sorts everything. A good mailroom guarantees three things. Every resident gets only their own mail. One resident who receives a thousand parcels at once cannot block everyone else’s deliveries. And the building manager can tell how much work each resident causes, so shared costs can be split fairly. Multi-tenant telemetry isolation asks the same three questions about monitoring data: who may see it, who may slow others down, and who pays for what.
1Why Multi-Tenancy Exists
Before talking about isolation, we need to understand why anyone shares a telemetry system in the first place.
What a tenant is
A tenant is a group that owns a set of data and expects it to be kept apart from other groups. In a company that sells software to other companies, a tenant is usually a customer. In a large organisation, a tenant might be a product team, a department or a business unit. In a platform team’s view, a tenant can even be an environment such as production or testing. The word is flexible, and the first job in any design is to decide exactly what it means in your setting.
Telemetry is the stream of traces, metrics and logs that systems produce about themselves. When many tenants produce telemetry, there are two simple ways to handle it. You can give every tenant a completely separate monitoring system, or you can let them share one. Separate systems are easy to understand and strongly separated, but they are expensive and slow to manage. A shared system is cheaper and easier to operate as one platform, but it demands careful design so that tenants do not harm or spy on each other.
Think of the difference between everyone owning a private swimming pool and a community swimming pool with marked lanes. Private pools give perfect privacy at a huge cost. A shared pool is far cheaper, but it needs lane ropes, rules and lifeguards so that one swimmer cannot splash everyone else out of the water.
Why organisations choose to share
Cost
Running one platform is far cheaper than running hundreds, because hardware, people and licences are shared.
Consistency
One platform means one set of standards, one upgrade routine and one place to fix problems.
Speed of Onboarding
Adding a tenant to a shared platform takes minutes, not weeks of building new infrastructure.
Platform Insight
A central team can see patterns across tenants, such as which kinds of workloads cause trouble, and improve the system for all.
Sharing is only safe when isolation is designed in from the start. Retrofitting isolation into a platform that was built as one big pool is possible, but it is painful, risky and often incomplete.
Who the stakeholders are
A multi-tenant platform serves several groups of people, and each of them cares about different things. Tenants want their data kept private, their dashboards fast and their bills fair. Platform engineers want a system that is simple to operate and does not wake them up at night. Security teams want clear evidence that data cannot leak. Finance teams want costs they can explain. Legal and compliance teams want proof that obligations about location and retention are met. A good design keeps all five groups in mind from the first day, because each can veto the platform if its needs are ignored.
| Stakeholder | Main question | What they need to see |
|---|---|---|
| Tenant | Is my data private and my service reliable? | Clear promises, usage reports, and fast, correct dashboards |
| Platform engineer | Can I run this safely and simply? | Small configuration, good monitoring and tested changes |
| Security team | Can one tenant reach another’s data? | Verified identity, server-side access checks and audit records |
| Finance team | Who caused which cost? | Per-tenant metering that can be explained and defended |
| Compliance team | Are legal obligations being met? | Evidence of location, retention, redaction and deletion |
2Core Concepts and Vocabulary
A shared vocabulary makes the rest of the guide much easier to follow.
Tenant
A customer, team or unit that owns data and expects it to be separate from others.
Isolation
The set of guarantees that keep one tenant’s data, performance and costs separate from another’s.
Noisy Neighbour
A tenant that uses so much of a shared resource that others suffer. The classic risk of sharing.
Tenant Identity
The label that says which tenant a piece of telemetry belongs to, such as an identifier carried as an attribute or a header.
Resource Attribute
A label attached to the source of telemetry, describing things such as the service, host or, in our case, the tenant.
Gateway Collector
A central Collector that receives telemetry from many sources, applies common rules and sends it onwards.
Routing
Choosing the destination or processing path for data according to a property such as its tenant.
Quota
A limit on how much a tenant may send or store, in volume, rate or number of distinct series.
An important fact about OpenTelemetry
OpenTelemetry is a set of standards and tools for producing and moving telemetry. It does not, on its own, define a built-in concept of tenancy. There is no single standard switch that says “this data belongs to tenant A and must never mix with tenant B”. Instead, multi-tenancy is something you build from the ordinary parts: attributes that carry identity, authentication that proves identity, processors and connectors that route and shape data, and backends that store data separately. Conventions for naming a tenant attribute also vary, so check the current guidance and choose a name your whole platform will use consistently.
Because tenancy is assembled rather than switched on, the quality of isolation depends entirely on how carefully the parts are assembled. Every gap between two parts is a place where data from one tenant could leak to another.
Telemetry has three signals, and tenancy touches each differently
Traces, metrics and logs behave differently, and so do their tenancy problems. Traces are made of spans linked together, so keeping a trace whole matters, and a single trace should never contain spans from two tenants. Metrics are numbers over time grouped into series, so the main danger is the sheer number of series a tenant can create. Logs are text with attributes, so the main danger is sensitive content hiding inside the message. A design that handles one signal well may handle another poorly, so review each signal separately.
Traces
Keep every span of a trace within one tenant and one processing group. Sampling decisions must never mix tenants.
Metrics
Limit the number of distinct series per tenant, and keep tenant labels on the resource rather than on every point.
Logs
Scan and redact message text and attributes, since logs are the signal most likely to contain personal data.
Linking signals without leaking across tenants
Users like to jump from a metric spike to a related trace to the logs of one request. This works because the signals share identifiers. In a multi-tenant system, the same tenant label must be present and trusted on all three signals, or a jump from one to another could cross a boundary. Treat the tenant label as part of the identity of every item, and make sure that links and lookups always include it.
Use exactly the same name and format for the tenant label across traces, metrics and logs, and across every environment. Inconsistent naming is a quiet cause of data that falls through routing rules or appears under the wrong tenant.
3What Isolation Actually Means
Isolation is not one thing. It is a bundle of separate promises, and a design can keep some and break others.
| Dimension | The promise | What failure looks like |
|---|---|---|
| Data isolation | A tenant can see only its own telemetry | A customer sees another customer’s traces or logs |
| Performance isolation | One tenant’s load cannot degrade others | A burst from one tenant causes delays or drops for everyone |
| Configuration isolation | Rules for one tenant do not alter another’s | A sampling change for one customer silently changes another’s data |
| Access isolation | Only authorised people reach a tenant’s data and settings | An engineer from one team edits another team’s dashboards |
| Cost isolation | Each tenant’s usage is measured and charged correctly | One tenant’s spike appears on another’s bill |
| Compliance isolation | Each tenant’s legal needs, such as location and retention, are respected | European data stored in the wrong region, or kept longer than agreed |
Why the dimensions must be considered separately
It is easy to believe that adding a tenant label to everything solves the whole problem. A label gives you a way to separate data when you query it, which helps with data isolation, but it does nothing for performance. A single noisy tenant can still overwhelm a shared queue even if every item carries a perfect label. Similarly, strong performance limits do nothing to stop someone from reading the wrong data. A serious design looks at each dimension and decides how it will be protected.
A hotel needs locks on doors, soundproof walls, separate bills and fire exits. A hotel that has excellent locks but thin walls keeps your belongings safe and your sleep poor. Each kind of protection solves a different problem, and guests expect all of them.
Threats that isolation must withstand
It helps to think like an attacker, or like a careless engineer, and list what could go wrong. A tenant might try to read another tenant’s data by guessing identifiers. A tenant might try to impersonate another by copying its label. A tenant might flood the platform on purpose to cause harm, or by accident through a bug. A platform operator might look at data they should not see. A configuration change might accidentally expose data. A shared cache or shared file might hold data from several tenants and return the wrong one. Each of these threats deserves a specific defence.
| Threat | Likely cause | Main defence |
|---|---|---|
| Impersonation | Trusting sender-supplied labels | Verify identity at the edge and overwrite labels |
| Cross-tenant reading | Missing server-side access checks | Enforce tenant access in the backend or trusted proxy |
| Flooding | Bug or attack from one tenant | Per-tenant limits and separate queues |
| Operator misuse | Wide access without oversight | Least privilege, approval and audit records |
| Accidental exposure | Unscoped rules or mixed batches | Reviewed, tested, tenant-scoped configuration |
| Shared cache leak | Cache keys that omit the tenant | Include the tenant in every cache key and test for it |
Query caches, lookup tables and result stores are shared by design. If any of them is keyed without the tenant, one tenant’s answer can be served to another. Review every cache and every shared store for a tenant component in its key.
4Tenancy Models: Silo, Pool and Bridge
Every multi-tenant design sits somewhere on a spectrum between full separation and full sharing.
Silo
Each tenant has its own dedicated collectors and storage. Separation is strongest, and cost is highest.
Pool
All tenants share the same collectors and storage, and data is separated by labels and rules. Cost is lowest, and the design must be careful.
Bridge
A mixture: some parts are shared, such as collectors, while others are separate, such as storage. Many real systems end up here.
flowchart LR
subgraph Silo
A1[Tenant A data] --> CA[Collector A] --> SA[Storage A]
B1[Tenant B data] --> CB[Collector B] --> SB[Storage B]
end
subgraph Pool
A2[Tenant A data] --> CP[Shared collector]
B2[Tenant B data] --> CP
CP --> SP[Shared storage with tenant labels]
end
Advantages of Sharing (Pool)
- Much lower cost, because hardware and effort are shared.
- Quick onboarding of new tenants through configuration alone.
- One platform to upgrade, secure and monitor.
- Better use of resources, since quiet tenants make room for busy ones.
- A single place to apply common rules such as redaction and limits.
Disadvantages / Trade-offs of Sharing
- A single mistake can expose one tenant’s data to another.
- Noisy neighbours can degrade the whole platform.
- Per-tenant legal needs are harder to satisfy in shared storage.
- Deleting one tenant’s data cleanly can be difficult.
- Large tenants may demand stronger guarantees than a pool can give.
Choosing a model
The right choice depends on the strength of the guarantees you owe and the money you can spend. A platform serving internal teams that trust each other may happily use a pool. A service holding regulated data for large customers may need silos for the largest of them. Many organisations adopt tiers: a shared pool for small tenants, with dedicated pipelines or storage for the few tenants who need or pay for stronger separation.
| Question | Silo | Pool | Bridge |
|---|---|---|---|
| Strength of separation | Strongest | Depends on configuration | Strong where it matters |
| Cost per tenant | High | Low | Medium |
| Effort to add a tenant | High | Low | Medium |
| Noisy neighbour risk | None between tenants | Real and must be managed | Reduced for separated parts |
| Operational complexity | Many stacks to run | One stack, complex rules | Both kinds of complexity |
Tiering in practice
Most successful platforms define a small number of tiers, rather than negotiating a unique design with every tenant. A basic tier might use the shared pool with modest limits and standard retention. A standard tier might use the shared pool with higher limits and longer retention. A premium tier might use a dedicated pipeline, or even dedicated storage, with strict guarantees. Defining tiers in advance makes the platform easier to explain, price and operate, and it gives tenants a clear path to upgrade as their needs grow.
Basic
Shared pool, modest limits, standard retention, shared support.
Standard
Shared pool with higher limits, longer retention and tenant-specific rules.
Premium
Dedicated pipeline or storage, strong guarantees and priority handling.
Regulated
Regional, separated deployment with strict redaction, audit records and deletion on exit.
Whichever tiers you choose, write them down in plain language, including what is promised about data separation, limits, retention, response times and deletion. Tenants will compare tiers carefully, and unclear promises lead to disputes later.
5Where Tenant Identity Comes From
Every isolation mechanism depends on knowing, reliably, which tenant a piece of telemetry belongs to.
The golden rule
If the label that names a tenant is supplied by the sender and trusted without checking, any sender can pretend to be any tenant. A careless or malicious client could stamp its data with another tenant’s identifier, filling that tenant’s storage with junk or causing its quota to run out. The safe design is for the platform to decide the tenant identity, based on something it has verified, such as the credentials presented when the data arrived.
Common sources of identity
Authentication Credentials
A token or certificate presented by the sender is checked at the edge, and the tenant is looked up from it. This is the most trustworthy source.
Network Location
Telemetry arriving from a particular network or cluster is assigned to a tenant. Useful inside a company, weaker across the internet.
Platform Metadata
Information from the orchestration platform, such as the namespace a workload runs in, mapped to a tenant by a trusted table.
Sender Claims
A label or header set by the sender. Acceptable only as a hint that is verified against a stronger source, never as the final answer.
| Source | Trust level | Typical use |
|---|---|---|
| Verified credentials at the edge | High | Customers sending data from outside |
| Platform metadata from a trusted system | Medium to high | Teams inside one orchestrated platform |
| Network location | Medium | Private networks with one tenant each |
| Claim made by the sender | Low | Only as a hint to be checked or overwritten |
If data arrives with a tenant label already attached by the sender, the gateway should replace it with the verified value, not add a second one. Leaving the sender’s label in place invites spoofing and confusing duplicates.
Baggage is a convenience, not a security tool
OpenTelemetry supports a mechanism called baggage, which carries small key and value pairs along with a request as it travels between services. It is tempting to put a tenant identifier there. Baggage is useful for passing context inside your own system, but it travels in ordinary request headers, may be forwarded to third parties, and can be forged by anyone who can send a request. Use it for convenience, such as letting downstream services label their telemetry, but never as proof of identity.
Identity for telemetry sent from inside a tenant’s own systems
Customers who send data to your platform usually run their own agents or collectors. Giving each customer a credential that is unique to them lets you identify the tenant reliably. Credentials should be easy to rotate and revoke, and should be scoped narrowly: the credential allows sending telemetry as that tenant and nothing else. Short-lived tokens, issued by an identity service and refreshed automatically, reduce the damage if one is stolen.
Within a single organisation, workloads often run in shared clusters, and identity can come from the platform: the namespace, the service account or a signed workload identity. Here the gateway reads trusted platform information and maps it to a tenant through a table. The table itself is critical, since an error in it sends data to the wrong tenant, so keep it under strict change control with review and testing.
Unique Credentials per Tenant
Never share a credential between tenants. Sharing makes it impossible to tell them apart or revoke one without the other.
Narrow Scope
A credential should permit sending as one tenant only, not reading, not administering and not sending as others.
Short Lifetimes
Tokens that expire quickly and renew automatically limit the usefulness of anything stolen.
Controlled Mapping Table
The table that maps identity to tenant is security-critical and needs review and version control.
6Architecture: Layers of a Multi-Tenant Pipeline
A well-formed design has clear layers, and each layer has a job in keeping tenants apart.
flowchart TD
A[Applications and agents of many tenants] --> G[Edge gateway: authenticate and stamp tenant]
G --> R[Routing layer: choose path by tenant]
R --> P1[Pipeline for tenant group 1]
R --> P2[Pipeline for tenant group 2]
R --> P3[Default or quarantine pipeline]
P1 --> E1[Exporter with tenant credentials]
P2 --> E2[Exporter with tenant credentials]
E1 --> S[Multi-tenant storage]
E2 --> S
P3 --> Q[Quarantine for investigation]
Sources
Applications and agents produce telemetry. They may add their own labels, but these are treated as hints.
Edge Gateway
The first platform-controlled stop. It checks credentials, decides the tenant, and stamps data with a trusted label.
Routing and Processing
Data is sent down the path suited to its tenant, where redaction, sampling and limits are applied.
Export
Data leaves for storage with the tenant identity expressed in the way the storage expects, such as a header or a separate endpoint.
Storage and Query
The backend keeps tenants apart and checks who may read what when people query.
The most important principle in this layout is that trust is established once, early and by the platform. Everything after the edge gateway relies on the trusted label rather than on anything the sender said. Each later layer can then make decisions confidently, and an auditor can examine the gateway to see how identity is determined.
The edge gateway is like the reception desk of an office building. Visitors show identification once at the entrance and receive a badge. After that, the lifts and doors inside trust the badge instead of asking every visitor for papers again.
Agents near the tenant and gateways in the middle
A common arrangement uses two kinds of collector. Agents run close to the applications, perhaps inside the tenant’s own environment, and do light work such as collecting and forwarding. Gateways run centrally and do the platform work of authentication, stamping, routing and limits. Keeping the security-critical work in the gateway means that tenants cannot change or bypass it by altering their own agents.
It is tempting to let agents add the tenant label, since they know which application they serve. This can be allowed as a convenience, but the gateway must still verify and overwrite. If the label is accepted from the agent without verification, a compromised or misconfigured agent can impersonate another tenant.
| Task | Agent near the tenant | Central gateway |
|---|---|---|
| Collect data from applications | Yes | No |
| Add hints such as environment name | Yes, as hints | May overwrite |
| Verify identity | No, it cannot be trusted to | Yes |
| Stamp the trusted tenant label | No | Yes |
| Enforce limits and redaction | Optional first pass | Yes, as the authority |
| Route to storage | No | Yes |
Treat everything that runs on the tenant’s side of the boundary as untrusted, even if your own team supplied it. The boundary is where trust begins, and the gateway is the checkpoint.
7The OpenTelemetry Building Blocks for Tenancy
The Collector offers a toolkit of components that, combined carefully, give you tenant isolation.
Names, options and availability change between versions, and some components live in the contributed set that must be included in your Collector build. Treat the list below as a map of ideas, and confirm details in the documentation for the version you run.
Receiver Metadata Passing
Receivers can be told to keep incoming request headers and make them available inside the Collector, so later steps can use values such as a tenant header or a token.
Authentication Extensions
Extensions that check tokens, certificates or passwords on incoming requests, rejecting senders who cannot prove who they are.
Attribute and Transform Processors
Components that add, replace or remove labels, including setting a trusted tenant label and deleting any label the sender supplied.
Routing Connector
A connector that sends data to different pipelines depending on a label or a header, with a default route for anything unmatched.
Batch Processor with Metadata Keys
Batching can be done separately for each value of chosen metadata, such as a tenant header, so batches never mix tenants.
Headers Setter Extension
An extension that sets headers on outgoing requests from information kept in the incoming context, so the tenant identity travels onwards to the backend.
Memory Limiter
A guard that refuses data when memory use is high, protecting the Collector from overload by any source.
Filter, Sampling and Redaction Processors
Components that drop unwanted data, keep a share of traces, and mask sensitive values, which can be applied differently for each tenant.
How the pieces fit together
The receiver accepts data and keeps metadata
It checks the sender through an authentication extension and retains the request headers.
A processor stamps the tenant label
The verified identity becomes a label on the data, replacing anything the sender claimed.
The routing connector picks a path
Data flows to the pipeline responsible for that tenant or group of tenants.
Per-tenant processing happens
Redaction, filtering, sampling and limits are applied according to that tenant’s rules.
The exporter sends data with tenant identity
The backend receives the data with the correct tenant header or credentials.
Avoiding common pitfalls with these components
Each building block has traps that catch people the first time. Keeping request headers in the context is useful but increases memory use, because the headers travel with the data through the pipeline, so retain only the headers you need. Batching per tenant metadata creates a separate batch for each distinct value, and with a very large number of tenants this can use a lot of memory, which is why the option has a limit on the number of distinct values it will track. Set that limit with care, since data for values beyond the limit may be handled differently from what you expect.
Routing connectors depend on matching rules, and a mistake in a rule can send data to the wrong place. Keep rules simple, test them with sample data, and always include a default route. Authentication extensions must be applied to every receiver that accepts outside data, including less obvious ones such as health or debugging endpoints, or an unauthenticated back door will exist.
Headers Kept Too Long
Keeping every header in the context wastes memory and may expose secrets. Keep only what is needed.
Unlimited Tenant Batches
Per-tenant batching uses memory for each tenant. Know the limit and what happens beyond it.
Unauthenticated Side Doors
Every receiver that accepts outside data needs authentication, not only the main one.
Untested Routing Rules
A small mistake in a rule misroutes data. Test each rule and keep a default.
8Stamping Tenant Identity at the Edge
The edge gateway is the most security-sensitive part of the design, so it deserves close attention.
sequenceDiagram
participant C as Tenant client
participant G as Edge gateway
participant A as Authentication check
participant T as Tenant table
participant P as Next pipeline stage
C->>G: Send telemetry with credentials
G->>A: Verify credentials
A-->>G: Verified identity
G->>T: Look up tenant for identity
T-->>G: Tenant identifier
G->>G: Remove any sender-supplied tenant label
G->>G: Add trusted tenant label
G->>P: Forward stamped telemetry
Step by step
When a request arrives, the receiver first checks the credentials through an authentication extension. Requests that fail the check are refused immediately, before any data is read. For requests that pass, the verified identity is used to look up the tenant, perhaps through a table kept in configuration or fetched from a trusted service. The gateway then removes any tenant label supplied by the sender and adds the verified one.
Where should the label go? Labels that describe the whole source, such as the tenant, belong on the resource, which is the top layer of the telemetry data model. Placing the tenant on the resource means that every item beneath it inherits the label, and it keeps the label out of the individual data points, where it would multiply the number of distinct series.
What to do with unknown or missing identity
Not every request will map cleanly. A credential may be valid but unknown to the tenant table. A trusted internal sender may omit its identity. The design must decide in advance what to do. The safest choices are to refuse data with no identity, or to send it to a clearly separate quarantine path where it is stored apart from any tenant and reviewed by the platform team. Silently assigning such data to a default tenant is dangerous, because it pollutes that tenant’s data and hides the problem.
Whatever you do with unidentified data, count it and alert on it. A steady trickle of unidentified telemetry is a sign of a broken sender, a misconfiguration or an attempt to probe the system.
Keeping Secrets Out of Logs
Tokens and keys used for authentication must never appear in logs, error messages or telemetry. Make sure that any debugging output, and any component that prints headers, is configured to hide sensitive values.
9Routing Data by Tenant
Once data carries a trusted label, the platform can choose what happens to it.
A routing connector sits between pipelines. It looks at a property of incoming data, such as a resource attribute or a request header kept in the context, and sends the data to a matching pipeline. A default route catches data that matches nothing, which is exactly where unidentified or unexpected data should go. Routing can be done for traces, metrics and logs in the same way, with separate pipelines for each signal.
One pipeline per tenant, or per group?
The most obvious design is a separate pipeline for every tenant. It is easy to understand and gives each tenant its own processing rules and exporters. But the number of pipelines grows with the number of tenants, and each pipeline consumes memory and adds start-up time. With hundreds or thousands of tenants this does not scale, and configuration files become enormous.
A more scalable design groups tenants by what they need. All small tenants with standard rules share one pipeline, and the tenant label travels with the data to the exporter. Medium tenants with special rules share a second pipeline. Only the few large or demanding tenants get a pipeline of their own. This keeps configuration small while preserving the ability to treat important tenants differently.
| Design | Scales to many tenants? | Per-tenant rules? | Configuration size |
|---|---|---|---|
| Pipeline per tenant | Poorly | Full freedom | Grows with every tenant |
| Single shared pipeline, label travels with data | Very well | Limited, same rules for all | Small and constant |
| Pipelines per tier or group | Well | Per group | Grows with the number of groups, not tenants |
flowchart TD
IN[Stamped telemetry] --> RC{Routing connector}
RC -->|Large tenants| PL[Dedicated pipeline each]
RC -->|Standard tenants| PS[Shared standard pipeline]
RC -->|Regulated tenants| PR[Regional pipeline]
RC -->|No match| PD[Default quarantine pipeline]
A routing setup with no default route silently discards or misplaces data that fits no rule. Always define a default path, and make that path safe, visible and monitored.
Handling tenants that move between groups
Tenants change over time. A small tenant grows into a standard one, and a standard tenant signs a contract that requires regional storage. Routing based on a table that maps tenants to groups makes such moves a matter of editing the table, rather than rewriting pipelines. The table should be versioned, reviewed and applied in a controlled way, since an incorrect edit can send a tenant’s data to the wrong group.
When a tenant moves, think about what happens to data in flight and to history. New data starts flowing to the new group as soon as the table changes. Data already stored in the old group remains there, and queries must be able to find it, either by searching both groups for a period or by copying the history across. Decide in advance which approach you use, and tell the tenant what to expect.
Prepare the new group
Make sure the new pipeline and storage are ready and tested with sample data.
Change the mapping
Update the table so new data flows to the new group, using reviewed, version-controlled changes.
Keep history readable
Search both groups for a transition period, or copy history across, according to the agreement.
Retire the old path
Once history has aged out or been moved, remove the old route and data, and record the change.
10Per-Tenant Processing
Tenants differ in what they send, what they may keep and what they want removed.
Redaction
Some tenants require that certain fields, such as personal identifiers, are masked or removed before storage.
Filtering
Drop noisy or unwanted data, such as health checks, to save cost and clutter.
Sampling
Keep a controlled share of traces. Different tenants may buy or need different shares.
Enrichment
Add helpful labels such as region or environment, drawn from trusted tables rather than the sender.
Sampling with tenants in mind
Sampling decisions interact with tenancy in interesting ways. A fixed percentage is fair in proportion, but it gives a small tenant very few traces and a large one a flood. A limit on traces per second gives each tenant a predictable ceiling, which protects the platform but may leave a very large tenant with thin coverage. Many platforms combine approaches, with a base rate for all tenants and higher or lower rates according to tier. Whatever the policy, make it explicit and visible to tenants, so that nobody is surprised by what they can and cannot see.
Tail-based sampling, which decides after a trace is complete, needs all spans of a trace in one place. In a multi-tenant system it must also keep tenants apart: the decision memory for one tenant’s traces must never be shared with another’s. Routing by tenant before tail sampling, and then routing by trace identifier inside each group, keeps both needs satisfied.
Head samplers such as the rate-limiting sampler can run inside tenants’ own applications to cut volume at the source, while the platform applies tail-based policies in a stateful tier. A load balancing exporter in front of that tier keeps whole traces together. Tenancy adds one more dimension to the same design.
Rules must be scoped
A rule meant for one tenant must apply to that tenant only. Conditions such as “drop spans with this name” need an extra condition that checks the tenant label, or they will delete data belonging to everyone who uses the same name. This is a classic source of cross-tenant damage, and it is easy to make when configurations are edited by hand. Prefer structures where each tenant’s rules live in their own clearly named section, and use automated checks to catch rules that lack a tenant condition.
Letting tenants control some of their own rules
Tenants often want a say in what is kept: which services are sampled more heavily, which fields are masked, which noisy endpoints are dropped. A safe way to allow this is a self-service layer that produces rules in a restricted form. The platform checks each request against the tenant’s tier and limits, wraps the rule in a condition that restricts it to that tenant, and applies it through the same reviewed process as any other change. The tenant never edits the shared configuration directly.
| Tenant request | Platform check | Result |
|---|---|---|
| Drop a noisy endpoint | Is the rule scoped to this tenant only? | Accepted, with a tenant condition added |
| Raise trace sampling rate | Is it within the tier’s limits? | Accepted or refused according to tier |
| Mask an additional field | Does it only remove data? | Accepted, since stricter redaction is always safe |
| Add a label from another tenant’s data | Does it reach outside the tenant’s own data? | Refused |
Giving tenants direct write access to shared configuration, even with good intentions, makes one tenant’s mistake into everyone’s incident. Provide a controlled interface that generates safe, scoped rules instead.
11Exporting to Multi-Tenant Backends
The last step is telling the storage system whose data it is receiving.
Many observability backends have their own idea of tenancy. A common pattern is that the backend expects a header on each request that names the tenant, and it stores and queries data separately for each header value. Several popular open-source systems for traces, metrics and logs follow this approach, so your platform can often rely on the backend to keep tenants apart in storage, provided it tells the backend the right tenant every time.
Three ways to carry the identity to the backend
Header from Context
The tenant is kept in the incoming context, and a headers extension copies it into a header on the outgoing request. Batches must be formed per tenant so that each request carries one tenant only.
Exporter per Tenant Group
Each routed pipeline has its own exporter configured with a fixed header or credential. Simple and explicit, but it multiplies exporters as groups grow.
Separate Endpoints
Different tenants are sent to different backend instances or regions. Strongest separation, used for silos or data residency.
If a single outgoing request contains data from two tenants but carries only one tenant header, one tenant’s data lands in the other’s storage. Make sure batching is done separately for each tenant, using the batching option that groups by chosen metadata, before any export that depends on a per-request header.
Credentials for storage
The platform needs credentials to write to the backend. Keep these separate from anything tenants can influence, store them securely, and give the exporter only the permission to write, never to read. If different tenants require different credentials, keep them in per-group exporters, and rotate them on a schedule.
| Approach | Isolation strength | Operational effort | Best for |
|---|---|---|---|
| Shared storage with tenant header | Medium to high, relies on backend | Low | Large numbers of small and medium tenants |
| Per-group exporters with fixed credentials | High | Medium | Tiers with different requirements |
| Separate backend instances | Highest | High | Regulated or very large tenants |
When the backend is unavailable for only one tenant
Backends sometimes fail for a single tenant, perhaps because that tenant has reached a storage limit or has been misconfigured on the backend side. If the exporter retries without limit, data for that tenant piles up, and if it shares a queue with others, their data waits too. The remedy is to separate queues by group, set sensible limits on retries and queue sizes, and make refusals visible both to operators and to the tenant concerned. Tenants should be told clearly when their data is being refused and why, so they can act.
Decide in advance what happens to data that cannot be delivered. Options include dropping it after a retry period, holding it in a limited disk queue, or sending it to a holding area. Dropping is simple but loses data. Holding is safer but costs resources and must be bounded. Whichever you choose, make the behaviour part of the tier’s promise, so that tenants know what to expect.
Drop After Retries
Simple and cheap, but data is lost. Suitable for lower tiers where occasional loss is acceptable.
Bounded Disk Queue
Holds data through short outages with a firm size limit, at the cost of storage and handling.
Holding Area
Sends undeliverable data to a separate store for later replay, useful for higher tiers with strong promises.
12Resource Isolation and Noisy Neighbours
Keeping data apart is useless if one tenant can still bring the whole platform to its knees.
Where tenants compete
Network and Ingest Capacity
A tenant sending a very large volume can saturate the gateway’s ability to accept data.
Memory and Queues
Shared queues fill up when one tenant’s data is slow to export, and then everyone’s data waits behind it.
Storage and Series Count
A tenant that creates millions of distinct metric series strains storage and slows queries for all.
Query Capacity
Heavy queries from one tenant can slow dashboards and alerts for others.
Tools for fairness
The first tool is a limit on how much each tenant may send, expressed as a rate, as a volume per period, or as a number of distinct series. The Collector’s core components include a memory limiter that protects the process as a whole, but a limit per tenant often needs additional help. Some contributed components and many API gateways can enforce per-tenant limits, and some backends enforce limits themselves for each tenant header. Check what is available for your versions, and place the limit as early as possible, so that excess data is refused before it consumes resources.
The second tool is separation of queues. If every tenant shares one exporter queue, a slow backend for one tenant can block all. Batching and queuing per tenant group means that a problem in one group fills only that group’s queue. The third tool is separate pipelines for tenants of different sizes, so that a very large tenant cannot crowd out small ones within a shared pipeline.
Measure each tenant’s usage
Know the rate, volume and series count each tenant produces, so limits can be set sensibly.
Set limits per tenant or tier
Agree on a rate, volume and series ceiling for each tier, and write them into the service agreement.
Enforce early
Refuse excess data at the edge, with a clear signal so the sender can slow down or retry.
Separate queues and batches
Keep tenants or tenant groups from sharing a single queue that one slow destination could block.
Alert before the limit is reached
Warn tenants and operators as usage approaches a limit, so surprises are rare.
A single careless label with unbounded values, such as a user identifier on a metric, can create millions of series. Limit the number of series per tenant, and tell tenants clearly when a limit has been hit and which labels caused it.
13Data Protection: Redaction, Residency, Retention and Deletion
Different tenants owe different obligations to their own customers, and the platform must honour them.
Redaction and minimisation
Telemetry often accidentally contains personal or secret data: names in messages, tokens in headers, addresses in queries. The platform should provide redaction rules that remove or mask such values before storage, and these rules should be applied at the gateway so that sensitive data never reaches shared storage. Tenants with stricter needs can have stricter rules. Where possible, encourage tenants to avoid recording sensitive data in the first place, because removal after the fact is never perfect.
Data residency
Some tenants must keep their data in a certain country or region. A pooled design in one location cannot satisfy this. The usual answer is routing by tenant to regional pipelines and regional storage, so that data never leaves the permitted area. This also affects the gateway: if raw data crosses a border on its way to the region where it is processed, the rule may already be broken. Place gateways in each region, or route at the very edge.
Retention
Tenants may buy, or be required to have, different retention periods. In a pooled store this requires the backend to support separate retention per tenant, or requires separate storage for tenants with different periods. Be clear about what retention means: how long data stays searchable, and how long it stays on disk, which may differ.
Deletion and the right to be forgotten
When a tenant leaves, or when a person asks for their data to be erased, the platform must be able to delete it. In shared storage where many tenants’ data is mixed in the same files, deletion can be hard or even impossible at fine detail. Design for deletion at the tenant level from the start, choosing storage that can remove a tenant’s data entirely, and document what can and cannot be removed at finer levels.
| Obligation | Typical technique | Hard part |
|---|---|---|
| Mask sensitive fields | Redaction rules at the gateway | Finding every place a sensitive value can hide |
| Keep data in a region | Regional gateways, pipelines and storage | Ensuring nothing crosses a border in transit |
| Different retention periods | Per-tenant retention or separate storage | Backends that cannot vary retention |
| Delete a tenant’s data | Tenant-level storage separation | Mixed files in pooled storage |
| Encrypt data | Encryption in transit and at rest, possibly with per-tenant keys | Key management at scale |
Encryption and key management
Data should be encrypted while it travels, between tenants and the gateway and between every internal stage, and while it rests in storage. Encryption in transit relies on certificates, which must be issued correctly, renewed before they expire and checked on both sides. Encryption at rest is often handled by the storage system. Some tenants ask for their own keys, so that they can revoke access to their data by withdrawing a key. This is a powerful form of isolation, but it adds the burden of managing many keys safely, and it must be designed carefully so that a lost key does not mean lost data.
Keep keys separate from the data they protect, rotate them on a schedule, and limit who can use them. Record every use of a sensitive key. If the platform serves regulated tenants, expect auditors to ask for evidence of each of these practices.
Encrypt every hop
Use encrypted connections from tenants to the gateway and between all internal stages.
Encrypt storage
Protect stored data with encryption, using per-tenant keys for tenants who require them.
Separate and rotate keys
Keep keys apart from the data, rotate them regularly and restrict who may use them.
Audit key use
Record who used which key and when, for later review and for auditors.
14Access Control on the Query Side
Isolation must hold when people read data, not only when machines write it.
Writing data under the correct tenant is only half of the story. The other half is making sure that people can read only what they are allowed to read. When someone opens a dashboard, runs a search or sets an alert, the platform must attach their tenant, or the set of tenants they are allowed to see, to every query, and the backend must enforce it. If this check is done only in the user interface, anyone who calls the backend directly can bypass it.
Principles for safe querying
Enforce on the Server
The backend or a trusted proxy in front of it must enforce tenant access. The browser can never be trusted to do so.
Least Privilege
People get access only to the tenants and actions they need, such as read-only access to one team’s data.
Separate Roles
Distinguish viewers, editors and administrators, and keep platform operators separate from tenant users.
Audit Access
Keep records of who read which tenant’s data and when, especially for platform staff with wide access.
The platform operator’s special position
The people who run the platform can usually see everything, which is both necessary and risky. Reduce the risk by limiting how many people have this power, requiring approval and recording for its use, and giving most operators tools that show health and volume without revealing content. Tenants, especially regulated ones, will ask how operator access is controlled, and a clear answer builds trust.
Dashboards, alerts and shared settings
Dashboards and alert rules are also tenant data. A dashboard that queries across tenants, or an alert rule that reads another tenant’s series, is a leak waiting to happen. Keep dashboards and alert rules organised by tenant, give each tenant its own space, and review any shared or template dashboards to ensure they do not pull data from outside the viewer’s allowed set.
Verify query isolation by logging in as a user of one tenant and trying to read another’s data through every route: the interface, the programming interface, saved links and exports. The test must fail to find anything, in every route.
15Cost Attribution and Usage Metering
Sharing saves money only if everyone can see, and pay, their fair part.
Cost isolation begins with measurement. For each tenant, the platform should record how much data it sent, how much was stored and for how long, and how much was queried. Without this, shared costs are split by guesswork, and tenants have no reason to be careful. With it, you can charge tenants, show them their own usage, and set limits based on facts.
What to measure
| Measure | Why it matters | Where to capture it |
|---|---|---|
| Items received per signal | Reflects ingest effort | At the gateway, after identity is stamped |
| Bytes received | Reflects network and storage cost better than counts | At the gateway or the export step |
| Items refused or dropped | Shows limit enforcement and billing disputes | At the point of refusal |
| Series count for metrics | Main driver of metric storage cost | In the backend or a counting step |
| Storage used and retention | Drives long-term cost | In the backend |
| Query load | Drives query cost and fairness | At the query layer |
Taking care with the meter itself
Counting per tenant means adding a tenant label to the platform’s own metrics, and if the number of tenants is large, this itself creates many series. Keep the number of labels small, count at coarse granularity, and consider summarising per tier for dashboards while keeping detailed figures in a separate store used for billing. Counting must be done after identity is verified, so that spoofed labels cannot shift costs to another tenant.
Showback Before Chargeback
Many platforms first show tenants their usage and cost without charging, a practice sometimes called showback. This builds understanding and trust, surfaces mistakes in measurement, and lets tenants reduce waste before money is involved. Once the numbers are trusted, charging can follow.
If a tenant disputes a bill, you need evidence. Keep usage records for longer than the billing period, protect them from alteration, and be able to explain how each figure was measured.
Preventing cost surprises for tenants
The best bill is one that holds no surprises. Send tenants usage reports on a regular schedule and warnings when they approach limits. Show which services, signals and labels contribute most to their volume, so they can reduce waste. Many surprise bills come from a single change, such as a new label with many distinct values or a debug setting left on, and an alert on sudden growth catches these early, often before the tenant notices.
Fair attribution also requires deciding what to do about shared costs that no tenant causes directly, such as the platform’s own monitoring or the cost of idle capacity kept in reserve. Some platforms spread these costs evenly, some in proportion to usage, and some absorb them as the cost of providing the service. Whatever is chosen, explain it clearly in advance.
16Monitoring the Platform Itself, per Tenant
A platform that promises isolation must be able to prove it is working.
The telemetry platform needs its own monitoring. Beyond the usual health figures such as memory, processor use and queue length, a multi-tenant platform needs views by tenant: how much each is sending, how much is being refused, how long their data takes to become searchable, and whether any limits are close. These views allow operators to spot a noisy neighbour early, answer questions from tenants, and verify that limits and routing behave as designed.
Ingest per Tenant
Volume received from each tenant over time, to spot sudden growth or silence.
Refusals per Tenant
Items rejected for authentication, limits or invalid data, which shows who is hitting limits.
Delay per Tenant
Time from arrival to being searchable, which shows whether one group is queuing behind another.
Unidentified Data
Amount of data landing in the default or quarantine path, a sign of broken senders or probing.
Service levels for tenants
Define what you promise: how quickly data is accepted, how long before it is searchable, how much availability to expect, and what happens when limits are exceeded. Different tiers may have different promises. Measure each promise from the tenant’s point of view, so that your reports match their experience. A promise that is not measured is only a hope.
Define the promise
State clearly what each tier gets, in terms a tenant can check.
Measure it per tenant
Record the relevant figures separately for each tenant or tier.
Alert on drift
Warn when a figure moves toward the promised limit, before it is broken.
Report openly
Share the results with tenants, especially after incidents, to build trust.
17Failure Modes and Anti-patterns
Most tenancy failures come from a small number of repeated mistakes.
Problem
Trusting a tenant label that the sender supplied, without verification.
Why It’s Harmful
Any sender can pose as any tenant, polluting data, consuming another tenant’s quota, or hiding its own activity.
Correct Approach
Verify identity at the edge, delete any sender-supplied tenant label, and stamp the trusted one.
Problem
Batching data from several tenants together before an export that carries a single tenant header.
Why It’s Harmful
Some tenants’ data is stored under another tenant’s name, which is a direct data leak.
Correct Approach
Batch separately by tenant, using the batching option that groups by chosen metadata.
Problem
Routing with no default path, or with a default that assigns unknown data to a real tenant.
Why It’s Harmful
Unmatched data is lost silently or pollutes a tenant’s records, and the problem stays hidden.
Correct Approach
Send unmatched data to a quarantine path, count it, alert on it and investigate.
Problem
Writing a processing rule for one tenant without a condition that restricts it to that tenant.
Why It’s Harmful
The rule affects everyone who matches it, silently deleting or changing other tenants’ data.
Correct Approach
Scope every rule by tenant, review changes, and test rules against sample data from several tenants.
Problem
Relying on the user interface alone to hide other tenants’ data.
Why It’s Harmful
Anyone who calls the backend directly bypasses the check and reads everything.
Correct Approach
Enforce tenant access on the server, in the backend or a trusted proxy, for every query.
Problem
Putting the tenant label on every metric data point instead of on the resource.
Why It’s Harmful
It adds to the number of distinct series, wastes memory and makes data harder to group.
Correct Approach
Place the tenant on the resource and keep data-point labels few and bounded.
18Scaling a Multi-Tenant Design
A design that works for ten tenants may break at ten thousand.
Scale in layers
The edge gateway scales by adding more identical collectors behind an ordinary load balancer, because authentication and stamping are stateless. The routing and processing layer can scale the same way where its work is stateless. The stateful parts, such as tail sampling, need data for each trace to arrive together, which is the job of a load balancing exporter that routes by trace identifier. In a multi-tenant system, combine this with tenant routing so that each stateful group serves a set of tenants and each trace stays whole inside its group.
Sharding tenants across fleets
Beyond a certain size, one fleet of collectors becomes too large or too risky. A single bad configuration affects everyone, and a single failure domain covers all tenants. Splitting tenants across several independent fleets, sometimes called cells, limits the blast radius of any one problem. Each cell serves a subset of tenants, and a thin routing layer in front of the cells sends each tenant to its home cell. A problem in one cell leaves the others untouched.
flowchart TD
T[All tenants] --> FR[Front routing layer]
FR --> C1[Cell 1: gateway, processing, storage]
FR --> C2[Cell 2: gateway, processing, storage]
FR --> C3[Cell 3: gateway, processing, storage]
Moving a tenant
Tenants grow, and a tenant that began small may need a bigger tier or its own cell. Plan the move in advance. Moving data while keeping history searchable and keeping live traffic flowing needs a routine: start sending new data to the new home, keep the old home readable for a period, and then retire it. Practising this on a small tenant before you need it for a large one is wise.
Capacity planning with tenants in mind
Capacity for a shared platform is planned differently from capacity for a single application. Total volume matters, but so does the shape of the distribution: a platform where a few tenants produce most of the data behaves very differently from one where volume is spread evenly. Study the distribution, and plan for the largest tenants separately, since their growth can consume capacity faster than the average suggests. Keep headroom for sudden growth from any tenant, and review the figures regularly as tenants join, grow and leave.
Plan also for the busiest moments. Many tenants share the same patterns, such as a morning rush or an end-of-month report, so their peaks may coincide. A platform sized only for the average will struggle at these moments, so size for the combined peak and test with realistic bursts.
| Planning question | Why it matters |
|---|---|
| How is volume distributed across tenants? | A few giants need separate planning from many small tenants |
| When do tenant peaks coincide? | Combined peaks, not averages, decide required capacity |
| How fast is each tier growing? | Growth rate decides how often to add capacity |
| What headroom is kept for surprises? | Sudden growth from one tenant should not cause loss for others |
19Testing and Verifying Isolation
Isolation that has never been tested is a belief, not a fact.
Spoofing Test
Send data claiming to belong to another tenant and confirm that the gateway overwrites the claim or rejects the request.
Cross-read Test
As a user of one tenant, try to read another’s data through every route, and confirm that nothing is returned.
Noisy Neighbour Test
Flood the platform from one test tenant and confirm that others see no loss or delay beyond the agreed limits.
Routing Test
Send sample data for each tenant group and verify that it arrives in the correct pipeline, with a default for unmatched data.
Redaction Test
Send data containing known sensitive values and confirm that none appears in storage.
Deletion Test
Remove a test tenant and confirm that all of its data, in all stores, is gone.
Canary tenants
Create a few synthetic tenants that exist only for testing. They continuously send known data, and automated checks read it back, verify that it arrived complete under the right tenant, and confirm that other tenants’ canary data is not visible. A failing canary check is an early warning that something in the platform has changed for the worse, even before any real tenant is affected.
Test configuration changes before they go live
Most isolation failures begin with a configuration change. Run each change in a test environment with sample data from several tenants, and compare the output with expectations. Automated checks can scan configurations for known dangers: rules without tenant conditions, pipelines without a default route, exporters with fixed credentials in the wrong group, and logging of sensitive headers.
Run isolation tests on every release and on a regular schedule in production using canary tenants. Isolation tends to erode through small changes, and routine tests catch the erosion early.
20Operating the Platform: Onboarding, Change and Governance
Technology alone does not deliver isolation. Routines and responsibilities do.
Onboarding a tenant
Agree the tier and limits
Decide the tenant’s rate, volume, series and retention limits, and any special obligations such as location.
Create identity and credentials
Issue credentials and add the tenant to the trusted table that maps identity to tenant.
Assign routing and rules
Place the tenant in the right group, with its redaction, sampling and filtering rules.
Verify with test data
Send sample data, confirm it arrives correctly and cannot be seen by others.
Go live and watch
Enable real traffic, and watch usage and refusals closely during the first days.
Offboarding
When a tenant leaves, disable credentials first, so no new data arrives. Then remove routing and rules, delete or return stored data according to the agreement, remove the tenant from billing and dashboards, and record that the work was done. Forgotten tenants, with live credentials and stored data, are a quiet source of risk.
Managing configuration safely
Keep tenant configuration in version control, generate it from a single source of truth where possible, and require review for changes. Automated generation reduces the risk of the copy-and-paste mistakes that cause cross-tenant effects. Give tenants self-service where safe, such as choosing their own sampling within limits, while keeping identity, routing and limits under platform control.
| Area | Owned by the platform | May be delegated to tenants |
|---|---|---|
| Identity and credentials | Yes | Rotation of their own credentials |
| Routing and limits | Yes | Requests to change tier |
| Redaction rules | Baseline rules | Additional stricter rules |
| Sampling | Allowed range | Choice within the range |
| Dashboards and alerts | Templates and safety checks | Their own content |
21Real-World Scenarios
A few typical situations show how the choices fit together.
A Software-as-a-Service Company Observing Its Customers
The company runs one product for thousands of customers and wants per-customer insight for support. It stamps each request with a customer label derived from authentication, places it on the resource, and uses a shared pool with a tenant header to the backend. Large customers with strict contracts get dedicated pipelines. Usage is metered per customer and shown in a dashboard for the support team.
An Internal Platform Team Serving Product Teams
A central team offers observability as a service to dozens of product teams. Tenants are derived from the platform namespace through a trusted table. Each team gets a limit on volume and series, a showback report, and its own dashboards. The platform uses a routing connector with a few groups instead of a pipeline per team.
A Managed Service Provider with Regulated Clients
A provider monitors systems for banks and hospitals. Each client must keep data in its own country and have it deleted on exit. The provider runs regional gateways and regional storage, routes by client, applies strict redaction, and keeps full audit logs of operator access. Because separation matters more than cost, the largest clients get silos.
A Company Separating Environments
A company treats production, staging and development as tenants so that test data never mixes with production data and test floods cannot affect production monitoring. A simple routing rule and separate limits achieve this with little extra cost.
graph LR
A[Small tenants] --> P[Shared pool with limits]
B[Growing tenants] --> M[Shared pool with higher tier]
C[Large or regulated tenants] --> S[Dedicated pipeline or silo]
P --> U[Usage metering and reports]
M --> U
S --> U
What the scenarios share
In every case, identity was established by the platform, routing used groups rather than one pipeline per tenant, limits protected everyone, and usage was measured per tenant. Where problems occurred in other organisations, one of these four foundations was missing.
A Team That Learned About Mixed Batches
A platform team exported to a backend using a single tenant header, but their batching combined data from several customers. For weeks, a small share of one customer’s data appeared in another’s searches. The problem was found only when a customer reported unfamiliar service names in their dashboard. Switching to per-tenant batching fixed it, and a cross-read test now runs on every release to prevent a repeat.
A Team That Tested With Canary Tenants
Another team created three synthetic tenants that send known data every minute, and an automated check reads it back and confirms that no tenant sees another’s data. When a configuration change accidentally removed a tenant condition from a filtering rule, the canary check failed within minutes, and the change was reverted before any real customer was affected.
22Best Practices and Common Mistakes
A checklist in plain language, drawn from everything above.
Define what a tenant is and what you promise
Write down the isolation dimensions you guarantee, and for which tiers.
Derive identity from verified credentials
Overwrite sender labels, and treat baggage and claims as hints only.
Place the tenant on the resource
Keep data-point labels few and bounded to protect storage.
Route by group and always define a default
Use tiers instead of a pipeline per tenant, and send the unknown to quarantine.
Batch and queue per tenant or group
Never mix tenants in a request that carries one tenant identity.
Limit early, measure always
Apply rate, volume and series limits at the edge, and meter usage per tenant.
Test isolation continuously
Use spoofing, cross-read, noisy-neighbour and deletion tests, with canary tenants in production.
The usual errors are trusting sender labels, mixing tenants in batches, forgetting the default route, writing unscoped rules, enforcing access only in the interface, and never testing isolation after the first launch. Each has caused real data exposure in real organisations.
A sensible order of work
Begin by writing the promises and choosing the model. Build the edge gateway and identity stamping first, since everything depends on it. Add routing with a default path, then export with tenant identity. Add limits and metering. Add redaction and per-tenant rules. Build monitoring and tests, and only then onboard real tenants, starting with a friendly few. Following this order gives you a trustworthy foundation before the stakes become high.
23Frequently Asked Questions
Short, plain answers to the questions people ask most often.
No single built-in switch exists. Multi-tenancy is assembled from attributes, authentication, processors, connectors, exporters and tenant-aware backends. Check the current documentation for any new conventions.
On the resource, the top layer of the telemetry model, so it applies to everything beneath it without multiplying labels on individual data points.
Not as proof. Use it at most as a hint, verify identity from credentials at the edge, and overwrite the header with the verified value.
Only for a few large or special tenants. For many tenants, route by group and let the tenant label travel with the data, to keep configuration and memory use manageable.
Apply limits on rate, volume and series count per tenant at the edge, separate queues and batches by group, and use pipelines of their own for the largest tenants.
Use the batching option that groups data by chosen metadata such as the tenant header, so each batch and each outgoing request contains one tenant only.
Refuse it or send it to a clearly separate quarantine path. Count it, alert on it and investigate, rather than assigning it to a default tenant.
It is convenient inside your own system but can be forged and may leak to third parties, so it must not be used as proof of identity or for security decisions.
Route tenants to regional pipelines and regional storage, and place gateways in each region so data does not cross borders before it is routed.
Choose storage that can remove a tenant’s data entirely, document what can be removed at finer detail, and test deletion with a synthetic tenant before it is needed.
Meter volume, series and storage per tenant after identity is verified, begin with showback reports, and keep the records long enough to settle disputes.
Not usually. Many backends support a tenant header and separate storage logically. Dedicated backends are for regulated or very large tenants who need the strongest separation.
Test it routinely with spoofing, cross-read, noisy-neighbour and deletion tests, and run canary tenants in production whose data is read back and checked automatically.
Tenant routing decides which group handles the data, sampling controls how much is kept, and a load balancing exporter keeps whole traces together inside each stateful group. They work together as stages of one design.
24Summary and Key Takeaways
Multi-tenant telemetry isolation lets many tenants share one observability platform without sharing their data, their performance or their bills. OpenTelemetry provides no single tenant switch, so isolation is built from ordinary parts: verified identity stamped at an edge gateway, a tenant label on the resource, routing by group with a safe default, per-tenant processing, exporters that carry tenant identity to the storage, limits that protect the platform, and metering that makes costs visible.
The strength of the whole design is the strength of its weakest gap. A single trusted-but-forged label, a single mixed batch, a single missing default route or a single query without a server-side check can undo everything else. Define what you promise, build trust once and early at the edge, scope every rule by tenant, and test isolation again and again, and a shared platform can be both economical and safe.
Key Takeaways
- Isolation has many dimensions — data, performance, configuration, access, cost and compliance each need their own protection.
- Tenancy is assembled, not switched on — OpenTelemetry offers building blocks, and the design determines how well tenants are kept apart.
- Verify identity at the edge — derive the tenant from authentication, overwrite sender labels, and treat claims and baggage as hints.
- Route by group with a safe default — avoid a pipeline per tenant, and send unmatched data to a monitored quarantine path.
- Never mix tenants in a batch — group by tenant before any export that carries a single tenant identity.
- Protect the platform from noisy neighbours — limit rate, volume and series per tenant early, and separate queues by group.
- Measure, test and govern — meter usage per tenant, test isolation continuously with canary tenants, and manage configuration through reviewed, scoped changes.