Before learning the tool, let us feel the problem it solves.

A busy company with many services

Picture an online food delivery company. It runs a hundred different services: one for the menu, one for orders, one for payments, one for finding drivers, and many more. Each service runs in many copies, called instances, spread over many machines and several cloud regions. When the telemetry from all these copies arrives in one place, millions of spans, numbers and log lines arrive together.

Now an engineer asks a simple question: “Why is the payment service slow in the Mumbai region since this morning’s release?” To answer, the data must carry certain facts. It must say which service produced it, which version of that service, which region, and ideally which machine or pod. Without these facts, the engineer sees a giant pile of anonymous numbers.

The pain of typing it all by hand

One answer is to ask every developer to add these facts by hand in every service. But that has serious problems.

• People forget. One team forgets the region, another spells the version differently, and the data becomes inconsistent.
• Facts change. The same code may run on a laptop today and in a container tomorrow, and the machine name changes every time it is deployed.
• Some facts are hard to know. Does your code know the ID of the cloud virtual machine it is running on? The name of the Kubernetes pod? The ID of the container? These facts live in the environment around the program, not in the program itself.
• It is repetitive. Two hundred services would each need the same code to look up the same facts.

flowchart LR
    subgraph NO["Without resource information"]
        A1["Span: slow call, 900 ms"] --> A2["Which service? Unknown"]
        A2 --> A3["Which region? Unknown"]
        A3 --> A4["Which version? Unknown"]
    end
    subgraph YES["With resource information"]
        B1["Span: slow call, 900 ms"] --> B2["Service: payments"]
        B2 --> B3["Region: ap-south-1"]
        B3 --> B4["Version: 2.4.1"]
    end
    

We must understand the luggage tag before we can understand the person who fills it in.

In OpenTelemetry, a resource is a set of facts that describes the thing that produces telemetry. That thing might be a service, a process, a container, a machine, or a function running in the cloud. The facts are stored as attributes, which are simple pairs of a name and a value, like “service.name equals payments” or “cloud.region equals ap-south-1”.

Three important properties

Resource versus span attributes

Beginners often confuse these two, so let us separate them with a table.

The short answer first, and then the details.

A resource detector is a small component that automatically discovers facts about the environment where your software is running, and turns those facts into resource attributes. It looks at the machine, the operating system, the process, the container, the cloud provider and the orchestration system such as Kubernetes. Then it hands back a bundle of attributes, ready to be added to the resource.

What detectors can discover

What detectors are not

• They are not instrumentation. They do not create spans or metrics. They only describe the sender.
• They are not magic. They can only discover what the environment makes available. If the environment hides a fact, the detector cannot find it.
• They do not describe individual requests. For per-request facts, you use span attributes.
• They are not the same as the Kubernetes enrichment tool in the Collector. We will explain this difference carefully in a later chapter, because it is a common source of confusion.

OpenTelemetry has agreed names for common facts, grouped into families. Knowing the families helps you read any trace.

The agreed names come from the semantic conventions, which are shared naming rules for telemetry. Using the same names everywhere means that dashboards, alerts and tools from different vendors understand your data without extra work. Here are the main families.

The most important attribute of all

If you remember only one resource attribute, remember service.name. It is the logical name of your service, such as “checkout” or “payments”. Almost every tracing and monitoring tool uses it as the main way to group data. If you do not set it, the SDK uses a placeholder that begins with the words “unknown service” followed by the name of the running program. Seeing “unknown service” in your tools is the classic sign that nobody set the service name.

Which ones can a detector find, and which must you provide?

This is a very practical question. Some facts are knowable from the environment, and some are choices that only your team can make.

Let us follow the story from the moment your program starts.

When your application starts and the OpenTelemetry SDK is set up, one of the first things it does is build the resource. It does this by running a list of detectors, one by one, collecting the facts each one finds, and then merging all the facts into one final resource. This resource is then handed to the providers for traces, metrics and logs, so every signal carries it.

sequenceDiagram
    participant App as Application starts
    participant SDK as OpenTelemetry SDK
    participant D1 as Default detector
    participant D2 as Environment detector
    participant D3 as Host and process detectors
    participant D4 as Cloud detector
    participant TP as Tracer, Meter and Logger providers
    App->>SDK: Initialise SDK
    SDK->>D1: Ask for SDK facts and default service name
    D1-->>SDK: telemetry.sdk and unknown service placeholder
    SDK->>D2: Ask for facts from environment variables
    D2-->>SDK: service.name and custom attributes
    SDK->>D3: Ask for host, OS and process facts
    D3-->>SDK: host.name, os.type, process.pid
    SDK->>D4: Ask for cloud facts
    D4-->>SDK: cloud.provider, cloud.region, host.id
    SDK->>SDK: Merge all facts into one resource
    SDK->>TP: Give the final resource to every provider
    

Automatic and manual detection

In many setups, detection happens without you doing anything. A few detectors are always on, such as the one that records the SDK’s own facts. Others are enabled by settings. In auto-instrumentation setups, such as agents, a sensible group of detectors is often turned on for you. In manual setups, you pick the ones you want when you build the SDK.

When several sources give an answer to the same question, there must be clear rules.

Imagine that the environment detector says the service is called “checkout”, while a cloud detector says nothing about the service name, and the default detector says “unknown service”. All three provide a value, or a placeholder, for the same attribute. The SDK needs a rule for which one to keep.

The general rule is simple: when two resources are merged, the values from the later or more specific one override the earlier one on conflicts. The default placeholders sit at the bottom, and explicit choices made by you sit at the top. In most SDKs, values you set through the standard environment variables take priority over automatically detected ones.

flowchart TD
    A["Lowest priority: SDK defaults and placeholders"] --> B["Automatic detectors: host, OS, process, container, cloud"]
    B --> C["Environment variable detector: OTEL_RESOURCE_ATTRIBUTES"]
    C --> D["Highest priority: OTEL_SERVICE_NAME and values set in code"]
    D --> E["Final resource"]
    

The order of detectors matters

Because later detectors can override earlier ones, the order in which you list detectors is part of your configuration. If two detectors can disagree, such as one that reads a host name from the operating system and another that reads a more precise host ID from the cloud, put the more trusted source later.

The simplest and most important detector. Almost everyone uses it.

An environment variable is a named setting that is handed to a program when it starts, a bit like a note stuck on the door before the program enters the room. OpenTelemetry defines standard names for the notes that describe your resource.

Why this approach is so popular

• No code changes. You set the values in your deployment, not in your program.
• Works in every language. The names are the same everywhere.
• Easy to automate. Deployment tools can fill in the version, the environment and the team for you, so every service is labelled the same way.
• Easy to change. You can fix a label by redeploying with a new setting, without rebuilding the application.

Each language ships its own set of detectors, with similar ideas but different names and defaults.

How the languages differ in style

• Java: Detectors are packaged as modules, and the Java agent turns on a sensible set automatically. Cloud ones are often switched on with a setting. Some detectors can be listed or hidden using provider settings.
• Python: The core package has the environment and process detectors, and cloud and container detectors come as separate packages. A setting can list which detectors to use at startup.
• JavaScript and Node.js: The core resources package offers environment, host, operating system, process and service instance detectors, and cloud detectors are separate packages. Some detectors work asynchronously, since they may need to wait for a reply.
• Go: The resource package offers options you choose when building the resource, such as ones for the host, operating system, process, container, and from the environment, and cloud detectors live in a separate contrib area.
• .NET: Packages for host, operating system, process, container and cloud environments are available as add-ons, and the environment settings are read by default.

How can a program learn which cloud region it is in? It asks a very friendly neighbour.

Cloud providers give every virtual machine, container task or serverless function a special local address, called a metadata service, that answers questions about itself. The address is only reachable from inside that machine. A cloud detector simply makes a small web request to this address and reads the reply, which includes facts such as the instance ID, region, zone and account.

sequenceDiagram
    participant App as Application with SDK
    participant CD as Cloud detector
    participant MS as Cloud metadata service
    App->>CD: Start detection at startup
    CD->>MS: Request token if the provider requires one
    MS-->>CD: Short lived token
    CD->>MS: Ask for instance ID, region and zone
    MS-->>CD: Answers
    CD->>CD: Turn answers into cloud.* and host.* attributes
    CD-->>App: Return attributes within the allowed time
    

The cloud families

Costs and cautions of asking the network

• Startup delay. A web request takes time. If the metadata service is slow or does not answer, startup may be delayed until a timeout. Good detectors use short timeouts.
• Failure must be harmless. If the program is running on a laptop and there is no metadata service, the detector must quietly return nothing. It must never stop the application from starting.
• Security tokens. Some clouds require a token, to protect the metadata service from abuse. Detectors handle this, but your network and permission settings must allow it.
• Containers and hops. In some container setups, the metadata service is only reachable if network settings allow an extra hop. If detection mysteriously finds nothing in a container, check this.
• Short-lived functions. In serverless functions that start often, every extra millisecond of startup is paid again and again. Prefer detectors that read environment variables over those that make network calls.

Inside a container, the program does not automatically know which pod or node it is in.

Kubernetes is a popular system for running many containers across many machines. Inside a container, a program sees its own small world. It does not directly know the name of its pod, the namespace, or the node. These facts are held by Kubernetes, not by the program, so a simple detector cannot discover them by looking around. We need a way to hand these facts in.

Three ways to get Kubernetes facts into the resource

flowchart TD
    K["Kubernetes pod definition"] --> D["Downward API passes pod name and namespace as environment variables"]
    K --> O["Operator injects settings when instrumenting"]
    D --> E["OTEL_RESOURCE_ATTRIBUTES includes k8s facts"]
    O --> E
    E --> S["SDK environment detector builds the resource"]
    S --> X["Telemetry carries k8s attributes"]
    K --> A["Kubernetes API"]
    A --> P["Collector Kubernetes attributes processor looks up the sender"]
    X --> P
    P --> Y["Telemetry enriched in the Collector"]
    

A very important difference: detecting versus looking up

This is a favourite source of confusion, so let us be careful. The two Collector tools sound similar but do different jobs.

The same idea, working inside the Collector pipeline.

In the Collector, the resource detection processor runs detectors and adds the facts it finds to the resource of the data that passes through. It is configured, not coded. You list the detectors you want, in order, and set a few options. It can add the same details to traces, metrics and logs.

The kinds of detectors it offers

The important settings

flowchart LR
    R["Receiver"] --> M["Memory Limiter"]
    M --> RD["Resource Detection"]
    RD --> F["Filter and Redaction"]
    F --> S["Sampling"]
    S --> B["Batch"]
    B --> X["Exporter"]
    

Resource detection normally sits early in the pipeline, right after the memory limiter. The reason is that later processors, such as filters and samplers, may want to use the labels it adds, for example “drop data from the test environment”. Adding labels first means the rules that come after can see them.

The same facts can often be found in either place. Let us choose wisely.

flowchart TD
    Q["Which fact do you need?"] --> A{"Does only the application know it?"}
    A -->|Yes| B["Set it in the SDK with environment settings"]
    A -->|No| C{"Can you change or restart the application?"}
    C -->|Yes| D["Detect in the SDK, or pass it in with the Downward API or Operator"]
    C -->|No| E{"Does the Collector run beside the application?"}
    E -->|Yes| F["Use the resource detection processor in the agent Collector"]
    E -->|No| G["Use a lookup, such as the Kubernetes attributes processor, or add fixed labels in the Collector"]
    

A sensible split of duties

• The application and its deployment own the identity: service name, version, environment and team.
• The agent Collector owns the local environment: host, cloud and node facts.
• The gateway Collector owns company-wide policies: adding or fixing labels, and removing sensitive ones.

Giving each layer one clear job avoids two layers fighting over the same attribute.

Follow one resource from the moment your program starts to the moment you read it on a dashboard.

How the resource looks in different backends

Different tools treat resource facts in different ways. Some show them as a panel of properties on each trace. Some use the service name as the main grouping key. Some store them in a separate table that you can join with other data. For example, when metrics are sent to a Prometheus-style system, the service name and instance become the standard “job” and “instance” labels, and the rest of the resource usually goes into a special information metric, which you can join in a query. Choose which facts to turn into direct labels with care, because each extra label can multiply the number of stored series.

A resource is a luggage tag, not a diary. Keep it stable.

Because a resource is created once and does not change, it is the wrong place for anything that changes while the program runs. Putting changing facts there leads to confusing data or forces you to restart programs just to update labels.

The churn problem

Some resource attributes change every time a program restarts, such as the process ID, the pod name and the container ID. That is correct, since a new pod really is a new sender. But in a metrics system, every new combination of labels creates a brand new series. A cluster that restarts pods often can produce a flood of short-lived series if these attributes are turned into labels. Keep these in the resource, but be cautious about promoting them to metric labels.

When twenty copies of a service run at once, each needs its own identity.

The service instance ID, written service.instance.id, is meant to be unique for each running copy of a service. It lets you answer questions such as “is only one of the twenty copies slow?” The guidance is that it should be unique within the same service name and namespace, and it should be stable for the life of that copy.

Where does it come from?

• You can set it through the standard environment settings, for example using the pod name.
• A detector may create it, often as a random unique value generated at startup, in languages that offer such a detector.
• A platform may provide it, such as a cloud instance ID or a container ID that you reuse.

Why this small field matters

In metrics systems, the instance ID is used to keep the numbers from different copies apart. If two copies share the same instance identity, their numbers can collide, overwrite each other or double count. If there is no instance identity at all, the backend cannot tell which copy produced a number.

Names are agreed standards. Standards evolve, and that has consequences.

The names of resource attributes come from the semantic conventions. These are carefully designed, but they are still maturing, and a few names have been renamed over time as the community improved them. For example, the attribute that records the deployment environment has been moving to a more specific name in newer versions. This is healthy for the project but can surprise teams whose dashboards depend on the old names.

Schema URLs

To help with this, a resource can carry a schema URL, a web address that names the version of the conventions used. A tool can use this to understand which version of the names a piece of data follows, and translate between versions. When two resources with different schema URLs are merged, the rules get more careful, and in some cases the merge is refused to avoid mixing incompatible names.

flowchart LR
    A["Old SDK sends deployment.environment"] --> C["Collector or backend"]
    B["New SDK sends the newer environment name"] --> C
    C --> D{"Are both names handled?"}
    D -->|Yes| E["Dashboards show all services together"]
    D -->|No| F["Some services vanish from filters and dashboards"]
    

How to stay safe

• Read the release notes before upgrading SDKs or agents, looking for renamed attributes.
• Test dashboards after an upgrade in a staging environment.
• Normalise in the Collector by adding a rule that copies old names to new names or the other way around, during migration.
• Pin versions across services, so that the whole company moves together instead of living with mixed names for months.

Every approach has a price. Let us count both sides honestly.

Key trade-offs

A luggage tag that says too much can be a problem. Be deliberate about what you reveal.

When labels go missing or look wrong, here is how to find out why.

Signals that something is wrong

• “Unknown service” in your tools: nobody set the service name for that program.
• Missing cloud or Kubernetes fields: a detector is not enabled, cannot reach its metadata source, or the platform hides the facts.
• All services showing the same host: detection is running in a central gateway instead of beside the sender.
• Services split in two on dashboards: two spellings or two naming versions are in use for the same service.
• Slow application start: a detector is waiting on a metadata service that never answers.
• Metric series exploding: restart-prone attributes such as pod name or process ID are being turned into labels.
• Values that look wrong or old: an attribute set by hand, or an old setting, is overriding the detected value.

A simple troubleshooting routine

Common symptoms and likely causes

Detection happens once, but when it happens matters.

Since detection normally runs once at startup, its cost is usually small and does not affect the speed of your requests. But there are cases where it matters, and they are worth understanding.

Habits worth copying and traps worth avoiding.

Good design patterns

Anti-patterns

A practical checklist for resources you can trust.

Best practices

• Always set the service name. It is the first thing every tool uses, and the most common reason for confusing dashboards.
• Set version and environment. They let you compare releases and separate production from testing.
• Provide a unique instance ID. It keeps copies of the same service apart in traces and metrics.
• Use the standard names. Follow the semantic conventions, and avoid inventing your own spellings.
• Standardise with templates. Let your deployment tools fill in labels, so developers cannot get them wrong.
• Decide where each fact is detected. One owner per attribute, with the Collector override setting matching that choice.
• Use short timeouts. Detectors that call the network must never be allowed to hold up your startup for long.
• Review before production. Print real resources in a test environment, and check them for privacy and correctness.
• Be selective with labels in metrics. Keep most resource facts in the resource, and promote only a short list.
• Watch naming changes. Read release notes, test dashboards and normalise old and new names during upgrades.
• Test it. Add an automatic test that checks the presence of required resource attributes for every service.

Common mistakes beginners make

• Believing that detectors set the service name for you. They usually cannot know it.
• Mixing up the resource detection processor with the Kubernetes attributes processor.
• Forgetting that a resource cannot change after the SDK starts.
• Assuming that every detector is on by default in every language.
• Using the resource for data that changes per request.
• Not checking what was actually exported, and trusting the configuration instead.
• Letting secrets slip into resource settings or captured command lines.
• Promoting too many resource attributes to metric labels.
• Ignoring conflicts between the SDK and the Collector.

How resource detectors show up in tools and platforms you may already know.

A typical journey for a growing company

Quick answers to questions beginners ask most.

If you remember only a few things, remember these.