OTLP Protocol Versioning in OpenTelemetry

OTLP Protocol Versioning in OpenTelemetry

A complete, plain-language guide to how the OpenTelemetry Protocol changes over time without breaking the applications, collectors and backends that depend on it, and how to upgrade a mixed fleet safely.

Imagine two neighbours who talk through a pair of tin cans connected by a string. For years they have agreed on how to speak: one short tug means “hello,” two tugs mean “dinner is ready.” One day a neighbour moves in next door, and the family decides to add a new signal, three tugs for “come over.” The old neighbours do not know the new signal, but if the new signal is just an extra, they can ignore it and everything still works. If the family had changed the meaning of the old tugs, though, everyone would be confused at once.

That small story is the heart of protocol versioning. OpenTelemetry moves telemetry between thousands of applications, collectors and backends, written by many different teams and upgraded at different speeds. The OpenTelemetry Protocol, called OTLP, must change over time to carry new kinds of information, and it must do so without breaking anyone. This post explains how it does that, what the version numbers really mean, and what you should do in your own systems, starting from the very beginning so that no earlier knowledge is needed.

1Why Protocols Need Versions at All

Before looking at OTLP itself, it helps to understand what a protocol is and why every protocol eventually has to change.

What a Protocol Is

A protocol is an agreed set of rules for exchanging information. It says what a message looks like, what each part means, in what order messages are sent, and what to do when something goes wrong. Two programs that follow the same protocol can talk to each other even if they were written by different people in different programming languages. The rules act like a shared language.

Simple Analogy

Think of a standard shipping label on a parcel. Every courier in the world agrees where the sender, the receiver and the weight are written. A parcel can pass through ten different companies and still arrive, because everyone reads the same label layout.

Why Protocols Must Change

The world never stands still. New kinds of data appear, mistakes in the first design are discovered, and better ways of doing things are invented. A protocol that never changed would soon be unable to describe what people need to say. But a protocol that changes carelessly breaks every program that already speaks it. This tension between “we must improve” and “we must not break anyone” is the reason versioning exists.

The Three Questions Versioning Answers

Question 1

What changed?

A version number gives people a short way to say which set of rules they mean, so conversations about changes stay precise.

Question 2

Is it safe?

The way a number moves tells you whether an upgrade is a small safe addition or a risky breaking change.

Question 3

Who must act?

Clear rules show whether senders, receivers or both need to be updated, and in what order.

Question 4

How long will it last?

Stability labels tell you whether a part of the protocol is settled or still being explored.

i
Remember

A version number is not decoration. It is a promise about what will and will not change. Learning to read that promise is the core skill in this whole topic.

The Real Cost of a Broken Protocol

It is easy to think of a protocol break as a small technical event, but in a large company it can mean hundreds of services losing their monitoring on the same afternoon. Engineers on call would lose traces just when a release was going out. Dashboards would turn blank, alerts would stop firing, and teams would spend hours working out whether their application was broken or merely invisible. Protocol versioning exists to make this kind of event extremely rare, and to make the unavoidable changes slow, visible and survivable.

Two Directions of Compatibility

Two terms appear again and again in discussions of versions, and they are easy to mix up. Backward compatibility asks whether a newer reader can understand data written by an older writer. Forward compatibility asks whether an older reader can cope with data written by a newer writer. A healthy protocol aims to offer both within a stable major version, because in real fleets old and new programs are always mixed together.

Term

Backward Compatible

New receivers keep understanding data from old senders, so you can upgrade receivers without waiting for senders.

Term

Forward Compatible

Old receivers keep working when new senders add information, ignoring what they do not understand.

Term

Breaking Change

A change that makes some existing sender or receiver misread or reject data, and so needs a major version.

Term

Deprecation

A formal notice that something still works but will be removed in a later major version, giving time to move.

Simple Analogy

Backward compatibility is like a new phone that can still play your old music files. Forward compatibility is like an old phone that can still open a new file by showing the parts it understands and skipping the rest. A good file format offers both.

2A Quick Tour of OTLP

To talk about versions of OTLP, we first need a clear picture of what OTLP carries and how it travels.

What OTLP Is For

OTLP stands for OpenTelemetry Protocol. It is the standard way for telemetry to travel from the place it is produced to the place it is processed or stored. Telemetry here means the signals that describe how software behaves: traces, which follow a request through many services; metrics, which are numbers measured over time; and logs, which are timestamped records of events. A newer signal for profiling is also being developed.

Who Talks to Whom

In a typical system, an application uses an OpenTelemetry library, called an SDK, to produce data. The SDK has an exporter that sends the data using OTLP to a Collector, which is a program that receives, processes and forwards telemetry. The Collector then sends the data onward, often again using OTLP, to a backend that stores and displays it. Many backends also accept OTLP directly, so you may skip the Collector in small setups.

flowchart LR
  A["Application with SDK"] -->|"OTLP"| B["Collector"]
  B -->|"OTLP"| C["Another Collector"]
  C -->|"OTLP"| D["Backend that accepts OTLP"]
  A -->|"OTLP directly"| D
        
FIG 1 — OTLP appears at every hop, so every hop must agree on its rules

The Ways OTLP Travels

TransportEncodingCommon Default PortNotes
OTLP over gRPCProtocol Buffers (binary)4317Efficient, supports streaming of calls and built-in status codes
OTLP over HTTPProtocol Buffers (binary)4318Easy to pass through proxies and load balancers
OTLP over HTTPJSON4318Human-readable, handy for debugging and some browsers
Simple Analogy

OTLP is like a shipping standard that allows parcels to travel by road, rail or air. The transport can differ, but the parcel layout stays the same, which is exactly why the rules for the contents matter more than the vehicle carrying them.

The Same Messages in Every Transport

An important point for versioning is that the messages are defined once. Whether you send them over gRPC or HTTP, in binary or JSON, you are sending the same structured data. A change to the message definition therefore affects every transport at the same time, so a single set of compatibility rules can protect everyone.

3
Main signals: traces, metrics, logs
2
Main transports: gRPC and HTTP
1
Shared message definition

What Each Signal Looks Like on the Wire

Understanding the kinds of information each signal carries helps you picture where new fields might be added in the future. Each signal has a few core ideas, and a large number of optional details around them. The optional details are where most additions happen, because adding another optional detail is the easiest kind of change to make safely.

SignalCore ItemsTypical Optional Details
TracesTrace and span identifiers, name, start and end time, parent linkAttributes, events, links to other spans, status, flags
MetricsMetric name, unit, kind of measurement and the data pointsAttributes on points, exemplars, aggregation temporality, flags
LogsTimestamp, severity and a body holding the messageAttributes, trace context, an event name, dropped counts

Metric Kinds

Metrics deserve a special mention, since they have several kinds of data point. A gauge records a value that can go up and down, such as temperature. A sum records a total that either only grows or can also fall. A histogram records how values spread across ranges, and an exponential histogram does the same with ranges that grow by a constant ratio and adapt their resolution. A summary type exists for compatibility with other systems. Adding a new kind of data point is a classic compatible change, because old receivers that do not know the new kind can ignore it.

Simple Analogy

Think of a doctor’s chart. The core items are the patient’s name and the date. The optional details are extra tests that can be added over time. A new test can be added to the chart without making the old charts unreadable, because the core stays the same.

i
Practical Tip

When a release note announces a new metric kind or a new optional detail, check whether your backend understands it. A pipeline can carry the new data perfectly while the backend quietly ignores it, which shows up as missing charts rather than as any error.

3The Many Layers Called “Version”

The word “version” is used for several different things in OpenTelemetry, and mixing them up is the number one cause of confusion.

Why One Word Causes Trouble

When someone says “we are on version 1.20,” they might mean the specification, the protocol definitions, a language SDK, the Collector or the naming rules for attributes. Each of these has its own number, its own release schedule and its own stability promise. Understanding which layer a number belongs to prevents many wrong assumptions about compatibility.

LayerWhat the Version DescribesWho Cares Most
OTLP protocol definitionThe shape of the messages exchanged on the wireAnyone sending or receiving OTLP, including backend vendors
Protocol package nameThe major generation of the message definitions, such as version oneCode generators and library authors
Endpoint pathThe version segment in the HTTP address of each signalProxy, gateway and firewall operators
OpenTelemetry specificationThe rules that SDKs and tools must followSDK authors and standards followers
Language SDK and APIA particular library for a particular programming languageApplication developers
Collector versionA build of the Collector program and its componentsPlatform and operations teams
Semantic conventionsThe agreed names and meanings of attributesDashboard and alert authors
Schema URLWhich version of the semantic conventions the data followsTools that translate between convention versions
Simple Analogy

Imagine a book. The edition number of the book, the version of its printing press, the dictionary used by the author and the typeface used for printing are all “versions,” yet they mean completely different things. Asking “which version?” without saying which one gets you nowhere.

Which Layers Does This Post Focus On?

The main subject is the OTLP protocol definition itself, together with its package names and endpoint paths. We will also separate it clearly from semantic conventions and schema URLs, since those are often mistaken for protocol versions. Later chapters explain how SDK and Collector upgrades connect to the protocol, because in daily work these are the versions you actually change.

!
Common Mistake

Assuming that a new SDK version means a new protocol version, or the other way around. A language SDK can release many times without the protocol changing at all, and a protocol change may reach SDKs slowly over many releases.

4How OTLP Is Defined: The Proto Files

OTLP is not described only in prose. It is defined precisely in files written in a schema language, and those files are the real source of truth.

What Protocol Buffers Are

Protocol Buffers, often called protobuf, is a way to describe structured data and to turn that data into compact bytes and back again. You write a definition file that lists the messages and their fields, and tools generate code in many programming languages from it. Because every language generates its code from the same definition, programs written in different languages can read each other’s messages without confusion.

Simple Analogy

A protobuf definition is like the blueprint for a standard form. Each box on the form has a number and a purpose. Anyone who prints the form from the blueprint gets an identical layout, so forms filled in by different people can always be read by anyone else.

Where the Definitions Live

The OpenTelemetry project keeps the OTLP definitions in a dedicated repository for protocol files, separate from the specification text and from the SDKs. The specification describes behaviour such as how to retry and how to encode JSON, while the proto files define the exact fields. SDKs and the Collector pull in the generated code for a chosen release of these files, so each build of a library is tied to a specific snapshot of the definitions.

The Shape of the Definitions

Group

Common Types

Shared building blocks such as attributes, values, and the description of the instrumentation scope that produced the data.

Group

Resource

A description of the entity producing telemetry, such as a service, a host or a container, attached to batches of data.

Group

Signal Data Models

Separate definitions for traces, metrics and logs, describing spans, data points and log records.

Group

Collector Services

The request and response messages and the service definitions used to export each signal from a sender to a receiver.

Nesting: How a Batch Is Built

A request to export data is built in layers. At the outside is an export request, which holds a list of resource groups. Each resource group holds a list of scope groups, and each scope group holds the actual items, such as spans, data points or log records. This nesting saves space, because a description of the resource is written once for many items rather than repeated on each one. It also matters for versioning, because a new field can be added at any level without disturbing the others.

flowchart TD
  R["Export request"] --> RG["Resource group: who produced this data"]
  RG --> SG["Scope group: which library produced it"]
  SG --> I1["Item: span, data point or log record"]
  SG --> I2["Item: span, data point or log record"]
        
FIG 2 — The layered structure of an OTLP export request
i
Practical Tip

When you want the final answer about any detail of OTLP, read the proto files and the protocol specification rather than blog posts. They are the authoritative sources, and they state the compatibility promises directly.

Reading a Definition Without Fear

Proto files look strange at first, but a handful of ideas explain almost everything in them. A message is a group of named fields, much like a form. Each field has a type, a name and a number. A field marked as repeated can appear many times, forming a list. A group marked as one-of means that exactly one of several choices is present at a time. Some numbers or names can be reserved, which means they are retired and must never be used again.

IdeaPlain MeaningVersioning Relevance
MessageA structured record made of fieldsNew messages can be added without disturbing old ones
Field numberThe permanent identity of a field on the wireMust never be reused for a different purpose
Repeated fieldA list of values of the same kindAdding entries to lists is naturally compatible
One-of groupExactly one choice among severalNew choices can be added, old readers ignore unknown ones
ReservedA number or name that is permanently retiredProtects against accidental reuse after removal
Deprecated markerA field that should no longer be usedStill works until a future major version

Comments Matter

The comments in the definition files are a rich source of information. They describe the meaning of each field, state its stability, and explain which values are valid. When you read a definition, read the comments around any field you plan to use. They often contain notes such as when a field was added or why a particular choice was made, which gives clues about how safe it is to depend on.

Simple Analogy

Reading a proto file is like reading a house plan. The drawing shows rooms and doors, while the notes in the margin explain which walls can be moved and which ones hold up the roof. The margin notes are where you learn what is safe.

5Semantic Versioning and Stability Levels

OTLP uses the familiar idea of semantic versioning, and it adds stability labels that tell you which parts you can rely on.

Semantic Versioning in Simple Words

Semantic versioning uses three numbers separated by dots, such as 1.4.2. The first is the major number, the second is the minor number and the third is the patch number. The major number increases when a change can break existing users. The minor number increases when new features are added in a compatible way. The patch number increases for small fixes that change no behaviour. Reading these numbers tells you how risky an upgrade is.

1

Patch change: 1.4.2 to 1.4.3

A small correction, such as a fixed comment or a documentation clarification. No one needs to change anything.

2

Minor change: 1.4.3 to 1.5.0

New optional fields or features were added. Existing senders and receivers keep working, and newer parties can use the extra information.

3

Major change: 1.5.0 to 2.0.0

Something incompatible was changed. Both sides must be updated together, or a bridge must be provided in between.

The Journey Before 1.0

In its early life, OTLP was labelled with numbers beginning with zero, such as 0.9 or 0.19. Under semantic versioning, a zero major number means the design is still being shaped and breaking changes may appear in any release. During that period names changed, some message types were removed or replaced, and fields were reorganised. Once the core signals reached a stable 1.0 line, the project committed to far stricter promises, which is the world most users now live in.

Simple Analogy

The zero era is like a building under construction, where walls move each week. Version one is the day the building opens. After that, you may add new rooms, but you do not knock down the walls people are already using.

Stability Labels

Beyond numbers, parts of OpenTelemetry carry a stability level. A stable part will not change in a breaking way within its major version. A part marked as in development may change at any time, including in ways that break compatibility. The core signals of traces, metrics and logs are stable in OTLP, while newer areas such as profiling have been developed under a development label, with their own package naming to make that clear.

LevelPromiseHow to Treat It
StableNo breaking changes within the major versionSafe to depend on in production
DevelopmentMay change or disappear without noticeUse for experiments, expect to adapt
DeprecatedStill works, but will be removed in a future major versionPlan to migrate away
!
Common Mistake

Treating a development-stage signal as if it were stable. Building a production dependency on something labelled as in development means that a routine upgrade could break your pipeline with no warning.

Reading Release Notes Like a Professional

Every release of the protocol definitions comes with notes, and reading them takes only minutes. Skim for four things. First, new fields or new messages, which show what you could use. Second, deprecations, which show what you should plan to leave. Third, changes to development-stage areas, which show where instability still lives. Fourth, anything described as a breaking change, which should be rare and always deserves a careful, team-wide discussion.

1

Check the number that moved

See whether the patch, minor or major number changed, to judge the likely size of the impact.

2

List new capabilities

Note which additions might benefit your teams, and which of your tools would need to support them.

3

List deprecations

Search your own code and configuration for anything that uses the deprecated parts.

4

Decide on timing

Choose whether to upgrade now, soon or later, and schedule it into the normal upgrade rhythm.

Upgrading Because You Want To, Not Because You Must

Because stable OTLP is additive, there is rarely an emergency reason to upgrade the protocol definitions on a particular day. This is a gift. It lets you choose upgrade moments that suit your team, such as quiet weeks, instead of reacting under pressure. The exceptions are security fixes in the software that implements the protocol, which should still be applied promptly even though the protocol rules themselves are calm.

i
Practical Tip

Distinguish clearly between “the protocol has a new release” and “the software I run has a security fix.” The first can wait for a convenient moment. The second should be scheduled as a normal maintenance priority.

6The Compatibility Rules for Stable OTLP

The promise that stable OTLP will not break you is backed by a short list of concrete rules about what changes are allowed.

What Is Allowed Within a Major Version

Within a stable major version, the project allows changes that cannot disturb existing users. New optional fields may be added to existing messages. New message types, new signals and new values for enumerations may be introduced. Documentation and comments may be improved. All of these are additive, which means that someone who ignores them still understands everything they understood before.

What Is Not Allowed

Changes that could confuse an older program are forbidden. An existing field may not be removed in a way that frees its identifying number for reuse. The meaning of a field may not be changed. A field may not change its type. Required behaviour may not be altered silently. When the project wants to retire something, it marks it as deprecated and keeps it in place, and the actual removal can only happen in a new major version.

Allowed in a Minor Release

  • Adding a new optional field to a message
  • Adding a new message type or signal
  • Adding a new value to an enumeration
  • Marking a field as deprecated
  • Improving comments and documentation

Forbidden Without a New Major Version

  • Reusing a field number for a different purpose
  • Changing the type or meaning of an existing field
  • Removing a field that senders still produce
  • Renaming a message in the generated package
Simple Analogy

Think of a numbered locker room. You may add new lockers at the end of the row, and you may put a “closed” sign on an old one. But you may never give locker number twelve to a new person while the old owner still has the key, or both will open the wrong door.

Why Reusing Numbers Is So Dangerous

In protobuf, the number of a field, not its name, identifies it on the wire. If a field number were reused for a new purpose, an old receiver would happily read the new data as if it were the old data, giving silently wrong results instead of an obvious error. Silent wrongness is the worst kind of failure, because nothing alerts you. That is why the project treats number reuse as a hard rule.

i
Practical Tip

When you see a field marked deprecated in the definitions, take it as a polite request to move away from it, not as a sign that it has stopped working. It will continue to function for the whole major version, which gives you time to migrate calmly.

A Quick Self-Check Before You Rely on Any Field

Before you build something important on a particular field, ask three short questions. Is the signal it belongs to marked stable, or still in development? Do the backends and tools you depend on actually read this field, or merely accept it? And will every older component on the data path behave sensibly if it never sees the field? Honest answers to these questions prevent most disappointment, because they reveal whether the field will really arrive and be used.

7How Protocol Buffers Make Compatibility Work

The compatibility promise is possible because of a few clever properties of the way protobuf reads and writes data.

Field Numbers on the Wire

When protobuf writes a message, it does not write field names. It writes each field’s number, a hint about the kind of value, and then the value itself. A receiver looks up each number in its own definition to learn what it means. This makes the messages small and fast, and it also means that names can be changed in generated code without affecting the bytes on the wire, as long as the numbers stay the same.

Unknown Fields

If a receiver meets a field number it does not know, because it was built from an older definition, it does not fail. Standard protobuf behaviour is to skip over fields it cannot understand. This is the key that makes forward compatibility possible. A newer sender can include extra information, and an older receiver simply ignores that extra information while still reading everything it does understand.

Missing Fields and Defaults

The opposite situation is also handled. If a sender is older and does not include a field that a newer receiver knows about, the receiver treats the field as absent and uses a sensible default value, such as zero or an empty string. This gives backward compatibility. Newer receivers keep working with older senders, as long as the new field was designed so that “absent” has a safe meaning.

SituationWhat Protobuf DoesName for the Property
Sender has a field the receiver does not knowReceiver skips it and continuesForward compatibility
Receiver expects a field the sender never writesReceiver uses the default valueBackward compatibility
Field number reused for another meaningReceiver misreads data without any errorBroken compatibility, so forbidden
Simple Analogy

Imagine reading a recipe card where the cook added a new line your copy does not have. You skip the line you do not recognise and keep cooking. If a line is missing from your copy, you assume the usual amount. You can still make dinner, though you may miss a small improvement.

The Limits of These Properties

Skipping unknown fields keeps the data flowing, but it does not give the old receiver the new information. If a new field carries something important, for instance a flag that changes how a span should be interpreted, an old receiver will process the span without knowing about it. This is acceptable only when the addition is designed so that ignoring it is harmless. Designers of new fields therefore ask a careful question: what happens if someone never reads this?

!
Common Mistake

Assuming that “the data was accepted” means “the data was fully understood.” An old receiver can accept a message and quietly discard the newest information in it. Success on the wire tells you nothing about whether every detail was used.

8JSON Encoding and Its Special Rules

OTLP can also travel as JSON, and the JSON form follows its own rules that matter for compatibility.

Why JSON Exists in OTLP

JSON is easy for humans to read and for tools in places such as web browsers to produce. OTLP over HTTP therefore allows the same messages to be written as JSON text instead of binary protobuf. This is helpful for debugging, quick tests and environments where binary encoding is awkward. It is not as compact or as fast as the binary form, so most production traffic still uses protobuf.

Rules That Differ From Plain Protobuf JSON

The standard way to turn protobuf messages into JSON has some defaults that were not ideal for telemetry, so OTLP defines its own adjustments. Trace and span identifiers are written as hexadecimal text rather than the base64 text that standard conversions would produce, because hexadecimal is how engineers recognise these identifiers. Enumerations are written as their integer values, not as names, which keeps messages compatible even when names change. Field names use lower camel case.

Rule

Hex Identifiers

Trace and span identifiers appear as hexadecimal strings, matching how they are shown in tools.

Rule

Integer Enumerations

Enumerated values use numbers, so a rename of a value name never breaks a message on the wire.

Rule

Camel Case Names

Fields use lower camel case, which gives one agreed spelling across all senders and receivers.

Rule

Ignore Unknown Fields

Receivers must skip fields they do not recognise, which gives JSON the same forward compatibility as binary.

Why This Matters for Versioning

These rules are part of the protocol’s compatibility story. Because enumerations travel as numbers and unknown fields are skipped, a JSON receiver built against an older definition continues to work when a sender starts using new fields or new enumeration values. If the rules had left names or strict validation in place, every addition could have broken JSON users. The project made these choices on purpose to keep additions safe.

Simple Analogy

Using numbers for enumerations is like labelling shelves by number instead of by a product name. Even if the shop renames the product, the shelf number stays the same, and nobody gets lost while looking for it.

i
Practical Tip

If you write a JSON receiver or a test tool, make sure it ignores unknown fields instead of rejecting them. A strict receiver that errors on unfamiliar fields will break the first time any sender uses a newer minor version of OTLP.

9No Handshake: What Happens When Versions Differ

OTLP does not ask the two sides to announce their versions, so everything depends on the compatibility rules working as designed.

No Version Negotiation

Some protocols begin with a greeting in which both sides say which version they speak and agree on the highest one they share. OTLP does not do this in the usual sense. A sender simply builds its message from the definitions it knows and sends it. A receiver simply reads whatever arrives using the definitions it knows. Compatibility comes from the design rules of the schema rather than from a conversation between the two programs.

Simple Analogy

It is like posting a letter. You do not call the recipient to agree which edition of the language dictionary to use. You simply write in a language you both know, trusting that new words will be understandable from context and that any unfamiliar new word will not ruin the whole letter.

The Four Meeting Cases

Because there is no handshake, it is useful to think through the four situations that can occur when a sender and a receiver were built from different minor versions of the same stable major version.

CaseSenderReceiverResult
1Same versionSame versionEverything is understood
2Newer minor versionOlder minor versionWorks, the receiver ignores new fields it does not know
3Older minor versionNewer minor versionWorks, the receiver uses defaults for fields never sent
4Different major versionAnything elseNo guarantee, needs a deliberate bridge or a joint upgrade

What You Gain and Lose in Case Two

Case two is the one most teams meet during rolling upgrades. A newer application sends data to an older collector. The data flows, so no alarm sounds. But any brand-new field is dropped at the first receiver that does not know it. If your goal in upgrading the application was to start sending a new field, you will not see its benefit until the collector, and every receiver after it, is upgraded too.

flowchart LR
  S["New SDK sends data with a new field"] --> C1["Old Collector: does not know the field"]
  C1 --> D["Field is dropped silently"]
  C1 --> B["Backend receives the older shape of data"]
        
FIG 3 — New information is lost at the first receiver that cannot read it
!
Common Mistake

Upgrading senders first and then wondering why a promised new capability never shows up in the backend. Because ignorance is silent, the missing information leaves no error to find. Check every hop on the path, not just the two ends.

How Designers Make New Fields Safe to Ignore

The rule that older programs ignore new fields only works if each new field is designed with that in mind. Designers therefore ask a set of questions before adding anything. They ask what an older receiver will do if it never reads the field. They ask whether the data would be misleading without the new field. They ask whether a sensible default exists for senders that never write it. If the answers are comfortable, the field is added as an optional extra. If not, the design is reworked.

A Safe Design

A new optional field gives extra detail about an item, and a missing value simply means “no extra detail.” Old receivers lose nothing they relied on, and new receivers gain a little more.

A Risky Design

A new field changes how an existing number must be interpreted. If an old receiver ignores the field, it reads the number the old way and produces wrong results with no error. Designs like this are avoided or delayed until a major version.

A Safer Rework

Instead of changing the meaning of an existing field, a new field is added with the new meaning, and the old field is deprecated. Old receivers keep using the old field correctly, and new ones prefer the new one.

The Add-Then-Deprecate Pattern

The safe way to replace an old design is a three-step dance. First add the better field alongside the old one. Second, mark the old one as deprecated and update documentation. Third, wait through a long period, even years, during which both exist. Only a future major version removes the old field. This slow dance may look cautious, but it is what allows thousands of independent teams to migrate on their own timelines.

Simple Analogy

It is like replacing a bridge. You do not demolish the old one on the day the new one opens. You let people cross either one for a long time, put up signs encouraging the new one, and close the old one only when almost nobody is left.

10Package Names and URL Paths: Where “v1” Appears

The text “v1” appears in several places in OTLP, and it is worth knowing exactly what each one tells you.

Version in the Package Name

In the proto files, each group of messages lives in a package whose name ends with a version segment, such as the first generation of the trace messages. This is how protobuf projects usually signal a major version. The segment changes only when a deliberately incompatible generation is created. It means that a future second generation could exist next to the first, so old and new programs can coexist while teams migrate.

Version in the HTTP Path

When OTLP travels over HTTP, each signal has its own address path, which begins with a version segment followed by the signal name, such as the path for traces, the path for metrics and the path for logs. The version segment in these paths follows the major version of the protocol. Proxies, gateways and firewalls often route by path, so the version in the path is a handy signpost for network teams.

Place 1

Package Name

Tells code generators which generation of message definitions are in use. Changes only for major, incompatible generations.

Place 2

HTTP Path

Lets network devices route traffic by signal and generation, and keeps old and new endpoints separate if ever needed.

Place 3

Release Tag

The full three-part number of the proto repository release, which is what changes with every minor addition.

Place 4

Library Dependency

The version of the generated protocol code that an SDK or the Collector pulls in at build time.

What a Second Generation Would Mean

If the project ever needed a truly breaking change, it would introduce new packages and new HTTP paths while keeping the old ones. Receivers could then offer both at the same time and let senders migrate at their own pace. This approach makes a large breaking change survivable, because it replaces a risky day of everyone switching together with a long, gentle transition.

Simple Analogy

A new road is built beside the old road before the old one is closed. Drivers move across over months, and when the old road finally closes, almost nobody is still using it.

i
Practical Tip

When configuring a proxy or load balancer for OTLP over HTTP, write rules for the path of each signal explicitly. If you ever need to support a future generation, you can add its paths beside the existing ones without disturbing current traffic.

11How OTLP Has Actually Evolved

Looking at real kinds of change shows the rules in action and builds an intuition for what future changes will look like.

Renames and Clean-Ups Before the Stable Line

During the zero-version era the project made several large corrections. A concept originally called instrumentation library was renamed to instrumentation scope, which better describes the part of code that produced the data. Some early metric message types were merged or removed once a simpler model was agreed. These changes were allowed because the protocol had not yet promised stability, and doing them early avoided carrying awkward names forever.

Additions After Stabilisation

Once the core signals became stable, changes took the form of additions. New optional fields were added to carry extra detail, new kinds of values were allowed in attributes, and new message types were introduced for richer metrics. Newer additions have included extra flags on spans and links to describe context more fully, and a name for log events. In each case, older programs that did not understand the addition simply ignored it.

Example Pattern: A New Optional Field

A field is added to carry extra information about a span. Old receivers skip it. New receivers read it and offer better features. No one is forced to upgrade, and the benefit grows gradually as more of the pipeline is upgraded.

Example Pattern: A New Metric Kind

A richer histogram type is added to the metrics definition. Existing metric types keep working unchanged, so senders that never use the new kind are unaffected, and the new kind is used where it brings value.

Example Pattern: A Deprecated Field

A field that has been replaced by a better design is marked deprecated but is never removed or reused within the major version. Teams migrate on their own schedule, and tooling can warn them.

The Design Habit Behind Additions

Behind each of these additions is a consistent habit. The proposal is discussed in the open, a careful look is taken at what happens if an older program ignores the new field, and the design is adjusted so that ignoring it is harmless. Where ignoring would cause harm, the change is delayed or reshaped. This habit is what makes the stable promise believable over many years.

Kind of ChangeEraEffect on Older Programs
Rename or remove messagesBefore the stable lineCould break, which is why it was done early
Add optional fieldAfter the stable lineIgnored safely
Add new message type or signalAfter the stable lineNot used by older programs, no harm
Deprecate a fieldAfter the stable lineContinues to work until a future major version
Simple Analogy

Early in a city’s life, planners can still move streets. After people have built houses along them, they only add side roads and new signs. Streets already in use are left where they are.

i
Practical Tip

Read the changelog of the proto repository when you upgrade. It lists each addition and deprecation in plain terms, which helps you decide whether a particular release brings anything you want to use.

12Experimental Signals and Development Stability

Not everything in OTLP is stable, and the project deliberately gives new areas room to change before promising permanence.

Why New Areas Start as Development

Designing a data model that will serve the world for a decade is hard to do on the first attempt. The OpenTelemetry project therefore introduces new signals in a development stage. In that stage, the shape of the data can still change as real users try it and report problems. Only after enough experience does the area move to stable and receive the full compatibility promise.

How Development Parts Are Marked

Development areas are labelled in the specification and in the definitions. They may live in packages whose names signal their status, so nobody can mistake them for the stable generation. Endpoints for such signals may exist, but they carry the warning that they can change. Treating the label seriously is the best protection against surprise breakage.

1

Proposal

An idea for a new signal or feature is described and discussed in the open, with its data model sketched out.

2

Development stage

Definitions are published with a development label. They may change, and early adopters give feedback.

3

Stabilisation

After the design settles and several implementations interoperate, the area is declared stable with a compatibility promise.

4

Long-term maintenance

Only additive changes are made. Retired parts are deprecated and kept until a future major version.

Simple Analogy

A development-stage signal is like a prototype car at a trade show. You can sit in it and tell the designers what you think, but you would not buy it to drive to work tomorrow, because the steering wheel might be moved before the car goes on sale.

Advantages of Trying Development Features

  • You gain early access to new capabilities
  • You can influence the final design with feedback
  • You learn about future needs before others do

Disadvantages / Trade-offs

  • No promise that your data will stay readable after an upgrade
  • Backends and tools may not support it yet
  • You may need to rework your setup when the design changes
!
Common Mistake

Enabling a development-stage signal across a whole production fleet. Try it first in a limited test environment, and keep it isolated so a future breaking change affects only the experiment.

13Schema URL and Semantic Convention Versioning

A different kind of versioning travels inside OTLP messages, and it describes the meaning of attribute names rather than the shape of the messages.

Shape versus Meaning

The OTLP definitions describe the shape of messages, such as “a span has a name, a start time and a list of attributes.” But the attributes inside are just names and values, and what those names mean is decided by a separate set of rules called semantic conventions. One version of those conventions might call an attribute by one name, while a later version renames it to improve consistency. The shape of the message did not change at all, but its meaning did.

Simple Analogy

The shape of a form is the layout of boxes. The meaning is what you are supposed to write in each box. If the form maker changes “Surname” to “Family name” but leaves the box in the same place, the form is the same shape and different in meaning to anyone with an old guide.

What a Schema URL Is

To help with this, OTLP messages can carry a schema URL on resource groups and scope groups. It is a web address that identifies which version of the semantic conventions the data follows. A tool that understands schemas can read the address, fetch the published list of renames between versions, and translate attribute names from one version to another. This keeps dashboards and alerts working even when different senders use different convention versions.

flowchart LR
  A["Sender with conventions version A"] --> M["OTLP message with schema URL A"]
  B["Sender with conventions version B"] --> N["OTLP message with schema URL B"]
  M --> T["Schema-aware tool translates names"]
  N --> T
  T --> D["Dashboards see one consistent naming"]
        
FIG 4 — Schema URLs let tools reconcile different attribute naming versions

Why This Is Not Protocol Versioning

QuestionOTLP Protocol VersionSemantic Convention Version
What does it describe?The structure of messagesThe names and meanings of attributes
What breaks if it changes badly?Parsing of messagesDashboards, alerts and queries
How is the change signalled?Release number and package namesSchema URL and convention release notes
Who is affected?Senders, receivers, vendorsAnyone who writes queries on attribute names
!
Common Mistake

Believing that a compatible protocol upgrade means dashboards will keep working. An SDK upgrade can rename attributes through a convention change while the protocol stays perfectly compatible, and your queries can silently stop matching. Review convention changes separately.

Surviving a Convention Rename in Practice

Semantic convention changes are the version changes most likely to hurt your dashboards, so it pays to have a routine for them. When a new SDK or instrumentation library release renames an attribute, the old name will stop appearing on new data while still existing on historical data. Queries that filter on only one name will see a sudden cliff in the charts at the moment of the upgrade.

1

Find out what is changing

Read the convention release notes and list attributes that are renamed, removed or given new meanings.

2

Search your own queries

Look through dashboards, alert rules and saved searches for the old names, so you know exactly what will be affected.

3

Bridge the gap

Update queries to accept both old and new names for a while, or add a collector rule that copies one name to the other for the transition period.

4

Roll out and watch

Upgrade in stages and compare charts before and after for each stage, looking for cliffs and gaps.

5

Retire the bridge

Once all services emit the new names and old data has aged out of your retention window, remove the temporary bridge.

Simple Analogy

A rename is like a shop changing the label on a shelf from “biscuits” to “cookies.” Customers with an old shopping list walk past the shelf without noticing it. A temporary sign saying “also called biscuits” bridges the gap until everyone has learned the new word.

!
Common Mistake

Updating the alert queries only after the upgrade, when alerts have already gone quiet. Alerts that depend on renamed attributes can stop firing without warning, which is the most dangerous outcome. Fix alert queries before the rollout, not after.

14Partial Success and Error Reporting Across Versions

How a receiver reports problems is part of the protocol, and it must also stay compatible as the protocol grows.

Full Success, Failure and Partial Success

When a sender exports a batch, the receiver replies. In the simplest case the whole batch is accepted. In another case the whole batch is rejected with an error. But a third case is common in real systems: some items are fine and some are not, for example because a few spans were malformed. For this, OTLP responses can include a partial success section that states how many items were rejected and gives a message explaining why.

Why Partial Success Helps Compatibility

Partial success lets a receiver say “I took what I could, and here is how many I could not use.” This is valuable during version mismatches. If a receiver refuses a few items because of a limit or validation rule, the sender can learn about it and the rest of the batch is not wasted. Because it is an optional part of the response, older senders that do not look at it continue to work without change.

Outcome

Full Success

Every item in the batch was accepted, and the response contains no complaint.

Outcome

Partial Success

The batch was accepted, but some items were rejected, with a count and a human-readable message.

Outcome

Failure

The whole request failed, and the response carries an error that tells the sender whether trying again can help.

Outcome

Warning

Some receivers use the partial success message with a zero count to pass on a warning without rejecting anything.

Simple Analogy

A post office might say, “We accepted nine of your ten parcels and one was refused because the address was missing.” You know exactly what to fix, and the nine good parcels are already on their way.

Reading Warnings as Early Signs of Drift

A rising count of rejected items after an upgrade is one of the best early signs of a version problem. For example, a newer sender may send something that a stricter receiver now rejects. Watching the rejection counts on your collectors and backends, and reading the messages that explain them, helps you catch mismatches before users notice missing data.

i
Practical Tip

Make sure your exporters log the partial success messages they receive. Many teams discover that months of data were being quietly rejected only because nobody ever read these responses.

15Retries, Throttling and Status Codes

Behaviour rules, not just message shapes, are part of the protocol, and they also need to stay stable as versions move.

Which Failures Are Worth Retrying

The OTLP specification tells senders which failures are temporary and which are permanent. A temporary failure, such as a receiver that is briefly unavailable or too busy, should be retried after a delay. A permanent failure, such as data that is invalid, should not be retried, since sending the same bad data again can never succeed. Both gRPC and HTTP have their own way of expressing this, using status codes that the specification maps to retry behaviour.

Throttling Signals

When a receiver is overloaded, it can ask senders to slow down. In gRPC it can include retry information that says how long to wait, and in HTTP it can use a standard header with a delay value. Senders are expected to respect these hints and to back off with growing delays and some randomness, so that thousands of senders do not all retry at the same instant.

SituationRetry?Typical Behaviour
Receiver temporarily unavailableYesWait and retry with growing delays
Receiver overloaded and asking for a pauseYesRespect the stated delay, then retry
Request invalid or malformedNoDrop the batch and report the error
Authentication or permission refusedNoFix credentials before sending again
Simple Analogy

If a shop door is locked because the staff are on a short break, you come back in ten minutes. If the shop tells you it does not sell what you want, coming back every ten minutes will never help. Knowing which situation you are in saves everyone’s time.

Why This Matters for Versions

Because these behaviours are written into the protocol, SDKs and collectors from different times act in predictable ways toward each other. An older sender that follows the retry rules handles a newer receiver’s throttling hints correctly. A newer sender treats an older receiver’s responses in the same standard way. Predictable behaviour under stress is just as important as predictable message shapes, since failures often arrive during upgrades.

!
Common Mistake

Retrying forever with no limits, or retrying permanent failures. Endless retries pile up memory on the sender and can turn a small version mismatch into a pipeline-wide outage. Always bound retries by time and by queue size.

16The Collector’s Role in Version Differences

The Collector sits in the middle of most pipelines, which makes it the natural place to absorb and manage version differences.

The Collector Has Its Own Internal Model

The Collector does not simply pass bytes through. It reads incoming data into its own internal representation, applies processors, and then writes the data out again using each exporter’s format. This means the Collector’s receivers and exporters are translation points. A receiver reads the incoming protocol version it knows, and an exporter writes the outgoing version it knows. Between them, the internal model keeps things consistent.

flowchart LR
  I["Incoming OTLP from many SDK versions"] --> R["OTLP receiver reads known fields"]
  R --> M["Internal data model"]
  M --> P["Processors"]
  P --> E["OTLP exporter writes the Collector's version"]
  E --> O["Outgoing OTLP to backends"]
        
FIG 5 — The Collector reads one version and writes another through an internal model

What This Means in Practice

Because of this reading and writing, a Collector acts like a normalising gateway. A fleet of applications using many different SDK versions can send to one Collector build, and the output has the shape that Collector knows. This is a big reason why platform teams like to put a Collector between applications and backends. They can upgrade the backend-facing side on their own schedule, without needing every application to move at the same time.

Information the Collector Cannot Pass Along

The same translation means that a field unknown to the Collector build is not carried through, because the Collector never stored it in its internal model. This is the same loss described earlier, and it explains why keeping the Collector reasonably current matters whenever applications start using new protocol features. The Collector sets the ceiling for how new the information flowing through it can be.

Normalising Gateway

A central Collector pool receives from hundreds of services using several SDK versions, and exports to the backend in a single consistent form. Backend teams see one stable shape.

Staged Upgrade Buffer

When a backend adopts a newer OTLP release, only the Collector exporters need to change. Applications can follow gradually without any cut-over day.

Version Ceiling

An application upgrade that introduces a new field has no effect on stored data until the Collector build is upgraded to a version that understands the field.

Simple Analogy

The Collector is like an interpreter at a meeting. Speakers use slightly different dialects, and the interpreter converts everything into one clear version for the listeners. But if the interpreter has never learned a new word, that word does not get passed along.

i
Practical Tip

Keep a short list of the Collector build versions in each environment, and the SDK versions sending to them. When something seems missing, comparing these two lists often shows immediately which side is behind.

Component Stability Is Another Layer

The Collector is built from many components, and each component carries its own stability level, separate from the stability of the protocol it speaks. A component might be labelled alpha, beta or stable, which tells you how likely its settings and behaviour are to change between releases. The OTLP receiver and exporter are among the most mature, but a processor or receiver you depend on for other work may be less settled. Reading these labels is part of managing versions in your own pipeline.

Stability LabelMeaning for Settings and BehaviourSuggested Use
AlphaMay change or be removed with little warningExperiments and learning only
BetaMostly settled, but changes are still possibleCareful use with testing before each upgrade
StableChanges follow a clear compatibility policySafe for production with normal upgrade care
DeprecatedScheduled for removalPlan migration away from it

Why the Collector Version Matters Even When the Protocol Is Stable

Even if the protocol never changes in a breaking way, a new Collector release can still change how a component’s settings are named or how a processor behaves. These are changes to the program, not to the protocol, and they follow their own release notes. When you upgrade a Collector, read both: the notes about the protocol definitions it bundles, and the notes about the components you use. A pipeline can break because of a renamed setting while the protocol is perfectly fine.

Simple Analogy

A language can stay the same while a translator’s office changes its opening hours and its rules for submitting documents. Your message is still understandable, but you may find the front door locked if you do not read the notice.

i
Practical Tip

Before each Collector upgrade, run its configuration check against your existing setup in a test environment. A quick validation catches renamed or removed settings long before they reach a production Pod.

17Upgrade Strategy: Receivers First, Senders Second

The order in which you upgrade matters more than the speed, and a simple rule keeps almost every rollout safe.

The Golden Rule of Ordering

Because older receivers ignore fields they do not know, the safest order is to upgrade the receivers before the senders. If receivers are upgraded first, they are ready to understand anything a newer sender might add. If senders are upgraded first, they may start sending information that the receivers silently drop. Following the flow of data backwards, from the backend toward the applications, keeps each step safe.

Simple Analogy

It is like opening a new lane on a motorway. You build the exit and the road at the destination first, and only then tell drivers to use the new lane. Telling drivers first sends them toward a road that is not ready.

1

Upgrade or verify the backend

Confirm the backend accepts the OTLP release you plan to use, including any new fields you intend to rely on.

2

Upgrade gateway collectors

Update the central Collector tier next, so it understands newer data and can write the version the backend expects.

3

Upgrade node-level agents

Move the agent collectors that sit close to the applications, since they send to the gateways you have already prepared.

4

Upgrade SDKs last

Roll new SDK versions through applications in stages, starting with low-risk services and watching for rejected data.

5

Review and clean up

After everything has moved, remove temporary compatibility settings and record the versions now in use.

Canaries and Gradual Rollouts

Even safe additions deserve a gentle rollout. Upgrade a small slice of the pipeline first, such as one collector replica or one low-risk application, and compare its outputs with those of the old version. Look at accepted counts, rejected counts, latency and the shape of the stored data. If everything looks the same or better, widen the rollout.

!
Common Mistake

Upgrading everything on the same day to “get it over with.” If a problem appears, you cannot tell which change caused it, and rolling back means undoing every layer at once. Stagger the layers and watch each one.

Always Have a Way Back

A good upgrade plan includes a rollback plan written before the first change. For collectors, this usually means keeping the previous image version and configuration ready, and being able to redeploy them quickly. For SDKs, it means being able to revert the dependency and redeploy the application. Because protocol additions are compatible, rolling back a sender or a receiver is usually safe, since the older version already understands everything it did before.

Living in a Mixed Fleet for a Long Time

Few organisations ever reach a moment when every component is on the same version. Plan for mixed fleets as the normal state of affairs. Make sure that dashboards and alerts do not depend on features that only some services can produce. When you do rely on a newer field, document which services provide it, and treat missing values as expected for services still on older versions.

Rollback Example: A Collector Misbehaves

After a gateway upgrade, rejected counts rise. The team redeploys the previous image within minutes, confirms counts return to normal, and then investigates in a test environment without time pressure.

Mixed Fleet Example: A Slow-Moving Team

One department upgrades its SDKs twice a year, while others upgrade monthly. The shared collectors handle both, and dashboards that need a new field are marked to say which services provide it.

Advantages of Mixed-Fleet Tolerance

  • Teams upgrade when they are ready, not on a single day
  • Problems affect only the slice that has changed
  • Rollbacks are simple and low-risk

Disadvantages / Trade-offs

  • Features available to some services are missing for others
  • Troubleshooting must consider several versions at once
  • An inventory is needed to avoid drifting out of the supported range
Simple Analogy

A mixed fleet is like a road with cars of many ages. The traffic rules are the same for all of them, so everyone gets home, even though the newest cars have features the oldest ones do not.

18Testing Compatibility Before It Hurts

Trust in compatibility rules is good, but testing them in your own environment is better, since your setup always has its own quirks.

Contract Tests Between Senders and Receivers

A contract test sends a known batch of data from a sender build to a receiver build and checks what comes out. Run it for each pairing that exists in your fleet: current sender to current receiver, newer sender to older receiver, and older sender to newer receiver. The test confirms that nothing breaks and shows exactly which fields survive, which makes the silent drops described earlier visible.

Replaying Real Traffic

Sample files captured from real production traffic are valuable test material, because they contain the odd shapes that handwritten tests miss. Capture a small, safely anonymised sample, store it with its version, and replay it against new collectors and backends in a test environment. Compare the results with those from the current version to spot changes in counts, attribute names or timestamps.

Test 1

Pairing Matrix

Try each combination of sender and receiver versions you run, and record the outcome in a simple table.

Test 2

Golden Files

Keep reference inputs and expected outputs, and flag any difference after an upgrade for human review.

Test 3

Replay Tests

Feed anonymised real traffic through the new pipeline in a test environment and compare to production results.

Test 4

Strictness Probes

Send messages with unknown fields to your own receivers and tools to confirm they ignore them without failing.

Checking Definitions Automatically

Teams that maintain their own protocol definitions, or that depend on a specific release, can use automated tools to compare two versions of the definitions and report any breaking change. The OpenTelemetry project itself uses automated checks of this kind on its protocol repository, which is part of why the stable promise holds. In your own pipelines, a check on dependency updates that flags protocol definition changes gives you early warning.

Simple Analogy

Contract tests are like a rehearsal before a concert. The musicians may all know their parts, but only by playing together do you discover that one of them has a different edition of the sheet music.

i
Practical Tip

Add one small contract test to your continuous integration that runs whenever a Collector or SDK version changes. A test that takes a minute to run can save a day of investigation later.

19Spotting and Fixing Version Drift

Version drift means that different parts of your fleet slowly end up on different versions, and it causes most real-world protocol surprises.

How Drift Happens

Nobody plans drift. It appears because teams upgrade at different speeds, some services are rarely touched, and some images are built once and run for years. After a year, a company may have a dozen different SDK versions in production and several Collector builds. Because the stable rules keep most combinations working, drift stays invisible until a specific mismatch finally matters.

Symptoms to Watch For

SymptomLikely Version CauseFirst Check
New feature never shows in the backendAn older receiver dropped the new fieldCompare versions on every hop of the path
Rising rejected counts after an upgradeA stricter receiver or a changed validation ruleRead the partial success messages
Dashboards suddenly empty for some servicesAttribute names changed through a convention updateCompare schema URLs and convention releases
Receiver errors on JSON from one toolTool uses wrong identifier encoding or strict parsingCheck hexadecimal identifiers and unknown field handling
Data from a development signal vanishesDefinitions changed between releasesRead the changelog for development-stage changes

Fixing Drift Gently

The fix for drift is awareness and a regular rhythm. Keep an inventory of the versions in use. Set a policy for how old a version may become. Offer teams an easy upgrade path through shared base images or automated dependency updates. Review the inventory on a schedule, and prioritise the oldest, busiest services first.

Simple Analogy

Drift is like the tiny differences that appear when a group of friends copies a recipe by hand over many years. Each copy is nearly the same, yet a few have lost a line, and one day dinner tastes wrong for no obvious reason.

i
Practical Tip

Attach SDK and Collector versions as resource attributes on your telemetry. Then you can query your own data to see which versions are running where, with no separate spreadsheet that goes out of date.

A Worked Investigation From Start to Finish

To see how these ideas fit together, follow a realistic investigation. A team upgrades one service to a newer SDK, expecting a new piece of span detail to appear in their tracing tool. Two weeks later, the detail is still missing. Nothing is failing, and nobody is sure where to look. The team follows a calm routine and finds the cause within an hour.

1

State the symptom clearly

“Spans from this service lack the new detail, although the SDK documentation says it should be present.”

2

Confirm the sender

Print the outgoing data from a test run of the service and check that the new detail is actually produced. It is.

3

Check each hop

Compare the version of the Collector at each hop with the version that introduced the field. The first gateway is two releases older.

4

Understand the cause

The older gateway does not know the new field and drops it. No error is produced, because skipping unknown fields is normal behaviour.

5

Fix and prevent

Upgrade the gateway, then add the receiver-first rule and a version inventory to the team’s upgrade checklist.

Simple Analogy

It is like adding a new ingredient to a recipe sent through three kitchens. If the middle kitchen has an older recipe card without the ingredient, the dish arrives without it, and no cook ever complained, because each followed their own card correctly.

i
Practical Tip

The most useful debugging habit is to inspect the data at each hop, from the sender onward, until the detail disappears. The place where it vanishes is almost always the place that needs upgrading.

20Anti-Patterns in Handling OTLP Versions

Naming common mistakes makes them easier to avoid in design reviews and upgrade plans.

ANTI-PATTERN-01 Avoid
Problem

Senders are upgraded first, because application teams move faster than platform teams.

Why It’s Harmful

New fields are dropped silently by older receivers, so the promised benefits never appear and nobody receives an error to investigate.

Correct Approach

Upgrade backends and collectors first, then agents, then SDKs, and verify each layer before moving to the next.

ANTI-PATTERN-02 Avoid
Problem

Strict receivers or tools reject any message that contains a field they do not recognise.

Why It’s Harmful

They violate the compatibility rules and break the first time any sender uses a newer minor version, turning a safe addition into an outage.

Correct Approach

Ignore unknown fields and unknown enumeration values, and log them at a low level so you can still notice them.

ANTI-PATTERN-03 Avoid
Problem

Development-stage signals are enabled across the whole fleet because the features look attractive.

Why It’s Harmful

The shape of the data may change without a compatibility promise, so a routine upgrade can break pipelines and lose data.

Correct Approach

Limit development-stage signals to test environments or a small isolated slice, and plan to adapt when they change.

ANTI-PATTERN-04 Avoid
Problem

Protocol versions and semantic convention versions are treated as one thing.

Why It’s Harmful

A protocol-compatible upgrade can still rename attributes and break queries, but teams relax because the protocol rules say it is safe.

Correct Approach

Review protocol changes and convention changes separately, and use schema URLs and release notes to find renamed attributes.

ANTI-PATTERN-05 Avoid
Problem

Nobody knows which SDK and Collector versions are running, and upgrades happen only when something breaks.

Why It’s Harmful

Drift grows unnoticed, troubleshooting is slow, and security fixes reach only some services.

Correct Approach

Record versions as resource attributes, keep an inventory, set a maximum age policy and review it regularly.

21Governance: A Version Policy for Your Organisation

Protocol compatibility is a shared promise, and the organisations that benefit most turn it into a short, clear policy.

What a Good Policy Contains

A useful policy fits on one page. It states which OTLP releases are supported by your backends and collectors, how many versions behind an SDK may be, who owns each layer’s upgrades, and how experimental signals are approved. It names the upgrade order and the testing required before a change is rolled out. Writing it down removes guesswork and speeds up decisions during busy periods.

Clause 1

Supported Range

List the oldest and newest SDK and Collector versions that are allowed to run in production.

Clause 2

Upgrade Order

State that receivers move first, then agents, then SDKs, with a stated pause between stages.

Clause 3

Experimental Rules

Require approval and isolation for any development-stage signal, with a named owner.

Clause 4

Evidence of Testing

Require a contract test result and a canary period before any protocol-affecting change goes wide.

Roles and Ownership

Decide who is responsible for what. A platform team usually owns the Collector tier and the backend relationship, and offers shared base images or libraries to application teams. Application teams own their own SDK upgrades within the supported range. A small group, perhaps a guild, watches upstream releases and announces changes that matter. Clear ownership prevents the common situation in which everyone assumes someone else is watching.

Simple Analogy

A version policy is like the traffic rules of a city. Most drivers follow them without thinking, and they make busy intersections work. Without them, even a city full of careful drivers produces accidents at every corner.

Advantages of a Written Policy

  • Decisions are faster and less controversial
  • Teams know what is expected of them
  • Security updates reach all services sooner
  • New staff learn the rules quickly

Disadvantages / Trade-offs

  • Needs regular review to avoid going stale
  • Can feel like bureaucracy if made too strict
  • Requires tooling or effort to measure compliance

Measuring Version Health

A policy is only useful if you can see whether it is being followed. A few simple measures give a clear view without much effort. Count how many services run each SDK version, and how far behind the newest release they are. Count how many collectors run each build. Track the rate of rejected and refused data across the pipeline. And track how many services use any development-stage signals. Reviewing these numbers monthly turns vague worries about drift into concrete, manageable lists.

Spread
How many versions are in use
Lag
How far behind the newest release
Rejects
Data refused by receivers

Communicating Changes Well

Most version problems are not technical failures but communication failures. A platform team upgrades a Collector without telling anyone, and application teams see unexplained changes. Prevent this with a short, regular announcement that states what will change, when, what to watch for, and whom to contact. Keep a running changelog of your pipeline’s versions, so anyone investigating a strange chart can see what changed that week.

i
Practical Tip

Post version changes in the same channel where teams already discuss incidents. When someone asks why a chart changed, the answer should be only one search away.

22Real-World Scenarios and Trade-offs

A few realistic situations show how the ideas in this guide play out, and where the trade-offs lie.

Scenario: A New Optional Field Arrives

A new OTLP release adds a field you want. You upgrade the backend, then the gateway collectors, then the agents, and finally the SDKs. At each stage you check that data keeps arriving unchanged. Only after the whole path understands the field do you start relying on it in dashboards.

Scenario: A Legacy Service Stuck on an Old SDK

One old service cannot be upgraded for months. Thanks to the compatibility rules, it keeps sending older data that every newer receiver can read. You lose the newest features for that service, but nothing breaks, and you record the exception in your inventory.

Scenario: A Convention Rename Breaks Dashboards

An SDK upgrade changes an attribute name through a new convention release. The protocol is fully compatible, yet charts that filter on the old name go blank. The team uses schema information and release notes to find the rename and updates the queries.

Scenario: Trying a Development-Stage Signal

A team wants to experiment with a development-stage signal. They isolate it in a test cluster with its own collector, accept that it may change, and keep production pipelines on stable signals only.

ChoiceBenefitCost
Stay on an older release longerFewer changes to manageMiss new capabilities and fixes
Upgrade promptly after each releaseEarly access to improvementsMore frequent testing work
Use a Collector as a normalising gatewayDecouples application and backend upgradesOne more tier to run and watch
Send directly from SDK to backendSimplest path with fewer partsEvery application must match the backend’s expectations
“Compatibility is a promise about what will not change, and your job is to make sure you keep your side of it.”

23Frequently Asked Questions

Short, direct answers to the questions people ask most often about OTLP versioning.

Q1Does OTLP have a version I must put in my configuration?

Not in the usual sense. You choose a transport and encoding, such as gRPC or HTTP with protobuf, but you do not declare a protocol version. The version is determined by the definitions built into your SDK or Collector, and the compatibility rules let different versions interoperate.

Q2Can a newer SDK send to an older Collector?

Yes, within a stable major version. The older Collector skips fields it does not know and processes the rest. The cost is that any brand-new information in the message is lost at that Collector, so you only get the full benefit once it is upgraded.

Q3Can an older SDK send to a newer Collector?

Yes. The newer Collector treats fields that were never sent as absent and uses default values. Features that depend on newer fields simply will not be present for that data, but nothing fails.

Q4What does the “v1” in the HTTP path mean?

It identifies the major generation of the protocol for that signal’s address. It changes only for deliberately incompatible generations. Minor additions do not change it, which is why proxy rules written for these paths keep working across many releases.

Q5Is the OTLP version the same as the OpenTelemetry specification version?

No. The specification, the protocol definitions, the SDKs and the Collector each have their own version numbers and release rhythms. They influence each other, but a number from one layer says little about another layer.

Q6What is the schema URL for?

It identifies which version of the semantic conventions the data follows, so tools can translate renamed attributes between versions. It is about the meaning of names, not the shape of messages, and it is separate from protocol versioning.

Q7Which signals are stable in OTLP?

Traces, metrics and logs are the core stable signals. Newer areas such as profiling have been developed under a development label. Always check the current specification for the exact status, since stability levels change as the project matures.

Q8Why is JSON encoding different from standard protobuf JSON?

OTLP adjusts a few defaults to suit telemetry. Identifiers are written as hexadecimal, enumerations as numbers and unknown fields must be ignored. These choices keep the JSON form readable and compatible as the protocol grows.

Q9How do I know if a receiver rejected part of my data?

Look for the partial success information in the response, which gives a count of rejected items and a message. Make sure your exporters or collectors log it, and watch the rejected counts on your pipeline dashboards.

Q10What is the safest upgrade order?

Upgrade receivers before senders. That means the backend first, then gateway collectors, then node agents, then SDKs, with checks and a pause after each stage. This order ensures new fields are never sent to something that cannot read them.

Q11Will there ever be an OTLP version two?

Only if a change is needed that cannot be made compatibly, and the project has designed the packages and paths so that a second generation could run beside the first during a long transition. Nothing in your current setup requires you to plan for that today.

Q12How do I keep track of versions across hundreds of services?

Attach SDK and Collector versions as resource attributes on your telemetry and query them, keep a simple inventory, and set a policy for maximum age. Automated dependency updates and shared base images also reduce drift without much manual work.

Q13Why do old receivers not simply reject messages with unknown fields?

Rejecting them would make every new field a breaking change, because any addition would instantly stop older receivers from working. By skipping what they do not understand, older receivers allow the protocol to grow gradually. This is a deliberate design choice, and it is the heart of the forward compatibility promise.

Q14What is the difference between deprecated and removed?

Deprecated means the item still exists and still works, but the project asks people to stop using it because it will be removed later. Removed means it is gone. Within a stable major version, items are only deprecated. Removal can happen only in a new major version, with a long warning period beforehand.

Q15Does compression affect versioning?

No. Compression, such as gzip, is applied to the bytes after the message is built and removed before the message is read. It is independent of the message definitions, so the compatibility rules for fields apply equally whether compression is used or not.

Q16Do SDKs always use the newest OTLP definitions?

Not always. Each SDK release bundles a particular snapshot of the definitions, and it may take time before a new field is supported in every language. This is why capabilities differ between languages, and why you should check the documentation of each SDK you use for the features it supports.

Q17What should a backend vendor do to stay compatible?

A backend should accept the stable signals, ignore unknown fields, return clear partial success information, follow the retry and throttling rules, and document which optional fields it uses. Following these simple habits lets customers send data from a wide range of SDK and Collector versions without surprises.

Q18Can I mix gRPC and HTTP in the same pipeline?

Yes. Because both transports carry the same messages, you can use gRPC on one hop and HTTP on another. A Collector can receive on one and export on the other. Versioning rules are the same either way, so you can choose transports for operational reasons such as network policy.

Q19How can I check which fields a receiver actually uses?

Send a test message that contains the field and look at what appears in the stored data or in the receiver’s debug output. Documentation helps, but a real test is the most reliable answer, since a receiver can accept a field without making any use of it.

Q20What is the single best habit for managing OTLP versions?

Upgrade in the order of receivers first, then senders, and check the data at each stage. This one habit prevents most silent data loss, and it works across every version change you will ever meet, big or small, protocol, collector or SDK.

24Summary and Key Takeaways

OTLP versioning is the set of rules that lets the OpenTelemetry Protocol grow without breaking the programs that use it. The protocol is defined in proto files and follows semantic versioning, with stable signals promising only additive changes within a major version. Protocol Buffers make this possible through numbered fields, skipping of unknown fields and sensible defaults for missing ones. The JSON form follows adjusted rules that preserve the same compatibility.

There is no version handshake, so safety comes from the schema rules, the package and path naming, the stability labels and your own upgrade discipline. Treat the protocol version, the semantic convention version and the SDK and Collector versions as separate things. Upgrade receivers before senders, test the pairings in your fleet, watch for rejected data, and keep a short, written version policy. If you do these simple things, protocol changes become routine events instead of emergencies.

Perhaps the most reassuring lesson is that the system was designed with human reality in mind. Teams upgrade at different speeds, services linger on old releases, and mistakes happen. The compatibility rules, stability labels and gentle deprecation process exist precisely so that ordinary organisations can keep their telemetry flowing through all of this. Learn the rules once, write them into a short policy, and you can treat each new release as an opportunity instead of a threat.

Key Takeaways

  • Many layers, many versions — separate the protocol, the specification, SDKs, the Collector and semantic conventions, because each has its own number and promise.
  • Stable means additive — within a major version only compatible additions are allowed, and old fields are deprecated, never reused.
  • Protobuf does the heavy lifting — unknown fields are skipped and missing fields get defaults, which gives both forward and backward compatibility.
  • No handshake — compatibility rests on the schema rules, so a successful response does not prove that every new detail was understood.
  • Watch the labels — development-stage signals may change at any time, so isolate them from production.
  • Meaning is versioned separately — schema URLs and convention releases describe attribute names, and they can break queries even when the protocol is compatible.
  • Receivers first — upgrade backends, collectors, agents and SDKs in that order, with canaries and checks between stages.
  • Test and record — run contract tests, read partial success messages, and track versions with resource attributes to prevent drift.