Segments and Subsegments

Every time a request enters a service that X-Ray is watching, that service records a segment — a JSON document describing the work that service did: when it started, when it finished, whether it errored, and what resource handled it. If that service then calls out to something else — a database, another microservice, an external HTTP API — each of those calls is recorded as a subsegment, nested inside the parent segment. A segment is the outer shell for one service’s contribution; subsegments are the individual actions inside that shell.

Trace ID

A trace is the complete collection of segments and subsegments that share one trace ID — a unique identifier generated the moment a request first enters the system, usually at the load balancer or API Gateway. That trace ID is passed downstream through HTTP headers (specifically the X-Amzn-Trace-Id header) so that every subsequent service knows which story it is contributing to. Without a shared trace ID, X-Ray would just have a pile of disconnected segments with no way to reassemble them into one request’s journey.

Annotations vs. Metadata

X-Ray lets you attach extra information to a segment in two different ways, and the distinction matters more than it looks. Annotations are indexed key-value pairs — small, simple values like a customer tier or an order ID — that X-Ray makes searchable, so you can filter traces by them later. Metadata is not indexed; it is meant for larger, more detailed objects — full request payloads, stack traces, configuration dumps — that you want attached for context but never need to search across thousands of traces.

Sampling Rules

Tracing every single request in a system doing tens of thousands of requests per second would be prohibitively expensive and would flood the X-Ray backend with redundant data. Sampling rules let you define what fraction of requests actually get traced — for example, always trace the first request per second, plus 5% of everything after that. X-Ray ships with a default sampling rule but production systems almost always define custom rules, often sampling more heavily on error-prone or business-critical paths (like checkout) and less heavily on high-volume, low-risk paths (like a health-check endpoint).

Service Graph

The service graph (rendered as the Service Map in the X-Ray console) is the visual, aggregated view built from many traces — a node-and-edge diagram showing every service in your system and the call patterns between them, color-coded by error rate and latency. It is not one trace; it is a statistical summary built from thousands of them, which is why it is so useful for spotting which service is degrading right now, even before you dig into any individual trace.

Trace Groups

A trace group is a saved filter expression — for example, “all traces on the checkout API with a response time above two seconds” — that X-Ray keeps evaluating continuously rather than as a one-off search. Once defined, a trace group gets its own dedicated statistics (average latency, error rate, request count) tracked over time, separate from the system-wide aggregate. Intermediate teams use trace groups to keep a permanent, focused lens on the handful of request paths that matter most, instead of re-typing the same filter expression in the console every time they want to check on it.

Segment Documents and Fault Versus Error Versus Fault-Throttle States

Every segment carries a status beyond simple pass/fail. X-Ray distinguishes an error (a client-side 4xx-class failure — the caller did something wrong) from a fault (a server-side 5xx-class failure — the service itself broke) and further flags a throttle state when a request was rejected specifically because a rate limit was exceeded. This three-way distinction matters because it changes where you look for a fix: an error spike often points at a client or an upstream contract change, while a fault spike points at the service’s own code, dependencies, or resource limits.

These four pieces are deliberately decoupled. The SDK never talks directly to the X-Ray API over the network — it always goes through the local daemon. This matters enormously for performance, which the next chapter covers in detail, but the short version is: your application code should never block waiting on a slow network call just to record telemetry about itself.

flowchart LR
    Client["Client Request"] --> APIGW["API Gateway"]
    APIGW --> Lambda["Lambda Function
(built-in X-Ray)"]
    APIGW --> ALB["Application Load Balancer"]
    ALB --> ECS["ECS Service
+ X-Ray SDK"]
    Lambda --> DDB[("DynamoDB")]
    ECS --> Daemon["X-Ray Daemon
UDP :2000"]
    Lambda -. "trace data" .-> XRayAPI["X-Ray API"]
    Daemon -->|"batched segments"| XRayAPI
    XRayAPI --> Store[("Trace Storage")]
    Store --> Console["X-Ray Console
Service Map / Traces / Insights"]
        

Notice that Lambda takes a shortcut: because AWS controls the entire execution environment, Lambda functions can send trace data without you running a separate daemon process at all — X-Ray support is a checkbox (“Active Tracing”) rather than infrastructure you manage. On EC2 or ECS, you own that daemon and are responsible for its resource footprint and availability, which is a meaningful operational difference between “serverless X-Ray” and “X-Ray on servers you manage.”

Where X-Ray Sits Relative to OpenTelemetry

AWS also offers the AWS Distro for OpenTelemetry (ADOT), a vendor-neutral instrumentation layer that can export trace data either to X-Ray or to other backends like a third-party observability platform. The relationship between the two is architectural, not competitive: ADOT replaces the instrumentation piece (the SDK layer), while X-Ray can remain the storage and visualization backend underneath it. Intermediate teams evaluating a move away from proprietary SDK lock-in often adopt ADOT specifically so that switching backends later doesn’t require re-instrumenting every service from scratch.

Understanding X-Ray at an intermediate level means understanding why it almost never slows down the requests it is measuring. The mechanism is deliberately asynchronous and non-blocking at every stage.

Why UDP, Not HTTP

The X-Ray SDK sends segment data to the local daemon over UDP, not TCP or HTTP. UDP is fire-and-forget: the SDK does not wait for an acknowledgment, does not retry, and does not block your application thread if the daemon happens to be slow or briefly unavailable. The trade-off is that a UDP packet can be silently dropped — meaning a trace can occasionally be incomplete — but that trade-off is intentional. X-Ray’s designers decided that losing an occasional trace is acceptable; slowing down real user traffic because a telemetry daemon is under load is not.

Buffering and Batching at the Daemon

Once the daemon receives a UDP packet, it doesn’t immediately forward it. It buffers incoming segments in memory and periodically flushes a batch to the X-Ray API, both to reduce the number of outbound API calls and to smooth out bursts of traffic. This buffering is also why a sudden daemon crash can lose whatever was sitting in the buffer at that moment — another reason X-Ray is built around the assumption of “mostly complete” data rather than guaranteed delivery.

Trace Assembly on the Backend

The X-Ray API does not require all segments for a trace to arrive at once, or even in order. Because every segment carries the same trace ID, the backend can receive a Lambda segment, an ECS segment, and a DynamoDB subsegment in any order, across a window of time, and still correctly assemble them into one coherent trace once enough pieces have arrived. This is what allows X-Ray to trace requests across services that have no direct network path to each other and no shared clock beyond the trace ID itself.

Context Propagation

For a trace to stay connected across service boundaries, the trace ID (and the ID of the currently active segment, called the parent ID) must travel with the request itself. X-Ray does this through the X-Amzn-Trace-Id HTTP header. Each SDK is responsible for reading that header on incoming requests and writing it on outgoing calls it makes to other traced services. If any single hop in the chain fails to propagate that header — a common issue when a request passes through an untraced proxy, a message queue, or a piece of legacy code — the trace effectively breaks in two, and X-Ray will show it as two separate, disconnected traces instead of one continuous story.

Reservoir and Rate: How a Sampling Rule Actually Decides

A sampling rule isn’t a single percentage — it’s a combination of two numbers. The reservoir is a fixed number of requests per second that are always traced, regardless of everything else, guaranteeing a minimum baseline of visibility even during quiet periods. The rate is the percentage applied to every request beyond the reservoir. A rule with a reservoir of 1 and a rate of 5% means: trace the first request each second unconditionally, then trace roughly one in twenty of everything after that. This two-part structure is what lets X-Ray guarantee some minimum trace coverage even for low-traffic services, while still keeping costs bounded during traffic spikes.

Subsegment Streaming for Long-Running Segments

A segment that accumulates an unusually large number of subsegments — common in services that fan out to dozens of downstream calls — can exceed the maximum size the daemon will buffer for a single UDP payload. To handle this, the SDK can stream completed subsegments to the daemon incrementally as they finish, rather than waiting to bundle everything into one document at the very end. This detail rarely matters for typical services, but it becomes relevant once you’re instrumenting fan-out-heavy code paths, like a service that queries dozens of downstream microservices in parallel to build a single aggregated response.

The central trade-off with X-Ray is the same one that shows up in almost every observability tool: completeness versus cost and performance overhead. Tracing 100% of requests would give you a perfect picture but would meaningfully increase both the data volume you pay for and the background load on every instrumented service. X-Ray’s sampling model is built around the assumption that a well-chosen statistical sample answers “is this system healthy and where is it slow” almost as well as full tracing would, at a fraction of the cost — but it does mean X-Ray is not the right tool if your requirement is “capture literally every single transaction for audit purposes.”

X-Ray Versus Rolling Your Own Correlation IDs

Before adopting X-Ray, some teams solve distributed tracing informally by generating a correlation ID at the edge and threading it through every log line by hand. That approach is cheap to start but scales poorly: there’s no automatic service map, no built-in latency breakdown per hop, and reconstructing a request’s full timeline means manually stitching together log entries from a dozen different services after the fact. X-Ray automates exactly that stitching, at the cost of adopting AWS-specific tooling and accepting its sampling-based completeness model rather than guaranteed full capture.

X-Ray was explicitly designed so that adding tracing does not become a scalability bottleneck of its own — a real risk with observability tooling, since a tracing system that can’t keep up with the traffic it’s watching is worse than no tracing at all.

Overhead on the Instrumented Service

Because the SDK writes to a local UDP socket rather than making a network call to AWS, the added latency per request is typically sub-millisecond — dominated by local serialization of the segment JSON, not network round trips. This is the architectural payoff of the daemon pattern discussed in Chapter 3: the expensive, potentially slow part (uploading to the X-Ray API) happens out of band, asynchronously, on the daemon’s own schedule.

Scaling the Daemon

On EC2 or ECS, the daemon itself scales with your application — typically one daemon per host or one daemon container per task, so daemon capacity naturally grows as you add more application instances. On Lambda, there is no daemon to scale at all, since AWS manages that path internally as part of the Lambda execution environment.

Scaling the Backend

The X-Ray API and storage layer are fully managed by AWS, meaning you never provision capacity for trace ingestion or storage yourself. That said, the service does enforce account-level rate limits on the ingestion API, which is one of the reasons sampling matters even for teams that could technically afford the cost of tracing everything — at very high request volumes, hitting those ingestion limits becomes a real constraint.

The practical scalability lesson for intermediate practitioners is that sampling rate, not raw request volume, is usually the first thing to tune when X-Ray-related costs or ingestion throttling become a problem — reducing your sampling percentage on high-volume, low-value paths (health checks, static asset requests) while preserving full or elevated sampling on business-critical, error-prone, or high-latency-risk paths gives you most of the diagnostic value at a fraction of the data volume.

Cold Starts and Tracing on Lambda

On Lambda, enabling Active Tracing adds a small amount of initialization work during a cold start, since the runtime needs to set up the tracing context alongside everything else it’s bootstrapping. In practice this overhead is small relative to the cold start itself, but it is a real, measurable factor for latency-sensitive Lambda functions operating close to strict SLA budgets — worth accounting for specifically during cold-start-focused performance tuning rather than assuming tracing is entirely free.

Backpressure Behavior

When the daemon’s internal buffer fills faster than it can flush to the X-Ray API — during a sudden traffic spike, for instance — it does not pause your application or apply backpressure upstream. It simply drops the newest segments once the buffer is full, favoring application throughput over trace completeness during exactly the moments when trace data would otherwise be most valuable. This is a known, accepted limitation: the busiest, most interesting moments in a system’s life are also the moments most likely to have gaps in trace coverage.

X-Ray’s reliability model is best understood by separating two very different questions: “is the X-Ray service itself highly available?” and “is my tracing data guaranteed to be complete?” The answer to the first is yes, in the way most managed AWS services are — it runs across multiple Availability Zones within a region with no single point of failure exposed to customers. The answer to the second is deliberately no, and that’s not a flaw; it’s the design.

What Happens When the Daemon Is Unreachable

If the X-Ray daemon on a given host crashes or is unreachable, the SDK does not retry indefinitely or block — UDP packets sent to a dead listener are simply dropped, and application requests continue processing normally, just without trace data for that window. This is a deliberate reliability trade-off: application availability is never sacrificed for the sake of observability availability.

What Happens When the X-Ray API Is Degraded

If the daemon can’t reach the X-Ray API — due to a regional service issue or a networking problem — it buffers what it can in memory and drops data once buffers fill, rather than writing to local disk and risking resource exhaustion on the host. Again, the failure mode is graceful degradation of observability, not a cascading failure of the application itself.

Multi-Region Considerations

X-Ray is a regional service — traces from us-east-1 and eu-west-1 are entirely separate and are not automatically stitched together. For globally distributed applications that span multiple regions, this means engineers need a deliberate strategy (often centralizing analysis through cross-region CloudWatch dashboards or exporting to a third-party backend) if they want unified visibility across regions rather than per-region silos.

Reliability of the Console Versus Reliability of the Data Plane

It’s worth separating the X-Ray console’s own availability from the underlying trace-ingestion pipeline’s availability. Even during a rare console outage, the ingestion path — SDK to daemon to API to storage — typically continues functioning independently, meaning trace data keeps accumulating even if nobody can view it at that exact moment. This separation of “can I write telemetry” from “can I currently look at telemetry” is a common pattern across AWS’s observability services, and it’s a useful mental model when triaging whether an X-Ray issue is actually blocking incident response or merely inconvenient.

Graceful Degradation as a Design Philosophy

Every reliability decision covered in this chapter traces back to one underlying philosophy: observability tooling should never become a new source of production incidents. A daemon that blocked the application when it couldn’t reach AWS, or an SDK that retried failed uploads indefinitely and consumed growing memory, would turn a monitoring tool into a liability. X-Ray’s designers consistently chose to sacrifice trace completeness rather than risk application stability, and understanding that trade-off is what separates an intermediate user from someone who treats X-Ray output as infallible.

IAM Permissions

Sending trace data requires the execution role of your Lambda function, EC2 instance profile, or ECS task role to carry xray:PutTraceSegments and xray:PutTelemetryRecords permissions. Reading trace data — through the console or API — is governed separately by read-oriented X-Ray permissions, meaning you can architect a system where application roles can only write telemetry and cannot read anyone else’s trace data, which is a meaningful separation for multi-tenant environments.

Sensitive Data in Segments

Because annotations and metadata are entirely developer-controlled — you decide what gets attached to a segment — X-Ray creates a real risk of accidentally logging sensitive data: customer emails, payment details, authentication tokens. Unlike some logging pipelines, X-Ray has no built-in redaction layer, so the responsibility for keeping personally identifiable information and secrets out of trace data falls entirely on the engineers instrumenting the code.

Encryption

Trace data is encrypted in transit between the daemon and the X-Ray API, and encrypted at rest using AWS-managed keys by default. Accounts with stricter compliance requirements can configure X-Ray to use a customer-managed KMS key instead, giving them control over key rotation and revocation for trace data specifically.

Network Boundaries

For workloads running inside a VPC without direct internet access, reaching the X-Ray API requires either a NAT gateway or, more commonly in security-conscious architectures, a VPC endpoint for X-Ray — which keeps trace traffic entirely on the AWS private network rather than traversing the public internet at all.

Resource-Level Access Control

X-Ray supports resource-based conditions in IAM policies, allowing organizations to restrict which trace groups, filter expressions, or even specific service names a given role can query. In larger organizations, this is how a platform or SRE team can grant application teams read access scoped narrowly to their own services’ traces, without opening visibility into every other team’s request data flowing through the same shared X-Ray account.

Audit Trail via CloudTrail

Configuration changes to X-Ray itself — creating or modifying sampling rules, changing encryption settings — are recorded in AWS CloudTrail like any other AWS API action. This matters for security reviews: it means changes to what gets traced, and how sensitively that data is protected, are themselves auditable events rather than invisible configuration drift.

X-Ray Insights

Rather than requiring an engineer to manually notice a spike in error rate on the service map, X-Ray Insights runs automated anomaly detection over trace data and proactively surfaces issues — a sudden increase in fault rate on a specific service, or a latency regression that started at a specific timestamp — along with the traces that best illustrate the anomaly, cutting down the time between “something is wrong” and “here is exactly where.”

CloudWatch Integration

X-Ray can emit derived metrics — request counts, error rates, and latency percentiles per service — directly into CloudWatch, which means teams can build CloudWatch Alarms on tracing-derived data, not just raw application logs. This is a common intermediate-level pattern: use X-Ray for deep, per-request investigation, but use the CloudWatch metrics it produces for always-on alerting.

Log Correlation via Trace ID

Many teams deliberately inject the active X-Ray trace ID into their application’s structured log lines. This doesn’t happen automatically — it requires the application code to read the current trace ID from the X-Ray SDK’s context and include it as a field in every log statement — but once in place, it lets engineers move seamlessly from “I found a suspicious trace” to “show me every log line associated with that exact request,” which is often the single highest-leverage habit an intermediate X-Ray user can adopt.

Alarming on Trace-Derived Metrics Versus Raw Application Metrics

A subtlety worth internalizing: metrics X-Ray derives from traces (like per-service latency percentiles) are built only from sampled requests, while metrics your application emits directly (through a custom CloudWatch metric or a StatsD-style client) reflect every single request regardless of sampling. At low sampling rates, trace-derived metrics can behave noisily for low-traffic services simply because the underlying sample size is small. For latency-sensitive alarming on lower-volume services, many teams intentionally alarm on directly emitted application metrics and reserve X-Ray-derived data for investigation rather than the primary trigger.

Infrastructure as Code

In production environments, the daemon deployment (as a container definition, DaemonSet manifest, or EC2 user-data script) and the IAM permissions it needs are almost always defined through infrastructure-as-code tooling rather than configured by hand, so that every new host or task automatically comes with tracing wired up consistently rather than depending on someone remembering to add it.

Hybrid and Multi-Account Setups

Larger organizations running multiple AWS accounts (common in a landing-zone or AWS Organizations setup) typically configure X-Ray per account, since trace data does not automatically cross account boundaries. Centralized visibility across accounts usually means either a cross-account IAM role that lets a central observability team query each account’s X-Ray API, or exporting derived metrics into a centralized CloudWatch or third-party dashboard.

Blue/Green and Canary Deployments

During a canary rollout, engineers often annotate segments with a deployment or version identifier specifically so that traces from the new canary version can be filtered and compared side by side against the stable baseline version — turning “does the canary look healthy” from a subjective judgment call based on aggregate error rates into a direct, trace-level comparison of latency and fault behavior between the two versions running simultaneously.

Pattern: Tiered Sampling

Rather than a single flat sampling rate across the whole system, mature X-Ray setups define multiple sampling rules ordered by priority — near-100% sampling on checkout, payment, and authentication paths; a low, fixed percentage on high-volume, low-risk paths like static content or health checks. This concentrates tracing cost and volume where the diagnostic value is highest.

Pattern: Annotation-Driven Filtering

Teams that consistently annotate segments with business-meaningful fields — customer tier, order ID, feature flag state — turn X-Ray from a purely technical tool into one that can answer business questions directly: “show me every trace where a premium customer experienced an error in the last hour,” filtered straight in the console without needing a separate analytics pipeline.

Instrument at Boundaries, Not Just Entry Points

A common intermediate pitfall is treating X-Ray instrumentation as something you set up once at the front door — the API Gateway or load balancer — and then forget. The real diagnostic value comes from instrumenting every meaningful boundary a request crosses: database calls, calls to third-party APIs, internal service-to-service hops, and queue publish/consume points. Each additional boundary you make visible is one more place a future incident can be pinpointed instead of guessed at.

Treat Sampling Rule Changes Like Code Changes

Because sampling rules directly control what evidence exists after an incident, teams that manage them ad hoc through the console often find, mid-incident, that the exact traffic they need wasn’t being sampled at all. Managing sampling rules through the same infrastructure-as-code and review process as application code — with version history and rollback — turns “we should have traced that” into a rare exception instead of a recurring postmortem line item.