Observability is the ability to understand what is happening inside a system by looking at the information it gives out.

Let us start with the word itself. To observe means to watch carefully. Something is observable when you can learn about its inside state just by studying its outside signals. The idea comes from control engineering, a field that studies machines like thermostats and rockets. An engineer named Rudolf Kálmán formalised it in the early 1960s. Decades later, software engineers borrowed the word because their programs had become so complicated that nobody could hold the whole picture in their head.

In software, a system is observable when engineers can ask new questions about its behaviour without changing the code and shipping it again. Questions like “Why was checkout slow for customers in one city last Tuesday?” or “Which tiny step inside this request is eating all the time?” A system that lets you answer such surprising questions is observable. A system that only lets you answer questions you planned in advance is merely monitored.

Monitoring versus observability

Monitoring means collecting data about things you already know can go wrong, and raising an alarm when a number crosses a line. For example, “alert me if the website’s error rate goes above five percent.” Monitoring is essential, but it answers only known questions, sometimes called known unknowns.

Observability goes further. It prepares you for the unknown unknowns: the strange failures nobody predicted. When something odd happens, you can dig through rich data and follow the clues. Monitoring is the smoke alarm; observability is the investigation that follows the alarm.

Software used to be one big program on one machine. Today it is hundreds of small pieces talking across a network.

Twenty years ago, many applications were monoliths. A monolith is a single large program where every feature lives together, like one big shop under one roof. If it broke, you logged into the one server and looked around. The number of places a problem could hide was small.

Modern applications are usually built from microservices. A microservice is a small program that does exactly one job, such as handling payments, sending emails or storing user profiles. These small programs call each other over the network to complete a single customer action. They run in containers, which are lightweight packages that hold a program and everything it needs, and those containers are started, stopped and moved around automatically by platforms like Kubernetes.

The cost of not seeing

When a system is a black box, every outage becomes a guessing game. Engineers argue, restart random services and hope. This wastes time, and time is expensive. Studies of large companies regularly put the cost of downtime for big online businesses in the range of thousands of dollars per minute. Even for small teams, hours lost to guessing mean unhappy users and tired engineers.

Logs tell the story, metrics show the numbers, and traces draw the journey.

Before we study each pillar in depth, let us meet all three together. Each answers a different kind of question. None of them is enough alone, and that is exactly why people call them pillars: a roof needs all of them to stand.

flowchart TD
    A["Running Application"] --> B["Logs: what happened"]
    A --> C["Metrics: how much and how often"]
    A --> D["Traces: where the request went"]
    B --> E["Understanding and Action"]
    C --> E
    D --> E
        

A log is a written note a program makes about something that just happened.

What a log is

A log entry is a small piece of text with a time stamp. Programs write these notes as they work: “user signed in”, “payment accepted”, “database did not answer in time”. Collected together, they form a diary of everything the program did, in order.

Why logs exist

Programmers need a way to remember details that no number can capture. A counter can tell you that fifty payments failed. Only a log can tell you that one failed because the customer’s card had expired, another because the bank was unreachable, and a third because of a typing mistake in an address.

Where logs are used

Logs are used for debugging, which means hunting for the cause of a bug, for security investigations, for audits that prove who did what, and for understanding rare events that happen once in a million requests.

Three flavours of logs

Severity levels

Most logs carry a severity label so readers can ignore noise. Common levels run from the least to the most serious: debug for tiny details useful only to developers, info for normal events, warning for odd things that are not yet harmful, error for things that failed, and critical for situations that need instant attention. Choosing the right level is a small habit that saves enormous time later.

A metric is a number that is measured again and again so you can watch it change over time.

What a metric is

Where a log is a sentence, a metric is a number with a time stamp and some labels. Examples include “requests per second”, “memory used”, “average response time” and “number of users online”. Metrics are measured at regular intervals, often every ten or fifteen seconds, and stored as a series of points that can be drawn as a line on a chart.

Why metrics exist

Humans are excellent at spotting patterns in pictures and poor at reading millions of text lines. A chart of response time instantly shows a slow climb that would be invisible in a log file. Metrics are also tiny, so storing them for months or years costs little. This makes them perfect for trends, capacity planning and alerting.

Main types of metrics

Why averages can lie

Suppose ninety-nine customers get an answer in one tenth of a second and one customer waits thirty seconds. The average looks wonderful, yet one person had a terrible experience. That is why engineers look at percentiles. The “99th percentile” tells you how slow the slowest one percent of requests were. Percentiles reveal the pain that averages hide.

Two famous checklists

Experts have created simple recipes so you know which metrics to collect first. The first recipe is known as the four golden signals, popularised by Google’s site reliability engineers: latency (how long requests take), traffic (how much demand arrives), errors (how many requests fail) and saturation (how full your resources are). The second is called RED for services: rate, errors and duration. A close cousin called USE is for hardware and resources: utilisation, saturation and errors.

A trace follows a single request on its whole journey across every service it touches.

What a trace is

When a customer action enters your system, it receives a unique identifier, called a trace identifier. As the request moves from service to service, that identifier travels along with it. Every service records how long its own piece of work took and attaches the same identifier. Later, all those pieces are collected and assembled into one picture of the full journey.

Spans: the building blocks

Each piece of work inside a trace is called a span. A span has a name, a start time, a duration and some notes. Spans are arranged like a family tree: a parent span, such as “handle checkout”, contains child spans, such as “check stock” and “charge card”. The whole tree is the trace.

sequenceDiagram
    participant U as Customer
    participant W as Web Service
    participant C as Cart Service
    participant P as Payment Service
    participant D as Database
    U->>W: Place order
    W->>C: Read cart items
    C->>D: Fetch items
    D-->>C: Items returned
    C-->>W: Cart ready
    W->>P: Charge card
    P-->>W: Payment accepted
    W-->>U: Order confirmed
        

Context propagation

The magic that makes tracing work is context propagation. It simply means passing the trace identifier from one service to the next, usually inside the hidden headers of each network call. If even one service forgets to pass the identifier along, the trace breaks into pieces, like a chain with a missing link.

Sampling: you cannot keep everything

Recording a full trace for every request in a busy system would produce enormous amounts of data. So teams use sampling, which means keeping only a fraction. Head sampling decides at the start of a request whether to keep it. Tail sampling waits until the request has finished and then keeps the interesting ones, such as slow or failed requests. Tail sampling is smarter but needs more infrastructure to hold the data while waiting for the decision.

Each pillar has its own superpower and its own blind spot.

A common beginner question is “Which pillar is the most important?” The honest answer is that it depends on the question you are asking. The table below places them side by side so you can see how they differ.

The real power appears when you move smoothly from one pillar to the next while solving a problem.

Let us walk through a made-up but very realistic story. An online store notices that customers are complaining about slow checkouts on a busy evening.

flowchart LR
    A["Alert from Metrics"] --> B["Narrow down by service"]
    B --> C["Open slow Traces"]
    C --> D["Search Logs by trace ID"]
    D --> E["Find root cause"]
    E --> F["Fix and confirm with Metrics"]
        

An observability setup is a pipeline that collects, moves, stores and shows signals.

No matter which products you choose, almost every observability system has the same building blocks. Understanding them helps you read any vendor’s documentation.

OpenTelemetry: a shared language

For many years, every vendor had its own way of collecting signals, which locked teams into one product. In 2019, two open projects named OpenTracing and OpenCensus merged to form OpenTelemetry, which now lives under the Cloud Native Computing Foundation. It offers a common set of standards and tools for producing logs, metrics and traces, so you can change your storage or display product later without rewriting how your software is measured. For a beginner, it is the safest starting point.

Well-known tools you may meet

Prometheus, which was inspired by an internal Google system and started at SoundCloud, is a popular open-source metrics system. Grafana is widely used to draw dashboards. Jaeger, created at Uber, and Zipkin, which grew out of Twitter, are open-source tracing tools, and both were influenced by a Google research paper called Dapper. For logs, many teams use the Elastic stack or Grafana Loki. Cloud providers also offer built-in services, such as Amazon CloudWatch and AWS X-Ray, Google Cloud Operations and Azure Monitor.

flowchart LR
    A["Apps and Services"] --> B["Agents and SDKs"]
    B --> C["Collector"]
    C --> D["Log Store"]
    C --> E["Metric Store"]
    C --> F["Trace Store"]
    D --> G["Dashboards and Search"]
    E --> G
    F --> G
    E --> H["Alert Rules"]
    H --> I["On-call Person"]
        

Signals have a life: they are born, travel, are stored, are used and finally retire.

It helps to picture the life of a single signal from the moment your program creates it to the moment it is deleted.

The three pillars become far more useful when they share common labels.

If logs, metrics and traces live in three separate worlds with no links, you must switch tools and guess at matches. Correlation means giving all three the same identifying details so that one click takes you from a spike on a chart to the traces behind it and the logs inside them.

Observability data is most useful when it is tied to promises you make to your users.

Collecting data is not the goal. The goal is a healthy product. To connect the two, teams use three related terms. A Service Level Indicator, or SLI, is a measurement of something users care about, such as the percentage of requests answered successfully. A Service Level Objective, or SLO, is the target you set for that measurement, such as “99.9 percent of requests succeed each month.” A Service Level Agreement, or SLA, is a formal contract with customers that usually includes penalties if the target is missed.

Alert on symptoms, not causes

A good alert tells a human that users are being hurt right now. A bad alert fires every time a machine’s processor gets busy, even when nobody notices. Too many bad alerts cause alert fatigue, where people begin ignoring the pager. Aim to wake someone up only for problems that need a human decision, and send everything else to a quiet report.

Observability data is powerful, which also makes it sensitive.

Logs and traces can accidentally hold email addresses, names, order details or secret keys. Because this data is copied into many systems and viewed by many engineers, it deserves the same care as a customer database.

Experienced teams repeat a handful of habits that make observability easier.

Structured logging

Write logs as labelled fields instead of loose sentences. Machines can then count, filter and group them. A line that clearly states the service, the severity, the user’s region and the outcome is far more useful than a paragraph of prose.

Correlation identifiers everywhere

Create an identifier when a request first arrives and pass it through every service and into every log line. This single habit is the cheapest way to connect your three pillars.

Standard dashboards per service

Give every service the same starter dashboard showing rate, errors and duration. New engineers then know where to look no matter which service they are investigating.

Health checks and heartbeats

A health check is a tiny test endpoint that a platform calls to ask “are you alive and ready?” Heartbeats are regular signals proving a background job is still running. Both produce simple metrics that make silent failures visible.

Synthetic monitoring

Synthetic monitoring means running fake users on a schedule, such as a robot that logs in and buys a test item every minute from different regions. It tells you the product is broken even when no real customer is online to complain.

Observability as part of the release process

Before launching a new feature, teams ask what they will watch to know it works. Adding the measuring points while building, not afterwards, produces better data and avoids blind launches.

Some habits look harmless at first but cause pain later.

A short checklist you can follow from day one.

A gentle starting path

If you are new, do not try to build everything at once. Begin with metrics for your most important service, because they give quick value at low cost. Add structured logs next, so you can explain the spikes you see. Introduce tracing when you have several services and start to ask “which one is slow?” This step-by-step approach keeps effort and cost under control while your skills grow.

Observability brings large rewards, but it is not free.

Balancing cost and insight

Every extra signal costs money to move and store, yet every missing signal may cost hours during an outage. The art is choosing wisely. Keep cheap, broad data such as metrics for a long time. Keep expensive, detailed data such as full traces and verbose logs for a short time, or sample them. Let the business importance of each service guide how much detail you keep.

Well-known companies shaped the tools and ideas we use today.

The three pillars are a foundation, not a ceiling.

As the field matures, practitioners have added other helpful signals. None of them replaces the original three, but you will hear about them often, so it is wise to meet them now.

They are logs, metrics and traces. Logs are time-stamped records of individual events, metrics are numbers measured over time, and traces follow one request through many services.

No. Monitoring watches for known problems and raises alarms. Observability gives you rich, connected data so you can investigate problems you did not predict.

Metrics are usually the best first step because they are cheap, quick to set up and excellent for alerts. Add structured logs next, then traces once you have multiple services.

Not always. A single small program may do well with logs and a few metrics. Tracing becomes truly valuable when a request crosses several services.

The identifier is what ties together the work done in different services. Without it, there would be no way to know that separate pieces of work belong to the same customer action.

A log describes an event in one place. A trace describes the whole path of a request across places, including how long each step took.

Averages can hide the experience of the slowest users. Percentiles show how slow the slowest group really was, which better matches what unhappy customers feel.

It is an open standard and toolset for producing logs, metrics and traces. You do not strictly need it, but it prevents vendor lock-in and is widely supported, so it is a sensible default.

Keep summary metrics for a long time because they are small. Keep detailed logs and traces for a shorter period, based on how far back your investigations usually reach and what laws require.

It adds a small overhead, usually minor when done well. Sampling, batching and efficient libraries keep the cost low, and the faster fixes it enables usually repay it many times over.

It means a label can take a huge number of different values. In metrics, this multiplies the number of stored series and can make the system slow and costly. Put such detail in logs or traces instead.

Everyone who builds the software. Platform teams provide the tools, but developers must add the measuring points and keep them useful, because they understand their own code best.

Modern software is a web of many small services, and no single person can keep all of it in their head. Observability is the answer: a way of designing systems so that they explain themselves. Logs record the detailed story of individual events, metrics reveal the health and trends through numbers, and traces draw the journey of a request across services. Each pillar alone has blind spots, but together, and especially when linked by shared identifiers, they let teams move from “something is wrong” to “here is exactly why” in minutes instead of hours.

Start small, measure what your users feel, protect the sensitive data you collect, and grow your practice step by step. With these habits, you will spend less time guessing in the dark and more time building things people enjoy using.