Designing a Real-Time Call Quality Monitoring Platform

Designing a Real-Time Call Quality Monitoring Platform

Designing a Real-Time Call Quality Monitoring Platform

A ground-up walkthrough of how Zoom, Google Meet, Microsoft Teams, Webex, Slack Huddles, Discord, Amazon Chime and every large VoIP or video platform ingests millions of per-second signals from every endpoint on Earth, turns them into a live picture of “is this call good right now?”, alerts on the exact minute a region starts to burn, and hands engineers — and machine-learning models — the evidence they need to root-cause an outage before customers even open a support ticket.

01

The Big Idea, in One Breath

A real-time call quality monitoring platform is the nervous system of a modern voice or video product. It watches every packet, every jitter buffer, every codec switch on every device — then compresses that torrent of low-level telemetry into a single, honest answer to one question: “How does the call sound and look, right now, to this human being?” It then rolls those answers up to rooms, meetings, tenants, ISPs, regions and continents, so the platform can react to a bad-weather day on the internet in seconds rather than after the next quarterly review.

Underneath the friendly “Excellent / Good / Poor” badge on the caller’s screen there is a full-blown streaming data platform: a client SDK that samples WebRTC and native stack metrics, a hardened edge that authenticates and rate-limits millions of writers, a durable event backbone, stateful stream-processing jobs that compute MOS, freeze rate, glitch counts and packet-loss bursts on the fly, a time-series and columnar analytics store, an anomaly detection layer, an alerting fabric, and a set of dashboards and APIs that operators, on-call engineers, ML models and customer support all lean on.

Analogy

Think of the platform as air-traffic control for conversations. Every call is a flight, every endpoint is an aircraft transponder pinging its altitude and heading, and the tower stitches those pings into a live radar picture. When the picture shows turbulence over Frankfurt, the tower does not wait for pilots to file a complaint — it re-routes traffic and dispatches engineers. Your monitoring platform does exactly the same thing, but at the scale of hundreds of millions of daily calls.

< 10 s
Telemetry to
live dashboard
< 60 s
Regional issue
to page on-call
10M+
Concurrent
reporting endpoints
02

What a Call Quality Monitoring Platform Really Is

Before touching any wire diagram, we need a precise picture of what this system does — and, just as importantly, what it does not. The word “monitoring” is overloaded in this industry, so let us pin down the exact contract this platform is signing with the rest of the product.

2.1 A Working Definition

A real-time call quality monitoring platform is a distributed system that, given a live communication session (audio, video, screen share, streaming) and the endpoints participating in it, guarantees:

  • every endpoint’s network, media and device telemetry is captured, normalised and made durable within perceptual real-time,
  • an objective and subjective quality score is computed continuously for every participant, every stream and every leg,
  • anomalies at any granularity — call, room, tenant, ISP, region, build, codec — are detected and correlated within seconds,
  • every score, event and alert is queryable and joinable to business context (customer, plan, region, meeting type) for as long as the business needs it,
  • none of this monitoring itself degrades the very calls it is watching.

2.2 What It Is Not

!Common confusion

Call quality monitoring is not the same as call recording, media routing, transcription or CDR (call detail record) billing. It sits next to those systems, borrowing signals from them, but its purpose is diagnostic, not archival. Confusing the two leads to storage bills that dwarf the entire product.

Scope

Live and Retro

It reports on calls happening right now, but it also feeds a historical warehouse so trend analysis, SLA reporting and ML training all read from the same source of truth.

Signals

Network & Media & UX

It fuses transport metrics (RTT, jitter, packet loss), media metrics (freeze rate, resolution, framerate, MOS), and user-experience metrics (join failures, ratings) into a single opinionated score.

Users

Multi-Persona

Its consumers are on-call engineers, product managers, customer support, ML models, executives and end users — each needing the same truth at a different granularity.

Boundary

Read-Mostly

Except for a narrow control channel (feature flags, sampling rate changes) it is fundamentally a read-heavy analytics system. Writes come from clients, reads come from humans and automation.

03

Why It Matters

Any product that ships audio or video is one bad packet-loss burst away from an angry tweet. Real-time monitoring is what turns a fragile media pipeline into a business that can promise, and honour, a five-nines conversation experience.

What you gain

  • Seconds-fast MTTR. Regional issues are visible before the first support ticket lands.
  • Objective SLAs. Contracts with enterprise buyers can be tied to real, measured MOS and drop rates.
  • Product-level insight. Which codec, which build, which OS, which network works best — and where to invest next.
  • Adaptive media. The stack can react to live telemetry — switching codecs, lowering resolution, re-routing through a closer TURN server — before humans notice.
  • ML fuel. Every call becomes a labelled training example for the next generation of bandwidth estimators, echo cancellers and audio enhancers.

What it costs

  • Storage & egress. High-frequency telemetry from tens of millions of endpoints is the largest single line item in most VoIP platforms.
  • Client CPU & battery. Every metric collected on a mobile handset is a millisecond stolen from the call itself.
  • Alert fatigue. A naive threshold system will page the on-call at 03:00 every time a coffee shop’s Wi-Fi hiccups.
  • Privacy surface. The same telemetry that helps engineers can, if mishandled, reveal who called whom, from where and for how long.
iRule of thumb

Every metric you collect must earn its keep. If nobody can point to a dashboard, alert or ML model that uses a metric within the next 90 days, do not ship it. Telemetry that nobody reads is a bill you pay forever.

04

The Signals: What We Actually Measure

A monitoring platform is only ever as good as the signals it collects. This chapter walks through the three layers of telemetry every serious call platform gathers and shows how they combine into a single perceptual score.

4.1 Transport Layer

These are the raw, unavoidable physics of the internet — sampled directly from the WebRTC/ICE stack or the native SIP/RTP engine every second.

MetricWhat it capturesTypical sampling
RTT (round-trip time)End-to-end latency between endpoint and media server, in ms.1 Hz
JitterVariance in packet inter-arrival time.1 Hz
Packet loss %Fraction of RTP packets never delivered.1 Hz
Available bandwidthEstimator’s current guess of usable bitrate.1 Hz
Route / TURN relayWhich relay path is active; changes are events.on change

4.2 Media Layer

Above the transport, we care about what the codec is actually producing and what the decoder is actually reconstructing.

Audio

MOS-LQO, Freeze, Concealment

Objective mean opinion score, freeze rate, concealment ratio from the neural PLC, echo return loss, noise suppression aggressiveness.

Video

Framerate, Resolution, Freeze

Encoded and rendered framerate, resolution ladder used, freeze count and freeze duration, keyframe request rate, decode-time p95.

Screen

Text Legibility

Fraction of time text-heavy screen share stays above a legibility threshold, dropped tile count.

Sync

A/V Skew

Lip-sync offset between audio and video streams, sampled every few seconds.

4.3 Device & UX Layer

Even a perfect network cannot rescue a call where the microphone is muted at the driver level or the CPU is pinned by a background scan.

  • CPU %, thermal state, battery level, power mode.
  • Camera / microphone permission state, hardware id, sample rate.
  • Join success, join latency, in-call errors, user-submitted post-call rating.
Transport tells you what the network did to the packets. Media tells you what the codec did with them. UX tells you whether the human on the other end noticed.
05

Building Blocks of the Platform

With the signals defined, we can sketch the physical components that carry them from the microphone in someone’s living room to a graph on an engineer’s screen in another hemisphere.

Client SDKsampler + buffer Edge Ingestauth · rate limit Event BusKafka / Pulsar Stream JobsMOS · freeze · loss Hot TSDBlive dashboards Cold LakeParquet / Iceberg Analytics DWcolumnar SQL Anomaly / MLdetection · scoring Alertingpage · ticket Dashboards, APIs, Support Console, User Scoregrafana · sql · embedded widgets
Figure 5.1 — The path a single second of telemetry takes, from a laptop in Berlin to a dashboard in Bangalore.
1

Client SDK

Runs inside every native app, browser tab and SIP handset. Samples the media stack, applies privacy scrubbing, buffers on network hiccups, ships batched events over HTTPS or QUIC.

2

Edge Ingest

Global anycast fleet that terminates TLS, authenticates the endpoint, enforces per-tenant quotas, drops corrupt payloads and hands normalised events to the bus.

3

Event Bus

A partitioned, durable log (Kafka, Pulsar, Kinesis) keyed by call id so all events for a single conversation land on the same partition and stay in order.

4

Stream Jobs

Stateful Flink / Spark Structured Streaming jobs that compute rolling MOS, freeze rate, packet-loss bursts and derived call-level scores in windows of one to ten seconds.

5

Hot Store

A time-series database (Prometheus / M3 / VictoriaMetrics / Influx) that powers live dashboards with sub-second query latency but retains only the last few days.

6

Cold Lake & DW

Raw events land in a data lake as Parquet/Iceberg, and enriched aggregates land in a columnar warehouse (BigQuery, Snowflake, ClickHouse) for long-term analytics and ML.

7

Anomaly & ML

Detectors compare live windows to historical baselines by tenant, region, ISP, build and codec — and hand anomalies to alerting with rich context.

8

Delivery

Grafana-style dashboards, SQL endpoints, embedded “call quality” widgets inside the support console, and end-user badges in the client itself.

06

The Ingestion Pipeline in Detail

The ingestion pipeline is where most call-monitoring projects live or die. It has to accept a firehose of writes, tolerate arbitrary network weather, avoid double-counting, and stay cheap. This chapter breaks down the moving parts.

6.1 The Client Contract

The SDK is not a naive HTTP poster. It is a small, resilient distributed system in itself:

  • Sampling ladder. Non-critical metrics are sampled at 0.1–1 Hz; critical events (join, leave, disconnect, codec switch) are sent immediately.
  • Batch & compress. Events are grouped into 1–5 s batches, protobuf-encoded and gzipped before shipping.
  • Local ring buffer. If the network is unreachable, the SDK spills to an in-memory or on-disk ring so the last few minutes of pre-crash telemetry survive.
  • Idempotency keys. Every batch carries a monotonic sequence per session so the server can safely de-dup on retry.

6.2 The Edge

The edge fleet is a stateless, horizontally-scaled service behind an anycast IP.

1

Terminate

TLS 1.3 with 0-RTT resumption for repeat clients; QUIC for mobile networks where TCP handshakes are expensive.

2

Authenticate

Short-lived JWTs issued at call join; verified locally against a rotating JWKS cached in the edge.

3

Validate

Schema check, size cap, per-tenant rate limit, and drop of any field on the current PII deny-list.

4

Enrich

Add server-side facts the client cannot know: IP→ASN→ISP mapping, geo, edge region, ingest timestamp.

5

Publish

Write to the event bus, keyed by call id so all events for a conversation stay in the same partition.

6.3 The Bus and the Windows

Downstream of the bus, stream jobs run on the same partitions. Windows are almost always “event-time” with a small allowed lateness, so a mobile client whose radio spent 20 seconds re-connecting still contributes its telemetry to the correct minute.

iDesign choice

Keep the schema for raw events aggressively narrow — a handful of numeric fields plus dimensions. Rich, wide events feel convenient in year one and turn into a five-figure daily bill in year three.

07

Detection, Scoring & Alerting

Once telemetry is flowing, the platform has to answer three questions in real time: is any call bad, is any cohort of calls bad, and is anyone experiencing that badness who deserves to be alerted? Each question is a different detector.

7.1 Per-Call Scoring

The per-call score is a function of transport, media and UX signals combined over a rolling window. A typical, opinionated formula:

score.pseudo
call_score = w1 · MOS_audio
           + w2 · f_video_freeze
           + w3 · f_video_framerate
           + w4 · f_screen_legibility
           - p_join_failure
           - p_disconnect

Weights are learnt from post-call user ratings and validated against controlled experiments. The output is bucketed into Excellent / Good / Fair / Poor so humans can reason about it without staring at a decimal.

7.2 Cohort Detection

Aggregating per-call scores by tenant, region, ISP, client build, OS version and codec turns individual bad calls into incidents. A simple z-score against a 28-day baseline catches most real regressions; a Bayesian change-point detector catches the subtler ones.

Signal

Sudden Drop

MOS in AS-12345 falls three sigma below its own 28-day median for > 3 consecutive minutes.

Signal

Slow Bleed

Freeze rate on build 8.2.1 has been creeping up 4% week-on-week for a month.

Signal

Correlated

Join-failure spike on Android 14 correlates with a codec-switch event; strong hint at a client bug.

Signal

Contextual

Poor MOS restricted to a single enterprise tenant — likely a customer network problem, not ours.

7.3 Alert Fabric

Alerts are only useful if they wake the right human. The fabric routes each incident along three axes:

  • Ownership. Client bug → mobile team; media routing → SFU team; ISP-scoped → network relations.
  • Severity. Sev-1 pages instantly; Sev-3 lands quietly in a triage queue.
  • Suppression. Duplicate alerts, planned maintenance windows and known-flapping cohorts are collapsed.
!Danger

Never page on raw thresholds like “packet loss > 2%”. Some coffee shop somewhere on Earth is always losing packets. Page on anomalies relative to a cohort’s own baseline, or you will train your on-call to ignore every page you send.

08

End-to-End Flow of a Bad Call

Nothing crystallises an architecture like following a single event through it. Let us walk one bad call — a laptop in Berlin joins a Meet call with a Bangalore host, hits a bad Wi-Fi patch three minutes in, recovers, and rates the call two stars — from tap to on-call escalation.

Client SDK Edge + Bus Stream + Detect Humans join callt=0s wifi degradest=180s codec switcht=182s recoverst=220s user rates 2★t=end auth + enrich events on bus batched retries catch-up flush final events rolling MOS freeze burst z-score fires anomaly closes rating joins on-call paged support widget
Figure 8.1 — One bad call, five lanes: what the client, edge, stream jobs and humans each experience, minute by minute.
t = 0

Join

SDK opens an outbound stream to the edge, sends a “join” event tagged with build, OS, network type and codec capabilities.

t = 3 min

Bad patch

Packet loss climbs from 0.3% to 8% over 20 seconds. Rolling MOS drops from 4.4 to 2.6. Freeze rate jumps to 12%.

t = 3 min 2 s

Stream reacts

The stream job’s per-call detector marks the call “Poor”. In parallel, a cohort detector notices thirty-seven other calls in the same Berlin ASN degrading in the same minute.

t = 3 min 8 s

Alert

A single, correlated Sev-2 alert fires for ASN 3320 in Berlin, not thirty-seven separate pages. The on-call opens the pre-baked runbook link.

t = 3 min 40 s

Recovery

The user’s Wi-Fi recovers, MOS climbs back to 4.3, freeze rate drops to 0.4%. The anomaly detector closes the incident automatically.

t = end

Rating

The user rates the call two stars. The rating is joined to the call’s telemetry in the warehouse; the training set gains one more real-world label.

09

Quality Attributes: What This System Must Guarantee

A monitoring platform lives and dies by its non-functional properties. The functional feature list is short; the guarantees are where the engineering happens.

Latency

Under 10 seconds

From metric emitted on device to visible on dashboard. Anything slower and operators stop trusting the picture.

Freshness

Under 60 seconds

From anomaly onset to on-call page. Any longer and the customer notices first.

Availability

99.99% ingest

Monitoring must survive its own product’s outages — ideally on independent infrastructure, and always deployable through a separate change window.

Durability

Zero silent drops

Every event is either persisted or explicitly counted as dropped; there is no third state.

Cardinality

Bounded & policed

Per-tenant metric cardinality is capped; runaway labels are quarantined, never allowed to blow up the TSDB.

Cost

Predictable per call

Cost-per-minute-of-conversation stays flat as scale grows — no sudden hockey stick when a customer 10×’s their usage.

Privacy

Minimal & masked

No content, no user text, no raw IPs after enrichment — only what an on-call needs to root-cause.

Governance

Change-safe

Schema evolution is additive and versioned; historic dashboards keep working even as new fields land every week.

ADR-01

Ingest on independent infrastructure

Accepted

Context. The tempting shortcut is to co-locate the monitoring ingest fleet with the media plane, sharing edge PoPs and databases.

Decision. Run the monitoring ingest on independent PoPs, a separate anycast IP, and a separate cloud account with its own IAM boundary.

Consequence. When the media plane fails — and one day it will — the monitoring platform is still there to tell you why. The extra cost is real but small compared to a blind incident.

10

Pitfalls Every Team Hits

There are a handful of failure modes that every team building this platform meets, usually in year two, and the fix is never a heroic rewrite — it is a small, boring decision made early.

What tends to break

  • Cardinality explosion. Someone adds “user_id” as a label; the TSDB is measurably sad within an hour.
  • Retry storms. A regional outage restores; every SDK on Earth flushes its ring buffer in the same 30 seconds; the edge folds under the thundering herd.
  • Silent drops. A schema change adds a required field; older clients silently fail validation and their events vanish for weeks before anyone notices.
  • Clock skew. A phone with a wrong system clock ships events tagged in 2029; naive event-time windows never close.
  • Alert fatigue. Static thresholds catch every coffee shop and page the on-call twenty times a night.
  • Privacy drift. A field added “temporarily for debugging” makes it into cold storage and is still there three years later.

What actually helps

  • Server-side cardinality guards. The edge counts distinct label combinations per tenant and refuses new ones once a budget is exceeded.
  • Jittered retries. SDKs spread the flush of buffered events over a random window and exponential-backoff on 429s.
  • Contract tests. Every schema change is validated against a corpus of real historical payloads before it ships.
  • Ingest-time clock check. The edge rejects or shifts obviously wrong timestamps and tags them for investigation.
  • Baseline alerts. Every alert compares to a cohort’s own historical baseline, not a global threshold.
  • PII register. A living document lists every field, its purpose and its retention; anything not on it is not written.
11

How It Evolves

A call-quality platform is never “done”. Its architecture changes shape as the product grows, and the good news is that the evolution is fairly predictable.

V1

Diagnostic dashboards

A single team, a single region, a single dashboard. Metrics land in a TSDB; alerts are hand-authored. Good enough to tell you when things are bad; useless at telling you why.

V2

Multi-tenant & cohorts

Signals are enriched with tenant, region, ISP, build. Dashboards are sliceable. On-call rotations start to trust the pages.

V3

Warehouse & SLAs

Cold storage lands in a columnar warehouse; historical trends drive product decisions and enterprise SLAs.

V4

Anomaly detection

Statistical detectors replace static thresholds. The on-call rotation shrinks; the mean-time-to-detect drops from tens of minutes to under a minute.

V5

Closed loop

Detectors feed the media plane: a bad ISP triggers automatic re-routing through a different TURN region. The platform stops being a passive observer and starts steering the product.

V6

ML-first

Per-call MOS is estimated by a neural model trained on billions of real ratings; anomaly detection is a graph over tenants and topologies; the platform becomes indistinguishable from the product itself.

iEvolution rule

Every stage is cheaper if you built the previous stage on additive schemas. The teams that struggle most are those that shipped V1 with a rigid, tightly-coupled event format and now have to rewrite two years of history to reach V4.

12

Key Takeaways

If you remember only a handful of things from this walkthrough, remember these — they are the ones that separate a monitoring platform that scales from one that quietly bankrupts a team.

  • Model the human, not just the packet. The scoring layer must translate raw transport and media metrics into a perceptual score a person would give the call.
  • Own the ingest end-to-end. The SDK, the edge and the bus are one system; you cannot outsource any of them and keep the guarantees.
  • Independent infrastructure. Monitoring must survive its own product’s outages. Different PoPs, different accounts, different change windows.
  • Baseline before you threshold. Alerts must fire on anomalies relative to a cohort’s own past, not on absolute numbers.
  • Additive schemas forever. Every field is easy to add and painful to remove; design the schema and the retention register with a five-year horizon.
  • Cardinality is your quiet enemy. Cap it at the edge, watch it every day, and never let a single tenant ship a runaway label.
  • Close the loop. The endgame is not dashboards; it is a monitoring signal that automatically steers the media plane before humans intervene.
  • Privacy is a feature. Every field carries a purpose, a retention and a legal basis. If you cannot explain why a field exists, you cannot ship it.

A well-built call-quality monitoring platform is invisible when everything works and priceless the moment something breaks. Build it as if you will one day have to defend every field, every alert and every dashboard to an executive, an auditor and a very tired on-call engineer — because eventually, you will.