Designing a Real-Time Call Quality Monitoring Platform
A ground-up walkthrough of how Zoom, Google Meet, Microsoft Teams, Webex, Slack Huddles, Discord, Amazon Chime and every large VoIP or video platform ingests millions of per-second signals from every endpoint on Earth, turns them into a live picture of “is this call good right now?”, alerts on the exact minute a region starts to burn, and hands engineers — and machine-learning models — the evidence they need to root-cause an outage before customers even open a support ticket.
The Big Idea, in One Breath
A real-time call quality monitoring platform is the nervous system of a modern voice or video product. It watches every packet, every jitter buffer, every codec switch on every device — then compresses that torrent of low-level telemetry into a single, honest answer to one question: “How does the call sound and look, right now, to this human being?” It then rolls those answers up to rooms, meetings, tenants, ISPs, regions and continents, so the platform can react to a bad-weather day on the internet in seconds rather than after the next quarterly review.
Underneath the friendly “Excellent / Good / Poor” badge on the caller’s screen there is a full-blown streaming data platform: a client SDK that samples WebRTC and native stack metrics, a hardened edge that authenticates and rate-limits millions of writers, a durable event backbone, stateful stream-processing jobs that compute MOS, freeze rate, glitch counts and packet-loss bursts on the fly, a time-series and columnar analytics store, an anomaly detection layer, an alerting fabric, and a set of dashboards and APIs that operators, on-call engineers, ML models and customer support all lean on.
Think of the platform as air-traffic control for conversations. Every call is a flight, every endpoint is an aircraft transponder pinging its altitude and heading, and the tower stitches those pings into a live radar picture. When the picture shows turbulence over Frankfurt, the tower does not wait for pilots to file a complaint — it re-routes traffic and dispatches engineers. Your monitoring platform does exactly the same thing, but at the scale of hundreds of millions of daily calls.
live dashboard
to page on-call
reporting endpoints
What a Call Quality Monitoring Platform Really Is
Before touching any wire diagram, we need a precise picture of what this system does — and, just as importantly, what it does not. The word “monitoring” is overloaded in this industry, so let us pin down the exact contract this platform is signing with the rest of the product.
2.1 A Working Definition
A real-time call quality monitoring platform is a distributed system that, given a live communication session (audio, video, screen share, streaming) and the endpoints participating in it, guarantees:
- every endpoint’s network, media and device telemetry is captured, normalised and made durable within perceptual real-time,
- an objective and subjective quality score is computed continuously for every participant, every stream and every leg,
- anomalies at any granularity — call, room, tenant, ISP, region, build, codec — are detected and correlated within seconds,
- every score, event and alert is queryable and joinable to business context (customer, plan, region, meeting type) for as long as the business needs it,
- none of this monitoring itself degrades the very calls it is watching.
2.2 What It Is Not
Call quality monitoring is not the same as call recording, media routing, transcription or CDR (call detail record) billing. It sits next to those systems, borrowing signals from them, but its purpose is diagnostic, not archival. Confusing the two leads to storage bills that dwarf the entire product.
Live and Retro
It reports on calls happening right now, but it also feeds a historical warehouse so trend analysis, SLA reporting and ML training all read from the same source of truth.
Network & Media & UX
It fuses transport metrics (RTT, jitter, packet loss), media metrics (freeze rate, resolution, framerate, MOS), and user-experience metrics (join failures, ratings) into a single opinionated score.
Multi-Persona
Its consumers are on-call engineers, product managers, customer support, ML models, executives and end users — each needing the same truth at a different granularity.
Read-Mostly
Except for a narrow control channel (feature flags, sampling rate changes) it is fundamentally a read-heavy analytics system. Writes come from clients, reads come from humans and automation.
Why It Matters
Any product that ships audio or video is one bad packet-loss burst away from an angry tweet. Real-time monitoring is what turns a fragile media pipeline into a business that can promise, and honour, a five-nines conversation experience.
What you gain
- Seconds-fast MTTR. Regional issues are visible before the first support ticket lands.
- Objective SLAs. Contracts with enterprise buyers can be tied to real, measured MOS and drop rates.
- Product-level insight. Which codec, which build, which OS, which network works best — and where to invest next.
- Adaptive media. The stack can react to live telemetry — switching codecs, lowering resolution, re-routing through a closer TURN server — before humans notice.
- ML fuel. Every call becomes a labelled training example for the next generation of bandwidth estimators, echo cancellers and audio enhancers.
What it costs
- Storage & egress. High-frequency telemetry from tens of millions of endpoints is the largest single line item in most VoIP platforms.
- Client CPU & battery. Every metric collected on a mobile handset is a millisecond stolen from the call itself.
- Alert fatigue. A naive threshold system will page the on-call at 03:00 every time a coffee shop’s Wi-Fi hiccups.
- Privacy surface. The same telemetry that helps engineers can, if mishandled, reveal who called whom, from where and for how long.
Every metric you collect must earn its keep. If nobody can point to a dashboard, alert or ML model that uses a metric within the next 90 days, do not ship it. Telemetry that nobody reads is a bill you pay forever.
The Signals: What We Actually Measure
A monitoring platform is only ever as good as the signals it collects. This chapter walks through the three layers of telemetry every serious call platform gathers and shows how they combine into a single perceptual score.
4.1 Transport Layer
These are the raw, unavoidable physics of the internet — sampled directly from the WebRTC/ICE stack or the native SIP/RTP engine every second.
| Metric | What it captures | Typical sampling |
|---|---|---|
| RTT (round-trip time) | End-to-end latency between endpoint and media server, in ms. | 1 Hz |
| Jitter | Variance in packet inter-arrival time. | 1 Hz |
| Packet loss % | Fraction of RTP packets never delivered. | 1 Hz |
| Available bandwidth | Estimator’s current guess of usable bitrate. | 1 Hz |
| Route / TURN relay | Which relay path is active; changes are events. | on change |
4.2 Media Layer
Above the transport, we care about what the codec is actually producing and what the decoder is actually reconstructing.
MOS-LQO, Freeze, Concealment
Objective mean opinion score, freeze rate, concealment ratio from the neural PLC, echo return loss, noise suppression aggressiveness.
Framerate, Resolution, Freeze
Encoded and rendered framerate, resolution ladder used, freeze count and freeze duration, keyframe request rate, decode-time p95.
Text Legibility
Fraction of time text-heavy screen share stays above a legibility threshold, dropped tile count.
A/V Skew
Lip-sync offset between audio and video streams, sampled every few seconds.
4.3 Device & UX Layer
Even a perfect network cannot rescue a call where the microphone is muted at the driver level or the CPU is pinned by a background scan.
- CPU %, thermal state, battery level, power mode.
- Camera / microphone permission state, hardware id, sample rate.
- Join success, join latency, in-call errors, user-submitted post-call rating.
Building Blocks of the Platform
With the signals defined, we can sketch the physical components that carry them from the microphone in someone’s living room to a graph on an engineer’s screen in another hemisphere.
Client SDK
Runs inside every native app, browser tab and SIP handset. Samples the media stack, applies privacy scrubbing, buffers on network hiccups, ships batched events over HTTPS or QUIC.
Edge Ingest
Global anycast fleet that terminates TLS, authenticates the endpoint, enforces per-tenant quotas, drops corrupt payloads and hands normalised events to the bus.
Event Bus
A partitioned, durable log (Kafka, Pulsar, Kinesis) keyed by call id so all events for a single conversation land on the same partition and stay in order.
Stream Jobs
Stateful Flink / Spark Structured Streaming jobs that compute rolling MOS, freeze rate, packet-loss bursts and derived call-level scores in windows of one to ten seconds.
Hot Store
A time-series database (Prometheus / M3 / VictoriaMetrics / Influx) that powers live dashboards with sub-second query latency but retains only the last few days.
Cold Lake & DW
Raw events land in a data lake as Parquet/Iceberg, and enriched aggregates land in a columnar warehouse (BigQuery, Snowflake, ClickHouse) for long-term analytics and ML.
Anomaly & ML
Detectors compare live windows to historical baselines by tenant, region, ISP, build and codec — and hand anomalies to alerting with rich context.
Delivery
Grafana-style dashboards, SQL endpoints, embedded “call quality” widgets inside the support console, and end-user badges in the client itself.
The Ingestion Pipeline in Detail
The ingestion pipeline is where most call-monitoring projects live or die. It has to accept a firehose of writes, tolerate arbitrary network weather, avoid double-counting, and stay cheap. This chapter breaks down the moving parts.
6.1 The Client Contract
The SDK is not a naive HTTP poster. It is a small, resilient distributed system in itself:
- Sampling ladder. Non-critical metrics are sampled at 0.1–1 Hz; critical events (join, leave, disconnect, codec switch) are sent immediately.
- Batch & compress. Events are grouped into 1–5 s batches, protobuf-encoded and gzipped before shipping.
- Local ring buffer. If the network is unreachable, the SDK spills to an in-memory or on-disk ring so the last few minutes of pre-crash telemetry survive.
- Idempotency keys. Every batch carries a monotonic sequence per session so the server can safely de-dup on retry.
6.2 The Edge
The edge fleet is a stateless, horizontally-scaled service behind an anycast IP.
Terminate
TLS 1.3 with 0-RTT resumption for repeat clients; QUIC for mobile networks where TCP handshakes are expensive.
Authenticate
Short-lived JWTs issued at call join; verified locally against a rotating JWKS cached in the edge.
Validate
Schema check, size cap, per-tenant rate limit, and drop of any field on the current PII deny-list.
Enrich
Add server-side facts the client cannot know: IP→ASN→ISP mapping, geo, edge region, ingest timestamp.
Publish
Write to the event bus, keyed by call id so all events for a conversation stay in the same partition.
6.3 The Bus and the Windows
Downstream of the bus, stream jobs run on the same partitions. Windows are almost always “event-time” with a small allowed lateness, so a mobile client whose radio spent 20 seconds re-connecting still contributes its telemetry to the correct minute.
Keep the schema for raw events aggressively narrow — a handful of numeric fields plus dimensions. Rich, wide events feel convenient in year one and turn into a five-figure daily bill in year three.
Detection, Scoring & Alerting
Once telemetry is flowing, the platform has to answer three questions in real time: is any call bad, is any cohort of calls bad, and is anyone experiencing that badness who deserves to be alerted? Each question is a different detector.
7.1 Per-Call Scoring
The per-call score is a function of transport, media and UX signals combined over a rolling window. A typical, opinionated formula:
call_score = w1 · MOS_audio
+ w2 · f_video_freeze
+ w3 · f_video_framerate
+ w4 · f_screen_legibility
- p_join_failure
- p_disconnectWeights are learnt from post-call user ratings and validated against controlled experiments. The output is bucketed into Excellent / Good / Fair / Poor so humans can reason about it without staring at a decimal.
7.2 Cohort Detection
Aggregating per-call scores by tenant, region, ISP, client build, OS version and codec turns individual bad calls into incidents. A simple z-score against a 28-day baseline catches most real regressions; a Bayesian change-point detector catches the subtler ones.
Sudden Drop
MOS in AS-12345 falls three sigma below its own 28-day median for > 3 consecutive minutes.
Slow Bleed
Freeze rate on build 8.2.1 has been creeping up 4% week-on-week for a month.
Correlated
Join-failure spike on Android 14 correlates with a codec-switch event; strong hint at a client bug.
Contextual
Poor MOS restricted to a single enterprise tenant — likely a customer network problem, not ours.
7.3 Alert Fabric
Alerts are only useful if they wake the right human. The fabric routes each incident along three axes:
- Ownership. Client bug → mobile team; media routing → SFU team; ISP-scoped → network relations.
- Severity. Sev-1 pages instantly; Sev-3 lands quietly in a triage queue.
- Suppression. Duplicate alerts, planned maintenance windows and known-flapping cohorts are collapsed.
Never page on raw thresholds like “packet loss > 2%”. Some coffee shop somewhere on Earth is always losing packets. Page on anomalies relative to a cohort’s own baseline, or you will train your on-call to ignore every page you send.
End-to-End Flow of a Bad Call
Nothing crystallises an architecture like following a single event through it. Let us walk one bad call — a laptop in Berlin joins a Meet call with a Bangalore host, hits a bad Wi-Fi patch three minutes in, recovers, and rates the call two stars — from tap to on-call escalation.
Join
SDK opens an outbound stream to the edge, sends a “join” event tagged with build, OS, network type and codec capabilities.
Bad patch
Packet loss climbs from 0.3% to 8% over 20 seconds. Rolling MOS drops from 4.4 to 2.6. Freeze rate jumps to 12%.
Stream reacts
The stream job’s per-call detector marks the call “Poor”. In parallel, a cohort detector notices thirty-seven other calls in the same Berlin ASN degrading in the same minute.
Alert
A single, correlated Sev-2 alert fires for ASN 3320 in Berlin, not thirty-seven separate pages. The on-call opens the pre-baked runbook link.
Recovery
The user’s Wi-Fi recovers, MOS climbs back to 4.3, freeze rate drops to 0.4%. The anomaly detector closes the incident automatically.
Rating
The user rates the call two stars. The rating is joined to the call’s telemetry in the warehouse; the training set gains one more real-world label.
Quality Attributes: What This System Must Guarantee
A monitoring platform lives and dies by its non-functional properties. The functional feature list is short; the guarantees are where the engineering happens.
Under 10 seconds
From metric emitted on device to visible on dashboard. Anything slower and operators stop trusting the picture.
Under 60 seconds
From anomaly onset to on-call page. Any longer and the customer notices first.
99.99% ingest
Monitoring must survive its own product’s outages — ideally on independent infrastructure, and always deployable through a separate change window.
Zero silent drops
Every event is either persisted or explicitly counted as dropped; there is no third state.
Bounded & policed
Per-tenant metric cardinality is capped; runaway labels are quarantined, never allowed to blow up the TSDB.
Predictable per call
Cost-per-minute-of-conversation stays flat as scale grows — no sudden hockey stick when a customer 10×’s their usage.
Minimal & masked
No content, no user text, no raw IPs after enrichment — only what an on-call needs to root-cause.
Change-safe
Schema evolution is additive and versioned; historic dashboards keep working even as new fields land every week.
Ingest on independent infrastructure
AcceptedContext. The tempting shortcut is to co-locate the monitoring ingest fleet with the media plane, sharing edge PoPs and databases.
Decision. Run the monitoring ingest on independent PoPs, a separate anycast IP, and a separate cloud account with its own IAM boundary.
Consequence. When the media plane fails — and one day it will — the monitoring platform is still there to tell you why. The extra cost is real but small compared to a blind incident.
Pitfalls Every Team Hits
There are a handful of failure modes that every team building this platform meets, usually in year two, and the fix is never a heroic rewrite — it is a small, boring decision made early.
What tends to break
- Cardinality explosion. Someone adds “user_id” as a label; the TSDB is measurably sad within an hour.
- Retry storms. A regional outage restores; every SDK on Earth flushes its ring buffer in the same 30 seconds; the edge folds under the thundering herd.
- Silent drops. A schema change adds a required field; older clients silently fail validation and their events vanish for weeks before anyone notices.
- Clock skew. A phone with a wrong system clock ships events tagged in 2029; naive event-time windows never close.
- Alert fatigue. Static thresholds catch every coffee shop and page the on-call twenty times a night.
- Privacy drift. A field added “temporarily for debugging” makes it into cold storage and is still there three years later.
What actually helps
- Server-side cardinality guards. The edge counts distinct label combinations per tenant and refuses new ones once a budget is exceeded.
- Jittered retries. SDKs spread the flush of buffered events over a random window and exponential-backoff on 429s.
- Contract tests. Every schema change is validated against a corpus of real historical payloads before it ships.
- Ingest-time clock check. The edge rejects or shifts obviously wrong timestamps and tags them for investigation.
- Baseline alerts. Every alert compares to a cohort’s own historical baseline, not a global threshold.
- PII register. A living document lists every field, its purpose and its retention; anything not on it is not written.
How It Evolves
A call-quality platform is never “done”. Its architecture changes shape as the product grows, and the good news is that the evolution is fairly predictable.
Diagnostic dashboards
A single team, a single region, a single dashboard. Metrics land in a TSDB; alerts are hand-authored. Good enough to tell you when things are bad; useless at telling you why.
Multi-tenant & cohorts
Signals are enriched with tenant, region, ISP, build. Dashboards are sliceable. On-call rotations start to trust the pages.
Warehouse & SLAs
Cold storage lands in a columnar warehouse; historical trends drive product decisions and enterprise SLAs.
Anomaly detection
Statistical detectors replace static thresholds. The on-call rotation shrinks; the mean-time-to-detect drops from tens of minutes to under a minute.
Closed loop
Detectors feed the media plane: a bad ISP triggers automatic re-routing through a different TURN region. The platform stops being a passive observer and starts steering the product.
ML-first
Per-call MOS is estimated by a neural model trained on billions of real ratings; anomaly detection is a graph over tenants and topologies; the platform becomes indistinguishable from the product itself.
Every stage is cheaper if you built the previous stage on additive schemas. The teams that struggle most are those that shipped V1 with a rigid, tightly-coupled event format and now have to rewrite two years of history to reach V4.
Key Takeaways
If you remember only a handful of things from this walkthrough, remember these — they are the ones that separate a monitoring platform that scales from one that quietly bankrupts a team.
- Model the human, not just the packet. The scoring layer must translate raw transport and media metrics into a perceptual score a person would give the call.
- Own the ingest end-to-end. The SDK, the edge and the bus are one system; you cannot outsource any of them and keep the guarantees.
- Independent infrastructure. Monitoring must survive its own product’s outages. Different PoPs, different accounts, different change windows.
- Baseline before you threshold. Alerts must fire on anomalies relative to a cohort’s own past, not on absolute numbers.
- Additive schemas forever. Every field is easy to add and painful to remove; design the schema and the retention register with a five-year horizon.
- Cardinality is your quiet enemy. Cap it at the edge, watch it every day, and never let a single tenant ship a runaway label.
- Close the loop. The endgame is not dashboards; it is a monitoring signal that automatically steers the media plane before humans intervene.
- Privacy is a feature. Every field carries a purpose, a retention and a legal basis. If you cannot explain why a field exists, you cannot ship it.
A well-built call-quality monitoring platform is invisible when everything works and priceless the moment something breaks. Build it as if you will one day have to defend every field, every alert and every dashboard to an executive, an auditor and a very tired on-call engineer — because eventually, you will.