Imagine two people on a video call — one speaking Japanese in Tokyo, the other speaking Spanish in Madrid. A few years ago, this call would have needed a human interpreter sitting in on the conversation, translating sentence by sentence, with awkward pauses and delays. Today, systems like Microsoft Teams’ live translated captions, Google Meet’s real-time subtitles, Zoom’s AI Companion translation, and Skype Translator do this automatically, converting speech from one language to text or speech in another language, in near real time, while the call is still happening.

This is one of the hardest problems in applied real-time systems, because it sits at the intersection of three very different engineering disciplines: low-latency real-time media transport (the same technology that powers video calls), streaming machine learning inference (speech recognition and translation models that must produce partial results within milliseconds), and large-scale distributed systems (because this has to work for millions of simultaneous calls without falling over).

A Brief History

Real-time speech translation research goes back decades, but it only became practical for consumer products in the last ten years, driven by three parallel advances. First, automatic speech recognition (ASR) moved from rigid, dictionary-based systems to deep learning models that could handle natural, messy, overlapping speech. Second, machine translation moved from statistical phrase-based translation to neural machine translation (NMT), which produces far more fluent and context-aware output. Third, and most importantly for this system, both ASR and NMT learned to work in a streaming fashion — producing partial, continuously-refined output as audio arrives, instead of waiting for someone to finish speaking before processing anything.

Skype Translator, launched around 2014, was one of the first mainstream attempts at real-time speech-to-speech translation in a video call. It proved the concept was possible but suffered from noticeable delay and translation quality issues. Since then, transformer-based architectures, better streaming inference techniques, and dramatically cheaper GPU inference have made the experience good enough for daily use in products like Google Meet, Microsoft Teams, and Zoom.

In this tutorial, we will design such a system from scratch: a real-time language translation layer that sits on top of a live video call, converting spoken language from one participant into captions or synthesized speech that another participant can understand, with the smallest latency the underlying technology allows, at the scale of millions of concurrent call-minutes per day.

Before drawing any boxes and arrows, we need to be precise about what “real-time” and “minimal added latency” actually mean here, because these constraints shape every architectural decision that follows.

2.1 Functional Requirements

2.2 Non-Functional Requirements

• Low latency: The end-to-end delay from a word being spoken to its translation appearing for the listener should ideally be under 2 to 3 seconds, and the “added” latency on top of the existing call’s audio/video delay should be minimized as much as possible — every extra millisecond in the pipeline directly hurts conversational flow.
• High accuracy: Poor transcription or mistranslation in a business or medical context can cause real harm, so quality cannot be sacrificed carelessly for speed.
• Massive scale: The system must support millions of concurrent call-minutes, with translation requests arriving continuously as small audio chunks, not as large batch jobs.
• High availability: A translation outage should degrade gracefully (e.g., fall back to captions only, or turn translation off) rather than dropping the underlying call.
• Privacy and security: Voice data is sensitive biometric and personal information; it must be encrypted in transit, processed with strict data handling policies, and, ideally, not stored longer than necessary.
• Multilingual coverage: Support dozens of source and target languages, including low-resource languages, with acceptable quality.
• Cost efficiency: Running GPU-backed ASR and translation models for millions of concurrent minutes is expensive; the design must be efficient about when and how much compute it uses.

2.3 Why This Is Architecturally Hard

The fundamental tension in this system is between latency and accuracy. Speech recognition and translation models generally produce better results when they can see more context — for example, waiting until the end of a sentence before translating it, because word order and meaning can change based on what comes later in the sentence. But waiting for a full sentence adds delay that breaks the feel of a live conversation. The entire design revolves around resolving this tension using streaming models, incremental re-translation, and careful latency budgeting across every hop in the pipeline.

At a high level, the system has two parallel planes: the existing real-time media plane (audio and video packets flowing between participants through the video calling infrastructure) and a new translation plane that taps into the audio stream, runs it through a speech-to-text-to-translation pipeline, and pushes results back out to viewers as captions or synthesized audio.

Two design decisions stand out immediately and are worth explaining, because they are exactly the kind of thing an interviewer will probe.

3.1 Decision 1: Tap, Don’t Interrupt

The translation plane receives a duplicated copy of the audio stream; it never sits in the critical path of the original call’s audio delivery. This means that even if the entire translation pipeline crashes, the underlying video call between participants continues completely unaffected — participants just lose captions or dubbed audio temporarily. This “tap” pattern (sometimes called a side-car or shadow pipeline) is a core reliability principle: never let an auxiliary feature become a single point of failure for the core product.

3.2 Decision 2: Separate Data Channel for Results

Translated captions and metadata are delivered to clients over a separate low-overhead data channel (e.g., WebRTC data channels or a WebSocket connection) rather than being muxed into the audio/video stream itself. This keeps the translation plane decoupled from the media transport layer, and lets us update or restart translation services without touching the video/audio pipeline.

Let’s go through every major building block in depth, the way an interviewer would expect for a senior-level system design discussion.

Let’s trace exactly what happens, frame by frame, from the moment a speaker utters a word to the moment a listener sees or hears the translation.

5.1 Audio Framing and Buffering

Raw audio captured by the microphone is typically sampled at 16 kHz or 48 kHz and split into small frames (commonly 20 milliseconds each, matching typical WebRTC audio packetization). These frames are batched into slightly larger windows (for example 100 to 300 milliseconds) before being handed to the ASR model, because feeding a model one 20 ms frame at a time is inefficient; there’s a deliberate trade-off here between “how small a chunk we send” (lower latency) and “how efficient inference is” (batching improves GPU utilization).

5.2 Endpointing

The system needs to know when a speaker has finished a phrase or sentence, a process called endpointing. This can be done acoustically (a pause in speech beyond some threshold, say 300–700 milliseconds of silence) or semantically (the ASR/MT model recognizes the current text forms a grammatically complete unit). Endpointing quality directly affects perceived latency and correctness: cut too early and you truncate meaning; wait too long and you add dead air.

5.3 Incremental ASR Decoding

As audio frames stream in, the ASR model’s decoder (commonly an RNN-Transducer or streaming Conformer-Transducer) emits token hypotheses continuously, using techniques like beam search constrained to a small number of candidates to stay fast. Early tokens are marked “unstable” and may be revised; once enough right-context has been seen and the model’s confidence stabilizes (or an endpoint is detected), tokens are marked “final” and are no longer revised. This partial/final distinction is exactly what lets captions appear instantly while still being corrected if the model changes its mind.

5.4 Sentence Segmentation for Translation

Because translating word-by-word produces poor quality (translation needs some syntactic context), the pipeline groups the ASR’s evolving token stream into translation units — typically clauses or short phrases bounded by punctuation predictions or a rolling word-count/time window. The MT model translates each unit as soon as it looks “translatable,” and re-translates it if the ASR later revises earlier words.

5.5 Streaming Neural Machine Translation

The MT model, often a transformer encoder-decoder or a decoder-only model fine-tuned for translation, is configured for simultaneous translation: rather than reading the entire source sentence before producing any output (as classic MT does), it uses a “wait-k” style policy — waiting for k source words before starting to emit target words, then continuing to consume source tokens and emit target tokens in an interleaved fashion. This lets translation begin well before the speaker finishes their sentence, at the cost of occasionally needing to revise earlier translated words when new source context arrives (particularly important for languages with different word orders).

5.6 Optional Text-to-Speech Synthesis

If voice dubbing is enabled, translated text segments are handed to a streaming TTS model (commonly based on neural vocoders paired with acoustic models like FastSpeech-style or diffusion-based architectures tuned for low-latency streaming synthesis) which generates audio incrementally as text arrives, rather than waiting for a full sentence.

5.7 Delivery and Client-Side Rendering

Translated captions (with timing and stability metadata) are pushed to listener clients over a low-latency channel. The client renders partial captions immediately (often shown in a lighter/greyed style) and updates them smoothly as final text arrives, avoiding jarring text replacement. For audio dubbing, the client’s audio mixer ducks the original speaker’s volume and plays the synthesized translated audio, buffered just enough to stay reasonably in sync with the video.

Let’s walk through the full lifecycle of a translation session, from call setup to teardown.

6.1 Session Initialization

When a call starts and a participant enables live translation, the client sends a request to the Session Orchestrator specifying the participant’s spoken language (or “auto-detect”) and each listener’s desired caption/audio language. The orchestrator allocates or reserves capacity on the ASR and MT service pools for this call and registers the call’s audio tap with the nearest regional media edge node.

6.2 Steady-State Streaming

While the call is active, audio frames flow continuously from the media edge into the ASR service. The ASR service maintains per-speaker streaming state (a “decoding session”) so that context from previous frames informs the recognition of new frames, without needing to resend earlier audio. Transcripts stream out to the MT service, which similarly maintains per-segment translation state, and translated output streams to the orchestrator, which fans it out to every listener who requested that target language — importantly, if three listeners in a call all want the same target language, the translation is computed once and multicast, not recomputed per listener.

6.3 Handling Turn-Taking and Overlap

Multi-party calls often have overlapping speech. The system runs a separate ASR/MT pipeline instance per active speaker’s audio track (not per listener), so overlapping speakers are transcribed and translated independently and in parallel; the orchestrator then interleaves or queues captions for display based on timestamps, and the client UI typically shows the most recent speaker’s caption prominently while others queue briefly.

6.4 Speaker or Participant Changes

When a new participant joins, the orchestrator spins up a new per-speaker pipeline binding on demand; when a participant leaves or stops speaking for an extended period, their ASR/MT session is torn down or moved to an idle, low-resource state after a timeout, freeing GPU capacity for other calls.

6.5 Session Teardown

When the call ends, the orchestrator releases all allocated ASR/MT/TTS session resources, flushes any short-lived buffers, and, per data retention policy, ensures raw audio and intermediate transcripts used only for real-time processing are not persisted beyond what is explicitly needed (e.g., for abuse investigation or opt-in call recording).

Because the entire point of this system is “minimal added latency,” we need to think about latency the way a performance engineer would: as a budget, allocated in milliseconds across each stage, with a target total and a plan for what happens when a stage runs over budget.

7.1 A Representative Latency Budget

Summed, a captions-only pipeline commonly lands in the 400 ms–1 second range for the first partial caption to appear, with finalized captions trailing the actual speech by roughly 1–2 seconds. A voice-dubbing pipeline typically adds another half a second to a full second on top of that, which is why most products ship captions before they ship full audio dubbing — the perceptual bar for “does this feel synced” is much stricter for audio than for text.

7.2 Latency Reduction Techniques

• Regional GPU placement: Deploy ASR/MT inference clusters in the same regions as the video calling infrastructure’s media edge nodes, so the extra network hop for translation is a few milliseconds rather than crossing continents.
• Chunked, streaming inference: Both ASR and MT models are specifically chosen/trained for streaming (causal or limited-lookahead architectures) rather than full-sequence models that require the entire input before producing output.
• Dynamic batching with latency-aware scheduling: GPU inference servers batch multiple concurrent calls’ audio frames together to improve throughput, but batching windows are capped (for example, 10–20 ms) so batching does not itself become a latency source.
• Model quantization and distillation: Serving smaller, quantized (e.g., int8) versions of ASR/MT models trades a small amount of accuracy for significantly faster inference and higher throughput per GPU.
• Speculative partial translation: Beginning translation on partial, not-yet-final ASR output (accepting that some words may need re-translation) rather than waiting for finalized transcripts.
• Wait-k policies tuned per language pair: Language pairs with very different word orders (e.g., English-to-Japanese) may need a slightly larger “k” (more lookahead) for acceptable quality, while similar-order pairs (e.g., English-to-Spanish) can use a smaller k for lower latency — this is a per-language-pair tuning knob.

7.3 Trade-off: Stability vs. Speed

An important, often-overlooked trade-off is caption stability. A model tuned to show output as early as possible will revise its guesses more often, causing visible “flicker” as words change on screen — annoying for readers. A model tuned to wait longer before showing output is more stable but feels laggy. Production systems expose this as a tunable parameter and often default to a middle ground: show partials quickly but apply a short stabilization delay (for instance, 300 ms) before locking a word as final, so it only flips once or twice rather than many times.

The system must support millions of concurrent call-minutes — meaning, at any given moment, potentially hundreds of thousands of simultaneous speech streams needing continuous ASR and MT inference. This is fundamentally a GPU capacity and scheduling problem layered on top of a real-time streaming problem.

8.1 Horizontal Scaling of Inference Clusters

ASR and MT services run as pools of stateless-per-request (but session-aware) inference workers behind a scheduling layer. Each worker hosts one or more model replicas on a GPU and can serve many concurrent streaming sessions by batching their audio/text micro-requests together. As demand grows, more worker instances are added; a scheduler routes new sessions to workers with available capacity, ideally within the same region as the call to keep latency low.

8.2 Session Affinity

Streaming ASR and MT are stateful — each active speaker’s session carries decoder state (partial hypotheses, translation context) between consecutive audio chunks. This means, unlike typical stateless web requests, we cannot freely load-balance every request to a random worker; the system needs session affinity, routing all frames for a given speaker’s session to the same worker instance (or replica group) for the session’s duration, typically implemented via consistent hashing or a session-to-worker mapping maintained by the orchestrator.

8.3 Autoscaling Based on Concurrent Sessions, Not Just CPU

Traditional autoscaling triggers off CPU or request-rate metrics, but the right signal here is concurrent active speech sessions per region, since that maps directly to GPU memory and compute needs. The system tracks this metric per region and scales GPU worker pools ahead of demand using predictive scaling based on historical call volume patterns (e.g., scaling up before typical business-hours peaks in each region).

8.4 Language-Pair Sharding

Loading every possible language model onto every GPU worker is wasteful. Instead, MT workers are often sharded by language pair or language family, with popular pairs (e.g., English-Spanish, English-Mandarin) getting dedicated, heavily-replicated capacity, while less common pairs share a smaller pool of general multilingual models. Modern multilingual NMT models that can handle many language pairs within a single model (rather than one model per pair) also help reduce the total number of distinct models that need to be hosted and kept warm.

8.5 Handling Traffic Spikes

Global events, holidays, and regional business hours create predictable and unpredictable spikes in concurrent call volume. The system uses a combination of pre-provisioned baseline capacity, fast-autoscaling burst capacity (accepting a brief cold-start latency penalty during rapid scale-up), and graceful degradation — for example, temporarily switching some sessions to a smaller/faster model variant during extreme load rather than rejecting translation requests outright.

Translation is an enhancement layered on top of a call, not the call itself — so the guiding reliability principle is: a translation failure must never break the underlying call. Beyond that, the system still needs its own resilience story so that translation itself stays usable.

9.1 Graceful Degradation Ladder

Rather than a binary “translation works or the whole feature dies,” the system is designed with fallback levels: full voice dubbing degrades to captions-only if TTS capacity is unavailable; live captions degrade to a “translation temporarily unavailable” indicator if ASR/MT capacity is exhausted, while the call itself continues uninterrupted; and if language identification fails, the system falls back to the participant’s previously known or manually selected language rather than blocking output entirely.

9.2 Failover for Stateful Sessions

Because ASR/MT sessions carry state, a worker failure mid-session is handled by periodically checkpointing lightweight session state (recent transcript context, not raw audio) to a fast in-memory store, so a session can be quickly rehydrated on a healthy replica with only a brief, usually imperceptible, gap in output rather than a full session restart.

9.3 Multi-Region Redundancy

Each region’s worker pools are deployed across multiple availability zones, and the global orchestrator can reroute a call’s translation session to a healthy nearby region if an entire region degrades, accepting a temporary latency increase in exchange for continued availability.

9.4 Circuit Breakers and Load Shedding

Under extreme load or partial outages, the orchestrator applies circuit breakers around the ASR/MT/TTS services: if a service’s error rate or latency crosses a threshold, new session requests are temporarily rejected or downgraded (e.g., disabling TTS while keeping captions running) rather than piling up requests that will fail anyway and risk cascading the failure into a healthy adjacent service.

9.5 Health Checks and Warm Standby Capacity

Because GPU workers can take non-trivial time to cold-start (loading multi-gigabyte models into GPU memory), the system maintains warm standby capacity in each region rather than relying purely on reactive autoscaling, which would be too slow to absorb sudden spikes without visible degradation.

Voice is biometric data, and transcripts of private conversations are highly sensitive. Security here is not an afterthought — it shapes core architecture decisions.

10.1 Encryption in Transit

The duplicated audio stream sent from the media edge to the translation plane, and the resulting captions/audio sent back to clients, must be encrypted end-to-end between infrastructure hops (e.g., using SRTP for media and TLS for data channels), matching the security posture of the underlying call itself.

10.2 Minimal Data Retention

Raw audio and intermediate transcripts used purely for real-time processing should be held only in memory for the duration needed to produce output, then discarded — not written to persistent storage — unless the user has explicitly opted into call recording or transcript saving, in which case that data falls under separate, clearly disclosed retention and consent policies.

10.3 Tenant and Session Isolation

In a multi-tenant deployment (e.g., a platform serving many enterprise customers), ASR/MT worker scheduling must guarantee that one customer’s audio, transcripts, or model customizations (like custom vocabulary for industry jargon) are never accessible to another tenant’s session, typically enforced through strict per-session context isolation and access-controlled routing at the orchestrator layer.

10.4 Consent and Transparency

Participants should be clearly notified when live translation (and any associated processing or optional recording) is active in a call, since some jurisdictions have legal requirements around notifying participants that their speech is being processed or recorded, and because trust in the product depends on users understanding what is happening to their voice data.

10.5 Model and Prompt Injection Considerations

If custom vocabulary lists, glossaries, or user-supplied context are fed into ASR or MT models to improve domain-specific accuracy (e.g., a company’s product names), that input should be validated and sandboxed so it cannot be used to manipulate model behavior in unintended ways or leak information across sessions.

10.6 Compliance

Depending on target markets, the system needs to account for regulations such as GDPR (EU), which governs processing of voice biometric data and requires clear legal basis and data subject rights, and similar regional data protection and telecommunications regulations, which may require in-region data processing (data residency) rather than routing audio to a distant region purely for latency or cost reasons.

Because quality here is partly subjective (does this feel natural?) and partly objective (is the latency within budget?), monitoring needs to cover both technical health and translation quality.

11.1 Latency Metrics

11.2 Quality Metrics

• Word Error Rate (WER): Standard ASR accuracy metric, comparing recognized text to ground truth, tracked per language and per audio condition (e.g., noisy vs. clean).
• BLEU / COMET scores: Standard machine translation quality metrics, though production systems increasingly rely more on human evaluation and user-facing signals (like caption correction rates, if users can flag bad translations) since automated metrics don’t fully capture conversational fluency.
• Caption revision rate: How often a partial caption changes before finalizing — a proxy for both model confidence and the flicker/stability trade-off discussed earlier.

11.3 System Health Metrics

• GPU utilization and memory pressure per worker pool, per region.
• Concurrent active sessions per region, compared against provisioned capacity.
• Error rates and timeout rates for each pipeline stage.
• Session failover/rehydration counts, as a signal of underlying infrastructure instability.

11.4 Distributed Tracing

Every audio chunk and its downstream transcript/translation carries a trace ID that follows it through VAD, ASR, MT, and delivery, allowing engineers to reconstruct the full path and timing of any individual utterance when debugging a specific reported issue — critical in a pipeline with this many hops, since a naive aggregate latency dashboard alone would not reveal where a specific slow-down happened.

11.5 Real User Monitoring (RUM)

Client applications report perceived experience metrics (caption render time, audio sync drift, dropped caption events) directly from real user sessions, since server-side metrics alone cannot capture last-mile network conditions or client rendering delays.

12.1 Regional GPU Clusters

ASR, MT, and TTS inference workloads run on GPU-backed compute clusters deployed in multiple geographic regions, co-located as close as possible to the video calling infrastructure’s media edge nodes to minimize the network leg of the latency budget. Kubernetes (or a similar container orchestration platform) is a natural fit for managing these worker pools, using GPU-aware node scheduling and custom autoscalers driven by the concurrent-session metrics discussed earlier rather than generic CPU-based autoscaling.

12.2 Model Serving Infrastructure

Models are served through a dedicated inference-serving layer optimized for streaming workloads — supporting persistent, stateful streaming connections (rather than simple request/response), dynamic batching with bounded wait windows, and fast model loading/versioning so new model releases can be rolled out with canary and rollback support without disrupting active sessions.

12.3 Blue-Green and Canary Deployments

Because this system serves live, ongoing sessions rather than stateless request/response traffic, deployments need to be session-aware: new model or service versions are rolled out to a small percentage of new sessions first (canary), while existing in-flight sessions on the old version are allowed to complete naturally (drain) rather than being abruptly cut over, avoiding a jarring mid-call model switch that could momentarily disrupt caption quality or continuity.

12.4 Edge Presence

Beyond regional GPU clusters, lightweight edge components (VAD, audio framing, and the data-channel delivery path) benefit from being deployed even closer to users — via CDN-adjacent edge points of presence — so the “last mile” network legs on both the capture and delivery sides are as short as possible, since these legs are pure network latency with no compute trade-off to exploit.

12.5 Infrastructure as Code and Multi-Cloud Considerations

Given GPU capacity constraints and cost differences across cloud providers, many production deployments of this kind of system are built to run across multiple cloud providers or a hybrid of cloud and dedicated GPU capacity, using infrastructure-as-code tooling to keep regional deployments consistent and using an abstraction layer in the scheduler so the orchestrator does not need to know which underlying cloud a given worker pool runs on.

13.1 What Actually Needs Persistent Storage

Much of this system is deliberately stateless-in-storage — raw audio and live transcripts flow through memory and are not written to a database by default. The things that do need persistent storage are: call and session metadata (participants, chosen languages, session start/end times) for orchestration and billing; user/tenant configuration such as custom vocabularies, glossaries, and language preferences; model version and deployment metadata; and, only if explicitly opted in, saved transcripts or recordings.

13.2 Session Metadata Store

A low-latency key-value or document store (such as a distributed store like Redis for hot session state and a durable document database for longer-lived call metadata) tracks each active call’s participants, language settings, and which worker instances are currently bound to which speaker sessions — this is read and written frequently during a call’s lifetime and must stay fast and consistent enough for the orchestrator to route audio correctly in real time.

13.3 Caching Strategies

13.4 Load Balancing

Two distinct load balancing problems exist here. First, connection-level load balancing routes each new session to an appropriate regional cluster and, within that cluster, to a worker with available capacity — this needs to be session-aware (sticky) because of the stateful streaming decoding discussed earlier, unlike typical round-robin load balancing for stateless HTTP requests. Second, GPU-level batching within a worker acts as a form of micro load-balancing, grouping many small concurrent inference requests together to make efficient use of each GPU’s parallel compute, with careful tuning so this batching does not introduce unacceptable latency for any individual session.

13.5 Why Not a Traditional Relational Database for Hot Path Data

Session-routing and streaming decoder state changes many times per second per active call and must be read with sub-millisecond latency by the orchestrator; a traditional relational database, even a fast one, is the wrong tool for this specific hot path, which is why an in-memory store is used there, with relational or document databases reserved for less latency-sensitive metadata like billing records, historical call summaries, and tenant configuration.

14.1 Service Boundaries

This separation allows each to scale independently (MT scales with number of target languages requested; ASR scales with number of active speakers) and to be owned, deployed, and upgraded by different teams without tightly coupling release cycles.

14.2 Protocol Choices

For the continuous, low-latency, bidirectional streaming needed between VAD/ASR/MT/TTS stages, a persistent streaming protocol such as gRPC bidirectional streaming (over HTTP/2) is a natural fit internally between services, since it avoids the overhead of repeated connection setup and supports efficient binary framing of audio and text chunks. For delivering results to end-user clients already connected via WebRTC, a WebRTC data channel or a WebSocket connection is used, matching the persistent, low-latency needs of the use case rather than a request/response REST API, which would be a poor fit for continuously streaming partial captions.

14.3 Public-Facing API for Developers

For third-party developers who want to embed live translation into their own video products (a realistic productization of this system), a higher-level API is exposed: a session-creation endpoint to start a translation session bound to a call, a streaming endpoint (WebSocket or gRPC stream) to send audio and receive captions/audio, and configuration endpoints to set source/target languages, custom vocabulary, and dubbing vs. captions mode. This public API sits behind the same orchestrator and inherits the same latency and reliability characteristics as the first-party product.

14.4 API Contract Considerations

Because output is inherently incremental, the API contract must explicitly model partial versus final results (rather than a single “translation” field), include timing/sequence metadata so clients can correctly order and replace partial results, and include confidence or stability indicators so client applications can choose how aggressively to display still-evolving text.

14.5 Backward Compatibility and Versioning

As ASR/MT/TTS models evolve, the orchestration and delivery layer’s external API contract is kept stable and versioned independently of internal model versions, so that upgrading the underlying translation model does not require every client application to change its integration code — the API describes stable concepts like “partial caption event” and “final caption event,” not model-specific internals.

16.1 Best Practices

16.2 Common Mistakes

Designing real-time translation for live video calls is fundamentally an exercise in managing a latency budget across a chain of increasingly sophisticated streaming machine learning models, while never letting that added complexity threaten the reliability of the underlying call. The winning architecture taps a copy of the audio stream rather than sitting inline, uses streaming-native ASR and MT models that emit and revise partial results incrementally, and fans out translated output efficiently to potentially many listeners with different language preferences in the same call.

“Correctness and speed pull against each other at the model level, not just the infrastructure level — the interviewer’s real question is whether you naturally reach for streaming-first thinking, session-aware scaling, and graceful degradation.”

Interviewers asking this question are typically probing whether a candidate can reason about a system where “correctness” and “speed” pull against each other at the model level, not just at the infrastructure level — and whether the candidate naturally reaches for streaming-first thinking, session-aware scaling, and graceful degradation rather than treating this as just another CRUD-style microservices design problem.