A noise-cancelling & background music system is the pipeline that takes the raw sound picked up by a microphone, separates the human voice from the noise, and optionally mixes in a curated background track — all in a few tens of milliseconds, on a laptop, phone or embedded device — before that audio is packetised, sent across the internet and heard by a remote listener as if they were in a quiet studio with you.

The hard part is that voice and noise share the same air. A door slam, a crying baby, keyboard clicks, HVAC hum and a barking dog all arrive at the microphone as one waveform. The system has to reverse-engineer that mix in real time, keep the voice, drop the rest, and then very gently add music underneath — without letting the music itself get suppressed by its own noise cancelling.

Before we design it, we need to pin down what “noise cancelling” and “background music” actually mean in a real-time voice platform. They are two closely related but distinct problems, and the temptation to conflate them is what produces the most frustrating bugs.

2.1 A Working Definition

A noise-cancelling & background music system is a real-time audio pipeline that, for every capture buffer coming out of the microphone, guarantees:

• the human voice is preserved with high fidelity, correct timbre and no artefacts,
• non-voice sound (fans, keyboards, traffic, dogs, sirens, room reverb) is suppressed by a target amount,
• optional background music or ambient audio can be layered below the voice on the send stream without being suppressed itself,
• the whole chain adds only a few tens of milliseconds of extra latency — well inside the perception of “real-time,”
• the pipeline can be enabled per participant, per meeting or per device, with graceful fallback when compute budget is tight.

2.2 Where You Encounter It

2.3 What It Is Not

It is not the codec, not the packetiser, and not the network jitter buffer. It sits upstream of all of those — on the audio-capture side of the client, and mirrored on the render side when needed. It also is not a mastering suite: broadcast production tools have quality and latency budgets orders of magnitude different from a real-time meeting.

Audio quality is the single most under-rated driver of user retention on voice platforms. Video can pixellate and users forgive; audio drops or garbles and users leave. Noise cancelling and controlled background audio are what turn a “phone call” into a “meeting.”

3.1 The Business & Human Problem

• Meetings from anywhere. Cafes, cars, kitchens, open-plan offices — without NC, remote work is only possible in perfect rooms.
• Cognitive load. Every distracting noise eats a slice of the listener’s attention; ten minutes of a barking dog fatigues the whole call.
• Accessibility. Hearing-impaired participants rely on clean voice for lip-sync and transcription; noisy audio breaks captions and translation.
• Brand & presence. Contact centres, sales teams, teachers — the audio quality is the brand impression. Background music is part of that brand.
• Downstream ML. Speech-to-text, summarisation, meeting notes and translation all depend on clean voice; noise degrades every AI feature that follows.

3.2 What Makes It Uniquely Hard

Every design choice hangs off one boring but sacred timeline: how a sample enters the microphone and exits the speaker on the other end. Get the ordering wrong and echoes, robot voices and missing music are the inevitable result.

4.1 The Capture Path

1. Microphone & driver — samples at 16, 32 or 48 kHz; hardware may add its own AGC/AEC/NR that the app must control.
2. Framing — audio is grouped into 10 or 20 ms frames.
3. AEC (acoustic echo cancellation) — removes what the local speaker is playing so it does not leak back to the far end.
4. NC (noise cancelling) — suppresses non-voice content in each frame.
5. Voice Activity Detection (VAD) — decides if a frame contains speech; drives DTX and mixer gates.
6. Auto Gain Control (AGC) — normalises voice level.
7. Background music mixer — ducks the music behind the voice, applies loudness normalisation.
8. Encoder — Opus or platform codec at target bitrate.
9. Packetiser + jitter-safe transport — ships frames over the network.

4.2 The Render Path

1. De-packetiser + jitter buffer.
2. Decoder.
3. Optional receive-side NC for participants whose devices did not run send-side NC.
4. Mixer — combines multiple participants; the local user hears everyone.
5. AEC reference signal — fed back to the capture path’s AEC to cancel local echo.
6. Speaker / driver.

4.3 The Timing Budget That Rules Everything

A production noise-cancelling & background-music system is a small constellation of focused components. Each has one narrow job; the leverage is in how they compose — and in the strict discipline of who owns which sample.

Noise cancelling is one of the oldest problems in signal processing, and one of the most quietly transformed by deep learning. Understanding the three generations lets us pick the right one for the right device.

6.1 Classical Spectral Subtraction

Estimate a noise floor in the frequency domain during silence, and subtract it from every subsequent frame. Cheap, deterministic, well-understood — and terrible on non-stationary noise like a dog barking or a keyboard.

• Pros: Microseconds per frame, no ML dependency, easy to reason about.
• Cons: Musical noise artefacts, poor performance on transient sounds, hurts voice under aggressive settings.
• Fits: Embedded devices, fallback path when compute is starved.

6.2 Statistical / Wiener Filters

Model speech and noise as random processes; compute a Wiener filter per frame that minimises expected error. Better than spectral subtraction on stationary noise; still struggles with speech-shaped interference and transients.

6.3 Deep Learning: RNNoise, DTLN & Transformers

6.4 A Simplified Frame-Level API

struct NCFrame {
  float[]  in;         // 160 samples @ 16 kHz = 10 ms
  float[]  reference;  // AEC reference from render path
  float    localSNR;   // hint from VAD/AGC
};

Frame ncProcess(NCFrame f, NCModelHandle model) {
  // returns cleaned frame + confidence + speech probability
  auto spec = fft(f.in);
  auto mask = model.infer(spec, f.reference);   // gain per frequency band
  auto out  = ifft(apply(mask, spec));
  return { samples: out, speechProb: model.vad(), suppressedDb: model.gainStats() };
}

6.5 Choosing the Right Model at Runtime

Adding music to a voice stream sounds simple. It is not. The music must survive the noise canceller (which is trained to kill everything that is not voice), stay legally licensed, duck under speech automatically, and never introduce clicks, pops or drift across a long call.

7.1 Where the Music Enters the Chain

The most consequential decision is where the music is injected. There are three canonical options:

Most production systems inject music post-NC, pre-codec on the send path, and additionally support a separate track for high-fidelity broadcasts.

7.2 Voice-Aware Ducking

7.3 Music Library, Rights & Delivery

• Pre-licensed catalogue: royalty-free packs, licensed collections, or user-uploaded tracks with responsibility statements.
• Per-region blocklists to comply with jurisdictional rights.
• Pre-decoded and cached on the device to avoid on-the-fly decoding stalls.
• Live-stream royalty reporting when broadcasts include licensed music.

7.4 Music vs Voice: The Uncomfortable Truth

Enough abstraction. Let us follow one audio frame — 20 ms of a video meeting with a barking dog in the background and a lo-fi track playing beneath the speaker’s voice — from microphone to remote listener.

Real-time audio has a very particular set of non-functional targets. It must be simultaneously low-latency, high-quality, energy-efficient, and gracefully degrading — on hardware ranging from a flagship laptop to a cheap Android phone in an underground metro.

9.1 Voice-Quality Targets

Every real deployment gets bitten by the same handful of subtle bugs. Knowing them turns weeks of firefighting into a paragraph in a design review.

10.1 Ten Traps We’ve All Fallen Into

10.2 The Trade-offs You Cannot Avoid

Real-time speech enhancement is one of the fastest-moving subfields in applied ML. The trajectory from spectral gates to personal voice models has taken a decade, and the next decade will keep raising the bar.

11.1 Adjacent Systems That Plug In

Noise cancelling and background music are two of the most quietly transformative features in modern communication. Every design choice here serves one goal: make a remote conversation feel like a shared, calm room — without ever calling attention to the machinery that makes it possible.