The math of listening.
How a pile of 16-bit numbers becomes "someone is talking." A low-level walk from raw samples to the FFT to a working noise gate — every number here came from a live run on Meatball's actual microphones.
samples → DC-block → window → FFT → magnitude/dB → { profile · denoise · gate } → profitSamples — what sound is to a computer
A microphone's converter measures air pressure R times a second — Meatball records at 8 or 16 kHz. Each measurement is a 16-bit signed integer (s16le), a number between −32768 and +32767. The first thing we ever do is normalise it to a float in [−1, 1]: x = int16 / 32768.
Two facts decide everything downstream. Nyquist: a sample rate R can only represent frequencies up to R/2— at 8 kHz that's a 4 kHz ceiling, fine since speech lives ~300–3400 Hz. And the DC-offset trap: a cheap converter adds a constant bias. We measured −531 counts on one dongle — a pure 0 Hz component reading as a healthy −35 dBFS "signal," which fooled me for an hour into thinking a dead mic was alive. So we always subtract the mean (x −= x.mean()); the true floor underneath was −76 dBFS, 40 dB lower. Subtract the mean, always.
From time to frequency — the DFT, and why the FFT
Fourier's idea: any signal is a sum of sine waves. The Discrete Fourier Transform asks "how much of each frequency is in these N samples?"
X[k] = Σ x[n] · e^(−2πi·kn/N)
n = 0 … N−1Each X[k] is a complex number: its magnitude is how much of that frequency is present, its phase is where the wave sits in time. Bin k maps to frequency f = k·R/N, so frequency resolution is R/N — bigger N means finer bins but a longer frame (more latency). At N=1024, 8 kHz:
frequency resolution = R/N = 7.812 Hz per bin
frame length = N/R = 128.0 msThe DFT as written is O(N²). The FFT (Cooley–Tukey, 1965) computes the exact same numbers in O(N log N)by recursively splitting the sum into even and odd samples — for N=1024, about 100× fewer operations. That's the difference between real-time and not.
Which FFT — the real one
Audio is real-valued. The DFT of a real signal is Hermitian-symmetric — X[N−k] = conj(X[k]) — so the top half of the spectrum is a mirror image, pure redundancy. We use the real FFT (numpy.fft.rfft): it returns only the unique N/2 + 1 bins (0 Hz to Nyquist), about 2× faster and half the memory. Every script in the rig uses it.
X = np.fft.rfft(frame * window) # 513 complex bins for N=1024
freqs = np.fft.rfftfreq(N, 1/R) # the Hz value of each binThe magic dust — windowing
The FFT secretly assumes your N samples repeat forever. A real chunk doesn't loop cleanly, so the jump at the wrap-around edge smears one tone's energy across many bins — spectral leakage. The fix: multiply the frame by a window that tapers to zero at both ends. We use the Hann window. Live proof, a 440 + 1200 Hz test signal:
no window: 6.07 % energy smeared across the spectrum
Hann window: 0.03 % ← 200× cleanerCost: the main peak gets a hair wider. Worth it every time. (A footnote that teaches a lot: the 440 Hz tone landed in the 437.5 Hz bin — 440 isn't an exact multiple of 7.8125 Hz, so it sits between bins and spreads to its neighbours. Bins are discrete; the world isn't.)
Reading the bins — magnitude, power, dBFS
Magnitude |X[k]| is the amplitude at that frequency; power is its square. We report in dBFS (decibels relative to full scale): 20·log10(amp), where 0 is the max and everything else is negative. The DC-blocked floor sits near −76 dBFS; speech towers ~40 dB above it. And RMS— the loudness of a frame — equals the total spectral energy (Parseval's theorem), so the gate can watch a cheap time-domain number and "see" exactly what the FFT shows.
Special sauce — spectral subtraction
Sample the quiet room for ~15 s and average |X[k]| over every frame: that's a noise fingerprint. Boll's 1979 idea — every live frame is speech plus that same noise, so subtract the noise's magnitude and keep the original phase:
X = rfft(frame · hann) # complex spectrum
|S| = max(|X| − α·|N|, β·|X|) # subtract noise mag, floored
S = |S| · e^(i·phase(X)) # reattach the original phase
y = irfft(S); overlap-add # back to the time domainα is how hard you scrub; β is a spectral floor that stops musical noise(the warble of bins flickering on and off). Here's the part nobody tells you — run live on the hissy QuickCam, target "the quick brown fox":
RAW floor −22 dBFS "the quick brown BOX …" ✗
α=1.0 β=0.15 gentle floor −32 dBFS "the quick brown FOX …" ✓ fixed it
α=1.5 β=0.08 floor −39 dBFS "the quick brown BOX …" ✗
α=2.5 β=0.02 greedy floor −55 dBFS "(nothing)" ✗ destroyedIt's a U-curve. The objective was never minimum noise floor — crush it 33 dB and you also crush the speech into artifacts the recogniser can't read. The sweet spot shaves ~10 dB, just enough to flip box → fox, and stops. The best-sounding result (lowest floor) was the worst transcript. Optimise the right metric.
Profit — the voice gate
Now the box can decide on its own when someone's talking. All time-domain — measure each ~30 ms frame's RMS in dBFS, smooth it with a one-pole filter, and run a tiny state machine with two thresholds:
db = 20·log10( rms(DC-blocked frame) )
ema = a·db + (1−a)·ema # smooth, a ≈ 0.4
OPEN when ema > open_thresh # a turn begins → record → transcribe
CLOSE when ema < close_thresh for ~0.5 s # hangover: pauses don't cut you offTwo thresholds (hysteresis) so a level hovering at the edge doesn't chatter the gate; a hangoverso a breath mid-sentence doesn't end your turn. Both come straight from the fingerprint: open = floor + 12, close = floor + 6dBFS. On Meatball's best ear that's open −28, close −34.
And that's the whole arc: samples became a spectrum, the spectrum became a fingerprint, the fingerprint set the gate — and a junk-pile in a garage can hear you coming.