Part II: Pipeline

Research extension: denoising

The requirement is to identify a measurable bottleneck, implement one improvement, and compare it against the baseline. The bottleneck is acoustic noise. The improvement is spectral subtraction. It did not help, and this page shows how that was established.

Outcome Noise raises WER by +0.045 (95% CI +0.014 to +0.077). Denoising changes it by +0.001 (CI -0.014 to +0.017): no measurable effect. Aggressive settings make it worse.

The bottleneck

Additive noise degrades recognition: corpus WER 0.2928 (clean) to 0.3377 (noisy, 0 dB SNR), CER 0.0745 to 0.0964. The effect is real by the paired bootstrap, even though the clean and noisy renderings differ (see pairing).

An earlier noisy corpus at 8 dB SNR showed no real effect (WER 0.2913 vs. 0.2928 clean), so there was no bottleneck to fix. The noise was increased to 0 dB specifically to create one worth addressing. That is a deliberate experimental choice, disclosed here and in the paper.

The fix: magnitude spectral subtraction

denoise.py implements a classical technique (Boll, 1979), not a novel one. What is evaluated is whether it narrows the clean/noisy gap for this pipeline, at what cost, on this data. It runs before ASR on every input when asr.denoise: true.

1. STFT of the mono signal (window = min(1024, length), scipy.signal.stft)
2. magnitude |S| and phase of every bin
3. noise floor N[f] = 10th percentile of |S[f, t]| over time, per frequency bin
4. cleaned = max(|S| - oversubtraction * N, floor * |S|)
5. keep the original phase; inverse STFT; trim or pad to the input length
6. if peak > 1, scale down to avoid clipping

Parameter	Default	Effect
`noise_percentile`	10.0	Which low percentile of each bin's magnitude over time counts as noise. Lower = closer to the quietest frames.
`oversubtraction`	1.5	How aggressively the floor is subtracted.
`floor`	0.05	Minimum fraction of the original magnitude kept per bin, so subtraction cannot zero out frames or invert the spectrum. Limits "musical noise".

Inputs shorter than 8 samples pass through unchanged. Invalid input (not 1-D, empty, non-finite, out-of-range parameters) raises ValueError.

A property worth knowing The noise estimate assumes each frequency bin has quiet frames somewhere in the clip, as speech does. A sustained pure tone has none, so its own energy becomes the "noise floor" and gets crushed. The first version of the unit test used a constant tone and failed for exactly this reason; the test now uses a bursty tone with silent gaps, which matches speech.

The ablation

Both runs read the identical corpus_noisy/ audio and differ only in asr.denoise (config.yaml vs. config_denoised.yaml).

Metric	Noisy, baseline	Noisy, denoised	Change	95% CI
Corpus WER	0.3377	0.3391	+0.0014	-0.0144 to +0.0174
Corpus CER	0.0964	0.0992	+0.0028	-0.0041 to +0.0099
Mean ASR time (s)	2.054	2.026	-0.028
Mean response F1 (ASR input)	0.1548	0.1547	-0.0001
Correct answers (of 129)	0	0	0

Result: no improvement. Both error rates moved by a hair in the wrong direction, well inside the noise. Latency overhead was negligible (denoising is cheap next to ASR and LLM inference).

Was the default just badly tuned?

Before accepting a flat result, a quick ASR-only sweep (same checkpoint, no LLM stage, so it runs fast) compared the default against a more aggressive setting on the first 25 questions (all capitals):

Setting	Mean WER, 25 questions
No denoising (baseline)	0.5433
Mild: `oversubtraction=1.5, floor=0.05` (the shipped default)	0.5300
Aggressive: `oversubtraction=3.0, floor=0.02`	0.5567

On this slice mild subtraction helped slightly (about -2.4% relative) and aggressive subtraction made things worse than no denoising. That fits spectral-subtraction theory: over-subtraction leaves isolated spurious spectral peaks ("musical noise") that a neural recognizer can handle worse than the original stationary noise.

The default is the "mild" setting, so the slice result does not contradict the full 129-sample ablation. It indicates the effect is small, inconsistent across subsets, and not distinguishable from noise, while aggressive tuning is counterproductive. With only 25 questions this is suggestive, not established. No further tuning was done on purpose: sweeping until something wins on a small set would be cherry-picking.

Is the filter even working?

A direct waveform check on one sample confirms it does what it should at the signal level:

Quantity	Noisy	Denoised
RMS energy	0.228	0.177
Mean squared error to an independently synthesized clean rendering of the same text	0.094	0.073

(The comparison clip is a separate TTS rendering of the same words, so this is a rough proxy, not a true SNR.) So the negative result is about whether that cleanup survives through the recognizer into fewer errors, not about the code being broken. A Whisper-style model is trained on varied noisy audio and may already be fairly robust to this kind of stationary noise.

Takeaway

A measurable bottleneck was found and quantified with an interval.
An improvement was implemented, unit-tested and evaluated on identical audio.
It did not help, and over-tuning hurt. That is reported as a negative result rather than tuned away.
Given Part II's other finding (the LLM, not ASR, limits answers), the larger lever is a bigger LLM, not a better front end.

Run it

python evaluate.py --dataset data/test_noisy.json --config config.yaml          --output-dir outputs/evaluation/noisy_baseline --warmup 1
python evaluate.py --dataset data/test_noisy.json --config config_denoised.yaml --output-dir outputs/evaluation/noisy_denoised --warmup 1
python bootstrap.py --a outputs/evaluation/noisy_baseline/report.json --b outputs/evaluation/noisy_denoised/report.json --metric wer