Start

Quickstart

From a clean checkout to a transcript, a test run, and the full reproduction of the reported results.

Requirements

Python 3.10 or newer (the reported run used 3.13.4).
Internet on first use, to download model weights from Hugging Face (roughly 1 to 3 GB in total).
A GPU is optional but strongly recommended. The reported run used an RTX 4060 Laptop (8 GB, about 6.5 GB free). CPU works but is slow, mostly because of the LLM stage.
Telugu text in the console: on Windows, printing Telugu to a cp1252 console raises UnicodeEncodeError. Write output to a file or set PYTHONIOENCODING=utf-8.

Install

python -m pip install -r requirements.txt

GPU builds of PyTorch requirements.txt lists plain torch. For an NVIDIA GPU, install the CUDA build of PyTorch for your system first, then run the command above. The reported run used torch 2.11.0+cu128 and transformers 4.57.6 (the pin allows >=4.45,<6).

Run the pipeline on one recording

No sample audio is bundled. Use any real Telugu WAV or FLAC file (other formats depend on your libsndfile).

python pipeline.py --audio "path/to/question.wav"

It prints JSON and saves it to paths.output from the config (default outputs/result.json). Fields: audio_path, transcript, response, asr_seconds, llm_seconds, total_seconds. See pipeline output.

Or from Python:

from pipeline import SpeechPipeline, load_config

pipeline = SpeechPipeline.from_config(load_config("config.yaml"))
result = pipeline.run("path/to/question.wav")
print(result.transcript, result.response, result.total_seconds)

Run the tests

All 30 tests are offline: no model downloads, no GPU.

python -m unittest discover -s tests -v

Use -s tests (not python -m unittest tests.test_x), because the test folder is not a package. Details on the tests page.

Check your setup

python verify_task2.py --config config.yaml --output-dir outputs/verification

Checks required files, dependency versions, configuration, device and output folders, then runs the real evaluator on three dummy samples with injected stand-ins and fixed expected values. It writes verification.log, checks.json and summary.md. The timings in its mock table are synthetic by design. It does not prove model quality or memory fit.

Reproduce every reported number

Run from the repository root, in order. The synthesized corpora and outputs/ are not committed, so these commands regenerate them.

# 1. synthesize the Telugu corpus (clean + 0 dB noisy)
python synthesize_corpus.py --clean-out corpus_clean --noisy-out corpus_noisy --snr-db 0

# 2. build evaluator manifests
python prepare_data.py --source corpus_clean --output data/test_clean.json --limit 0 --seed 42
python prepare_data.py --source corpus_noisy --output data/test_noisy.json --limit 0 --seed 42

# 3. evaluate three conditions (each loads both models once)
python evaluate.py --dataset data/test_clean.json --config config.yaml --output-dir outputs/evaluation/clean --warmup 1
python evaluate.py --dataset data/test_noisy.json --config config.yaml --output-dir outputs/evaluation/noisy_baseline --warmup 1
python evaluate.py --dataset data/test_noisy.json --config config_denoised.yaml --output-dir outputs/evaluation/noisy_denoised --warmup 1

# 4. plots, hallucination rate, confidence intervals
python plot_results.py --input outputs/evaluation/clean/report.json --output-dir outputs/plots/clean
python plot_results.py --input outputs/evaluation/noisy_baseline/report.json --output-dir outputs/plots/noisy_baseline
python plot_results.py --input outputs/evaluation/noisy_denoised/report.json --output-dir outputs/plots/noisy_denoised
python hallucination.py --reports outputs/evaluation/clean/report.json outputs/evaluation/noisy_baseline/report.json outputs/evaluation/noisy_denoised/report.json --output outputs/evaluation/hallucination.json
python bootstrap.py --a outputs/evaluation/noisy_baseline/report.json --b outputs/evaluation/noisy_denoised/report.json --metric wer

Expect small differences Evaluation is greedy and deterministic for fixed audio, but the TTS step is not seeded, so a fresh synthesis renders slightly different audio. Your absolute numbers will be close to, not identical to, those on the results page. Each of the three evaluations takes roughly 20 to 25 minutes on the reference GPU.

Rebuild the paper

cd reports
pdflatex report.tex
pdflatex report.tex

Run it twice for cross-references. The figures read the plots under outputs/plots/, so regenerate those first. See Reproducibility for the full environment.