Reference

Reproducibility

Everything needed to rerun the experiments and know how close a rerun should land.

Environment of the reported runs

OS	Windows 11 (PowerShell and Git Bash)
Python	3.13.4
PyTorch	2.11.0+cu128
transformers	4.57.6 (allowed range `>=4.45,<6`)
Other dependencies	numpy >=1.26, scipy >=1.11, soundfile >=0.12, PyYAML >=6, matplotlib >=3.8, packaging >=23 (see `requirements.txt`)
GPU	NVIDIA GeForce RTX 4060 Laptop, 8 GB total, about 6.5 GB free
Precision	float16 on CUDA (`dtype: auto`)
Decoding	Greedy for both ASR (`num_beams: 1`) and LLM (`do_sample: false`), plus one warm-up pass
LaTeX (for the PDFs)	MiKTeX 25.12, `pdflatex`

Models and revisions

Role	Model	Revision
ASR	`vasista22/whisper-telugu-small`	`main` (unpinned; see Limitations)
LLM	`Qwen/Qwen2.5-1.5B-Instruct`	`989aa7980e4cf806f80c7fef2b1adb7bc71aa306`
TTS (data generation)	`facebook/mms-tts-tel`	`main`

What is and is not deterministic

Step	Deterministic?	Detail
Question text and reference answers	Yes	Fixed lists and arithmetic in `synthesize_corpus.py`.
TTS audio	No	The TTS model's duration prediction is not seeded, so a fresh synthesis renders different audio for the same text.
Noise addition	Yes, given the clean audio	NumPy generator seeded with 42.
Dataset subset selection	Yes	Seeded (42); with `--limit 0` every sample is used.
ASR and LLM output	Yes, for fixed audio and hardware	Greedy decoding. Not guaranteed bit-identical across hardware or library versions.
Latency numbers	No	Wall-clock time varies run to run and across machines.
Bootstrap intervals	Yes	Seeded (0 by default).

Expectation for a rerun Because the TTS audio differs on a fresh run, expect WER and CER in the same neighborhood as the reported values, not identical digits. The qualitative findings (noise hurts, denoiser flat, LLM never correct) should hold; the exact decimals will not.

Commands

The full ordered command list is on the Quickstart. In short: synthesize the corpus at --snr-db 0, build both manifests with --limit 0, run evaluate.py three times (clean; noisy with config.yaml; noisy with config_denoised.yaml), then plots, hallucination.py and bootstrap.py.

What is and is not in the repository

Committed	Git-ignored (regenerate or obtain locally)
All source, tests, both configs, `README.md`, `requirements.txt`, `AGENTS.md`, `DraftTask1.md`, the `reports/` sources and PDFs, this `docs/` site	`outputs/` (evaluation reports, plots), `data/` (manifests), `corpus_clean/`, `corpus_noisy/`, `.cache/` (model weights), `tmp/`, `__pycache__/`, `.venv/`, and the local copies of the two source papers (`moshi.pdf`, `qwen omni.pdf`)

The result plots and diagrams on this site are copies stored in docs/assets/img/ so the site works without the ignored folders.

Commit history

Commit	Date	Message
`5df1efa`	2026-10-24	docs: add unified ACL-style report with bootstrap confidence intervals
`c83d51e`	2026-10-24	feat: add explicit correctness and hallucination rate to Task 2
`b205d73`	2026-10-24	feat: finalize Task 1 report and run real Task 2 evaluation
`2e87b8b`	2026-10-23	feat: verify Task 2 setup and mock metrics
`4fa0cde`	2026-10-23	feat: plot evaluation quality and latency
`965bb5c`	2026-10-23	feat: prepare reproducible Telugu test datasets
`02c63c6`	2026-10-23	docs: record incremental commit workflow
`56fcb35`	2026-10-23	feat: add paired speech pipeline evaluation
`e3dfd04`	2026-10-23	feat: add Telugu speech pipeline

Check the repository for anything newer than this table.

Rebuilding the PDFs

cd reports
pdflatex report.tex   # twice, for cross-references
pdflatex report.tex
pdflatex task1.tex
pdflatex task2.tex

The paper's Part II figures include outputs/plots/*.png; task2.tex does too. task1.tex is self-contained (its diagrams are TikZ). The sources contain no Telugu script because plain pdflatex cannot render it.