Modules and CLI

Every Python file in the repository, what it exposes, and how to call it. All scripts are run from the repository root. About 1,990 lines of code in 11 modules, plus 8 test files.

asr.py

Whisper-compatible ASR with local audio decoding and synchronized timing. Inputs supported by libsndfile (WAV, FLAC) work without FFmpeg.

Functions

• resolve_runtime(device, dtype) -> (torch.device, torch.dtype): resolves auto, validates the accelerator and precision. Shared with llm.py.
• synchronize(device): waits for queued CUDA/MPS work before a clock read.

class ASRWrapper

ASRWrapper(model_name="openai/whisper-large-v3", *, device="auto", dtype="auto",
           cache_dir=None, revision="main", language="telugu",
           chunk_length_s=30, batch_size=1, generation=None, denoise=False)

• Loads the processor and model, rejects non-Whisper architectures, builds a Hugging Face automatic-speech-recognition pipeline.
• language: a language name forces it and defaults task to transcribe. None drops both from the generate kwargs (needed for the Telugu fine-tune).
• transcribe(audio_path) -> (text, elapsed_seconds): reads audio, averages channels to mono, applies denoising if enabled, resamples to the model's rate, runs under torch.inference_mode(). Raises FileNotFoundError or ValueError (empty or non-finite audio).

llm.py

class LLMWrapper

LLMWrapper(model_name="Qwen/Qwen2.5-7B-Instruct", *, device="auto", dtype="auto",
           cache_dir=None, revision="main",
           system_prompt="Answer clearly and concisely in Telugu script.",
           max_input_tokens=2048, generation=None)

• Requires a tokenizer with a chat template (system and user roles); otherwise raises.
• generate(prompt) -> (response, elapsed_seconds): builds the chat messages, tokenizes, rejects prompts over max_input_tokens or overflowing the model context, generates, and decodes only the newly generated tokens. No history is retained.
• Sets pad_token_id from the pad or EOS token if missing. Drops temperature and top_p when do_sample is false.

pipeline.py

• PipelineResult (frozen dataclass): audio_path, transcript, response, asr_seconds, llm_seconds, total_seconds.
• SpeechPipeline(asr, llm): accepts any objects with the same methods (used for testing). from_config(config) loads both models. run(audio_path) runs both stages and raises on an empty transcript.
• load_config(path): safe YAML load, validates that sections are mappings, resolves paths.audio / output / cache_dir relative to the YAML's directory.

python pipeline.py [--config config.yaml] [--audio FILE] [--output FILE]

--audio and --output override the config, resolved from the current directory. Prints the JSON and saves it. Exits with status 1 on failure.

denoise.py

spectral_subtract_denoise(audio, sample_rate, *, noise_percentile=10.0,
                          oversubtraction=1.5, floor=0.05) -> np.ndarray

Returns a denoised float32 copy of a 1-D mono waveform. Validates input and parameters. See the extension page for the algorithm.

evaluate.py

Metric functions (standard library only)

Dataset and evaluation

• EvaluationSample(id, audio_path, ground_truth_transcript, ground_truth_response)
• load_dataset(path): nonempty JSON array; required keys audio_path, ground_truth_transcript, ground_truth_response; optional unique string id (default: row number); audio paths resolve from the JSON's directory. Extra fields such as response_source are tolerated.
• analyze_error_propagation(pipeline, sample, quality_scorer=...): the paired comparison for one sample; returns the per-sample record.
• aggregate_results(rows): corpus WER/CER/RTF plus mean, p50, p95 distributions, degraded/improved fractions and exact-match rates.
• evaluate_dataset(pipeline, samples, *, quality_scorer, quality_name, fail_fast): sequential loop that records failures and excludes them from aggregates.
• save_report(report, output_dir): writes report.json, samples.csv, failures.csv, wer_vs_quality.csv.

CLI

python evaluate.py --dataset DATA.json [--config config.yaml]
                   [--output-dir outputs/evaluation] [--warmup N] [--fail-fast]

The CLI refuses to run if llm.generation.do_sample is true, because the paired comparison needs deterministic decoding. It loads both models once, prints the summary JSON, and exits 1 if any sample failed (partial reports are still saved).

prepare_data.py

Builds the evaluator JSON from a local audio folder (with .txt and .response.txt sidecars) or a CSV, TSV or JSON manifest. Works only with local files; it does not download datasets or detect the language.

python prepare_data.py --source SRC [--output data/test.json] [--limit 100] [--seed 42]
                       [--response-mode require|placeholder]
                       [--audio-column audio_path]
                       [--transcript-column ground_truth_transcript]
                       [--response-column ground_truth_response] [--overwrite]

• --limit 0 selects every sample; otherwise a seeded random subset (default 100).
• --response-mode placeholder fills only missing answers with a fixed Telugu test string and flags the dataset as invalid for response-quality scoring.
• Refuses to overwrite existing output without --overwrite, never overwrites source audio or labels, and fails on missing audio, duplicates or empty labels.
• Writes <output>.json and <output>.metadata.json (source, counts, seed, mode, placeholder count).

synthesize_corpus.py

python synthesize_corpus.py [--clean-out corpus_clean] [--noisy-out corpus_noisy]
                            [--snr-db 8.0] [--limit 0] [--skip-clean]

• Constants: TTS_MODEL_NAME = "facebook/mms-tts-tel", TARGET_SNR_DB = 8.0 (the default; the reported runs use --snr-db 0), NOISE_SEED = 42.
• build_qa_pairs() returns the 129 (id, question, answer) triples. add_noise(signal, snr_db, rng) adds Gaussian noise at a fixed SNR.
• --limit N synthesizes only the first N pairs. --skip-clean still synthesizes each clean waveform in memory but writes only the noisy copy.

plot_results.py

python plot_results.py --input report.json|samples.csv|wer_vs_quality.csv
                       [--output-dir outputs/plots] [--quality-label "Word overlap F1"]

Matplotlib only, rendered headless. Produces wer_vs_quality.png (both series against ASR WER) and latency_breakdown.png (stacked mean ASR, LLM and overhead, plus the clean-transcript LLM time as a separate bar). JSON input also reports the failed-sample count; CSV input cannot. Undefined points are omitted with counts in the legend; malformed numbers and incomplete latency records are rejected. It accepts only evaluation outputs, not the prepared dataset JSON. Existing plot files in the output directory are replaced.

hallucination.py

python hallucination.py --reports R1.json [R2.json ...] [--output OUT.json]

• extract_key_fact(audio_path, reference_answer): capital name or sum word, chosen by the audio file prefix.
• accepted_forms(audio_path, key_fact): the word, plus the digit form for sums.
• classify_response(response, key_fact_or_forms): correct / abstained / hallucinated.
• fact_rates(samples, response_field) and analyze_report(path): counts and rates for the ASR-transcript arm and the human-transcript arm. The condition name in the output is the report's parent folder name.
• Constant: UNCERTAINTY_MARKERS (three Telugu phrases).

bootstrap.py

python bootstrap.py --a REF.json --b COMPARE.json [--metric wer|cer]
                    [--resamples 10000] [--seed 0]

paired_bootstrap(edits_a, units_a, edits_b, units_b, *, resamples, seed, level) returns difference (B minus A), ci_low, ci_high, samples. compare_reports(path_a, path_b, metric) pairs rows by sample ID and errors if the two reports do not cover the same IDs.

verify_task2.py

python verify_task2.py [--config config.yaml] [--output-dir outputs/verification]

Checks: required source files exist and parse; installed dependencies match requirements.txt ranges; configuration types and ranges (including Telugu ASR settings and deterministic decoding); device availability; write access to the cache, output and default data, outputs/evaluation, outputs/plots folders. Then it generates exactly three silent dummy WAVs and runs the real SpeechPipeline and evaluate_dataset with injected stand-ins whose outputs and clocks are synthetic, comparing results with fixed expected values for perfect, substitution and deletion cases plus insertion, empty-reference and Unicode-normalization checks.

Outputs: verification.log, checks.json (PASS / FAIL / WARN / UNAVAILABLE with independent setup-readiness and mock-pass flags), summary.md, and a mock/ folder. Exit status 0 when no FAIL or UNAVAILABLE checks remain; warnings (such as an unpinned revision) do not fail the run.