Part II: Pipeline
Pipeline and configuration
A cascaded system: Telugu speech in, a Telugu text answer out, with a Whisper-family recognizer feeding an instruction-tuned LLM. Both models stay loaded and are reused across a whole evaluation.
Architecture
Speech inputWAV / FLAC
→
Denoiseoptional, asr.denoise
→
ASRvasista22/whisper-telugu-small
→
LLMQwen2.5-1.5B-Instruct
→
Telugu text response
| File | Role |
|---|---|
config.yaml | Models, revisions, device, precision, decoding, paths. |
asr.py | ASRWrapper: decodes audio with soundfile, averages to mono, optionally denoises, resamples to the model's rate, transcribes with a Hugging Face ASR pipeline. Timing is synchronized. |
llm.py | LLMWrapper: applies the chat template (system + user), generates, returns only the newly generated text. Stateless: no history is kept between calls. |
pipeline.py | SpeechPipeline composes the two, rejects an empty transcript before calling the LLM, and exposes the JSON CLI. |
denoise.py | Spectral subtraction used by the ASR stage when enabled. See Denoising extension. |
What is timed
asr_seconds: audio decoding, mono mix, optional denoising, resampling and transcription. Excludes model loading.llm_seconds: prompt tokenization, generation and decoding. Excludes model loading.total_seconds: wall clock around both stages, so it can exceed their sum slightly (pipeline overhead).- CUDA work is synchronized before each clock read. The first inference may include kernel initialization; the evaluator therefore runs
--warmup 1by default in the documented commands.
Model choices
| Stage | Used | Original default | Why changed |
|---|---|---|---|
| ASR | vasista22/whisper-telugu-small | openai/whisper-large-v3 | Large-v3 plus a 7B LLM do not fit in about 6.5 GB free VRAM. The general openai/whisper-small was tried and rejected: see below. |
| LLM | Qwen/Qwen2.5-1.5B-Instruct | Qwen/Qwen2.5-7B-Instruct | Memory (about 3 GB fp16 vs. about 15 GB). |
Why not the general whisper-small
Even with Telugu forced,
openai/whisper-small transcribed this corpus's Telugu audio into Devanagari-looking text and looped on short clips. The first full clean run under it scored corpus WER 2.56 and CER 2.69, and was discarded. The Telugu-only fine-tune fixed it (WER 0.293). Full story in the decision log.Why
language: null
The Telugu fine-tune ships a generation_config.json that predates transformers' language/task validation. Passing either to generate() raises ValueError: The generation config is outdated..., even for valid values (reproduced on transformers 5.15.0 and 4.57.6). ASRWrapper therefore treats a falsy language as "pass neither language nor task". Because the checkpoint is Telugu-only, nothing is lost. For a general multilingual Whisper checkpoint, set language: telugu and task defaults to transcribe.Config reference: config.yaml
Relative filesystem paths resolve from the YAML file's directory. CLI path overrides resolve from the current directory.
paths
| Key | Default | Meaning |
|---|---|---|
paths.audio | data/sample_te.wav | Default input for pipeline.py when --audio is not given. No file is bundled. |
paths.output | outputs/result.json | Where the pipeline's JSON result is saved (UTF-8, Telugu preserved). |
paths.cache_dir | .cache/huggingface | Hugging Face model cache. |
runtime
| Key | Default | Meaning |
|---|---|---|
runtime.device | auto | auto, cpu, cuda, cuda:N or mps. auto prefers CUDA, then MPS, then CPU. Errors if a requested accelerator is unavailable. |
runtime.dtype | auto | float32, float16 or bfloat16. auto is float16 on CUDA, float32 elsewhere. float16 on CPU is rejected. |
asr
| Key | Value used | Meaning |
|---|---|---|
asr.model_name | vasista22/whisper-telugu-small | Any Transformers-compatible Whisper (or IndicWhisper) checkpoint or local directory. Non-Whisper architectures are rejected. |
asr.revision | main | Model commit. Left unpinned for this checkpoint; see Limitations. |
asr.language | null | See the warning above. A value like telugu forces the language and sets task. |
asr.chunk_length_s | 30 | Chunking window for the ASR pipeline. Must be positive. |
asr.batch_size | 1 | Must be at least 1. The wrapper handles one request at a time. |
asr.denoise | false | Spectral-subtraction front end. true only in config_denoised.yaml. |
asr.generation.max_new_tokens | 256 | Decoder cap. (Transformers also warns that max_length is set; max_new_tokens wins.) |
asr.generation.num_beams | 1 | Greedy. Must be a positive integer. |
asr.generation.do_sample | false | Must be false for paired evaluation; evaluate.py enforces it for the LLM and verify_task2.py for both. |
llm
| Key | Value used | Meaning |
|---|---|---|
llm.model_name | Qwen/Qwen2.5-1.5B-Instruct | A causal instruction model with a chat template supporting system and user roles. Base models need their own prompt adapter. |
llm.revision | 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 | Pinned commit. |
llm.system_prompt | "You are a helpful assistant. Answer the user's question clearly and concisely in Telugu using Telugu script. If the request is unclear, ask for clarification. Do not invent facts." | The prompt the hallucination test measures against: it explicitly allows abstaining. |
llm.max_input_tokens | 2048 | Longer prompts raise an error rather than being silently truncated. |
llm.generation.max_new_tokens | 256 | Response budget. Prompt plus budget must fit the model's context. |
llm.generation.do_sample | false | Greedy, for a repeatable baseline. |
llm.generation.temperature, top_p, repetition_penalty | 0.7, 0.9, 1.05 | Used only when sampling. When do_sample is false, temperature and top_p are dropped before generation. |
The two config files
config_denoised.yaml is identical to config.yaml except for asr.denoise: true (and slightly shorter comments, with asr.revision: main noted as unpinned). It exists so the baseline-vs-improved ablation differs in exactly one setting.
Output schema
pipeline.py prints and saves one JSON object (PipelineResult):
| Field | Type | Meaning |
|---|---|---|
audio_path | string | Resolved path of the input file. |
transcript | string | ASR output. |
response | string | LLM answer to the transcript. |
asr_seconds | float | See what is timed. |
llm_seconds | float | |
total_seconds | float |
These records are evaluation inputs, not scores. WER and response quality need reference labels, which is what the evaluator adds.
Built-in guards
- An empty ASR transcript raises an error and the LLM is not called.
- Audio must be finite and non-empty. Stereo is averaged to mono. Audio is resampled to the model's actual rate (
scipy.signal.resample_poly), not just relabeled. - Over-long prompts and prompt-plus-budget overflow raise errors rather than truncating Telugu text or dropping chat-control tokens.
- Each pipeline instance is intended for sequential use.