Requirements
What the report is required to cover, and a checklist mapping each requirement to the page that meets it.
Scope
The report has two parts. Part I is an analysis of speech language-model architectures with a proposed improvement. Part II is an implemented, evaluated Indic speech pipeline. The focus throughout is research reasoning: identifying gaps, forming hypotheses, designing experiments, and reporting results honestly, rather than only implementing existing models.
| Area | Speech foundation models, Speech LLMs, ASR, multilingual (Indic) speech understanding |
|---|---|
| Part I | Comparative architectural analysis of two recent Speech LLMs, with diagrams and a proposed improvement |
| Part II | A reproducible Indic speech pipeline with its own evaluation infrastructure, evaluated on at least 100 examples |
| Deliverables | A PDF report and a repository with a README of reproduction steps |
Part I: architecture analysis
Select and critically analyze two recent Speech LLM architectures, at least one with speech-to-speech generation. Deliver a comparative analysis with diagrams and a proposed improvement.
| Requirement | Where it is met | Status |
|---|---|---|
| Two recent Speech LLMs | Moshi and Qwen2.5-Omni. Why these two | Met |
| At least one speech-to-speech | Both generate speech: Moshi end to end, Qwen2.5-Omni via its Talker | Met |
| 1. Speech encoder and representation learning | Axis 1 | Met |
| 2. Semantic vs. acoustic representations | Axis 2: Moshi's split quantizer, Qwen's Thinker/Talker conditioning | Met |
| 3. Speech-to-LLM alignment (adapters, projectors, cross-attention) | Axis 3 | Met |
| 4. Streaming vs. non-streaming | Axis 4 | Met |
| 5. Autoregressive vs. non-autoregressive generation | Axis 5 | Met |
| 6. End-of-speech detection and stopping criteria | Axis 6 and four decisions | Met |
| 7. Training objectives and data requirements | Axis 7 | Met |
| Research question: the main architectural limitations preventing low-latency, natural, real-time conversation | Three ranked limitations | Answered |
| Diagrams | Architecture and timing, conversational events, experimental design | Met |
| A proposed improvement | Generation-state-aware yielding controller | Met Specified, not run |
Part II: implementation and evaluation
Build an Indic pipeline (speech input, ASR, language model, text response) from pretrained models, with the evaluation infrastructure written in the repository, evaluated on at least 100 examples.
| Requirement | Where it is met | Status |
|---|---|---|
| Indic speech-to-text-response pipeline | Pipeline: Telugu Whisper fine-tune to Qwen2.5-1.5B-Instruct | Met |
| Own evaluation infrastructure | evaluate.py (Levenshtein WER and CER from scratch), prepare_data.py, plot_results.py, hallucination.py, bootstrap.py. No evaluation library is used. Modules | Met |
| At least 100 examples | 129 per condition. Dataset | Met |
| ASR WER and CER | Corpus and per-sample, with mean, median and p95. Results | Met |
| Response correctness and hallucination rate, with clearly defined criteria | Word-overlap F1, exact match, and a three-way correct / abstained / hallucinated classification. Definition, results | Met |
| Processing latency, including real-time factor | ASR, LLM and total seconds with mean, p50 and p95, plus pipeline and ASR-only RTF. Definition | Met |
| At least two linguistic or acoustic conditions | Clean vs. 0 dB SNR noisy, with paired bootstrap intervals | Met |
| Research extension: a measurable bottleneck and one improvement, compared with the baseline | Noise raises WER (the bottleneck); spectral-subtraction denoising is the improvement. Extension | Met Improvement did not help |
| Working code, evaluation results, ablation analysis | The repository; results; the baseline vs. denoised ablation and a parameter sweep | Met |
Deliverables
| Item | Location |
|---|---|
| Single PDF report covering both parts | report.pdf |
| Longer standalone write-ups | task1.pdf, task2.pdf |
| Code, configs, tests and README with reproduction steps | GitHub repository; see also Quickstart |