Speech AI Report

An architectural analysis of two speech LLMs, a proposal for conversational floor control, and a reproducible Telugu speech-to-text-response pipeline with its own evaluation code, results and limitations.

Author: Banuru Mohammed Fawaz. These pages document everything in the repository: the analysis, the code, the data, the metrics, the results and the decisions behind them.

Headline results

All Part II numbers are measured on 129 synthesized Telugu utterances per condition, with 0 failed samples in every run. Part I contains no experiments.

0.293
Corpus WER, clean
CER 0.075
0.338
Corpus WER, noisy (0 dB SNR)
+0.045, 95% CI +0.014 to +0.077
+0.001
WER change from denoising
95% CI -0.014 to +0.017, no effect
0 / 129
Correct answers from speech
3 / 129 from perfect transcripts
0
Abstentions in 774 answers
The LLM never says "I don't know"
4.15
Pipeline real-time factor, clean
ASR alone: 0.50. The LLM is about 88% of time.
The one-line takeaway The ASR stage is fine and real-time. The small 1.5B LLM is the dominant failure, and it fails confidently rather than abstaining. Noise measurably hurts ASR, and a classical denoiser does not fix it.

The two parts

Part I: Architecture analysis

Moshi vs. Qwen2.5-Omni along seven axes, three ranked limitations, and why "end of speech" is really four decisions.

Part I: Yielding proposal

A falsifiable experiment: a small controller on frozen Moshi that uses generation state to continue, hold or yield. Specified, not run.

Part II: Pipeline

Telugu speech to ASR to LLM to text response, the full config reference, and how timing is measured.

Part II: Results

Clean vs. noisy tables, latency distributions, hallucination rate, confidence intervals and all six plots.
Keep the two parts separate Part II measures a cascade, which has no floor-control decision to evaluate. It does not test Part I's hypothesis, and these docs never present it as if it did.

Deliverables

FileWhat it is
report.pdfThe single ACL-style paper combining both parts (11 pages).
task1.pdfLonger standalone Task 1 write-up (13 pages) with the three diagrams.
task2.pdfLonger standalone Task 2 write-up (7 pages) with all plots.
GitHub repositoryAll code, configs, tests and reports.

Where to find things

I want to...Go to
Run itQuickstart
See how each requirement is metRequirements
Understand a config keyConfig reference
Know how a metric is definedMetrics and evaluation
Look up a function or CLI flagModules and CLI
Learn why a decision was madeDecision log
See what could be wrong or missingLimitations
Find a fileRepository map

Press / or Ctrl+K anywhere to search every page.