Metrics and evaluation

The evaluation infrastructure is written from scratch in this repository. No evaluation library is used: edit distance, error rates, latency accounting, response scoring, hallucination classification and bootstrap intervals are all written in this repository.

Text normalization

Every reference and hypothesis passes through the same function before scoring: Unicode NFC normalization, then whitespace collapsed to single spaces. Case and punctuation are kept (Telugu has no case; punctuation differences count). NFC matters for Telugu because vowel signs and combining marks can be encoded in more than one equivalent way.

WER and CER

WER = word_edits / reference_words
CER = character_edits / reference_characters

word units      = normalized text split on whitespace
character units = Unicode code points of the text with whitespace removed
edits           = unit-cost insertions + deletions + substitutions (Levenshtein)

• The edit distance is a two-row dynamic program, O(N*M) time and O(min(N, M)) memory, implemented from scratch.
• Both are ratios, not percentages, and can exceed 1 because of insertions (the discarded whisper-small run reached a corpus WER of 2.56).
• CER counts code points, not grapheme clusters, so a Telugu consonant plus vowel sign counts as two units. This is applied identically to reference and hypothesis.
• An empty reference gives an undefined rate (None); dataset loading rejects empty references.

Corpus vs. per-sample

Corpus WER/CER (the headline) = total edits over all samples divided by total reference length, so longer questions weigh more. The report also gives per-sample mean, median (p50) and 95th percentile, using linear interpolation between ranks.

Latency and real-time factor

Audio duration is read from file metadata (frames / sample rate) without decoding samples. The extra "clean transcript" LLM call used for the paired comparison is excluded from pipeline latency and RTF, and reported separately (clean_llm_seconds).

Response quality

The default score is multiset word-overlap F1 in [0, 1], a transparent lexical proxy:

overlap = sum over words of min(count in reference, count in hypothesis)
F1      = 2 * overlap / (|reference words| + |hypothesis words|)

both empty -> 1;  exactly one empty -> 0;  word order is ignored

It is not semantic correctness: a valid paraphrase or a different inflection can score low, and a wrong answer that shares words with the reference can score higher than it deserves. That is why the stricter fact test exists. A custom scorer (higher is better, finite, in [0, 1]) can be passed programmatically and is then applied to both arms.

Exact-match rates (normalized equality with the reference answer) are also reported.

The paired error-propagation comparison

Each question is answered twice by the same LLM, with deterministic decoding and no chat history, and both answers are scored against the same human reference answer:

• quality_drop = clean_response_quality - asr_response_quality. Positive means ASR errors hurt; negative means the ASR arm scored better.
• degraded_fraction and improved_fraction: share of samples where the drop is above or below zero (tolerance 1e-12).
• response_agreement_f1: F1 between the two arms' answers.
• The clean-arm answer is a baseline, not the scoring truth. The truth is always the human-written reference.

This isolates ASR's downstream effect. When the two arms score about the same, the LLM, not the recognizer, is what limits quality.

Correctness and hallucination rate

The requirements ask for "response correctness and hallucination rate using clearly defined evaluation criteria". Because each reference answer states exactly one fact, a response can be classified against it with a rule that needs no human judge. hallucination.py does this post hoc from saved report.json files, with no model inference.

correct_rate       = correct / samples
hallucination_rate = hallucinated / samples

It is computed for both arms (answer from ASR transcript; answer from human transcript), so ASR-induced and LLM-induced failures can be told apart.

Confidence intervals

bootstrap.py gives a 95% paired bootstrap interval for the difference in corpus WER or CER between two systems:

1. Pair the two reports' rows by sample ID (same question in both).
2. Resample the question indices with replacement, 10,000 times, with a fixed seed.
3. For each resample, recompute each system's corpus rate (sum of edits / sum of reference units) and take the difference.
4. Report the point difference and the 2.5th and 97.5th percentiles.

An interval that contains 0 means the difference is not distinguishable from question-sampling noise. It does not capture speaker or recording variation, since the question set has none.

Failure handling

A sample whose pipeline raises (for example an empty ASR transcript) is recorded in failures.csv and excluded from every aggregate. samples_failed and failure_rate sit next to the metrics, because a successful-only average can look better than the truth. The CLI still saves partial reports and exits with status 1; --fail-fast stops at the first failure instead. In the reported runs, 0 of 129 samples failed in every condition.

Evaluator outputs