> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ai-coustics.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How to evaluate audio processing

> Compare paired recordings, downstream accuracy, agent behavior and runtime cost with a reproducible procedure.

Choose a metric for your task: transcription accuracy for Quail models, speech boundaries for ai-coustics VAD or listening quality for Rook Multi Speaker. Evaluate agent outcomes separately from audio quality.

## Define the comparison

Record the task, model ID, SDK/binding versions and settings. Use a baseline such as enhancement disabled and change one factor at a time. Keep channel selection, resampling, downstream model, language, prompts and decoding settings fixed.

Use authorized recordings covering clean and quiet speech, noise, reverb, competing voices and transport artifacts. Separate calibration from held-out evaluation data. Identify fixtures by ID or hash in reports.

Label the target speaker for Quail Voice Focus or all intended speakers for multi-speaker enhancement. This determines whether removing background speech counts as an error.

## Use real call recordings

The [ai-coustics English test-call dataset](https://huggingface.co/datasets/ai-coustics/aic_test_calls_en) contains 83 clips from four speakers, about 35 minutes in total, with real acoustic and transmission conditions. Audio is mono, 16 kHz, signed 16-bit PCM. Use the staff-reviewed transcripts for speech-to-text (STT) comparisons; there is no clean audio reference.

The dataset is published under **CC BY-NC 4.0**. Review the [dataset card and license information](https://huggingface.co/datasets/ai-coustics/aic_test_calls_en/blob/3825a2dc1a1146c17627fb92a022a8d43c0dd71c/README.md) before use. These four speakers do not represent every deployment; add held-out recordings for your languages, speakers and conditions.

### Prepare one recording and its transcript

Use Python 3.11 or newer. This example downloads the approximately 61 MB `eval` split at a fixed revision, verifies its checksum and extracts one complete 31.838 s recording without resampling or trimming. Create an isolated environment:

```bash theme={null}
python -m venv .venv
source .venv/bin/activate
python -m pip install 'pyarrow==24.0.0'
```

Save as `prepare_call.py`:

```python theme={null}
import hashlib
import io
import json
from pathlib import Path
from urllib.request import urlretrieve
import wave

import pyarrow.parquet as pq

DATASET = "ai-coustics/aic_test_calls_en"
REVISION = "3825a2dc1a1146c17627fb92a022a8d43c0dd71c"
PARQUET_SHA256 = "fb4fdc4c705b2d4ba94daf29bed5be6a18400a1b0deec9bdc30ec0c752b079a3"
output = Path("test-call")
output.mkdir(exist_ok=True)
parquet = output / "eval.parquet"
urlretrieve(f"https://huggingface.co/datasets/{DATASET}/resolve/{REVISION}/eval.parquet", parquet)
if hashlib.sha256(parquet.read_bytes()).hexdigest() != PARQUET_SHA256:
    raise RuntimeError("Dataset download checksum mismatch")
row = next(pq.ParquetFile(parquet).iter_batches(batch_size=1)).to_pylist()[0]
if row["id"] != "323e13ea":
    raise RuntimeError("Unexpected dataset record")
audio = row["mix"]["bytes"]
with wave.open(io.BytesIO(audio), "rb") as source:
    if (source.getnchannels(), source.getsampwidth(), source.getframerate()) != (1, 2, 16000):
        raise RuntimeError("Expected mono, 16 kHz, signed 16-bit PCM WAV")
    frames = source.getnframes()
(output / "input.wav").write_bytes(audio)
(output / "reference.txt").write_text(row["transcript"] + "\n", encoding="utf-8")
manifest = {
    "dataset": DATASET, "revision": REVISION, "split": "eval", "id": row["id"],
    "speaker_id": row["speaker_id"], "index": row["index"], "license": "CC BY-NC 4.0",
    "sample_rate": 16000, "frames": frames,
    "audio_sha256": hashlib.sha256(audio).hexdigest(),
    "transcript_sha256": hashlib.sha256(row["transcript"].encode("utf-8")).hexdigest(),
}
(output / "provenance.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
print(f"Saved {row['id']}: {frames / 16000:.3f} s to {output / 'input.wav'}")
```

Run `python prepare_call.py`. Expect `Saved 323e13ea: 31.838 s to test-call/input.wav`. The folder also contains the matching `reference.txt`, `provenance.json` and downloaded `eval.parquet`. Keep that revision and record ID fixed for both sides of the comparison.

Complete the [Python quickstart setup](/reference/sdk/language-bindings/python), then use `test-call/input.wav` as `input.wav` in the quickstart's working directory, skipping its fixture-download step. This recording contains 509,402 samples; the quickstart should report that processed sample count and write an aligned `enhanced.wav`. Use the original and enhanced files in the paired procedure below.

Use `reference.txt` for this clip's `reference` field in `transcripts.json`, and actual STT output for `baseline` and `enhanced`. Keep the full recording aligned with its transcript and check which speakers the reference includes.

## Prepare paired audio

1. **Keep the original.** Save the decoded, channel-selected input with its actual sample rate and sample count.
2. **Process the same recording.** Use the [SDK quickstart](/models/get-started/sdk-quickstart) with checked errors. Preserve the exact model artifact and parameters.
3. **Align the output.** Flush buffered output, remove the queried SDK delay and trim to the original duration using the [latency guide](/reference/concepts/latency#align-file-comparisons).
4. **Check the pair.** Confirm matching rates, expected duration, finite samples and no unexplained clipping or silence. Listen to selected segments and inspect failures before scoring.

For a first setup check, use the [downloadable clean and noisy speech fixture](/guides/test-audio). Its 3.505 s recording and synthetic noise are illustrative, not a representative benchmark. The Tyto batch example requires a longer recording under its stated input policy.

Report processing failures, bypasses and exclusions separately; do not discard difficult samples after scoring.

## Measure speech-to-text quality

Transcribe both versions with the same STT provider, model and settings. Compare against a human-reviewed transcript, using a fixed normalization policy for punctuation, case, numbers and filler words.

Word error rate (WER) is:

$$
\mathrm{WER} = \frac{S + D + I}{N}
$$

Here, `S` is substitutions, `D` deletions, `I` insertions and `N` reference words. Lower is better, and WER can exceed 100%. Empty-reference clips have no per-clip WER; track their false insertions separately.

Save a JSON array as `transcripts.json`, with one object per clip and string fields `id`, `reference`, `baseline` and `enhanced`. Those fields must contain actual transcripts from the paired run. Save the following as `score_wer.py`:

```python theme={null}
import json
import sys


def errors(reference, hypothesis):
    # Deliberate policy: lowercase, whitespace tokenization, punctuation retained.
    ref, hyp = reference.lower().split(), hypothesis.lower().split()
    # Each cell holds substitutions, deletions and insertions.
    previous = [(0, 0, j) for j in range(len(hyp) + 1)]
    for i, word in enumerate(ref, start=1):
        current = [(0, i, 0)]
        for j, candidate in enumerate(hyp, start=1):
            s, d, ins = previous[j - 1]
            substitution = (s + (word != candidate), d, ins)
            s, d, ins = previous[j]
            deletion = (s, d + 1, ins)
            s, d, ins = current[j - 1]
            insertion = (s, d, ins + 1)
            current.append(min([substitution, deletion, insertion], key=sum))
        previous = current
    return previous[-1], len(ref)


rows = json.load(open(sys.argv[1], encoding="utf-8"))
for variant in ["baseline", "enhanced"]:
    totals, words, silent_insertions = [0, 0, 0], 0, 0
    for row in rows:
        counts, n = errors(row["reference"], row[variant])
        if n == 0:
            silent_insertions += counts[2]
            continue
        totals = [a + b for a, b in zip(totals, counts)]
        words += n
    print(json.dumps({
        "variant": variant, "reference_words": words,
        "substitutions": totals[0], "deletions": totals[1], "insertions": totals[2],
        "wer": sum(totals) / words if words else None,
        "empty_reference_insertions": silent_insertions,
    }))
```

Run:

```bash theme={null}
python score_wer.py transcripts.json
```

The script sums errors and reference words to calculate corpus WER. It lowercases and splits on whitespace, retaining punctuation. Apply any replacement normalization consistently and retain paired per-clip results for subgroup analysis and confidence intervals.

Report absolute WER, insertion/deletion changes and the difference from baseline. Distinguish percentage-point change from relative change. Break results down by language and acoustic condition; an overall improvement can hide a regression for quiet speech or a particular speaker group.

## Tune and compare again

Use the tuning guide for your task: [Quail Voice Focus](/models/voice-focus/voice-focus-for-voice-ai-systems) for primary-speaker isolation or [Quail Multi Speaker](/models/speech-enhancement/speech-enhancement-for-voice-ai-systems) to preserve multiple speakers. Change one setting at a time, then repeat the paired comparison with the same recordings, transcripts and STT configuration. Check speech retention and agent behavior alongside WER.

## Measure VAD and agent behavior

Use labeled speech intervals and the same original audio to evaluate voice activity detection (VAD). Measure missed speech, false triggers, speech-start delay and speech-end delay. Match timestamps to the VAD's prediction delay and the framework's start/stop thresholds.

The test-call dataset's [`mix__vad__human` labels](https://huggingface.co/datasets/ai-coustics/aic_test_calls_en/blob/3825a2dc1a1146c17627fb92a022a8d43c0dd71c/README.md#human-vad-labels) mark the foreground speaker at one label per 10 ms. Background talkers are left unmarked. Use these labels for primary-speaker activity; an all-speech detector can correctly detect background speech that these labels exclude. Keep the labels aligned with the untrimmed recording.

For a voice agent, replay the same scenarios through both complete pipelines. Record false interruptions, missed interruptions, premature endpointing, time to the first response and task completion. Include competing speech and user corrections. Hold the downstream model/settings fixed and repeat runs when responses are nondeterministic.

Check the signal routing required by your framework; evaluate turn-taking alongside transcription.

## Evaluate human listening

For Rook Multi Speaker and human-facing audio, run matched-level listening comparisons with randomized presentation order. Evaluate speech clarity, artifacts, speaker identity and fatigue on representative recordings. Keep listening ratings separate from WER and avoid changing gain between variants in ways that bias the comparison.

## Use Tyto as an additional signal

[Tyto](/models/audio-insight/tyto) helps rank audio segments and inspect degradation dimensions. Calibrate risk thresholds against observed failures in your application. Record the analysis window, interval, startup handling and aggregation method.

Overlapping windows are correlated samples. Do not treat each one as an independent call or infer a causal network fault from a high packet-loss dimension. Validate non-English traffic and exclude the neutral `speaker_loudness` dimension when choosing the largest degradation score.

## Measure runtime cost

Run the [performance comparison](/reference/concepts/performance#run-a-bounded-comparison) on target hardware, first with one stream, then two and finally the intended concurrency under deployment resource limits.

Record processing-call duration, frame-duration ratio, algorithmic/buffering delay, queue depth, dropped frames, CPU and peak memory. Check overload and recovery. A fast offline throughput run does not establish real-time scheduling behavior.

## Save the result

Use one report per fixed configuration:

| Field              | Record                                                                                               |
| :----------------- | :--------------------------------------------------------------------------------------------------- |
| **Provenance**     | Date, fixture hashes, consent/source, package/core/model versions and model artifact hash            |
| **Configuration**  | Host rate, block size, channel selection, parameters, STT/agent settings and normalization           |
| **Environment**    | CPU model, OS/architecture, runtime, resource limits and concurrent streams                          |
| **Quality**        | Baseline/enhanced WER and error counts, stratified results, labeled VAD metrics and listening method |
| **Agent behavior** | Interruptions, endpointing, response latency and task outcomes over repeated scenarios               |
| **Runtime**        | Timing percentiles, frame ratios, queue/drop behavior, CPU, peak memory and checked SDK errors       |
| **Uncertainty**    | Sample count, variation/confidence intervals, exclusions and untested conditions                     |
| **Decision**       | Accepted trade-offs, regressions and the configuration selected for a deployment trial               |

Record untested conditions and reuse the fixtures and acceptance criteria for upgrades and [deployment readiness](/production/deployment).