How to use and tune VAD

ai-coustics VAD provides dedicated voice activity detection (VAD) for real-time audio. Choose VAD Multi Speaker when speech from any participant matters, or VAD Voice Focus when only the primary speaker should drive activity. Both use the processing and tuning workflow below.

The current model IDs are vad-ms-2.1-xxs-16khz and vad-vf-2.0-s-16khz. Earlier IDs and naming changes are listed in renamed models. Preserve the exact ID when reproducing a deployment.

What is VAD?

VAD identifies speech in an audio stream. Voice agents use it for turn-taking, endpointing and gating downstream speech-to-text (STT). Both models run independently of enhancement. Compare whose speech they detect under model differences.

Use cases

VAD supplies speech activity, not a complete semantic end-of-turn decision. Your pipeline decides how to combine it with transcription, turn detection and interruption policies.

Gate downstream work. Use activity to decide which audio to send to STT, while preserving enough leading and trailing audio to avoid clipping words.
Drive turn-taking. Evaluate the speech-start and speech-stop events used by your voice agent. Check quiet speech, pauses and competing speakers.
Measure the trade-off. Track missed speech and false activity separately. Any cost or transcription benefit depends on the audio and downstream provider.

How it works

The VAD is a standalone object with its own lifecycle, independent of enhancement:

Load a dedicated VAD model, vad-ms-2.1-xxs-16khz or vad-vf-2.0-s-16khz
Create a VAD from that model and initialize it for your sample rate and block size
Feed each mono audio block to the VAD
Query its context to see whether speech was detected

The model produces a speech probability for each processed block, which the SDK compares against the configured sensitivity threshold. Processing does not modify your audio, so the VAD can read the same buffer you pass to an enhancement processor. The following Python 3.11 example uses aic-sdk==3.3.0 and numpy==2.2.6. Save it as vad.py, set AIC_SDK_LICENSE in the same terminal and run python vad.py input.wav with a mono, 16 kHz, signed 16-bit PCM WAV. See the Python setup instructions for the environment and key setup. The example loads VAD Multi Speaker. To use VAD Voice Focus, replace vad-ms-2.1-xxs-16khz with vad-vf-2.0-s-16khz in the model download call. The processing API is the same for both models.

vad.py

import os
import sys
import wave

import aic_sdk as aic
import numpy as np

key = os.environ.get("AIC_SDK_LICENSE", "").strip()
if not key:
    raise RuntimeError("Set AIC_SDK_LICENSE before running vad.py")
if len(sys.argv) != 2:
    raise SystemExit("Usage: python vad.py input.wav")

with wave.open(sys.argv[1], "rb") as source:
    if (source.getnchannels(), source.getsampwidth(), source.getframerate()) != (1, 2, 16000):
        raise ValueError("Use mono, 16 kHz, signed 16-bit PCM WAV audio")
    audio = np.frombuffer(source.readframes(source.getnframes()), dtype="<i2")
    audio = audio.astype(np.float32) / 32768.0
if not len(audio):
    raise ValueError("Use a nonempty recording with speech and pauses")

model = aic.Model.from_file(aic.Model.download("vad-ms-2.1-xxs-16khz", "models"))
config = aic.ProcessorConfig.optimal(model)
vad = aic.Vad(model, key, config)
context = vad.get_context()
try:
    print("model:", model.get_id())
    print("prediction_delay_samples:", context.get_prediction_delay())
    print("block_start_sample,probability,speech_detected")
    for offset in range(0, len(audio), config.block_size):
        block = np.zeros(config.block_size, dtype=np.float32)
        part = audio[offset:offset + config.block_size]
        block[:len(part)] = part  # Pad only the last incomplete block.
        vad.process(block)
        print(f"{offset},{context.raw_vad_probability():.6f},{context.is_speech_detected()}")
finally:
    vad.terminate_session()

The output has one row per block. Inspect probabilities and decisions against speech and pauses in your recording. Align rows using the reported prediction delay: block_start_sample is the uncompensated input position. Final-block padding does not flush all delayed predictions. Successful processing alone does not establish detection accuracy. If model loading fails, check the download path and SDK/model compatibility. If construction or processing fails, inspect the SDK error and key entitlement, correct the problem and rerun. Do not treat all-false decisions as a successful quality test without checking the input and error logs.

Between SDK 0.22.0 and 0.24.0, a dedicated VAD model was the only option: the earlier energy-based VAD derived from an enhancement processor had been removed. Core SDK 0.25.0 brought energy-based detection back under a new API, EnergyVadContext, so an enhancement processor can report speech again without a separate VAD model. A dedicated VAD model remains the more accurate choice. See Migrate to the dedicated VAD for the differences.

The VAD keeps a history of the audio it has seen, so reset it through its context at a discontinuity, such as a seek or a new call. In a long-running integration, call this while the VAD is still active and no processing call is in flight:

context.reset()

This clears the VAD’s state and immediately drops the published decision and probability, so queries cannot return stale values from the previous stream. The VAD stays initialized. See streams and state.

Model differences

Model	How it works	Recommended use
VAD Multi Speaker (`vad-ms-2.1-xxs-16khz`)	Predicts speech probability directly from the input audio, for any audible speaker.	Speech activity from multiple participants; evaluate false detections on noise and music.
VAD Voice Focus (`vad-vf-2.0-s-16khz`)	Targets speech probability for the primary speaker.	Single-target-speaker turn-taking where interfering speakers should not trigger the VAD.

Both models run independently of enhancement. Neither requires Quail Voice Focus, Quail Multi Speaker or Rook Multi Speaker to be present in the signal chain.

Using VAD alongside enhancement

Run the VAD and the enhancement processor side by side on the same original input block, rather than chaining them:

vad.process(audio)                  # reads the block, does not modify it
enhanced = processor.process(audio) # returns the enhanced block

Feeding the VAD the enhanced output means detecting speech in audio that no longer matches what the VAD model expects, and it stacks the processor’s delay on top of the VAD’s own prediction delay. The two delays are independent and are reported separately: the processor’s context reports how far the enhanced audio lags its input, and the VAD’s context reports how far the published prediction lags the same input.

VAD parameters

Read the three parameter defaults from the loaded model through its VAD context. Use these model-specific values as your tuning baseline.

Sensitivity

The probability threshold used to decide whether speech is detected.A block probability above this threshold triggers a speech-detected decision. Higher thresholds require more model confidence and detect speech less aggressively. Confidence does not guarantee a correct classification.Read raw_vad_probability() on the Python VAD context to see the model’s probability before thresholding and post-processing. This is the most practical way to choose a threshold from recorded audio.

Range: 0.0 to 1.0
Default: model-specific

This range applies to a dedicated VAD model. On an energy-based VAD, sensitivity is an energy threshold in the range 1.0 to 15.0 and higher values detect speech more aggressively. That was the behavior before SDK 0.22.0, and it returns in core SDK 0.25.0 for EnergyVadContext. The two scales are not interchangeable: energy values are not valid probabilities and the direction is inverted. See Retune sensitivity.

Speech hold duration

Stabilizes speech-to-silence transitions. When the current probability is at or below the sensitivity threshold, speech remains detected while at least 50% of retained frames were above threshold. This history spans approximately speech_hold_duration * 2 seconds. Above-threshold decisions instead use the consecutive-frame requirement set by minimum speech duration. With a 0.5 s hold duration and a full history of speech, detection continues for approximately 0.5 s after speech ends. Further speech changes the history and can extend the hold.

Effective timing is rounded to the nearest model window, such as a multiple of 15 ms. get_parameter() returns the stored requested value, not the effective timing.

Unit: seconds
Range: 0.0 to 300x the model window length. For the two 15 ms models listed here, this is 4.5 s
Default: model-specific

Minimum speech duration

Sets how long speech must be present before detection, stabilizing silence-to-speech transitions.

Effective timing is rounded to the nearest model window, such as a multiple of 15 ms. get_parameter() returns the stored requested value, not the effective timing.

Unit: seconds
Range: 0.0 to 1.0
Default: model-specific

API reference by language

Use your binding’s reference for initialization, parameter values and prediction queries.

Language	Create a detector	Parameters	Raw probability
Python	`Vad()`	`VadParameter`	`raw_vad_probability()`
Node.js	`new Vad()`	`VadParameter`	`getRawVadProbability()`
C	`aic_vad_create()`	`AicVadParameter`	`aic_vad_context_get_raw_vad_probability()`
C++	`aic::Vad::create()`	`aic::VadParameter`	`get_raw_vad_probability()`
Rust	`Vad::new()`	`VadParameter`	`raw_vad_probability()`
WebAssembly	`new Vad()`	`VadParameter`	`getRawVadProbability()`

Best practices

Choose whose speech should count. Use VAD Multi Speaker for activity from any speaker or VAD Voice Focus for a primary-speaker task.
Tune against labeled audio. Read model defaults through get_parameter and evaluate sensitivity and timing with speech, silence, quiet utterances and interfering speakers.
Feed original audio. Run dedicated VAD beside enhancement on the same input signal. Preserve mono format and process blocks in order.
Align the timeline. Read the VAD’s prediction delay separately from the enhancement processor’s audio delay.
Keep session state separate. Use one VAD per independent stream. Reset at a discontinuity and terminate after the stream is finished.

Framework integration

Framework adapters can override SDK timing defaults or use a different state machine. The LiveKit quickstart requires both the session VAD and its RoomIO frame processor. The Pipecat quickstart uses Pipecat VADParams and explains why a transport enhancement filter feeds enhanced audio to downstream VAD. Do not copy timing values between frameworks without checking their units and behavior.

Third-party VADs

A third-party VAD may respond differently to enhanced audio. Evaluate false starts, missed speech and end-of-turn latency before choosing an enhancement model or strength for that detector. Enhancement delay is part of that input path. Use Quail Voice Focus when the preprocessing task requires primary-speaker isolation, or Rook Multi Speaker when the output is also intended for human listening. No enhancement level guarantees the best performance for every third-party detector. Dedicated ai-coustics VAD does not require an enhancement model and should receive original audio.

Choose a model

Speech enhancement

Audio insight

Voice activity detection

Human listening

Older models

What is VAD?

Use cases

How it works

Model differences

Using VAD alongside enhancement

VAD parameters

API reference by language

Best practices

Framework integration

Third-party VADs

​What is VAD?

​Use cases

​How it works

​Model differences

​Using VAD alongside enhancement

​VAD parameters

​API reference by language

​Best practices

​Framework integration

​Third-party VADs

What is VAD?

Use cases

How it works

Model differences

Using VAD alongside enhancement

VAD parameters

API reference by language

Best practices

Framework integration

Third-party VADs