LiveKit Quickstart

This guide is for the LiveKit plugins maintained by ai-coustics: ai-coustics-livekit-plugin for Python and @ai-coustics/livekit-plugin for Node.js.For the official LiveKit-owned ai-coustics plugins, follow LiveKit’s noise and echo cancellation documentation.

The ai-coustics plugins connect the public ai-coustics SDK to LiveKit Agents. They provide:

Processor for real-time speech enhancement through LiveKit’s audio frame processor interface.
FrameProcessorChain for running frame processors in sequence.
VAD for shared voice activity inference through vad.processor and LiveKit’s streaming VAD interface.
Analyzer for periodic audio-quality analysis.

The ai-coustics-maintained plugins always use your ai-coustics SDK key and let you provision the model files yourself. LiveKit-owned plugins use LiveKit Cloud authentication and metering when used with LiveKit Cloud, or an ai-coustics SDK key in self-hosted environments.

These packages replace LiveKit’s livekit-plugins-ai-coustics and @livekit/plugins-ai-coustics packages.Do not install both the ai-coustics and LiveKit-owned plugins in the same project. The two Python distributions provide the same livekit.plugins.ai_coustics import path, and objects from the two implementations cannot be mixed.

Prerequisites

Before you start, you need:

A LiveKit agent project. You can create one from a LiveKit starter template.
An ai-coustics SDK key from the developer platform.

Setup guide

Create a LiveKit agent project

If you do not already have an agent, install the LiveKit CLI, authenticate it, and create a starter project:

lk cloud auth

lk agent init my-agent --template agent-starter-python
cd my-agent

lk agent init my-agent --template agent-starter-node
cd my-agent

This project creation flow uses LiveKit Cloud. If you already have an agent project or use a self-hosted LiveKit deployment, skip lk cloud auth and configure your own LiveKit connection. The ai-coustics plugin requires a separate ai-coustics SDK key in either case.

Install the ai-coustics plugin

uv add ai-coustics-livekit-plugin

pip install ai-coustics-livekit-plugin

pnpm add @ai-coustics/livekit-plugin

If you are migrating from a LiveKit-owned plugin, follow the migration guide. The old package must be removed first:

uv remove livekit-plugins-ai-coustics
uv add ai-coustics-livekit-plugin

pip uninstall livekit-plugins-ai-coustics
pip install ai-coustics-livekit-plugin

pnpm remove @livekit/plugins-ai-coustics
pnpm add @ai-coustics/livekit-plugin

Configure your SDK key

Add your ai-coustics SDK key to the backend environment used by your agent:

.env.local

AIC_SDK_LICENSE=your-sdk-key

Keep this key on the server. Do not expose it in a browser or mobile client.

Add speech enhancement and VAD

Load model weights once per worker process. Then create a new Processor and VAD for each agent session.

from livekit.agents import Agent, AgentSession, inference, room_io
from livekit.plugins import ai_coustics


# Optional provisioning step: download models before starting the worker.
# Choose compatible model IDs at https://artifacts.ai-coustics.io/.
# ai_coustics.Model.download("quail-vf-2.2-l-16khz", "./models")
# ai_coustics.Model.download("vad-2.1-xxs-16khz", "./models")

# Module scope: load each provisioned model once per worker.
enhancement_model = ai_coustics.Model.from_file(
    "/path/to/enhancement-model.aicmodel"
)
vad_model = ai_coustics.Model.from_file("/path/to/vad-model.aicmodel")


# Inside your agent entrypoint: create stateful objects for this session.
processor = ai_coustics.Processor(model=enhancement_model)
processor.get_context().set_parameter(
    ai_coustics.ProcessorParameter.EnhancementLevel,
    0.8,  # Balanced speech preservation and suppression; range: 0.0 to 1.0.
)

vad = ai_coustics.VAD(
    model=vad_model,
    vad_parameters=ai_coustics.VADParameters(
        # Probability threshold followed by durations in seconds.
        # See https://docs.ai-coustics.com/models/voice-activity-detection/vad.
        sensitivity=0.5,
        speech_hold_duration=0.25,
        minimum_speech_duration=0.05,
    ),
)
frame_processor = ai_coustics.FrameProcessorChain(vad.processor, processor)

session = AgentSession(
    vad=vad,
    # Example LiveKit Inference pipeline; keep your existing providers if configured.
    stt=inference.STT("deepgram/nova-3", language="multi"),
    llm=inference.LLM("openai/gpt-4.1-mini"),
    tts=inference.TTS("cartesia/sonic-3"),
)

await session.start(
    agent=Agent(instructions="You are a helpful voice assistant."),
    room=ctx.room,
    room_options=room_io.RoomOptions(
        audio_input=room_io.AudioInputOptions(
            noise_cancellation=frame_processor,
        ),
    ),
)

import { inference, voice } from '@livekit/agents';
import {
  Model,
  FrameProcessorChain,
  Processor,
  ProcessorParameter,
  VAD,
} from '@ai-coustics/livekit-plugin';

// Optional provisioning step: download models before starting the worker.
// Choose compatible model IDs at https://artifacts.ai-coustics.io/.
// Model.download('quail-vf-2.2-l-16khz', './models');
// Model.download('vad-2.1-xxs-16khz', './models');

// Module scope: load each provisioned model once per worker.
const enhancementModel = Model.fromFile('/path/to/enhancement-model.aicmodel');
const vadModel = Model.fromFile('/path/to/vad-model.aicmodel');

// Inside your agent entrypoint: create stateful objects for this session.
const processor = new Processor({ model: enhancementModel });
processor
  .getContext()
  // 0.8 balances speech preservation and suppression; valid range: 0.0 to 1.0.
  .setParameter(ProcessorParameter.EnhancementLevel, 0.8);

const vad = new VAD({
  model: vadModel,
  vadParameters: {
    // Probability threshold followed by durations in seconds.
    // See https://docs.ai-coustics.com/models/voice-activity-detection/vad.
    sensitivity: 0.5,
    speechHoldDuration: 0.25,
    minimumSpeechDuration: 0.05,
  },
  // Unlike the vadParameters durations, these options use milliseconds.
  prefixPaddingDuration: 500,
  maxBufferedSpeech: 60_000,
});
const frameProcessor = new FrameProcessorChain(vad.processor, processor);

const session = new voice.AgentSession({
  vad,
  // Example LiveKit Inference pipeline; keep your existing providers if configured.
  stt: new inference.STT({ model: 'deepgram/nova-3', language: 'multi' }),
  llm: new inference.LLM({ model: 'openai/gpt-4.1-mini' }),
  tts: new inference.TTS({ model: 'cartesia/sonic-3' }),
});

await session.start({
  agent: new voice.Agent({
    instructions: 'You are a helpful voice assistant.',
  }),
  room: ctx.room,
  inputOptions: {
    noiseCancellation: frameProcessor,
  },
});

vad.processor must be installed in the RoomIO audio path whenever VAD is used. The chain runs it on the original microphone audio before Processor enhances the frame. VAD streams then read the metadata attached to that frame, so the SDK VAD model runs only once per audio block.For VAD without enhancement, configure vad.processor directly as noise_cancellation or noiseCancellation. For enhancement without VAD, configure processor directly.

Run the agent

uv run python src/agent.py console

pnpm dev

Your LiveKit agent now uses the ai-coustics plugin for speech enhancement and voice activity detection.

Model provisioning

Downloading and loading are separate operations. Model.download stores a model in a directory you choose and returns its resolved path. Model.from_file in Python or Model.fromFile in Node.js loads an existing model file into memory. The quickstart leaves the optional download calls commented out so you can run them during provisioning. For production, download models while building your container or preparing the deployment. Make the resulting files available to the worker and load them directly:

# Optional provisioning step:
# ai_coustics.Model.download("quail-vf-2.2-l-16khz", "/models")

enhancement_model = ai_coustics.Model.from_file("/path/to/model.aicmodel")

// Optional provisioning step:
// Model.download('quail-vf-2.2-l-16khz', '/models');

const enhancementModel = Model.fromFile('/path/to/model.aicmodel');

These plugins do not integrate with the LiveKit download-files command. Provision ai-coustics models explicitly with Model.download or supply model files yourself.

Choose models

The plugins do not embed a fixed selection of models. You can load any model compatible with the underlying ai-coustics SDK.

Purpose	Recommended model ID
Foreground-speaker enhancement	`quail-vf-2.2-l-16khz`
Voice activity detection	`vad-2.1-xxs-16khz`
Audio-quality analysis	`tyto-1.1-l-16khz`

Enhancement, VAD, and analysis models are different model types and cannot be substituted for one another. See the models reference for other model IDs and the VAD guide for parameter ranges.

Audio-quality analysis

To monitor the incoming audio with Tyto, create an Analyzer for each session and place its collector before VAD and enhancement:

# Optional provisioning step:
# ai_coustics.Model.download("tyto-1.1-l-16khz", "./models")

# Module scope: load the analysis model once per worker.
analysis_model = ai_coustics.Model.from_file("/path/to/analysis-model.aicmodel")

# Inside your agent entrypoint:
analyzer = ai_coustics.Analyzer(
    model=analysis_model,
    analysis_interval=5.0,  # seconds
)


@analyzer.on("analysis_result")
def on_analysis(event: ai_coustics.AnalysisEvent) -> None:
    print(event.result.risk_score)


frame_processor_with_analysis = ai_coustics.FrameProcessorChain(
    analyzer.collector,
    frame_processor,
)

import { Analyzer } from '@ai-coustics/livekit-plugin';

// Optional provisioning step:
// Model.download('tyto-1.1-l-16khz', './models');

// Module scope: load the analysis model once per worker.
const analysisModel = Model.fromFile('/path/to/analysis-model.aicmodel');

// Inside your agent entrypoint:
const analyzer = new Analyzer({
  model: analysisModel,
  analysisInterval: 5, // seconds
});
analyzer.on('analysisResult', (event) => {
  console.log(event.result.riskScore);
});

const frameProcessorWithAnalysis = new FrameProcessorChain(
  analyzer.collector,
  frameProcessor,
);

Pass frame_processor_with_analysis or frameProcessorWithAnalysis to RoomIO as shown in the setup guide. The plugin does not log analysis results automatically; handle them in the event callback. Constructing an Analyzer without adding its collector to the audio path does not feed it any audio.

Authentication

AIC_SDK_LICENSE is the default authentication method. You can instead pass the key directly when constructing a component:

processor = ai_coustics.Processor(
    model=enhancement_model,
    license_key="your-sdk-key",
)

const processor = new Processor({
  model: enhancementModel,
  licenseKey: 'your-sdk-key',
});

The custom plugins do not support Auth.livekit_cloud() or LiveKit Cloud metering. Generate an SDK key on the ai-coustics developer platform.

Lifecycle

Every call to Model.from_file in Python or Model.fromFile in Node.js loads a separate copy of the model into memory. Ideally, call it only once for each model file in a worker process. You can then reuse the resulting Model object to create any number of Processor, VAD, or Analyzer instances of the corresponding model type. The Model objects can be shared across sessions, but each concurrent room needs its own instance of every stateful component it uses: Processor, VAD, and Analyzer. LiveKit RoomIO closes a directly configured frame processor or FrameProcessorChain when its input stream closes; a chain closes the components it contains.

Next steps

Plugins on GitHub

Browse the Python and Node.js plugins and their examples.

Speech enhancement models

Choose between Quail and Quail Voice Focus models.

ai-coustics VAD

Configure noise-robust voice activity detection.

Tyto: Audio Insight

Understand and act on audio-quality analysis results.

Migration guide

Migrate from LiveKit-owned packages to the plugins maintained by ai-coustics.

Get started

Voice Focus

Audio Insight

Voice Activity Detection

Speech Enhancement

Perceptual SE

Older Models

Prerequisites

Setup guide

Model provisioning

Choose models

Audio-quality analysis

Authentication

Lifecycle

Next steps

Plugins on GitHub

Speech enhancement models

ai-coustics VAD

Tyto: Audio Insight

Migration guide

​Prerequisites

​Setup guide

​Model provisioning

​Choose models

​Audio-quality analysis

​Authentication

​Lifecycle

​Next steps

Plugins on GitHub

Speech enhancement models

ai-coustics VAD

Tyto: Audio Insight

Migration guide

Prerequisites

Setup guide

Model provisioning

Choose models

Audio-quality analysis

Authentication

Lifecycle

Next steps