Voice Activity Detection (VAD) for AI Voice Agents

VAD is the first stage of audio processing - it determines when the user is speaking and when they've finished. Getting VAD right is crucial for natural conversations.

Why VAD Matters#

Without proper VAD:
─────────────────────────────────────────────────────────────
User: "I want to check my order status" [background noise...]
Bot:  [Waits for noise to stop... 3 seconds pass]
      "I'd be happy to help! What's your order number?"
User: [Frustrated by delay]

With optimized VAD:
─────────────────────────────────────────────────────────────
User: "I want to check my order status"
Bot:  [Detects end of speech in 200ms]
      "I'd be happy to help! What's your order number?"
User: [Smooth conversation]

Silero VAD#

We use Silero VAD, a state-of-the-art neural network-based VAD that runs locally:

Advantages#

• High accuracy: 99%+ on clean speech
• Low latency: ~10ms per frame
• No cloud dependency: Runs entirely on-device
• Small footprint: ~2MB model file

How It Works#

Audio Input (8kHz, 16-bit PCM)
        │
        ▼
┌───────────────────┐
│   Silero ONNX     │
│   Neural Network  │
│                   │
│ Input: 512 samples│
│ Output: 0.0-1.0   │
│ (speech prob)     │
└───────────────────┘
        │
        ▼
Speech Probability
        │
    ┌───┴───┐
    │       │
   <0.8    ≥0.8
    │       │
 Silence  Speech

Configuration#

Basic Configuration#

cfg := silero.DetectorConfig{
    ModelPath:            "./silero_vad.onnx",
    SampleRate:           8000,          // Match telephony audio
    Threshold:            0.8,           // Speech probability threshold
    MinSilenceDurationMs: 200,           // Silence before end-of-speech
    LogLevel:             silero.LogLevelError,
}

detector, err := silero.NewDetector(cfg)

Configuration Parameters#

Threshold Tuning#

Threshold = 0.5 (too sensitive)
─────────────────────────────────────────────────────────────
Detects: speech, breathing, background noise, air conditioning
Result: False positives, bot interrupts itself

Threshold = 0.9 (too strict)
─────────────────────────────────────────────────────────────
Detects: Only loud, clear speech
Result: Misses quiet speakers, soft endings

Threshold = 0.8 (balanced)
─────────────────────────────────────────────────────────────
Detects: Normal speech, ignores most noise
Result: Good for most telephony use cases

Per-Agent Configuration#

Configure VAD per agent based on use case:

{
  "agent": {
    "name": "Call Center Agent",
    "vadConfig": {
      "threshold": 0.75,
      "minSilenceDurationMs": 250,
      "volumeThreshold": 0.02
    }
  }
}

Speech Detection Events#

The VAD emits events that drive the pipeline:

type VADEvent struct {
    Type      VADEventType
    Timestamp time.Time
    Duration  time.Duration
}

type VADEventType int

const (
    SpeechStart  VADEventType = iota  // User started speaking
    SpeechEnd                          // User stopped speaking
    Interruption                       // User spoke during bot output
)

Event Flow Example#

Timeline (ms):  0    100   200   300   400   500   600   700   800
User Audio:     [─────████████████████████████─────────────────────]
                      │                      │
Events:           SpeechStart            SpeechEnd
                    (100ms)               (500ms)
                                         + 200ms silence
                                         = 700ms event fired

Volume Threshold#

Filter out low-level background noise before VAD processing:

// Calculate RMS volume
func calculateVolume(samples []int16) float32 {
    var sum float64
    for _, sample := range samples {
        sum += float64(sample) * float64(sample)
    }
    rms := math.Sqrt(sum / float64(len(samples)))
    return float32(rms / 32768.0) // Normalize to 0.0-1.0
}

// In VAD processing
func (d *Detector) processAudio(audio []byte) {
    volume := calculateVolume(audio)

    if volume < d.VolumeThreshold {
        // Skip VAD processing for very quiet audio
        return
    }

    // Proceed with neural network inference
    probability := d.model.Infer(audio)
    // ...
}

Volume Threshold Guidelines#

Interruption Detection#

VAD also enables interruption handling when the user speaks during bot output:

type InterruptionHandler struct {
    callback func()
    enabled  bool
}

func (h *InterruptionHandler) OnVADEvent(event VADEvent) {
    if event.Type == SpeechStart && h.enabled {
        // User is speaking while bot is outputting
        h.callback() // Trigger interruption
    }
}

// In call provider
provider.SetInterruptionHandler(&InterruptionHandler{
    callback: func() {
        provider.ClearPlayback()     // Stop current audio
        llm.CancelGeneration()       // Cancel LLM
        // Process new user input
    },
    enabled: agent.AllowInterruptions,
})

STT Mute Filter#

Control when audio is sent to STT based on VAD and bot state:

type STTMuteFilter struct {
    muteStrategies []MuteStrategy
}

type MuteStrategy interface {
    ShouldMute() bool
}

// Mute during bot speech (echo cancellation)
type MuteDuringBotSpeech struct {
    user *session.User
}

func (m *MuteDuringBotSpeech) ShouldMute() bool {
    return m.user.IsBotSpeaking()
}

// Mute during function execution
type MuteDuringFunctionCall struct {
    user *session.User
}

func (m *MuteDuringFunctionCall) ShouldMute() bool {
    return m.user.IsExecutingFunction()
}

Best Practices#

1. Match Sample Rate#

Ensure audio sample rate matches VAD configuration:

// ❌ Wrong - sample rate mismatch
cfg := silero.DetectorConfig{
    SampleRate: 16000, // VAD expects 16kHz
}
// But telephony sends 8kHz audio - will cause detection issues

// ✅ Correct
cfg := silero.DetectorConfig{
    SampleRate: 8000, // Matches telephony audio
}

2. Handle Edge Cases#

// Very short utterances (< 200ms)
if speechDuration < 200*time.Millisecond {
    // Might be noise, cough, or "uh-huh"
    // Consider waiting for more speech
}

// Very long silence after speech
if silenceDuration > 3*time.Second {
    // User might be thinking or distracted
    // Consider sending a prompt
}

3. Adaptive Thresholds#

Adjust thresholds based on call quality:

func adaptThreshold(initialNoiseLevel float32) float32 {
    if initialNoiseLevel > 0.1 {
        return 0.85 // Noisy environment - be more strict
    }
    return 0.8 // Normal threshold
}

Debugging VAD Issues#

Common Problems#

Logging VAD Events#

detector.SetLogLevel(silero.LogLevelDebug)

// Output:
// [VAD] Frame 1: prob=0.12, speaking=false
// [VAD] Frame 2: prob=0.45, speaking=false
// [VAD] Frame 3: prob=0.89, speaking=true, event=SpeechStart
// [VAD] Frame 4: prob=0.92, speaking=true
// ...
// [VAD] Frame N: prob=0.15, speaking=true
// [VAD] Frame N+1: prob=0.08, speaking=false
// [VAD] Silence duration: 200ms, event=SpeechEnd

Next Steps#

• Interruption Handling - Handle user barge-in
• Turn Detection - Advanced turn-taking
• Latency Optimization - Reduce response time