STT - Whisper Sidecar

GPU-accelerated or CPU-only speech-to-text sidecar built from the official whisper.cpp container image.

Getting started

Requirements:

• Docker with Compose
• just
• uv for diarization dependency updates
• NVIDIA Container Toolkit for CUDA deployments

Create local configuration:

cp .env.local.example .env.local

Download configured model:

just download-model large-v3-turbo

Start CUDA sidecar with rootful Docker:

just up-cuda

For rootless Docker with NVIDIA CDI:

just up-cuda-cdi

Use just up as an alias for just up-cuda. Use just up-cpu for CPU inference. See Run with NVIDIA CUDA for host setup and troubleshooting.

CUDA startup runs a model-loading preflight. Keep enough VRAM free for the configured model before starting the service.

Submit audio:

just inference data/audio/jfk.wav

Include speaker turns with the optional pyannote facade:

just up-speakers-cuda-cdi
just inference-speakers data/audio/conversation.wav auto 2 2

Use just up-speakers-cuda with rootful Docker or just up-speakers-cpu without a GPU. See Add speaker turns for token setup and response details.

Call service directly from another Compose service:

curl http://whisper:9199/inference \
  --form file=@recording.m4a \
  --form language=auto \
  --form response_format=json

Stop sidecar:

just down

Documentation

• Architecture
• Run with NVIDIA CUDA
• Load and unload models
• Add speaker turns
• API reference
• Configuration reference
• Measured model requirements