VibeVoice
github.com/microsoft/vibevoice- Category
- AI Tools
- Rank
- No. 96Tools index
Previous survey · No. 101 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- @microsoft
- GitHub
- 53.9k stars
- Date
About
Open-source voice AI framework that includes advanced speech recognition (ASR) for 60-minute audio transcription with speaker diarization, text-to-speech (TTS) for 90-minute multi-speaker synthesis, and real-time streaming TTS. Operates at ultra-low 7.5Hz frame rate for efficient long-form audio processing.
What it does
VibeVoice is a research-focused family of models for turning recordings into structured transcripts and turning text into spoken audio. Its recognition model tracks speakers, timing, and content together. Its synthesis architecture combines continuous speech tokenization, language-model context handling, and diffusion-based acoustic generation. A smaller variant targets interactive speech output.
Why it's ranked here
The strongest case is its unusually complete recognition workflow: long recordings, speaker attribution, timestamps, multilingual code-switching, custom terminology, fine-tuning, and scalable serving. The package is less complete as a general speech toolkit. The original long-form synthesis code was removed, and the documentation conflicts over whether live token feeding is implemented in the real-time model.
What's good
Recognition produces structured speaker, time, and text segments instead of a plain transcript. Custom hotwords can improve names and technical vocabulary. It supports more than 50 languages without an explicit language setting, including code-switching. The serving plugin adds an OpenAI-compatible endpoint, streaming responses, continuous batching, and data or tensor parallel deployment.
Tradeoffs
The maintainers describe the models as research and development software, not ready for commercial or real-world use without further testing. Recommended recognition setup assumes Docker, NVIDIA GPUs, CUDA, FFmpeg, and substantial model downloads. Long-form multi-speaker synthesis installation is disabled. Real-time synthesis supports one speaker, primarily English, and can struggle with very short or symbol-heavy input.
How to use it well
Use it for research prototypes that transcribe meetings, interviews, or other long recordings where speaker identity and timing matter. Add hotwords for domain vocabulary, then serve recognition through the compatible API when throughput matters. Use the real-time model for experimental narration or spoken model responses. Do not treat it as production-ready voice infrastructure or a tool for music, effects, and overlapping speech.
Technical notes+
The Python package is configured in pyproject.toml with setuptools, Python 3.10 or newer, core Torch and Transformers dependencies, and an optional streaming TTS extra. vllm_plugin/__init__.py registers VibeVoice configuration, tokenizer, processor, and model classes through the vLLM plugin entry point. vllm_plugin/model.py replaces vLLM audio loading with FFmpeg-backed decoding and integrates acoustic and semantic tokenizers. vllm_plugin/inputs.py accepts paths, bytes, NumPy arrays, or path lists, normalizes 24 kHz audio, and enforces a configurable duration ceiling. finetuning-asr/lora_finetune.py supplies PEFT-based LoRA training, while demo/vibevoice_realtime_demo.py starts a Uvicorn web application.
Observed
- License
- MIT License
- Primary language
- Python
- Packaging
- Setuptools package requiring Python 3.10 or newer, installable from the cloned repository with pip editable mode
- Interfaces
- Python library, command-line demo scripts, Gradio interface, WebSocket demo, and OpenAI-compatible vLLM API
- Model distribution
- Pretrained model weights are linked through Hugging Face
- Platform support
- Package metadata declares operating-system independence, while recommended deployments use NVIDIA GPU Docker containers
- Training surface
- Repository includes LoRA fine-tuning and adapter inference support for speech recognition
Read from README.md, pyproject.toml, docs/vibevoice-asr.md, docs/vibevoice-tts.md, docs/setup_gradio_demo.md, docs/vibevoice-vllm-asr.md, docs/vibevoice-realtime-0.5b.md, vllm_plugin/model.py, vibevoice/__init__.py, vllm_plugin/inputs.py, vllm_plugin/__init__.py, demo/vibevoice_realtime_demo.py, finetuning-asr/lora_finetune.py, finetuning-asr/inference_lora.py, demo/vibevoice_asr_gradio_demo.py.
What it can do
Transcribe long-form audio with speaker identification
Audio file up to 60 minutes → Text transcript with speaker labels
Generate multi-speaker synthetic speech
Text and speaker specifications → Audio file up to 90 minutes with multiple voices
Stream text-to-speech in real-time
Text input stream → Live audio speech output
Convert speech to continuous tokens
Audio speech data → 7.5Hz speech tokens
Process single-pass audio transcription
60-minute audio recording → Complete transcription without chunking
Intel on VibeVoice
Tags
Tech Stack
Featured in Intel
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.