PersonaPlex
github.com/nvidia/personaplex- Category
- AI Tools
- Rank
- No. 430Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- nvidia
- GitHub
- 10.5k stars
- Date
About
A real-time, full-duplex speech-to-speech conversational AI model that enables persona control through text-based role prompts and audio-based voice conditioning. Trained on synthetic and real conversations, it produces natural, low-latency spoken interactions with consistent personas.
What it does
PersonaPlex is a self-hosted spoken-agent engine that listens while it talks. You shape behavior with a written character brief and select a voice embedding. It includes a browser experience for live conversations and a batch workflow that converts recorded input into synchronized audio and text output.
Why it's ranked here
The project covers more than a model checkpoint. It supplies streaming inference, browser audio handling, offline evaluation, ready-made voices, and practical prompting examples. That makes it a credible foundation for experiments with interruption, backchannels, turn taking, customer service roles, and casual conversation. Deployment still demands substantial model infrastructure.
What's good
The live server streams Opus audio and generated text over WebSockets. The browser audio processor adapts its buffer after underruns and drops old packets when latency grows. Offline runs accept deterministic seeds, expose sampling controls, produce text alongside speech, and force output duration to match the input recording.
Tradeoffs
Setup requires the system Opus development library, gated model-weight access, and Hugging Face authentication. GPU memory may require CPU offloading plus an extra dependency, while pure CPU execution is presented only for offline evaluation. The documented role training centers on one assistant pattern, customer service scenarios, and casual English conversations. Prompts outside that distribution are explicitly experimental.
How to use it well
Use it for research prototypes and self-hosted voice agents where interruption handling, conversational timing, role prompts, and controlled voices matter. Start with the supplied prompt patterns and voice embeddings, test recorded conversations offline with fixed seeds, then move successful cases into the live browser flow. It does not provide business integrations, telephony, authentication, or production orchestration.
Technical notes+
moshi/moshi/server.py runs an aiohttp WebSocket service, decodes incoming Opus packets, advances streaming Mimi and language-model state frame by frame, and sends audio plus tokenizer pieces under a single asyncio lock. moshi/moshi/offline.py mirrors that pipeline for WAV input, supports CUDA or CPU offload, and trims or pads generated PCM to the source duration. moshi/moshi/models/lm.py defines eight audio token streams at 12.5 Hz and contains streaming token generation helpers. client/src/audio-processor.ts implements adaptive playback buffering, underrun recovery, fades, and packet dropping. client/vite.config.ts configures an HTTPS Vite client with an optional queue API proxy.
Observed
- Code license
- MIT license
- Model weights license
- NVIDIA Open Model license
- Primary implementation
- Python inference core with a TypeScript and React browser client
- Installation
- Local pip installation from the moshi subdirectory; system Opus development library required
- Authentication
- Model access requires accepting the Hugging Face license and supplying a Hugging Face token
- Interfaces
- HTTPS browser UI with WebSocket streaming, plus an offline command-line WAV workflow
- Compute support
- CUDA and CPU devices are implemented; CPU layer offload requires accelerate
- Packaged voices
- Eighteen labeled voice embeddings across natural and varied female and male groups
Read from README.md, client/vite.config.ts, client/postcss.config.js, client/tailwind.config.js, client/src/env.ts, client/src/app.tsx, moshi/moshi/server.py, moshi/moshi/offline.py, moshi/moshi/__init__.py, client/src/modules.d.ts, moshi/moshi/client_utils.py, client/src/audio-processor.ts, moshi/moshi/models/lm.py, moshi/moshi/modules/conv.py, moshi/moshi/modules/rope.py.
What it can do
Convert speech to speech with persona control
Spoken audio and text-based role prompts → Natural spoken responses in specified persona
Condition voice characteristics through audio examples
Audio samples for voice conditioning → Speech output matching the conditioned voice type
Generate real-time conversational responses
Live spoken input → Low-latency spoken responses
Maintain consistent persona throughout conversation
Role-specific text prompts and ongoing dialogue → Coherent persona-based conversational behavior
Support assistant role conversations
User queries and requests → Assistant-style spoken responses
Handle customer service scenarios
Customer inquiries and service requests → Customer service representative responses
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.