Perplexity open sources Lily, an Apple silicon engine built for hybrid agents
- Source
- x.com
- Date
Today we’re open-sourcing Lily, the local inference engine we built for hybrid compute in Perplexity Computer. Lily is specialized for Qwen3.6-35B-A3B on Apple silicon, built so on-device compute doesn’t bottleneck Computer tasks. Read more: https://t.co/OnowTP3ql6
Hybrid compute splits work between cloud models and a local model on the Mac. Local inference must keep pace with the rest of the task. MLX-LM is a general-purpose framework, while Lily is purpose-built for this inference workload.
Lily treats prefill and decode as fundamentally different workloads. Prefill processes many prompt tokens at once and can reuse weights across them, while decode generates one token at a time with much less reuse, making memory traffic and bandwidth more important.
We benchmark Qwen3.6-35B-A3B on an M5 Max MacBook Pro. Across ten prompt lengths and ten decode contexts, Lily was faster than MLX-LM, averaging 1.23× higher prefill throughput and 1.35× higher decode throughput while keeping output quality effectively unchanged.
Context
Perplexity open-sourced Lily, the local engine it built for Perplexity Computer's hybrid compute mode, which splits tasks between cloud models and a model running on the user's own Mac so local inference does not bottleneck the rest of a task. Lily is specialized for Qwen3.6-35B-A3B on Apple silicon and treats prefill, which processes many prompt at once and can reuse weights across them, as a separate workload from decode, which generates tokens one at a time and is limited more by memory bandwidth than compute.
In Perplexity's own on an M5 Max MacBook Pro, across ten prompt lengths and ten decode contexts, Lily averaged 1.23 times the prefill throughput and 1.35 times the decode throughput of the general-purpose MLX-LM framework, with output quality it describes as effectively unchanged. These are Perplexity's own reported figures rather than an independently run benchmark.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Checking sign-in…
Loading comments…