Today we’re open-sourcing Lily, the local inference engine we built for hybrid compute in Perplexity Computer.
Lily is specialized for Qwen3.6-35B-A3B on Apple silicon, built so on-device compute doesn’t bottleneck Computer tasks.
Read more: https://t.co/OnowTP3ql6
Hybrid compute splits work between cloud models and a local model on the Mac. Local inference must keep pace with the rest of the task.
MLX-LM is a general-purpose framework, while Lily is purpose-built for this inference workload.
Lily treats prefill and decode as fundamentally different workloads.
Prefill processes many prompt tokens at once and can reuse weights across them, while decode generates one token at a time with much less reuse, making memory traffic and bandwidth more important.
We benchmark Qwen3.6-35B-A3B on an M5 Max MacBook Pro.
Across ten prompt lengths and ten decode contexts, Lily was faster than MLX-LM, averaging 1.23× higher prefill throughput and 1.35× higher decode throughput while keeping output quality effectively unchanged.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Hybrid setups stall when on-device inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → lags the cloud half. Lily, now open source, averaged 1.23x prefill and 1.35x decode throughput over MLX-LM for Qwen3.6-35B-A3B on an M5 Max with output quality effectively unchanged.
Read the full thread on X ↗
