Big model. 2.4T params, 95B active. 4.89TB. Surprised they didn't release a 4-bi

A 2.4T-parameter open release is not locally runnable as shipped: 4.89TB of weights with no 4-bit quant from the lab, and even 4-bit needs roughly four 512GB machines.

• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

• postIs the Rush to Build New LLM Inference Engines Fragmenting the Ecosystem?
• post@Jason @Lons @eisokant @ape Correction: Should actually be closer to 50 tok/sec
• postApple markets a four-Mac cluster for trillion-parameter local inference
• postLocal inference crosses over: the M5 Ultra and the 1,000x claim

• articleServe Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300 NVL72Michelle Horton
• postA 2-bit 1.8B model in 600MB, tuned for Arm SME2 siliconTencent Hy
• clipMixed 2/4/8-bit quantization puts a 2B model at ~2.9 bits per weightAI Engineer

Checking sign-in…

Loading comments…