Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
Source
Tanya Lenz
Author
Tanya Lenz
Date
Terms in this piece · Glossary
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
A 10x-plus throughput jump on dropless mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → training changes the cost-per-tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → math for anyone training or fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → large mixture-of-experts models on NVIDIA hardware.
articleWhen to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
articleBuilding a Memory-Driven Agent with NVIDIA NemoClaw
articleNVIDIA PAIR Virtual Inference Router Expands Available Compute on Your Local Network
articleCo-Designing AI Models Using Speculative Decoding for Faster LLM Inference
