Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading

When HBM binds before compute does, offloading activations to host memory can beat recomputing them by up to 57% — and unlock batch sizes that were previously impossible — provided the transfers actually overlap.

• mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.

• articleBuild Applications on NVIDIA BlueField Faster with NVIDIA DOCA Agent Skills
• articleAI Native by Design: Lessons Learned from Building NVIDIA TensorRT Model Connect
• articleNVIDIA Open Agent Safety Platform: A Reference for Continuous In-Silicon Agent Monitoring
• articleAccelerating a ROS 2 Node with an AI Agent and NVIDIA Isaac ROS

• articleEnhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor ParallelismMichelle Horton
• articleEngrams Embedding Entendre: Codesign for Efficient DRAM/SSD OffloadingBryan Shan
• articleEfficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon et al.

Checking sign-in…

Loading comments…