HPC-Ops open-sources the kernels behind Tencent's inference serving
- Source
- x.com
- Date
🚀🚀🚀 We are excited to open-source HPC-Ops, Tencent HY's production-grade LLM inference operator library designed to unlock peak performance on mainstream inference cards. Highlights: 🔹 30% Throughput Boost: Delivers 30% QPM improvement for Tencent HY models and 17% for DeepSeek in production. 🔹 Hardware-Optimized: Built from scratch with CUDA and CuTe to maximize GPU utilization, addressing the pain point where mainstream libraries fail to reach hardware peak. 🔹 SOTA Kernel Performance: -Attention: Up to 2.22x speedup over FlashInfer/FlashAttention -GroupGEMM: Up to 1.88x speedup over DeepGEMM -FusedMoE: Up to 1.49x speedup over TensorRT-LLM 🔹 Production-Ready: Comprehensive operator coverage including FusedMoE, GroupGEMM, and multi-node communication with clean abstraction for easy customization. Already powering Tencent's large-scale inference services. Unleash your hardware potential today! 🔗 GitHub:
Context
Tencent's HPC-Ops is a library of low-level GPU operations used to run language models, including and the grouped matrix calculations behind models. These kernels implement recurring pieces of , giving serving engineers components they can integrate and benchmark within their own systems. Library documentation.
The launch post reports a 30% improvement in queries per minute for Tencent's Hunyuan models in production, a whole-service figure Tencent attributes to itself. It separately reports kernel benchmarks, such as up to 2.22 times the speed of FlashInfer and FlashAttention for attention. Those are different scopes: speeding up one operation does not establish the same gain for an entire service. The repository's README lists benchmark scopes and baselines, and notes that performance varies by case. Launch report, benchmarks.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleExplorationBench: Feedback and Self-Designed Probes Drive Agent Discovery
postHy Image3.5 preview is live. 🚀 Professional-grade image generation, +30% win…
postTencent Hunyuan's EvolveScaler trips up frontier models on evolving facts
postTencent Hunyuan open-sources AuK, a unified speech generation and editing model
Checking sign-in…
Loading comments…