🚀🚀🚀 We are excited to open-source HPC-Ops, Tencent HY's production-grade LLM inference operator library designed to unlock peak performance on mainstream inference cards.
Highlights:
🔹 30% Throughput Boost: Delivers 30% QPM improvement for Tencent HY models and 17% for DeepSeek in production.
🔹 Hardware-Optimized: Built from scratch with CUDA and CuTe to maximize GPU utilization, addressing the pain point where mainstream libraries fail to reach hardware peak.
🔹 SOTA Kernel Performance:
-Attention: Up to 2.22x speedup over FlashInfer/FlashAttention
-GroupGEMM: Up to 1.88x speedup over DeepGEMM
-FusedMoE: Up to 1.49x speedup over TensorRT-LLM
🔹 Production-Ready: Comprehensive operator coverage including FusedMoE, GroupGEMM, and multi-node communication with clean abstraction for easy customization.
Already powering Tencent's large-scale inference services. Unleash your hardware potential today!
🔗 GitHub:
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Serving throughput is mostly kernel quality; these are the operators already running Tencent's production inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →, with measured margins over FlashAttention, DeepGEMM and TensorRT-LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →.
