Vibeleaderboard
← All Intel
Intel / article

Tencent open-sources Hy4 flash, a 12B-active 1M-context MoE

Source
github.com
Author
Tencent Hy
Date

Context

Tencent has published the weights of Hy4 flash, so anyone with enough hardware can download and run the model it had already been selling as a cheap API. The release, documented in its GitHub model card, includes the instruction-tuned model, a smaller FP8 version that stores numbers at lower precision, and the pretrained base model for further training.

The design is built around cost. Hy4 flash is a model: its network is split into 512 specialised sub-networks, and for each token a router activates only 16 of them plus one shared expert. That is why a model with about 220 billion parameters uses only about 12 billion for each token it processes. Roughly 64 billion of the parameters sit in learned lookup tables for two- and three-token sequences, which are fetched by a hash lookup rather than computed. Tencent also mixes in a cheaper attention scheme that mostly looks at nearby text, and says these changes together cut the memory the model holds for its to a sixth of Hy4's, which matters when the context window runs to a million tokens.

Tencent's performance claims come from its own tests. On WorkIndex, an internal benchmark of WorkBuddy office tasks such as making slides and repairing spreadsheets, Hy4 flash scored 42.1 against 51.4 for the full Hy4, at a twentieth of Hy4's cost per task. In a blind test with 85 Tencent users, it scored 2.79 out of 4 against 2.81 for Claude Haiku 5.5. Tencent recommends it for everyday productivity work and still recommends Hy4 for long-horizon software engineering. Its API pricing was announced two days earlier.

Open weights here do not mean an unrestricted license. The Tencent Hy License is based on Apache 2.0 but requires businesses that sell model access as a service to get a separate license from Tencent if their total revenue exceeds 50 million US dollars a year or if they use the model in a product with more than 100 million monthly users. Purely internal use, where neither the model nor its outputs reach third parties, is exempt. Running the model is also not a laptop job: Tencent's serving examples spread the FP8 version across eight GPUs.

Key takeaways · AI-distilled
  • Hy4 flash has 220B total parameters, about 64B of them learned bigram and trigram lookup (Engram), but activates about 12B per token. It takes text and images, supports a 1M-token context, and ships as open weights under the Tencent Hy License.
  • Tencent says interleaving sparse and sliding-window attention at 1:3, running experts in a lower-dimensional latent space, and adding Engram cut the to 1/24 of Hy3 and 1/6 of Hy4. Reinforcement learning also halved average end-to-end output tokens.
  • On Tencent's internal WorkIndex of WorkBuddy-style tasks, Hy4 flash scored 42.1 versus 39.7 for Hy4 preview and 51.4 for Hy4, at 1/13 the cost per task of Hy4 preview and 1/20 of Hy4. Tencent still recommends Hy4 for long-horizon software engineering.
  • In a blind test with 85 internal WorkBuddy users across 259 tasks, Hy4 flash scored 2.79/4 for satisfaction against 2.81 for Claude Haiku 5.5, with the lowest cost per task (RMB 0.26) and the shortest median completion time (6.7 minutes) of the models evaluated.
  • Its built-in 0.85B drafter, DFly2, reuses the main model's KV cache and sparse-attention selections, so each draft token attends to at most about 2K keys regardless of context length and the drafter needs no KV memory of its own.
Terms in this piece · Glossary
  • mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
More from Tencent Hy
Recommended reads
Comments

Checking sign-in…

Loading comments…