The pelican-SVG test has lost its correlation with model quality: GLM-5.2 draws a better pelican than Claude Fable 5 or GPT-5.6, and the test never touches agentic tool useA model's ability to call external functions — run code, search the web, edit files — instead of only generating text.Full definition →, which is what matters most for today's models.
One prompt still reveals cost and verbosity. Kimi K3 burned 13,241 reasoning tokens to emit 3,417 tokens of answer, so a single pelican cost 25 cents, because the model currently ships with only a 'max' reasoning effort setting.
You can smoke out a hidden system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition → by counting tokens. A 10-token prompt billed as 95 input tokens on Kimi K3, and 'hi' billed 86, implying roughly 85 tokens of prepended instructions the model refused to reveal.
The test's real value is procedural: it forces you to actually run the model, proves it can emit valid SVG with basic spatial sense (a big deal for small local models), and yields a comparable artifact across releases in one family.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
Why it matters
The pelican-on-a-bicycle SVG test — the informal benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → that reveals model differences the official leaderboards miss.
Key quotes
“It started out as a joke on how absurdly difficult it is to compare these models, but then for the first year it turned out to have a surprising correlation to how good the models actually were.”
“The biggest limitation of the pelican is that it doesn't touch at all on the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.”
“The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic's Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date.”
“The model consumed 13,241 reasoning tokens to output 3,417 tokens of response. This is expensive - the pelican cost 25 cents!”
