Intel / article

The Pelican Benchmark (SVG Pelican on a Bicycle)

Source: simonwillison.net
Author: Simon Willison
Date: Published Jul 16, 2026

Context

Simon Willison's pelican-on-a-bicycle prompt is useful as a first encounter with a model. Asking it to generate an SVG drawing lets him check that he can run the model, inspect its output and observe the cost and reasoning effort of that particular attempt. His assessment of the test explains why he still uses it.

Its usefulness stops short of ranking general capability. A single drawing request does not exercise reliable across a growing conversation, which Willison identifies as a central requirement for agents. The test gives a concrete first impression while leaving those longer workflows to separate evaluation.

Key takeaways · AI-distilled

The pelican-SVG test has lost its correlation with model quality: GLM-5.2 draws a better pelican than Claude Fable 5 or GPT-5.6, and the test never touches agentic tool calling, which is what matters most for today's models.
One prompt still reveals cost and verbosity. Kimi K3 burned 13,241 reasoning tokens to emit 3,417 tokens of answer, so a single pelican cost 25 cents, because the model currently ships with only a 'max' reasoning effort setting.
You can smoke out a hidden by counting tokens. A 10-token prompt billed as 95 input tokens on Kimi K3, and 'hi' billed 86, implying roughly 85 tokens of prepended instructions the model refused to reveal.
The test's real value is procedural: it forces you to actually run the model, proves it can emit valid SVG with basic spatial sense (a big deal for small local models), and yields a comparable artifact across releases in one family.

Terms in this piece · Glossary

tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.

Key quotes

“It started out as a joke on how absurdly difficult it is to compare these models, but then for the first year it turned out to have a surprising correlation to how good the models actually were.”

“The biggest limitation of the pelican is that it doesn't touch at all on the thing that matters most for today's model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.”

“The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic's Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date.”

“The model consumed 13,241 reasoning tokens to output 3,417 tokens of response. This is expensive - the pelican cost 25 cents!”

Read the source simonwillison.net