Vibeleaderboard
← All Intel
Intel / article

Inkling 2 Small: An Inkling of Profanity

Source
vals.ai
Date

Context

Vals AI, which evaluates AI models on public and private benchmarks, read the reasoning that Thinking Machines' Inkling 2 Small writes to itself during tasks, text most users never see, across 2,655 runs on 13 benchmarks. The model wrote "fuck" in 50 of those runs. It wrote the word 316 times in all, but 241 of them came from two runs that got stuck in a loop. In the larger one, on a task to determine an asteroid's orbit, a single reply ran past 500,000 characters while cycling through the same few sentences, and that run alone accounts for 185.

The swearing comes late in long runs. No run under 100 steps contained it, against 29 of the 370 runs of 500 steps or more, and 95 of the 112 replies that swore came in the second half of a run. The five benchmarks with the longest runs, averaging 338 to 1,180 steps, are the five where the model swore most, in 3% to 10% of runs, while on every other the rate was 1% or less. Vals cautions that run length and benchmark overlap, so it cannot say length alone drives the behavior. The examples it shows include the model cursing after realizing its own cleanup command had been killing its jobs, and again as it gave up on a task it could not solve.

Profanity turns out to be a poor warning sign. Of the 48 graded runs with swearing, 46 failed, but about 75% of all graded runs fail anyway, and 96% of failed runs never swore, so watching for the word would flag only about 4% of failed runs. The finding covers one model, run through Thinking Machines' API at default settings, and counts only the model's own reasoning and messages, not tool calls or tool results. Eight of the 13 benchmarks are private, and Vals reports those only as counts.

Key takeaways · AI-distilled
  • Vals AI scanned the reasoning and messages of Thinking Machines' Inkling 2 Small across 2,655 runs on 13 benchmarks. It wrote "fuck" in 50 runs, 316 times in total, and 241 of those came from two runaway runs.
  • The swearing clusters late in long runs: no run under 100 steps swears, 29 of the 370 runs of 500+ steps do, and 95 of 112 swearing replies fall in a run's second half. Length and benchmark overlap, so Vals cannot say length alone drives it.
  • Profanity is a poor failure signal. 46 of 48 graded swearing runs failed, but about 75% of all graded runs fail, and 96% of failed runs never swore.
  • The largest count came from a loop: one Terminal-Bench Science reply over 500,000 characters cycled the same sentences, and the run kept looping for 37 more steps until it filled the , accounting for 185 of the 316 instances.
  • Vals scanned only the model's own output, not tool calls or results, counted variants like "fucking", and excluded programming-language names, the gene FucK and censored forms such as "F*CK", which one pixterm run wrote 787 times.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Read the source www.vals.ai
Recommended reads
Comments

Checking sign-in…

Loading comments…