Needle

Why it's ranked here

The case is scope discipline backed by real mechanism. Instead of shrinking a chat model, the authors gave up open-ended text entirely: the model answers only with calls, a byte-level grammar constrains every token, and the engine holds back calls it rates under 0.1 confidence. One trained network can be cut to any depth from 2 to 20 layers, so a team picks a size per device. It is Apache 2.0 licensed and the runtime install needs only the Hugging Face client. The headline benchmark numbers ship as images and on the vendor site, so treat them as the vendor's claims until tested on your own tools.

What's good

Refusal is designed in rather than hoped for: an off-topic request yields no call, optional arguments the user never stated are dropped, and a repair step grounds values like dates, names and on/off verbs in the request text. A tool can be a decorated function, a Pydantic model or a raw JSON schema. The depth ladder is concrete: smaller cuts always keep the first and last blocks and add middle ones by splitting the widest gap, so every size is a nested slice of the same weights. Local fine-tuning runs on CPU, CUDA or Apple Metal, and keeps training dependencies out of the runtime install.

Tradeoffs

Telemetry is on by default and posts to a remote endpoint: event name, package version, OS, CPU architecture, Python version and a random install ID kept in your home directory. Environment variables switch it off, and it skips CI runs. Size figures disagree across the project's own text, from 8 to 35 megabytes for weights and 29 to 121 million parameters, so measure on your target. Local fine-tuning leaves the confidence head untrained, so tuned models report no confidence. Synthetic training data comes from a third-party model through OpenRouter. The bundled inference tests accept almost any response shape, so they prove little about accuracy.

How to use it well

Fits an app or device with a fixed, known set of actions: smart-home controls, a phone assistant, filling records from notifications or invoices. Write one tool per action with names a user would actually say, then pick a confidence threshold per product: act above it, ask for confirmation below it. Past five declared tools, only the five best matches reach the model each turn, so keep catalogues tight. It does not hold conversations or write free text, so pair it with a chat model if the product needs either. Turn telemetry off before shipping to end users.

Observed

License

Apache-2.0

Language

Python, requires 3.9 or newer; models trained in JAX

Install

pip install cactus-needle; training extras behind [train], with CUDA and Apple Metal variants

Runtime dependency

huggingface_hub only; engine and weights download from Hugging Face on first use

Interfaces

Python library plus a needle CLI (run, finetune, generate-data, build, platform, download, fetch, playground, whistle)

Deployment

Prebuilt engine under 1 MB per platform folder, e.g. macos-arm64 and linux-arm64

Telemetry

On by default; off with NEEDLE_TELEMETRY=0, DO_NOT_TRACK, or in CI

Fine-tuning

Local LoRA on attention projections; hosted full-model tuning needs a Cactus API key

Output format

Grammar-constrained JSON tool calls with reasoning and a confidence score; no free-text generation

Tests

pytest suite in the tests directory, inference tests gated on the downloaded engine