Jeff

Why it's ranked here

The case rests on measured numbers with the method written down. Across eight adapter tasks the authors report 94.6% mean accuracy for Jeff with its adapter against 86.6% for a 27B model answering directly, at 0.25 seconds per decision versus 8.1 on an Apple M4 Max, with fixed held-out samples and the fallback threshold chosen before test rows were scored. The code is MIT licensed, the request format is frozen for the current base, and the serving code refuses inputs it cannot handle rather than quietly cutting them. The caveat: every figure is self-reported, and the 27B baseline ran with step-by-step reasoning switched off.

What's good

Adapters are bound to the exact base they were trained on: each records a hash of the base weights, and the server refuses a mismatch, so an old adapter cannot silently run on a new base. Inputs over the trained length of 8,192 tokens are rejected, never truncated. Option counts above what a model was trained on are refused with a message telling you to shortlist. The prompt puts the fixed question and options before the changing part, so a server can reuse that work across requests on one screen. A bundled kit checks training rows, splits them by family, and fails when a training row overlaps an evaluation row.

Tradeoffs

The current base gave up zero-shot accuracy to favour adapters: on the 100-option contract clause task without an adapter it falls from 66.0% on the previous base to 7.4%, and the authors say to use the older base for zero-shot work. Adapters only run on the base they were trained for. Adapter licences differ: the sanctions and security-alert adapters are non-commercial because of their source data. Training data is not released. The model answers one request at a time; by default a request that arrives while it is busy is turned away at once. Emotion detection stays weak at 60.5% with its adapter.

How to use it well

Use it as a fast first filter in an agent or support pipeline where the choices are known in advance: injection guarding, ticket routing, picking which tool an agent calls, or checking whether an answer is backed by its sources. Pick the adapters you need rather than relying on the bare base, and send uncertain cases to a larger model, which is how the authors get their best numbers. Check each adapter's licence before commercial use. With no matching adapter, the kit lets you train one, and on small data the Jeff base gives it a head start. It is not for multi-step reasoning or writing.

Observed

License

MIT (code); some adapters carry CC BY-NC 4.0 or data-source terms

Language

Python, requires 3.12 or newer

Packaging

pyproject.toml with hatchling build backend, installed via uv

Interfaces

HTTP decision API (FastAPI), Python client, 20 CLI entry points including jeff-serve and jeff-kit

Platform support

NVIDIA GPUs (cuda extra), Apple silicon via MLX (mac extra), CPU fallback in code

Base models supported

Qwen3.5 backbone, plus Gemma 3, Gemma 4 E2B, Phi-4-mini and ModernBERT variants

Option limit

1 to 255 options per question

Input limit

8,192 tokens by default; longer inputs rejected, not truncated

Training data

Not released; sources listed in docs/data-sources.md