Index / tool

Jeeves

Category: AI Tools
Rank: No. 1144Tools index
Previous survey · No. 1135 · 7 November 2026
Pricing: Open Source
Type: TOOL
Use case: Models: Train & Run
Interfaces: API
GitHub: 440 stars
Date: Published Sep 29, 2026

About

Jeeves is a 9B decision model built on Qwen3.5-9B with LoRA and a pointer head that reasons before it decides. It handles yes/no, multiple-choice and rating questions in one request through a Jev-compatible API, with a diffusion drafter for speed. The repository reports 0.889 on held-out test data versus 0.822 for Kev-9B and 0.857 for Jev, and includes training code and data.

What it does

Jeeves answers structured questions about a piece of text: is it true or false, which of these labels fits, where does it sit on a rating scale. Every answer comes back with probabilities. It is an open-weights 9-billion-parameter model you host yourself on one GPU or a large Apple Silicon Mac. When its first guess is less than 90% confident, it writes out a reasoning chain before answering; otherwise it answers immediately. A small companion model proposes several tokens at once to make that reasoning faster. The server accepts the same request format as Jev, a hosted classification API, so existing Jev clients can point at it.

Why it's ranked here

The README's own table puts Jeeves v2 ahead of Jev, Kev-9B and JevK5 on a held-out test set (0.905 against 0.857 for Jev) and on the public JevBench tiers, and it never answered an unknowable question with 0.9 confidence. Those are the authors' numbers, with the Kev and Jev columns copied from Kev's publications rather than rerun. What lifts it past a benchmark claim is the package around it: MIT license, released weights in bf16 and FP8, the full training code, and a data build that pins each public source to an exact revision and hash so the training set can be rebuilt.

What's good

Cost follows difficulty: a confident question returns in about 20 ms on an H100, and only the unsure ones pay for reasoning. Several questions about the same text go in one request and reason in parallel. The API rejects unknown fields and out-of-range options with an error instead of guessing, so a broken client fails loudly. The Python client is a drop-in for Jev's own SDK, so switching is an import change. The README ships checks that compare the fast GPU and Metal kernels against reference math, and it is specific about Mac memory: 21 GB of weights plus 28 GB of default caches.

Tradeoffs

Hardware is the gate. You need a CUDA GPU or a Mac with 48 GB or more, and FP8 needs compute capability 8.9 or newer. The 1.2 second median with reasoning hides a long tail: the README's own three-question example took 8.1 seconds. Jev still wins on the transfer tasks, 0.840 against 0.770 on 10-way MMLU-Pro, and v2's calibration error on JevBench is worse than v1's or JevK5's. Dependencies are pinned to exact, very recent library versions, and the Python client wraps Jev's SDK, so it moves when that package moves.

How to use it well

It fits a pipeline that routes, tags or gates text (support tickets, moderation, checking agent inputs for prompt injection, which is in its training mix) and today falls back to a large reasoning model whenever a cheap classifier is unsure. Run the server on your own GPU, keep the default confidence threshold, and cap reasoning length per request to bound latency. Ask for the reasoning text while tuning question wording, then turn it off. On a 48 GB Mac, shrink the caches as the README shows or use the FP8 weights. It is a decision model, not a chat or writing model.

Technical notes+

inference/api.py validates every request: unknown option or question fields raise errors, choice and score questions cap at 255 options, and choice confidence is the top probability rescaled against chance. inference/engine.py refuses fp8 unless CUDA capability is at least (8, 9) or Metal kernels are on, merges projections that share an input into one GEMM on CUDA, and packs questions from several requests into shared rows. model/model.py falls back to pure PyTorch gated delta rule code when flash-linear-attention is absent; QWEN35_KERNELS=0 disables fused kernels. train.py has three stages: sft, cispo and drafter. prep/prep.py checks rebuilt splits against sha256 manifests; prep/injection.py pins Hugging Face dataset revisions. jevbench.py computes 10-bin top-label ECE.

Observed

License: MIT (LICENSE)
Language: Python 3.12 (README.md); SDK requires Python 3.10+ (sdk/pyproject.toml)
Interface: Self-hosted HTTP server accepting Jev's request format (inference/api.py)
Python client: jeeves-sdk, a drop-in wrapper over typesafe-sdk (sdk/pyproject.toml)
Platform support: CUDA in bf16 or FP8 (capability 8.9+); Apple Silicon via MPS with 48 GB or more (README.md)
Training code: SFT, CISPO and drafter distillation stages included (train.py)
Weights: Released on Hugging Face in bf16 and FP8 (README.md)

Read from README.md, requirements.txt, LICENSE, train.py, model/model.py, inference/engine.py, inference/api.py, prep/prep.py, prep/injection.py, jevbench.py, sdk/README.md, sdk/pyproject.toml, sdk/jeeves_sdk/__init__.py.

Tech Stack

Python

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.