NullPrime
github.com/bmiedlar/nullprime- Category
- Developer Tools
- Rank
- No. 2047Tools index
- Pricing
- Open Source
- Type
- TOOL
- GitHub
- 1 stars
- Latest release
- v1.0
- Date
About
NullPrime is a dataset of 10- and 15-digit integers, half prime, filtered so none has a small prime factor, which removes the divisibility pattern that learners could otherwise pick up. Standard learners memorize the training set and score at chance on new numbers, so the dataset works as a negative control for uncertainty estimates, abstention and LLM calibration. The release includes a pre-registered test of eight language models and a script that generates fresh numbers.
At a glance
NullPrime is a dataset and test harness of integers labeled prime or composite, filtered so no number has a factor below a bound B. That removes the divisibility signal learners normally pick up, so standard classifiers score at chance on held-out numbers while the label stays exactly checkable. It serves as a negative control for uncertainty estimates, abstention and LLM calibration.
Good fit
Researchers checking whether a model, uncertainty method or selective-prediction setup admits it cannot know an answer. The included LLM protocol, pre-registered on OSF, compares a plain question, an allowed "cannot determine" option, a code tool and reasoning on or off.
Consider something else when
You want to measure mathematical skill or a hard primality problem. The README says Miller-Rabin answers primality exactly in microseconds, and with a code tool most models it tested got 89-100% correct, so the benchmark says little about raw capability.
Start here
Download the dependency-free make_numbers.py and run python make_numbers.py 50 15 for 50 fresh 15-digit numbers, half prime, with answers written to answers.csv. Ask your model whether each is prime and count confident wrong answers. For a full run, use inspect eval src/nullprime/llm_eval.py@nullprime -T condition=plain --model <provider/model>.
Cost and requirements
Code is MIT and data is CC-BY-4.0. The number generator needs only Python. The LLM evaluation uses Inspect, model API keys in .env, and Docker for the code-tool condition. CSV splits for 10-digit and 15-digit configs download directly from the repository.
Limits to know
Published numbers may end up in future training data, so the README recommends generating fresh ones and asks trainers to filter its canary string. The chance-level result holds for the given features (last digits and small-prime remainders). The author lists prior work showing the same effect; the contribution is a ready-made control and protocol. On Windows, Inspect 0.3.281's Google batch upload needed a local one-line fix.
Sources checked 2026-10-31. Source-based assessment.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.