jev-align
github.com/sutro-sh/jev-align- Category
- Developer Tools
- Rank
- No. 1153Tools index
Previous survey · No. 1144 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- Interfaces
- CLI
- GitHub
- 315 stars
- Latest release
- v0.1.7
- Date
About
jev-align is an experimental CLI from Sutro for building calibrated AI classifiers on top of TypeSafe's Jev model. It surfaces the most uncertain predictions for a human to label, then uses the GEPA optimization method to iteratively refine the underlying classification function from that accumulated feedback.
What it does
A Python command-line tool for turning a yes/no, pick-one, pick-several or rubric-scoring question into a reusable text classifier. You point it at a CSV, Parquet or JSONL file, it runs every row through Jev, a hosted judging model, and shows you the rows it is least sure about plus a random spot check. Your answers and optional reasons feed GEPA, a prompt-evolution optimizer, which rewrites the task wording. You see the before and after score and a diff, then accept or reject. The result can be called from Python in production or published to a public registry.
Why it's ranked here
The loop is honest about what a number means. A better training score never auto-accepts a rewrite, a held-out split is reported separately, and the label-free squash mode is labelled dangerous because it can win by giving every row the same answer. Apache 2.0, installable from PyPI, with tests covering metrics, data loading and row selection. Against that: it is alpha, published at version 0.1.6, and every evaluation depends on a paid Jev provider.
What's good
Concrete safety choices run throughout. The production recorder keeps only uncertain calls plus a small audit sample, caps each record and file, and drops rather than blocks when its queue fills, so logging never slows the request path. Cached predictions are discarded whenever the task wording, backend, columns or source file hash change. Publishing requires an explicit public-data confirmation, excludes unlabeled rows and credentials, and pulls verify a SHA-256 digest. The bundled agent guide tells coding assistants never to invent labels or accept a rewrite for the user.
Tradeoffs
You need two sets of credentials: one for Jev through TypeSafe, Vercel AI Gateway or Cloudflare Workers AI, and one for the reflection model that writes rewrites. Each guided round asks for only 5 to 20 labels, and the training score is computed on those same accumulated labels, so it is not a generalization estimate unless you turn on the held-out split. The public registry has no private or team option, and unpublishing cannot recall copies already pulled. Interactive installs check PyPI for updates at startup unless disabled.
How to use it well
Best for a team with a few hundred to a thousand unlabeled rows and a fuzzy category definition they want sharpened, such as routing support tickets or grading agent traces. Pick explicit columns rather than concatenating everything, so IDs and timestamps do not distract the judge. Turn on the held-out split before trusting the score. Use capture in production, then resume and label what it caught. Do not use squash mode for anything real, and skip publishing if your labeled rows are sensitive. It does not train a model of your own.
Technical notes+
pyproject.toml declares hatchling, Python >=3.11, Apache-2.0, console scripts (jeva and jev-align) bound to jev_align.cli:app, and pins gepa[full] <0.2 plus typesafe-sdk. src/jev_align/jev.py wraps AsyncTypeSafeClient with model jev-1.13.0, a semaphore of 16, and 4 retry attempts with jittered backoff on transient errors. src/jev_align/capture.py runs a daemon writer thread with a 256-slot queue, 64 KiB record cap, 64 MiB file cap, 0.8 ambiguity threshold and 5 percent audit rate. src/jev_align/session.py invalidates pool caches on candidate, backend, source_sha256 or column changes. src/jev_align/optimizer.py pins one class-aware GEPA minibatch per run so F1 stays comparable. tests/test_core.py asserts four ambiguous rows plus one random pick.
Observed
- License
- Apache 2.0 (LICENSE, pyproject.toml)
- Language
- Python, requires 3.11 or newer
- Install
- PyPI package jev-align via uv tool install or pip
- Interfaces
- CLI (jeva / jev-align) and a Python library for loading saved functions
- Development status
- Alpha classifier in pyproject.toml
- Evaluation backends
- TypeSafe, Vercel AI Gateway, Cloudflare Workers AI
- Tests
- tests/test_core.py covers data loading, metrics and acquisition
- Input formats
- CSV, Parquet, JSONL
Read from README.md, pyproject.toml, LICENSE, AGENTS.md, src/jev_align/cli.py, src/jev_align/jev.py, src/jev_align/optimizer.py, src/jev_align/runtime.py, src/jev_align/session.py, src/jev_align/capture.py, tests/test_core.py.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.