Jinfer

Why it's ranked here

The case rests on scope backed by detail. The repository supports eighteen model families across chat, retrieval and speech, sixteen quantization and precision types, and hand-written matrix kernels for x86 from SSE3 to AVX-512, ARM NEON through I8MM, and Apple Metal, all under Apache 2.0 and published to Maven Central. Docs state operating limits precisely, down to which speculative-decoding workloads slow down. The headline claim, kernels competitive with llama.cpp, rests on benchmark charts and a methods page that were not part of what was read, so treat it as the project's own number until you reproduce it.

What's good

Structured output is enforced while tokens are generated, so a reply bound to a schema or a grammar cannot come back malformed, and the Python package accepts a plain dataclass or Pydantic class as that schema. The prompt cache can save a long system prompt to disk and reload it after a restart, and a cache miss falls back to recomputing rather than changing the answer. The local server binds to the loopback address by default, and the docs pair listening on any other interface with an API key. The model library is split per architecture, so an application can ship only the families it uses.

Tradeoffs

Jinfer needs JDK 25 and an incubator flag for the Java Vector API on every launch, which rules out older Java estates. The server runs one generation worker behind an admission gate, and generations on one engine are serialized by a lock, so it is a single-tenant tool, not a batching server for many users. The engine's own docs describe CPU kernels plus Metal on Apple; CUDA appears only in a separate tensor engine module. Version numbers are all below 1.0. Kokoro speech needs eSpeak installed, and the native binary needs ffmpeg to decode audio.

How to use it well

Pick it when a Java or Spring service should own its model in-process: retrieval with local embeddings and reranking, structured extraction, tool calling through LangChain4j or Spring AI, or offline transcription. Start with the small Liquid AI chat model from the examples, use the recommended 8-bit quantization, and add the native kernel module at runtime, falling back to the pure-Java one if native libraries cannot load. Set the offline switch in production so inference never checks upstream. For serving many concurrent users, or for GPU throughput outside Apple hardware, look elsewhere.

Observed

License

Apache License 2.0 (root LICENSE and pom.xml)

Language

Java; jinfer modules compile for Java 25, native jam kernels in C built with CMake

Packaging

Maven artifacts under com.qxotic with a BOM; per-architecture model modules plus an all-models aggregate

Interfaces

Java library, LangChain4j and Spring AI providers, CLI (fat JAR or GraalVM native executable), OpenAI-compatible HTTP server, C API shared library, Python package

Server endpoints

Chat completions, embeddings, rerank and audio transcriptions; default bind 127.0.0.1:54154, API key required for other hosts

Model format

GGUF checkpoints resolved from Hugging Face or other hosts, cached locally, with an offline mode

CPU and GPU support

x86 SSE3 to AVX-512-VNNI, ARM NEON, DotProd and I8MM, Metal on Apple Silicon; jam lists Linux, macOS and Windows

Testing

Default suite runs with no models or network; model-loading integration tests are opt-in and fixture-gated