Ollaya
ollaya.dev- Category
- Developer Tools
- Rank
- No. 1009Tools index
Previous survey · No. 1029 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- Interfaces
- API
- GitHub
- 1.5k stars
- Latest release
- v0.18.0
- Date
About
Ollaya is a local server that downloads and runs small open-weight "decision models" (such as Laya, Decider, NLI classifiers) to answer typed choice, score, and yes/no questions about text or JSON in a single forward pass, in milliseconds, entirely on the user's own hardware. It exposes an API compatible with TypeSafe's hosted decision-model service, so the existing TypeSafe SDK can point at a local Ollaya server instead of the cloud API, and its published benchmarks show much lower latency than the hosted alternative.
What it does
Ollaya is Ollama for classifiers. You hand it a customer message, ticket or JSON blob plus a set of typed questions (pick a label, rate on a scale, yes or no), and a pulled model returns a probability for every option without writing any prose. One binary acts as command line, background daemon and per-model worker process. It mirrors the TypeSafe wire format, so clients built for that hosted service switch over by changing one environment variable, and it also offers its own richer HTTP interface plus an MCP server for coding agents.
Why it's ranked here
A narrow job done with unusual rigor. Apache 2.0, written in Rust, and every model is checked question by question against its authors' reference code on each device before it ships; the README reports fp32 exports matching the PyTorch reference on all 2,383 questions per checkpoint. The catalogue is wide, over twenty model families from encoders near 120M parameters up to 27B decoders, and the HTTP contract is a written spec with examples checked by the test suite. The cost is a heavy install surface and a project still on a 0.x version.
What's good
Weights are never rehosted: the project publishes small graphs, about 3 MB each, that read the author's original files pinned to a commit and verified by sha256. Errors carry a stable machine-readable code alongside a plain string, shaped so Ollama clients, the TypeSafe SDK and scripts all parse them. The API refuses to download models implicitly, because a pull of hundreds of megabytes would outlive the SDK's 10 second timeout. Request bodies cap at 8 MiB and state at 65,536 tokens, both with named errors. Design decisions are written up with the measurements behind them, including the options that lost.
Tradeoffs
GPU support is large: the Linux CUDA 13 pack is about 1.1 GB compressed and 2 GB unpacked, and the Windows CUDA 12 zip sits close to GitHub's 2 GiB asset limit. Linux needs glibc 2.38 or newer and the host's OpenMP runtime for GGUF models. On a Mac the ONNX models run on the CPU only; GGUF models get Metal, and the Apple GPU engine for the rest is off by default and in no release. GGUF models answer one question at a time, sequentially. Model accuracy varies a lot, from 0.357 to 0.722 on the project's own typed-decision benchmark, so model choice matters more than the runtime.
How to use it well
Use it where a pipeline needs fast, repeatable labels: routing support tickets, flagging urgency or churn risk, moderating inputs, gating agent actions. Start with the English encoder on a CPU and move to the 4B Gemma fine-tune when you have an NVIDIA card. Save recurring question sets as presets so the command line, the API and the MCP server share them. Set an API key before exposing the port beyond localhost. It does not generate, summarize or embed text; its server answers Ollama's text routes with a not-found error pointing at the decision route.
Technical notes+
Cargo.toml defines a Rust 2024 workspace (rust-version 1.90, version 0.12.1, Apache-2.0) of eight crates under crates/, with the Tauri desktop app excluded; ONNX Runtime comes through ort pinned exactly at 2.0.0-rc.13, MCP through rmcp, and tokenizers uses the pure-Rust fancy-regex backend to avoid C dependencies. crates/ollaya/src/main.rs is one clap CLI whose hidden Runner subcommand is the per-model subprocess (ONNX graphs at fp16/fp32, a GGUF path with llama_dir, or MLX arch/weights) and whose comment states the CLI and daemon never load ONNX Runtime in-process. docs/api.md is the normative HTTP contract: /api/decide, /api/pull NDJSON streams, /v1/systemone and /v1/decisions wire-identical to TypeSafe, a sibling code field on every error, and 503 QUEUE_FULL past OLLAYA_MAX_QUEUE. docs/decisions/0003-llama-cpp-runtime.md picks libllama loaded via libloading from ggml-org's release binaries (b11146, sha256-pinned) with a thin FFI of about 30 functions, gated per device against stock llama-server. docs/decisions/0001-mlx-engine.md keeps the MLX engine behind a non-default cargo feature pending a macOS 14.0 minimum. docs/distribution.md lists deterministic tarballs and CUDA packs reused on upgrade when their FILES.sha256 matches. site/docs/cli.md documents server defaults such as 127.0.0.1:11435, five-minute keep-alive and three loaded models.
Observed
- License
- Apache-2.0
- Language
- Rust workspace of eight crates, edition 2024
- Interfaces
- CLI, HTTP daemon (native and TypeSafe-compatible routes), MCP server over stdio or HTTP
- Platforms
- Linux x86_64 and arm64, macOS on Apple silicon, Windows x64, Docker images for CPU and CUDA
- Install surface
- Shell and PowerShell install scripts, desktop app, GHCR container images
- Inference backends
- ONNX Runtime on CPU and CUDA; llama.cpp for GGUF models on CPU, CUDA, Vulkan and Metal
- Default bind address
- 127.0.0.1:11435, optional bearer API key
- Weights hosting
- Upstream author weights pinned to a commit and verified by sha256, not rehosted
Read from README.md, Cargo.toml, LICENSE, docs/api.md, site/docs/cli.md, docs/distribution.md, docs/decisions/0001-mlx-engine.md, docs/decisions/0003-llama-cpp-runtime.md, crates/ollaya/src/main.rs.
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.