- Category
- AI Tools
- Rank
- No. 1475Tools index
Previous survey · No. 1431 ·
- Listed in
- #6 Run models locally
- Pricing
- Open Source
- Type
- TOOL
- Builder
- docusealco
- GitHub
- 92 stars
- Date
About
Ruby FFI bindings for llama.cpp to run open-source LLMs like GPT-OSS, Qwen, Gemma, and Llama 3 locally with Ruby.
What it does
Rllama gives Ruby programs a local model runtime with two entry points: an interactive terminal chat and an application library. It loads GGUF models from disk or downloads them, then supports text generation, streaming, conversation history, embeddings, reasoning output, and structured tool calls.
Why it's ranked here
The scope is unusually complete for Ruby developers who want local inference. Model discovery and resumable downloads reduce setup work, while chat templates, sampling metadata, cache reuse, embeddings, and parsed tool calls cover more than basic completion. The native dependency and explicit resource cleanup keep it from feeling entirely Ruby-native.
What's good
It preserves model-specific behavior by reading sampling defaults and chat templates from GGUF metadata. Generation reports timing, throughput, token counts, seed, and resolved sampling values. Stateful conversations reuse the shared token prefix after history edits. Tool-capable templates produce structured calls, and embedding batches return normalized vectors by default.
Tradeoffs
The design depends on native llama.cpp libraries and a C++ compatibility shim, which adds platform and binary compatibility concerns. Models must fit local storage and available compute, with curated downloads ranging from hundreds of megabytes to tens of gigabytes. Callers must close models and contexts. Streaming reasoning exposes raw reasoning tokens rather than only final text.
How to use it well
Choose it for Ruby applications that need private, local generation, conversational state, embeddings, or model-driven tool selection. Start with the terminal chat to test a GGUF model, then move proven prompts into the library and retain a context for multi-turn work. It does not provide hosted inference, remote scaling, or tool execution itself.
Technical notes+
lib/rllama.rb exposes the public module and a Result struct. lib/rllama/model.rb owns model loading, metadata defaults, tokenization, generation, embeddings, and cleanup. lib/rllama/context.rb handles decoding, streaming, message history, KV-cache rollback, batching, reasoning parsing, and structured tool calls. lib/rllama/loader.rb resolves local files, direct URLs, and Hugging Face GGUF paths, with resumable downloads and size verification. lib/rllama/cpp.rb declares the llama.cpp FFI surface and chooses native libraries by OS, architecture, and libc. lib/rllama/common.rb binds a native helper implemented in ext/llama_common.cpp for chat templates, parsers, grammars, and sampling. Rakefile pins a llama.cpp build and defines platform gem assembly.
Observed
- License
- MIT
- Primary language
- Ruby, with a C++ native shim and FFI bindings
- Packaging
- Ruby gem installable through Bundler or the gem command
- Interfaces
- Ruby library and interactive command-line chat
- Model format
- GGUF models from local files, direct URLs, or Hugging Face paths
- Platform support
- Native library selection covers macOS, Windows or MinGW, and GNU or musl Linux
- Capabilities
- Text generation, token streaming, stateful chat, embeddings, reasoning separation, and structured tool calls
Read from README.md, Gemfile, lib/rllama.rb, lib/rllama/cli.rb, lib/rllama/cpp.rb, lib/rllama/model.rb, lib/rllama/common.rb, lib/rllama/loader.rb, lib/rllama/context.rb, lib/rllama/version.rb, ext/llama_common.cpp, licenses/LICENSE-llama.cpp, LICENSE, Rakefile, .rubocop.yml.
What it can do
Run large language models locally
Open-source LLM model files (GPT-OSS, Qwen, Gemma, Llama 3) → Local LLM inference capability
Generate text responses from prompts
Text prompts or questions → AI-generated text responses
Execute LLM inference through Ruby code
Ruby code with llama.cpp FFI bindings → LLM predictions and completions
Load and initialize language models
Model files in llama.cpp compatible format → Ready-to-use model instance
Process natural language queries
Natural language questions or instructions → Contextual AI responses
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.