- Category
- AI Tools
- Rank
- No. 1961Tools index
Previous survey · No. 1956 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- JuliusBrussee
- GitHub
- 121 stars
- Date
About
LoRA/QLoRA fine-tune of Gemma 4 31B that natively speaks caveman-mode — a style-transfer fine-tuning recipe using Unsloth.
What it does
Cavegemma makes a large language model answer with terse, fragment-heavy technical prose without relying on a persistent style prompt. It removes pleasantries, articles, filler, and hedging while attempting to preserve meaning, identifiers, commands, error strings, and fenced code. Users can load either complete merged weights or attach the smaller adapter to the base model.
Why it's ranked here
The project pairs a clear behavioral goal with unusually candid measurements and reproducible training machinery. Its holdout results show substantial savings for dialogue and questions, with strong semantic similarity and nearly exact fenced-code preservation. The value is narrower for debugging and refactoring, where outputs shrink only slightly. That makes Cavegemma a credible specialist experiment, not a general compression solution.
What's good
The evaluation separates dialogue, questions, debugging, and refactoring instead of hiding variation behind one average. Across 193 holdout pairs, outputs used 27 percent fewer tokens, semantic similarity ranged from 0.91 to 0.98, and fenced code was preserved exactly 96.3 to 100 percent of the time. The data filter also rejects mutated code fences, excessive article density, duplicates, and overlong examples.
Tradeoffs
Compression falls well short of the training target for long debugging and refactoring answers, saving only 8 percent in each category. The repository attributes this to a permissive training filter that admitted rewrites as long as their sources. Review coverage is too small to support a claim, workflow evaluation lacks reference answers, and vision or audio behavior remains untested. There are no published GGUF or AWQ builds.
How to use it well
Use Cavegemma when concise technical chat or question answering matters enough to justify hosting a 31B model or its adapter. Start with the adapter when download size matters, then run the supplied holdout metrics against your own response categories and code patterns. It fits embedded hosts and agent frameworks where prompt control is weak. It does not replace factual evaluation, multimodal validation, broad code-review testing, or deployment-oriented quantization.
Technical notes+
pyproject.toml defines a Python 3.11 to 3.12 project with core evaluation dependencies and a separate train extra for Torch, PEFT, TRL, bitsandbytes, Accelerate, and Weights & Biases. data/build_corpus.py, data/synthesize.py, data/filter.py, and data/split.py form the corpus pipeline; synthesis can drive Claude or Codex command-line tools, while filtering enforces compression bands, article density, deduplication, length, dialogue integrity, and byte-exact fenced code. training/train_unsloth.py loads Gemma in 4-bit mode, unwraps the text tokenizer from Gemma4Processor, attaches LoRA adapters, trains with completion-only loss, resumes checkpoints, and saves evaluation metrics. eval/metrics.py computes token compression, article density, fenced-code matching, and MiniLM cosine similarity. eval/run_eval.py generates deterministic adapter outputs and applies category gates, while eval/judge.py optionally uses the Claude CLI for sampled judgments. scripts/infer.py provides a GPU smoke test, and scripts/push_to_hub.py uploads an adapter and generated model card.
Observed
- Repository license
- MIT, declared in pyproject.toml and the README badge.
- Base-model terms
- The generated model card says the adapter inherits the Gemma Prohibited Use Policy.
- Primary language
- Python.
- Python support
- Python >=3.11 and <3.13.
- Install surface
- pyproject.toml defines core dependencies, a GPU training extra, and uv development dependencies.
- Interfaces
- Python command-line scripts for corpus building, synthesis, filtering, training, inference, evaluation, judging, and Hub upload.
- Published formats
- A 62.5 GB bf16 merged model and a 534 MB LoRA adapter are linked from the README.
- Unavailable formats
- The README states that GGUF and AWQ builds are not provided.
Read from README.md, pyproject.toml, data/split.py, eval/judge.py, data/filter.py, eval/metrics.py, eval/run_eval.py, scripts/infer.py, data/synthesize.py, data/build_corpus.py, data/build_prompts.py, data/extract_seeds.py, scripts/push_to_hub.py, training/train_unsloth.py, data/sources/_util.py.
What it can do
Convert modern text to caveman speech style
Standard English text or prompts → Text written in caveman-style language
Generate responses in caveman dialect
User questions or conversation prompts → AI responses using primitive vocabulary and grammar
Perform language model inference with style transfer
Natural language queries → Contextually relevant responses in caveman communication style
Fine-tune language models using LoRA/QLoRA techniques
Base Gemma model and training data → Specialized model with caveman speech capabilities
Apply style transfer fine-tuning recipe
Source text dataset and target style parameters → Fine-tuned model optimized for specific communication style
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.