- Category
- Education
- Rank
- No. 459Tools index
Previous survey · No. 494 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run · Research & Education
- Builder
- jingyaogong
- GitHub
- 8.7k stars
- Latest release
- v2
- Date
About
Train a 65M-parameter vision-language model from scratch in just 2 hours — readable, didactic implementation for learning VLM internals.
What it does
MiniMind-V adds image understanding to a small language model. A frozen SigLIP2 encoder turns each image into 64 visual tokens, then a normalization and two-layer projection maps them into the language model’s hidden space. Those vectors replace image placeholders in the text sequence, after which ordinary autoregressive generation produces the answer.
Why it's ranked here
Its strongest case is transparency: the repository exposes model structure, Parquet data loading, pretraining, supervised fine-tuning, evaluation, checkpoint recovery, conversion, and a web interface. The documentation also corrects its own shorthand. The language backbone starts from existing MiniMind weights, and the frozen vision encoder raises dense inference size to about 160 million parameters.
What's good
The project makes cross-modal alignment concrete instead of hiding it behind a framework. It shows how patch features become language-compatible tokens and offers three freezing strategies: projection only, projection plus the first and last language layers, or all non-vision parameters. Training supports distributed execution, mixed precision, optional compilation, gradient clipping, experiment logging, and resumable checkpoints with atomic replacement.
Tradeoffs
This is not a fully independent pretraining stack. The default workflow requires separate MiniMind language weights, a separately downloaded SigLIP2 encoder, and external Parquet datasets. The vision encoder stays frozen and expects fixed 256 by 256 input with 32-pixel patches. The published two-hour claim covers one supervised fine-tuning epoch on a single NVIDIA 3090, not complete language and vision pretraining.
How to use it well
Use it to learn or modify the mechanics of a compact vision-language training pipeline, especially token replacement, projection alignment, freezing policies, and checkpointed fine-tuning. Start with released weights for inference, then run direct supervised fine-tuning; add projector pretraining only when you want a separate alignment stage. It does not replace a general model-serving platform or a strict end-to-end foundation-model pretraining recipe.
Technical notes+
model/model_vlm.py subclasses the MiniMind causal model, loads SiglipVisionModel, freezes its parameters, and projects 64 encoder states through LayerNorm, Linear, GELU, and Linear before replacing placeholder embeddings. dataset/lm_dataset.py reads image bytes and conversations from Parquet, applies chat templates, and masks labels outside assistant spans. trainer/train_pretrain_vlm.py defaults to projector-only training, while trainer/train_sft_vlm.py defaults to training the projector plus the first and last language layers; trainer/trainer_utils.py handles DDP setup, skipped batches during resume, and atomic checkpoint replacement. eval_vlm.py accepts native PyTorch or Transformers-format models, scripts/convert_vlm.py converts between those formats, and scripts/web_demo_vlm.py provides streaming Gradio chat with model discovery and switching.
Observed
- License
- Apache License 2.0
- Primary language
- Python
- Installation
- Clone the repository and install pinned dependencies from requirements.txt with pip
- Interfaces
- Command-line training and evaluation scripts, plus a Gradio web chat interface
- Model formats
- Supports native PyTorch checkpoints and Transformers-format model directories
- Training data
- Reads image bytes and conversations from Parquet datasets
- Acceleration
- Supports PyTorch DistributedDataParallel, bfloat16 or float16 mixed precision, and optional torch.compile
Read from README.md, requirements.txt, eval_vlm.py, model/model_vlm.py, dataset/lm_dataset.py, scripts/convert_vlm.py, model/model_minimind.py, scripts/web_demo_vlm.py, trainer/train_sft_vlm.py, trainer/trainer_utils.py, trainer/train_pretrain_vlm.py, LICENSE.
What it can do
Train a vision-language model from scratch
Training dataset with images and text → 65M-parameter vision-language model
Provide educational implementation of VLM architecture
User studying machine learning → Readable, didactic code demonstrating VLM internals
Execute rapid model training
Training configuration and data → Trained model in approximately 2 hours
Demonstrate vision-language model fundamentals
Learning objectives for VLM understanding → Working example of multimodal AI system
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.