Trinity-RFT
https://github.com/agentscope-ai/trinity-rft- Category
- AI Tools
- Rank
- No. 796Tools index
Previous survey · No. 788 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- Interfaces
- CLI
- Builder
- agentscope-ai
- GitHub
- 710 stars
- Latest release
- v0.6.0
- Date
About
General-purpose, scalable framework for reinforcement fine-tuning of large language models, with RLHF support.
What it does
Trinity-RFT splits model training into three coordinated parts. An explorer collects agent interactions, a trainer updates weights, and a buffer processes experience data. It supports online, offline, synchronous, asynchronous, on-policy, and off-policy workflows, plus multi-step agent training.
Why it's ranked here
The breadth is credible because the repository documents many training modes, algorithm implementations, agent workflows, data pipelines, benchmarks, and backend options. The modular split also gives researchers clear extension points. However, its alpha classification, substantial dependency surface, and specialized infrastructure make it a serious engineering commitment.
What's good
Rollout generation and weight training can run separately and scale independently. The data layer covers prioritization, cleaning, augmentation, replay, task mixing, and human participation. Researchers get implementations and examples for PPO, GRPO, DPO, SFT, CHORD, REC variants, distillation, and several other methods.
Tradeoffs
The base installation pulls in distributed computing, experiment tracking, databases, web frameworks, model tooling, and GPU monitoring. Python support is limited to versions 3.10 through 3.12. Several capabilities require optional backend packages, while Megatron guidance recommends source installations for parts of that stack.
How to use it well
Use it for research teams or agent developers who need configurable experiments spanning rollout collection, data processing, and weight updates. Start with a documented example, then customize algorithms or pipeline modules. It does not replace a general production model-serving platform or remove the operational work of distributed training.
Technical notes+
pyproject.toml defines a setuptools package named trinity-rft, requires Python >=3.10,<3.13, installs the trinity CLI through trinity.cli.launcher:main, and exposes optional extras for vLLM, SGLang, data processing, AgentScope, Tinker, multimodal work, and Megatron. trinity/cli/server.py adds a Flask GET endpoint that starts training from a configPath query parameter and runs with debug mode enabled. docker/mcp_server.py provides a stdio FastMCP server with sync_code, run_tests, and check_status tools for remote Docker and GPU test operations over SSH. trinity/buffer/__init__.py exports buffer reader and writer interfaces, while trinity/manager/__init__.py exports state management and synchronization classes. setup.py delegates package configuration to setuptools.
Observed
- License
- Apache Software License
- Primary language
- Python 3 only
- Python support
- Python 3.10, 3.11, and 3.12
- Packaging
- Setuptools package published under the name trinity-rft
- CLI interface
- Installs a trinity command
- Additional interfaces
- Includes a Flask HTTP training endpoint and a stdio MCP utility for remote testing
- Documentation
- Includes English and Chinese documentation sources
Read from README.md, setup.py, pyproject.toml, docs/README.md, docs/sphinx_doc/source/main.md, docs/sphinx_doc/source_zh/main.md, trinity/__init__.py, docker/mcp_server.py, trinity/cli/server.py, trinity/perf/__init__.py, trinity/buffer/__init__.py, trinity/manager/__init__.py, trinity/plugins/__init__.py.
What it can do
Fine-tune large language models using reinforcement learning
Pre-trained language model and training data → Fine-tuned language model
Implement reinforcement learning from human feedback (RLHF)
Language model and human feedback data → Human-aligned language model
Scale training across multiple compute resources
Training configuration and compute cluster → Distributed training job
Configure reinforcement learning training parameters
Training hyperparameters and policy settings → Training configuration file
Monitor training progress and metrics
Active training session → Training metrics and performance data
Apply reward modeling for preference learning
Model responses and preference rankings → Reward model for policy optimization
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.