- Category
- Developer Tools
- Rank
- No. 875Tools index
Previous survey · No. 2063 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run · Agent Building
- Builder
- @ant_oss
- GitHub
- 5.8k stars
- Latest release
- v2.1.0
- Date
About
AReaL is a large-scale asynchronous reinforcement learning system that connects foundation model training to agent-based applications, letting researchers run RL on LLM agents without building the distributed training plumbing themselves. It is Apache 2.0 licensed and ships documentation, benchmarks, examples, notebooks, and a Dockerfile, so a training run can be reproduced rather than reassembled from scratch.
What it does
AReaL trains language models with reinforcement learning (RL) while they act, not only while they answer. Generation and weight updates run out of step with each other, so the GPUs producing rollouts never sit idle waiting for a training step to finish. The newer design splits training, inference, agent hosting and weight syncing into separate services. An existing agent can be pointed at its RL endpoint by swapping the model address and key, and its traffic becomes training data. Ready recipes cover math, multi-turn math, countdown puzzles, coding agents, search agents and customer service.
Why it's ranked here
The case rests on breadth and reproducibility. The algorithm menu is long (GRPO, PPO, DAPO, REINFORCE, RLOO, GSPO and more), training runs on both FSDP and Megatron backends, and inference runs on SGLang or vLLM. The project reports a 2.77x speedup over synchronous systems at comparable or better quality, a self-reported figure from its own paper. Outside teams have trained agents on it, and it is Apache 2.0 licensed. Against that, the package metadata still declares itself pre-alpha.
What's good
The quickstart is short and concrete: one command trains a small model on a grade-school math set on a single node, downloading the data and model itself, and the same command scales to a Ray cluster by changing a few config values. The math reward checker enforces its timeout in a thread-safe way, avoiding signal-based timeouts that only work on the main thread. Heavy modules load lazily, so importing the package does not drag in every engine. A container recipe builds the full CUDA stack in a separate stage and ships a leaner runtime image.
Tradeoffs
This is heavy infrastructure. Core dependencies include Ray, Redis, LangChain, OpenHands, LiteLLM, two agent SDKs and three experiment trackers, plus pinned pre-release builds of the config libraries. Training extras target Linux on x86 with NVIDIA CUDA 12, and Python is limited to 3.11 and 3.12. SGLang and vLLM need conflicting torch versions, so switching backends means swapping in a separate project file and lockfile. The container pulls some tools at their latest version through piped install scripts, so two builds may differ. Speed claims such as industry-leading come from the project itself.
How to use it well
It fits a team with multi-GPU NVIDIA Linux machines that wants to run RL on reasoning or tool-using agents and compare algorithms without writing the rollout and weight-sync plumbing. Start with the single-node math recipe to confirm the stack installs, then move to the agentic examples. Install the prebuilt flash-attention wheel first, as the README says, to avoid compiling from source. Pick SGLang or vLLM up front, since they live in different lockfiles. It is not a fit for laptop experiments, macOS training or teams that only need supervised fine-tuning.
Technical notes+
pyproject.toml declares Apache-2.0, Python >=3.11,<3.13, Development Status 2 Pre-Alpha, and a console script areal mapped to areal.v2.cli.main:cli (areal/v2/cli/main.py just imports and calls it). The cuda extra bundles cuda-train (tms, megatron, kernels) with sglang; vLLM lives in a separate vLLM-specific project file because sglang and vllm pin incompatible torch/torchao versions, and override-dependencies force openai>=2.8.0 and torchao==0.15.0 over sglang's pins. megatron-bridge is sourced from a third-party fork's release wheel. areal/__init__.py exports RolloutController, StalenessManager, TrainController and WorkflowExecutor and lazily exposes DPOTrainer, PPOTrainer, RWTrainer and SFTTrainer from areal/trainer/__init__.py. areal/engine/__init__.py lazily maps FSDP and Megatron engines plus RemoteSGLangEngine and RemotevLLMEngine. areal/reward/__init__.py wraps math_verify parse/verify with parsing_timeout=None and enforces a 5 s default timeout via a single-worker ThreadPoolExecutor, returning 0.0 on timeout or exception. The Dockerfile is two-stage on an sglang runtime base, compiles apex, TransformerEngine, DeepGEMM, DeepEP and FlashMLA at pinned commits, then installs openclaw@latest and a zeroclaw latest-release script via curl | bash.
Observed
- License
- Apache-2.0 (declared in pyproject.toml and SPDX headers)
- Language
- Python, requires >=3.11,<3.13
- Declared maturity
- Development Status :: 2 - Pre-Alpha classifier in pyproject.toml
- Platform
- Classifiers declare POSIX Linux and NVIDIA CUDA 12; CUDA extras are restricted to linux x86_64
- Interfaces
- Python library (trainers, engines, workflows) plus an areal console script
- Inference backends
- SGLang by default; vLLM via a separate pyproject and lockfile
- Training backends
- FSDP and Megatron engines exported from areal/engine/__init__.py
- Packaging
- uv build backend; install via git clone and uv sync --extra cuda; Dockerfile with sglang and vllm variants
- Unpinned container tools
- Dockerfile installs openclaw@latest and a latest-release install script piped to bash
Read from README.md, pyproject.toml, Dockerfile, areal/__init__.py, areal/trainer/__init__.py, areal/trainer/ppo/__init__.py, areal/workflow/__init__.py, areal/engine/__init__.py, areal/v2/cli/main.py, areal/api/__init__.py, areal/reward/__init__.py, docs/requirements.txt.
Intel on AReaL
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.