Index / tool

RelateAnything

Category: AI Tools
Rank: No. 1040Tools index
Pricing: Open Source
Type: TOOL
Use case: Models: Train & Run
GitHub: 1.2k stars
Date: Published Sep 15, 2026

About

RelateAnything is an open-vocabulary scene-graph model that predicts relations between detected objects in real time, using a predicate vocabulary supplied at inference rather than fixed labels. The 53-million-parameter model runs at about 20 ms per frame on an A40 GPU and tracks relations across frames with a Kalman filter so edges survive brief detection dropouts.

What it does

You hand it a photo, a set of object regions from any detector, segmenter or person, and a list of relation phrases you care about. It returns ranked subject, relation, object triplets such as person riding horse. A DINOv3 vision backbone reads the image once, a sampler trims every possible object pairing down to a fixed budget, and each surviving pair is compared against text embeddings of your phrases. Because the phrases are data rather than a trained output layer, changing them needs no retraining. It never sees object class names, and it does not find objects itself.

Why it's ranked here

The numbers are unusually well argued for a research release. Run through one shared evaluator against the OvSGTR baseline, the recommended checkpoint roughly doubles transfer F1 on three scene-graph datasets (36.9 against 16.5 on VG150) without being given the object labels the baseline receives. It ships as a pip-installable Python package with seven checkpoints, an ONNX export, TensorRT and OpenVINO paths, and a browser demo. The documentation openly lists the mistakes that once inflated its own results. The ceiling on adoption is licensing: AGPL code and weights derived from a separately licensed vision model.

What's good

The vocabulary really is runtime data. The exported graph takes the phrase embeddings and per-phrase weights as inputs, so a deployed app can switch which relations it looks for between frames with no text encoder on the device. One scoring formula is shared by evaluation, the Python API and the exported graph, after the project found five diverging copies. Release tooling refuses to borrow thresholds across checkpoints and writes a deliberate gap instead. Calibration is documented with its numbers: a two-parameter fit cut expected calibration error from 0.176 to 0.004, and it never changes rankings. Masks are accepted through the same code path as boxes.

Tradeoffs

Code is AGPL 3.0, so running a modified version as a network service obliges you to publish its source; a commercial license means contacting the author. Weights carry the backbone vendor's license and the bundled demo detectors are AGPL too. Raw scores cluster near 1.0 and mean nothing until calibrated, and thresholds do not survive a vocabulary change. The shipped bank holds 243 curated phrases; anything outside it must be re-encoded where the text encoder lives. The A40 timing of about 20 ms per frame is a compiled PyTorch figure that includes the detector. Cross-frame tracking with Kalman filters lives in the video rendering tool, not in the model.

How to use it well

Fit it into a pipeline that already has boxes or masks: robotics perception, video analytics, or dataset annotation where you want relations you can rename on the fly. Start with the recommended small checkpoint, install the Platt calibration before setting any threshold, and pick the relatedness weight for your question, since keeping it raises recall while dropping it improves truthfulness on verified negatives. Use the two-graph mode when layout and interaction matter separately. Budget for the detector: the docs measure missed detections, not box noise, as the main cause of the drop from lab boxes to real ones. It is not a detector, a tracker, or a captioning model.

Technical notes+

README.md documents the API (set_vocabulary, predict with decompose=True) and seven DINOv3 checkpoints from 46.1 M to 245.7 M parameters. docs/architecture.md describes the path: SoftSpatialPool and a 19-feature GeoEncoder feed a two-stage PairSampler (400 then 128 pairs, recovering 99.79 % of positives), a RelationTransformer, then a VocabHead computing cosine(proj(r), W) with no learned text projection; scores fuse as sigmoid(a*(pred + w*pair) + b). relsgg/model/__init__.py only re-exports RelSGG. pyproject.toml pins torch~=2.13 and transformers~=5.14, requires Python 3.12+, and gates ultralytics behind an optional detector extra. docs/deployment.md gives 20.3 ms per frame on an A40 with torch.compile and detector, and 137 ms fp16 OpenVINO on 8 CPU threads. docs/pitfalls.md and docs/evaluation.md record the graph-constraint inflation of 12 to 19 R@K points. deploy/README.md places the per-track and per-relation Kalman filters in the video reel renderer. LICENSE is AGPL-3.0.

Observed

License: AGPL-3.0 code; commercial license by request; weights under the DINOv3 license
Language: Python, requires 3.12 or newer
Packaging: pip install from source (package relsgg), with optional extras for deploy, tensorrt, detector, hub and gradio
Interfaces: Python library; ONNX graph; command-line webcam and image demo; browser demo on ONNX Runtime Web
Runtimes: PyTorch CPU or CUDA, TensorRT 10, ONNX Runtime, OpenVINO; Jetson stated as not validated
Inputs: Image plus boxes or masks; no built-in object detector
Checkpoints: Seven, from 46.1 M to 245.7 M parameters

Read from README.md, pyproject.toml, LICENSE, docs/architecture.md, docs/quickstart.md, docs/results.md, docs/pitfalls.md, docs/deployment.md, docs/evaluation.md, relsgg/model/__init__.py, deploy/README.md.

Tech Stack

Python

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.