Intel / repo

rf-detr 1.12.0 — v1.12.0: One prediction API and INT8 exports

Source: github.com
Author: Borda
Date: Published Nov 6, 2026

Context

The main change in Roboflow's RF-DETR 1.12.0 is that the model's own Python path now matches an exported TensorRT engine for single images on an NVIDIA L4. RF-DETR is a family of models that find and label objects in images, and TensorRT is NVIDIA's inference compiler, the usual route to the lowest latency on its GPUs. In this release the model's inference() method captures a CUDA graph on NVIDIA hardware: it records the sequence of GPU operations for one prediction and replays it, instead of having the CPU launch each operation separately every time. Measured end to end, including preprocessing and postprocessing, the small model took 3.60 ms per image this way on the L4, against 3.57 ms for an FP16 TensorRT engine. In a separate with the Nano model on the same GPU, the CUDA graph cut median latency from 9.719 to 3.884 ms with identical COCO accuracy. On an Apple M4 Max the method compiles the model with PyTorch's inductor compiler instead, and the small model went from 20.0 ms with uncompiled predict() to 10.3 ms.

Code written for 1.11 has to change to use it. inference() now returns a separate predictor object rather than modifying the model in place, so calls that pass inplace=True fail and predictions go through the returned predictor. Each predictor holds its own copy of the weights until it is deleted. The same predictor class, RFDETRInference, also loads exported ONNX, TensorRT and OpenVINO files, and the older TRTInference and OpenVINOInference classes are deprecated and scheduled for removal in v1.15.0.

The release also adds INT8 export, which trades a little accuracy for speed and size. It quantizes the model to 8-bit integers, using a folder of images supplied by the user. On an RTX 5070 and a Tesla T4, the release notes report INT8 TensorRT engines running 1.08 to 1.41 times faster than FP16 engines and taking 25% to 39% less space, at a cost of 0.27 to 1.01 points of AP, the standard detection accuracy score, at batch 1.

Key takeaways · AI-distilled

RF-DETR 1.12.0 runs live models, native checkpoints and exported models through one RFDETRInference API with shared preprocessing and decoding. It reads folders, video files, cameras and RTSP streams, and stream=True yields results as they arrive.
On an RTX 5070 and a Tesla T4, INT8 TensorRT engines ran 1.08x to 1.41x faster than FP16 and were 25% to 39% smaller, at a cost of 0.27 to 1.01 AP points at batch 1, per the release benchmarks.
RFDETR.inference() now captures a CUDA graph on CUDA. In one benchmark on an L4 with RF-DETR Nano at FP16 and batch 1, CUDA graph replay cut median end-to-end latency from 9.719 ms to 3.884 ms with identical COCO AP.
RFDETR.inference() now returns an independent predictor that holds its own copy of the weights, and the inplace keyword is rejected, so 1.11.x code that called inference(inplace=True) and then predict() on the model fails with no deprecation shim.
An Affine augmentation config without rotate no longer applies a random rotation of up to 15 degrees, which changes training data for existing configs. Keeping the old behavior requires rotate=(-15, 15); the built-in presets are unaffected.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.

Read the source github.com