ONNX Runtime
github.com/microsoft/onnxruntime- Category
- Developer Tools
- Rank
- No. 326Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- microsoft
- GitHub
- 21.9k stars
- Latest release
- v1.30.0
- Date
About
ONNX Runtime is a cross-platform inference and training engine for machine learning models exported to the ONNX format, accepting models originally built in PyTorch, TensorFlow/Keras, scikit-learn, LightGBM, or XGBoost. It applies graph optimizations and dispatches work to whatever hardware accelerators are available on the target, so the same model runs on server CPUs, GPUs, mobile, and edge devices. It also accelerates multi-node NVIDIA GPU training for transformer models.
What it does
ONNX Runtime executes interoperable ONNX model graphs, applying graph transforms and assigning operations to available execution providers. It handles inference across varied compute targets and can also accelerate transformer training on multi-node NVIDIA GPU systems.
Why it's ranked here
The case is strong for production inference that must span hardware and operating systems. Broad operator coverage, pluggable execution providers, explicit compatibility policy, and bindings for several languages make it credible infrastructure. Its complexity and uneven accelerator support demand careful validation.
What's good
It supports CPU, CUDA, and DirectML operator kernels, with documented operator, data type, and opset coverage. Public APIs follow semantic versioning, models remain compatible back through opset 7, and execution settings expose threading and logging controls.
Tradeoffs
Accelerator behavior is not uniform. CUDA supports only three standard quantization operators by default, while TensorRT has limited INT8 coverage. Official builds enable consent-dependent telemetry by default on supported platforms. The former server feature is deprecated and unsupported.
How to use it well
Choose it when you already have an ONNX model and need controlled inference across desktop, server, mobile, or accelerator targets. Benchmark each execution provider, verify required operators, and tune threading before quantizing. Use external exporters or converters for model creation, and another serving layer for network deployment.
Technical notes+
setup.py builds non-pure Python wheels around native bindings and selects package variants including onnxruntime-migraphx, onnxruntime-openvino, onnxruntime-dnnl, onnxruntime-tvm, onnxruntime-vitisai, onnxruntime-acl, onnxruntime-cann, onnxruntime-qnn, and onnxruntime-webgpu; its manylinux path rewrites native-library preload handling. pyproject.toml uses setuptools.build_meta, while requirements.txt declares FlatBuffers, NumPy, Packaging, and Protobuf. docs/OperatorKernels.md is generated from registered kernels and records provider-specific operator, type, and opset coverage. docs/Privacy.md documents build-time, environment, and binding-level telemetry controls. docs/Server.md marks ONNX Runtime Server deprecated and unsupported.
Observed
- License
- MIT License
- Packaging
- Python wheels use setuptools, with separate accelerator-specific package variants.
- Library interfaces
- C, C++, Python, C#, and Java bindings are documented.
- Platforms
- Windows, Linux, macOS, Android, and iOS are supported or listed with preview status.
- Architectures
- X64, X86, ARM64, and limited ARM32 support are listed.
- Execution providers
- Generated kernel documentation covers CPU, CUDA, and DirectML providers.
- Compatibility
- Public APIs follow Semantic Versioning, and ONNX opsets back through version 7 are supported.
Read from README.md, setup.py, pyproject.toml, requirements.txt, docs/FAQ.md, docs/Server.md, docs/Privacy.md, docs/Roadmap.md, docs/Model_Test.md, docs/Versioning.md, docs/ABI_Dev_Notes.md, docs/PR_Guidelines.md, docs/Android_testing.md, docs/OperatorKernels.md, docs/cmake_guideline.md.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.