- Category
- AI Tools
- Rank
- No. 297Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- google-ai-edge
- GitHub
- 3.4k stars
- Latest release
- v2.2.0
- Date
About
Google's on-device ML runtime — successor to TensorFlow Lite for running models on mobile, embedded, and edge hardware.
What it does
LiteRT takes converted and optimized machine-learning models, selects available acceleration, manages tensor memory, and executes inference locally. Its compiled-model interface supports asynchronous execution and shared GPU buffers, while companion runtimes extend deployment to browsers and generative models.
Why it's ranked here
The case rests on unusually broad hardware reach and serious runtime engineering. CPU support spans every listed platform, with GPU and selected NPU paths layered behind a consistent interface. Automated accelerator selection, asynchronous execution, explicit memory planning, and substantial tensor tests make it credible for production inference, though several advertised integrations remain separate projects or future work.
What's good
The runtime reduces hardware-specific integration through automatic accelerator selection and a unified NPU interface. It supports efficient input and output buffers, asynchronous execution, tensor quantization metadata, aligned allocations, packed numeric types, and safe arena reuse. Tests cover buffer ownership, alignment, type conversion, graph relationships, broadcasting, cloning, and shape propagation.
Tradeoffs
The ecosystem is split across separate converter, quantizer, language-model, browser, and command-line projects. Some accelerator entries, including Apple neural hardware, WebNN, Samsung silicon, and Raspberry Pi, are marked as coming soon. Source builds described here target Linux and Android through Docker, with CMake or Bazel underneath. Hardware coverage therefore varies materially by platform.
How to use it well
Choose LiteRT when an application must run prepared models locally across mobile, desktop, web, or embedded targets and benefit from available accelerators. Convert and quantize models upstream, validate behavior on the exact target hardware, then integrate the appropriate runtime library. Use the separate language-model runtime for generative workloads and the browser runtime for WebGPU applications. It does not replace model training, conversion, or quantization tooling.
Technical notes+
README.md describes the compiled-model API, accelerator matrix, Docker builds, related projects, and a nightly Python CLI package. configure.py generates .litert_configure.bazelrc and still contains extensive TensorFlow-era build variables and prompts. Package.swift exposes a Swift library for iOS 15 and macOS 12 using prebuilt CLiteRT XCFramework archives, plus litert/swift/Tests. ci/setup_with_binary.py packages Python modules with native .so or .pyd extensions and depends on flatbuffers, ml_dtypes, and NumPy. tensor/buffer.cc implements aligned owning CPU buffers; tensor/tensor.cc stores tensor metadata, buffers, shapes, types, and quantization state. tflite/arena_planner.cc plans arena lifetimes and guarded in-place buffer sharing. tensor/buffer_test.cc, tensor/tensor_test.cc, tensor/datatypes_test.cc, and tensor/arithmetic_test.cc provide focused C++ coverage.
Observed
- License
- Apache License 2.0
- Languages
- Core implementation shown in C++, with Python and Swift package surfaces
- Library interfaces
- Runtime interfaces are described for C++, Kotlin, and JavaScript
- Python packaging
- Native Python packages include shared-library extensions and depend on flatbuffers, ml_dtypes, and NumPy
- Swift packaging
- Swift Package Manager library supports iOS 15 and macOS 12 through prebuilt XCFramework binaries
- Build systems
- Source build guidance covers Docker-based Linux and Android artifacts using CMake or Bazel
- Platform support
- CPU support is listed for Android, iOS, Linux, macOS, Windows, Web, and IoT
- Testing structure
- Repository text includes dedicated C++ tests for buffers, tensors, datatypes, and arithmetic graphs
Read from README.md, configure.py, Package.swift, tensor/buffer.cc, tensor/tensor.cc, litert/__init__.py, tensor/buffer_test.cc, tensor/tensor_test.cc, ci/setup_with_binary.py, tflite/arena_planner.cc, tensor/datatypes_test.cc, tensor/arithmetic_test.cc.
What it can do
Run machine learning models on mobile devices
Trained ML model and mobile device → Model predictions and inferences
Execute ML inference on embedded hardware
ML model and embedded system → Real-time inference results
Optimize model performance for edge devices
ML model and target hardware specifications → Optimized model with reduced latency and memory usage
Convert TensorFlow models for on-device deployment
TensorFlow model → LiteRT-compatible model format
Process real-time data streams with ML models
Streaming data and deployed model → Continuous inference results
Enable offline ML inference
Local model and input data → Predictions without internet connectivity
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.