LiteRT-LM
github.com/google-ai-edge/litert-lm- Category
- AI Tools
- Rank
- No. 181Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- google-ai-edge
- GitHub
- 6.4k stars
- Latest release
- v0.16.1
- Date
About
Google's production-ready inference framework for deploying Large Language Models on edge devices like smartphones, IoT devices, and wearables. Enables on-device AI without requiring cloud connectivity, with hardware acceleration support for GPUs and NPUs.
What it does
LiteRT-LM wraps model execution in a stateful conversation layer. A heavyweight engine holds model weights, while lightweight conversations manage history, prompt formatting, multimodal preprocessing, streaming responses, tool declarations, and parsed tool calls. It also supports constrained generation through regular expressions, JSON schemas, Lark grammars, or custom constraints.
Why it's ranked here
The case is strong because the framework combines practical application abstractions with broad integration choices. Its conversation layer handles difficult plumbing, including model-specific preprocessing and tool-call parsing. The documented deployment surface spans native, mobile, browser, scripting, and command-line use, while named deployments include Chrome, Chromebook Plus, and Pixel Watch. The maturity varies by language, which keeps the verdict positive but qualified.
What's good
The engine and conversation split lets applications load weights once and create lighter stateful sessions. Messages can mix text, images, and audio when the chosen model supports them. Responses can arrive synchronously or stream token by token. Tool declarations use JSON schemas, and constrained decoding can enforce valid structures. Prebuilt SDKs spare most application developers from compiling the C++ core.
Tradeoffs
The language surfaces do not share one maturity level. Swift and browser JavaScript are early previews, while Flutter is community supported. Building the core is substantially heavier than installing an SDK: Bazel is recommended, and the newer CMake route has complex dependency transformation. CMake verification is limited to a GNU toolchain on Debian-based Linux, with accelerated backend validation deferred. Engine initialization can also take several seconds.
How to use it well
Use it when an application needs stateful local conversations, streaming, multimodal input, structured output, or model-directed tool requests. Start with the prebuilt Python, Kotlin, Swift, or other supported SDK appropriate to the product. Use the command line for quick model trials and performance checks. Reserve source builds for core contributors or native developers needing custom embedded flags. Your application must still implement and execute any real-world tools requested by the model.
Technical notes+
README.md presents the litert-lm CLI, language APIs, supported platforms, and an OpenAI-compatible server exposed through the CLI. docs/api/cpp/conversation.md describes Engine as the heavyweight weight holder and Conversation as the lightweight stateful layer, with SendMessage and SendMessageAsync operating on ordered JSON messages. docs/api/cpp/constrained-decoding.md documents LLGuidance regex, JSON Schema, and Lark constraints plus externally supplied Constraint implementations. docs/api/cpp/tool-use.md assigns actual tool execution to application code. docs/litert_lm_builder.md documents the PyPI builder package and its archive building, inspection, and unpacking interfaces. docs/getting-started/build-and-run.md recommends Bazel for the C++ core, while docs/getting-started/cmake.md labels CMake newer and documents its Super-Build dependency normalization. Cargo.toml builds a Rust static library containing llguidance and parser support. Package.swift wraps prebuilt C libraries for iOS and macOS and defines separate Swift test targets.
Observed
- License
- Apache License 2.0
- Core implementation
- The source-build guide describes a C++ framework, with Rust used for parser and constraint-related components.
- Interfaces
- Command-line tool, native libraries, language SDKs, builder CLI and Python API, plus an OpenAI-compatible server through the CLI.
- Package surfaces
- Installable CLI through uv, Maven packages for Android and JVM, a Swift package using prebuilt binary targets, and a PyPI model-file builder.
- Platform support
- Android, iOS, web, Linux, macOS, Windows, JVM, and Raspberry Pi are documented.
- Build systems
- Bazel is recommended; CMake is newer, under active development, and verified for GNU tooling on Debian-based Linux.
- Language APIs
- Python, Kotlin, Swift, browser JavaScript, Flutter, and C++ APIs are documented.
Read from README.md, Cargo.toml, requirements.txt, __init__.py, src/lib.rs, docs/README.md, docs/litert_lm_builder.md, docs/getting-started/cmake.md, docs/getting-started/build-and-run.md, docs/api/cpp/tool-use.md, docs/api/cpp/conversation.md, docs/api/cpp/tool-use-antlr.md, docs/api/kotlin/getting_started.md, docs/api/cpp/constrained-decoding.md, Package.swift.
What it can do
Run large language models on smartphones
LLM model files and user prompts → AI-generated text responses
Deploy LLMs on IoT devices without internet
Pre-trained language models and device specifications → Local AI inference capabilities
Execute LLM inference on wearable devices
Compressed language models and user queries → Real-time AI responses on wearables
Accelerate AI processing using GPU hardware
LLM computations and GPU resources → Faster inference performance
Optimize LLM performance with NPU acceleration
Neural network operations and NPU hardware → Hardware-accelerated AI processing
Enable offline AI experiences across platforms
Language models and cross-platform deployment configurations → Platform-specific AI applications without cloud dependency
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.