Vibeleaderboard
← All Intel
Intel / post

Which small model actually runs well on an iPhone 17 Pro

Source
x.com
Date
Artificial Analysis@ArtificialAnlys

Which model should you run on the iPhone 17 Pro? Announcing our new intelligence and inference testing for small models on mobile devices: independent measurement of how capable small models are in typical on-device tasks, and how they perform on popular phones - in partnership with @liquidai We have partnered with @liquidai to deliver mobile device inference benchmarking, covering a range of models in 4-bit or lower precision on the iPhone 17 Pro and Galaxy S26 Ultra We’re publishing our phone-scale intelligence evaluation results in combination with @liquidai's inference performance benchmarks to give users and developers a holistic view of how small models are performing on phones Inference benchmarking is conducted in a controlled environment using Liquid AI’s inference benchmarking software, which Artificial Analysis has examined and is open-sourced on Liquid AI’s GitHub. Results cover end-to-end generation time, output speed, peak memory usage and other metrics. The inference benchmarking app, ‘Pipette’, is available to download for free on iOS and Android, allowing users to test a variety of models on their own devices We are ranking phone-scale model intelligence…

Read the full post on X

Context

On an iPhone 17 Pro, Artificial Analysis's new small-model benchmark puts Liquid AI's LFM2.5-2.6B level with the top scorer and far ahead of it on waiting time. In its August 24 results, LFM2.5-2.6B and Nanbeige4.2-3B share the best average score, 63, across five tests chosen for the work phone models do: (BFCL), instruction following (IFBench), factual reliability (AA-Omniscience), science questions (GPQA Diamond) and math (MATH-500). In a standard speed test, a 1,024- prompt and a 256-token reply, LFM2.5-2.6B took 8.0 seconds and 2.3 GB of memory; Nanbeige4.2-3B took 21.4 seconds and 4.0 GB, and the two 9-billion-parameter models just behind took over 25 seconds and 6.9 GB. Those times come from a fixed generation test, not from solving the capability questions.

The ranking rests on a deliberate limit. Artificial Analysis caps each model at 16,000 tokens of , because a phone lacks the memory to store a long conversation's working state, and it counts only models that fit in 8 GB after compression to 4 bits or fewer. That cap penalizes models that think at length. Qwen3.5 9B in reasoning mode hit the limit on 29% of its answers and finished fourth; with a 64,000-token cap, Ling 3.0 Tiny moves to first at 66. The evaluator argues the larger cap is impractical on a phone: at about 55 tokens per second, 64,000 tokens could take more than 20 minutes and a lot of battery, so it also publishes 64K and one-minute versions alongside the 16K headline.

One caveat bears on the headline result. The speed measurements use Liquid AI's own open-source benchmarking software, which Artificial Analysis says it examined, and Liquid AI is both the partner and the maker of the fastest top scorer. The scores and timings are still the most direct comparison on offer, and the post also credits LFM2.5-2.6B with the best instruction-following score on the iPhone and 79% non- on AA-Omniscience against 33% for Nanbeige4.2-3B. For anyone choosing a model to ship on a phone, the result says similar measured ability can come with very different waits, so speed and memory belong in the decision.

Terms in this piece · Glossary
  • tool use — A model's ability to call external functions — run code, search the web, edit files — instead of only generating text.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
More from Artificial Analysis
Recommended reads
Comments

Checking sign-in…

Loading comments…