Liquid AI Releases Pipette to Benchmark Mobile AI Models

Liquid AI, in collaboration with Artificial Analysis, has released Pipette, an Apache 2.0-licensed benchmarking suite designed to evaluate foundation models directly on edge hardware. Instead of testing models on server-class hardware, Pipette measures performance based on the specific combination of model, quantization, runtime, and device. The launch includes native iOS and Android apps, an interactive leaderboard, and a dataset containing over 10,000 results across more than 30 models, seven quantization formats, and devices like the iPhone 17 Pro, Galaxy S26 Ultra, and MacBook Pro with an M5 Max chip.

To ensure reproducibility, Pipette employs a strict protocol using greedy decoding, discarded warm-up runs, five repetitions, and thermal readiness gates to prevent hardware throttling from skewing results. The suite pairs these runs with quality evaluations using IFBench, GPQA Diamond, and MATH-500 across context lengths from 256 to 8,192 tokens. The architecture is split into pipette-mgmt for managing definitions, pipette-clients for running on-device inference, and pipette-scores for model-blind grading.

The initial data reveals how drastically model performance degrades on mobile hardware. For instance, at Q4_K_M on a Galaxy S26 Ultra, Granite-4.0-H-350M retains 78.4% of its decode throughput as context scales from 256 to 4,096 tokens, whereas Granite-4.0-350M retains just 33.8%. Additionally, at 2,048 input tokens on the same Galaxy device, the sparse LFM2.5-8B-A1B decodes 2.4 times faster than Qwen3.5-4B and 2.6 times faster than Ministral-3B-Instruct-2512. However, because all expert weights contribute to its memory footprint, LFM2.5-8B-A1B still peaks at 5.29 GiB of RAM despite only activating 1.5 billion of its 8.5 billion parameters per token.

Currently, Pipette lacks NPU and Android GPU support due to inconsistent kernel coverage, meaning Android runs use a CPU-based path while iOS runs use Metal. Consequently, cross-device comparisons are not recommended. For developers deploying small language models to mobile apps or laptops, Pipette provides a transparent, empirical dataset to help them choose the right quantization and model configuration without relying on vendor marketing.