Sending every request to a cloud LLM costs you speed, money and privacy. Here’s how to route routine tasks to on-device models and reserve the cloud for what needs it.

On-Device AI and SLMs: How to Architect Hybrid Inference That Cuts Cost, Latency and Data Exposure

Sending every AI request to a frontier cloud model is the default, and for most production features it is the wrong one. A smart reply, an intent check, a field extraction or a redaction pass does not need a trillion-parameter model and a network round trip. It needs an answer in a fraction of a second, at close to zero marginal cost, without the user's data leaving their device. Teams that discover this after launch usually find it on the same three dashboards: the cloud bill, the latency chart and the compliance review.

Modern phones, laptops and edge devices now ship with neural processing units (NPUs) and system-level model runtimes that can run small language models (SLMs) locally. The architecture that follows is hybrid inference: a routing layer sends routine, latency-sensitive or sensitive work to an on-device model, and escalates to a cloud LLM only when the task needs more reasoning than a small model can give. This article explains why cloud-only inference breaks down at scale, how to decide what runs where, the routing and privacy patterns that work in production, and the mistakes that sink mobile deployments. It builds on the principles in our small language models and edge AI guides. Written for engineering leads, product owners and architects shipping AI features on mobile and edge devices.

Why Cloud-Only Inference Breaks Down at Scale

Three structural problems appear when a prototype becomes a product.

Token economics at high frequency. Features like smart replies, OCR clean-up, intent classification and entity extraction fire many times per user session. Paying per-token cloud fees for narrow, repetitive tasks erodes unit margins quickly, because volume is high and the value of each call is small. Network-bound latency. A cloud call carries network overhead before the first token appears, and that overhead varies with connection quality. In transit, warehouses, clinics and field operations, cloud-only features feel sluggish or stop working entirely. Data egress and compliance exposure. Every request that leaves the device widens your compliance boundary. Regulations such as India's DPDP Act, the EU AI Act and HIPAA put obligations on how personal, financial and clinical data is transmitted and processed. Keeping sensitive parsing on the device shrinks what you have to defend.

If a task is narrow, frequent and sensitive, sending it to a frontier cloud model is paying a premium to lose speed and privacy.

The Decision Matrix: What Runs Where

Local and cloud models are not interchangeable. The useful question for each workload is how much context it needs, how deep the reasoning must go and how sensitive the data is.

Dimension

On-device SLM (NPU / GPU)

Cloud frontier LLM

Typical models	Gemini Nano (via Android AICore), Gemma-class and other open-weight SLMs	Frontier cloud APIs
Ideal tasks	PII redaction, entity extraction, summarisation, intent routing, local tool calling	Multi-document retrieval and synthesis, complex code generation, multi-step planning
Latency	Low and predictable, no network hop	Network and queue dependent, usually higher
Marginal cost	No API fee (uses the user's hardware)	Variable per-token spend
Offline use	Works fully offline	Requires connectivity
Data boundary	Stays on the device	Encrypted transit to a cloud endpoint

Latency and cost figures vary by device, model and region, so treat any published numbers as indicative and benchmark on the hardware your users actually own.

Route by what the task needs, not by what the biggest model can do.

Core Pattern 1: A Telemetry-Aware Routing Gateway

A production hybrid pipeline needs a client-side gateway that decides where each request goes. Unlike a server cluster, mobile hardware is variable: free memory, thermal state, battery level and whether the OS has the model ready differ from device to device and minute to minute.

The gateway should check, before dispatching:

Capability and readiness. Is the model present and loaded, and does the device's NPU or GPU delegate support it?
Thermal and battery state. Under heavy thermal load or low-power mode, degrade gracefully or send non-sensitive work to the cloud rather than freezing the interface.
Complexity and size limits. Set an input ceiling, such as a token threshold, above which a request escalates to the cloud. Single-turn, short instructions stay local.
Sensitivity. Requests containing personal or regulated data default to local execution, or to the sanitise-then-escalate pattern below.

An illustrative routing policy, in simplified Kotlin:

kotlin

// Illustrative only: adapt to your runtime and SDK.
fun chooseTier(req: AiRequest, device: DeviceState): Tier = when {
    req.containsSensitiveData && device.localModelReady -> Tier.ON_DEVICE
    !device.localModelReady || device.thermalThrottled   -> Tier.CLOUD_FALLBACK
    req.estimatedTokens > LOCAL_TOKEN_LIMIT               -> Tier.CLOUD
    req.needsDeepReasoning                                -> Tier.CLOUD
    else                                                  -> Tier.ON_DEVICE
}

Core Pattern 2: Managed Hybrid vs. Custom Runtimes

On Android there are two common implementation paths.

Managed hybrid routing. For teams standardising on Gemini, the Firebase AI Logic hybrid inference API offers routing profiles such as PREFER_ON_DEVICE, which tries on-device Gemini Nano first and falls back to a cloud model when the local model is unsupported or unavailable. This is the fastest route to production, and it lets you apply platform controls such as App Check and spend caps to the cloud fallback. Check the current SDK documentation for exact configuration, because these APIs are still evolving. Custom open-weight runtimes. When you need a fine-tuned domain model, custom quantisation (INT4 or INT8) or parity across Android, iOS and desktop, a runtime such as LiteRT-LM runs your own bundled models on NPU or GPU. This gives more control and more work: you own model delivery, versioning, evaluation and updates.

Choice

Best for

Trade-off

Managed hybrid (system model + SDK)	Fast delivery, Gemini-centric products	Less control over model behaviour and updates
Custom open-weight runtime	Domain-tuned models, cross-platform parity	You own packaging, evaluation and lifecycle

Core Pattern 3: The Local-First Privacy Sanitiser

One of the highest-value hybrid patterns uses both tiers in one workflow:

Local pre-processing. An on-device model scans the raw input, such as a support message or a clinical intake note, extracts structured entities and redacts names, phone numbers and account identifiers.
Anonymised cloud escalation. Only the sanitised, abstracted query goes to a cloud LLM for deep reasoning or retrieval across large knowledge sources.
Local re-hydration. The app maps the cloud response back to the user's real context before showing it.

You get frontier-level reasoning without sending raw identities to an external service.

The best privacy control is the data that never left the device.

One caution: on-device processing reduces exposure but does not remove your obligations. Redaction quality needs testing, consent and retention rules still apply, and regulated sectors should confirm the approach with their compliance teams.

Production Pitfalls to Avoid

Bundling model weights in the app package. Even a heavily quantised small model can take up a large share of a gigabyte or more. Deliver model files on demand through platform asset delivery or background downloads with Wi-Fi and storage checks, instead of inflating the base install. Sending the same prompt to both tiers. Small models need tighter instructions, explicit examples and constrained output, such as JSON-schema enforcement, to behave reliably. Maintain separate, regression-tested prompt templates for the local and cloud paths. Ignoring mobile lifecycle and memory pressure. Backgrounded apps lose memory aggressively. Scope the inference engine to a lifecycle-aware component and unload weights cleanly when the OS signals memory trouble, or you will ship out-of-memory crashes. No fallback and no measurement. Without a tested cloud fallback, devices that cannot run the model get a broken feature. Without per-tier metrics, you cannot tell whether routing is saving money or quietly hurting quality. This is where the practices in our AI observability guide apply directly: track latency, fallback rate, cost per request and quality by tier. Over-routing to the cloud "to be safe." It defeats the architecture. Define clear criteria for escalation and measure how often it happens.

Where Decision Models and Routers Fit

A routing gateway is itself a classification problem: given a request, choose a tier. Simple rules cover most cases, and a small, fast classifier can handle the rest. This is the same family of idea we covered in our article on decision-optimised models, where narrow, closed-set choices are handled by a model built for selection instead of generation. Keep the router cheap, deterministic where possible and observable, since it runs on every request.

A Readiness Checklist

Workloads inventoried and classified by frequency, reasoning depth and data sensitivity
Routing policy documented: what runs locally, what escalates, and why
Device readiness checks in place for model availability, memory, thermal state and battery
Cloud fallback implemented and tested for unsupported or offline-incapable devices
Separate, regression-tested prompts and output constraints for on-device and cloud models
Model delivery handled on demand, with versioning and an update path
Sensitive-data handling defined, including redaction testing and compliance review
Per-tier metrics tracked: latency, cost per request, fallback rate and output quality
Memory and lifecycle handling tested under background and low-memory conditions

Conclusion

Hybrid inference is quickly becoming the baseline architecture for AI features on mobile and edge devices, because it fixes problems that cloud-only designs cannot: recurring token cost on high-frequency tasks, latency tied to network conditions, and the compliance weight of moving user data off the device. The approach is not to replace cloud models but to give them a narrower, better-chosen job. Put the device NPU to work on routine, latency-sensitive and sensitive tasks, reserve the cloud for heavy synthesis, and measure both tiers so the routing stays honest as models and hardware change.

If your team is planning an on-device SLM pipeline or wants to audit what your current LLM routing is costing you, NetConsulate designs production-grade hybrid AI architectures across mobile, edge and cloud.

Planning on-device or hybrid AI and want the routing, privacy and cost model designed properly? Submit a proposal request and our team will respond with a tailored approach within 2 business days.