Gemma 4 on Android with LiteRT-LM: The Model Wasn't the Hard Part

Loading a local language model on Android sounds like the hard part. In our work bringing Gemma 4 to OpenAlly with LiteRT-LM, the trouble started after the model loaded.

An AI agent receives more than the sentence a person typed. Before it can answer, it may also receive operating instructions, conversation history, descriptions of the tools it can use, results returned by those tools, and enough empty space to write a useful response. On a small context window, those pieces compete for the same limited token budget.

Fitting an agent into the context window was harder than fitting the model into memory.

This is our dated engineering account. Plain conversation was good when the prompt fit. In OpenAlly's June–July 2026 tests, however, both Gemma 4 E2B and E4B performed badly at tool calling, especially when a request needed several calls across several turns. That difference shaped the work described below.

Google's Gemma release notes, Gemma 4 announcement, and Gemma 4 model card provide the release and architecture context. The rest of this article explains what those capabilities met inside a working Android agent.

Reproducibility starts with more than a product name. “LiteRT-LM” and “Gemma 4 E4B” can point to different commits, exports, quantisation choices, and compiled context limits over time.

OpenAlly's Android runtime uses the LiteRT-LM project. Our runtime is pinned to the exact LiteRT-LM commit 23f36e8, committed on 27 June 2026. Git describes that snapshot as v0.14.0-alpha.0-53-g23f36e8. It is an exact pre-release snapshot after v0.13.1, not a claim that OpenAlly uses a stable v0.14 package.

The underlying engine is pinned to the LiteRT core commit c6de975.

These are the packages OpenAlly is putting into its Android catalog now. The exact revisions and hashes make the model bytes reproducible instead of relying on a moving filename or a broad family label. The performance and tool-calling findings below come from our June–July 2026 Galaxy S26 Ultra test cycle.

The OpenAlly catalog currently gives E2B an 8,192-token window. E4B starts at 4,096 with an 8,192-token package ceiling. The app also uses 6 GiB and 8 GiB device-memory floors for E2B and E4B respectively. Those are OpenAlly safety gates, not universal Android requirements published by Google.

There are three context numbers worth keeping separate. Google's core model card lists a 128K architectural context for E2B and E4B. The current E2B and E4B LiteRT-LM package pages document support up to 32K. OpenAlly's present catalog then chooses an 8,192-token ceiling for these mobile packages. An architecture ceiling, a downloadable package ceiling, and an app's operating limit are not the same promise.

Tokens are small pieces of text that a model reads and writes. A context window is the maximum working space for those pieces during a request.

Imagine a 4,096-seat room. Your message may need only nine seats, but the agent's instructions, conversation, and tool descriptions enter first. A tool description carries a machine-readable name, purpose, argument fields, allowed values, and usage guidance. The model reads that schema before it chooses whether to call the tool.

On 21 July 2026, one OpenAlly prompt reported this budget against that same earlier 4,096-token export:

The 5,483 system tokens are already inside the 5,612 history tokens. Adding them again would double count them. The correct total is the recorded wire total: 8,446 tokens trying to enter a 4,096-token window.

As a ratio, the request was 206% of the window (8,446 divided by 4,096), and 4,350 tokens never got through the door. That is more than the entire window again, discarded before the model had written a single character of an answer. The nine tokens the person actually typed were competing for seats with a queue twice the size of the room.

That also explains why starting a fresh chat did not solve the underlying case. Most of the budget was spent before a long conversation had developed.

We ran the most useful control test: same phone, same model path, same runtime, but with all tools removed from the prompt. The model produced a real response.

That narrowed the fault to the request rather than the model: it could load and generate; what surrounded it was too large.

Removing tools is not a complete agent solution. It removes the schemas, which gives the model room to talk, but it also removes its ability to take actions. A useful on-device agent has to keep both sides of that trade:

• Select a smaller working set of relevant tools for each turn.
• Use terse schemas that preserve essential meaning.
• Reserve 1,024 tokens for the answer instead of filling every seat with input.
• Cap verbose tool results before they re-enter the conversation.
• Carry only the history that remains useful for the next decision.

The goal is fewer instructions competing at once, not fewer capabilities. A business agent may know hundreds of possible actions while showing the local model only the small set needed for the current request.

Google has published Android measurements for Gemma 4 E2B and E4B with LiteRT-LM. The table below reproduces those reference figures with their test shape: 1,024 input tokens for prefill, 256 output tokens for decode, a 2,048-token context, and warm caches. These are Google-published reference numbers, not OpenAlly measurements.

Those eight figures are published on the E2B and E4B LiteRT-LM package cards, at the exact revisions pinned above, and every row is measured on an S26 Ultra, though the cards do not name a regional variant. Google's LiteRT-LM performance write-up covers the same work in round numbers and sends readers to those cards for the per-device detail. The cards are also explicit about what the fastest column leaves out: time-to-first-token does not include load time, the benchmarks ran with caches enabled and initialised, and first-run latency and memory may differ. So 0.3 s is the distance between an already-loaded model and its first token, not the wait after a person chooses a model. The table still shows why accelerated prefill matters, because the phone reads a long prompt far faster before it starts writing. It does not promise the same speed for every phone, artifact, prompt, temperature, thermal state, or agent workload.

Our Galaxy S26 Ultra (SM-S948B) test used 12 GB of RAM. GPU-only inference ran, but the first interaction took roughly 10–15 seconds and throughput was poor. The model was running locally and producing output; it simply was not close to the responsiveness an everyday OpenAlly agent needs. For OpenAlly's daily agent workload, that observed speed was the result that mattered.

The 4.58 GB model file from the June test (an earlier export compiled for a 4,096-token window, not one of the packages pinned above) was only one part of the memory bill. At runtime, Android still needs memory for the operating system, OpenAlly, model working buffers, converted weights, the key-value cache that holds context, and the graphics or CPU backend.

In one June E4B load attempt, Android reported roughly 5.2 GB available memory and the process failed under memory pressure. That is one observation on one device and artifact, not a universal minimum-RAM rule. It taught us to consider available memory and runtime headroom rather than comparing the download size with the number printed on the phone's box.

The context limit created a separate failure. That earlier export had been compiled for 4,096 tokens. Forcing a 32,768-token setting could not create a larger model cache after the fact; it drove execution beyond the artifact's safe attention-mask bounds. The durable rule is to respect the context capacity packaged into the artifact.

GPU execution solved only one part of the problem. In our test, the model ran on the GPU path and still felt slow: roughly 10–15 seconds for the first interaction, followed by poor generation throughput. A local-agent benchmark has to measure the experience around the model, not just confirm that a GPU delegate opened.

Plain chat asks the model to read and respond. A multi-turn agent task may ask it to:

1. Choose the correct tool.
2. Produce valid structured arguments.
3. Read the tool result.
4. Decide whether another tool is required.
5. Preserve the user's intent across the growing history.
6. Finish with a useful answer rather than another unnecessary call.

Every result adds tokens. Every round creates another chance to choose the wrong tool, lose an argument, or stop early. In our June–July 2026 tests, general conversation was good once the prompt fit, but Gemma 4 E2B and E4B both performed badly at multi-turn tool calling. E2B regularly damaged tool names or arguments. E4B's extra capacity did not make the full OpenAlly chain dependable once schemas, returned data, and history accumulated. These were direct results from our Android tests in OpenAlly's tool-heavy agent environment.

Google's structured tool-use guide explains the expected call loop for the model. Its Gemma 4 edge-agent work also shows active progress toward useful local actions.

One lesson from Google's FunctionGemma fine-tuning guide is especially relevant: correct function-call syntax and dependable task intent are different skills. A model may know the shape of a call while still benefiting from examples that teach which tool sequence solves a real workflow.

The opportunity remains strong because the weak points are concrete rather than mysterious.

Larger safely packaged context gives instructions, selected tools, results, and answers more room, provided the memory cost remains suitable for a phone. Stronger accelerated backends can reduce prompt-reading time and make local interaction feel immediate. Better native tool support can improve structured calls without surrounding the model with so much explanatory text. Specialised routing and fine-tuning can teach a compact model the smaller set of actions that matter for a particular task.

Google is approaching on-device Gemma from more than one direction. LiteRT-LM gives developers direct runtime and packaging control. Android's Gemma 4 AICore developer preview represents a separate system-managed path. Those approaches can move at different speeds, but both increase the pressure for clearer backend reporting, dependable tool calls, and efficient context use.

OpenAlly's direction follows the evidence: keep normal conversation natural, select only relevant actions per turn, protect answer space, constrain tool-result growth, and measure the backend that actually ran. How an on-device route sits beside provider keys, your own server, and OpenAlly Cloud is set out in the trust and control guide.

Can Gemma 4 run fully on an Android phone?

Yes. Gemma 4 has official on-device packages and Android runtime paths. Whether a specific package runs well depends on the device, available memory, packaged context, backend support, and workload. Loading the model and running a multi-tool agent are different levels of difficulty.

Which Gemma 4 model does OpenAlly use on Android?

The selected official packages verified on 19 August 2026 are Gemma 4 E2B and E4B LiteRT-LM exports. Their exact repository revisions, byte sizes, and SHA-256 values appear in the pin table above. OpenAlly can choose the package that suits the phone's practical memory headroom.

Why do AI tools use so many tokens before they run?

The model needs a description of each available tool, including its purpose and structured arguments. Those schemas are prompt text. Fifteen detailed tools can consume thousands of tokens even when the model ultimately calls only one.

What worked well, and what failed, in OpenAlly's tests?

Plain conversation and general-purpose answers were good when the prompt fit. Tool use was the failure point: in OpenAlly's June–July 2026 tests, Gemma 4 E2B and E4B both performed badly at choosing and chaining tools for daily agent work. The context budget, schema load, tool results, and multi-turn decisions made that workload much harder than chat.

How fast was OpenAlly's Galaxy S26 Ultra test?

GPU-only inference ran, but the first interaction took roughly 10–15 seconds and throughput was poor. Google's E2B GPU reference reaches a first token in 0.3 s, but that figure excludes model load time and runs with caches already initialised, so it is not the wait a person feels after choosing a model. OpenAlly's field workload produced a much slower user experience.

On-device AI is often discussed as a model-size contest: can the weights fit, and how many tokens per second can the phone produce? An agent adds a more important question: after the instructions, tools, results, history, and reply reserve are counted, is there enough useful context left to do the job?

On 21 July, removing the tools turned a failed request into a real conversation on the same device. That single test changed the engineering focus: treat context as a strict product budget, rather than abandoning local models or pretending every tool could fit.

Gemma 4 and LiteRT-LM make serious local Android inference possible. Building a dependable local agent on top takes the less glamorous work too: exact package pins, verified backends, safe memory margins, compact schemas, selective tools, bounded results, and honest measurements.