↓ Skip to main content

Running Ollama on a Jetson Orin Nano: From Gemma 3 to Gemma 4 with GPU Acceleration

Author
Julien
Cloud & AI Infrastructure Engineer.
Dedicated to bringing AI to Ops & Ops to AI.

A few months ago I added an NVIDIA Jetson Orin Nano Developer Kit to my homelab. The idea was simple: a dedicated, always-on inference server for local LLMs, completely separate from my main Proxmox cluster. With 8GB of unified memory and an integrated CUDA-capable GPU, it sounded like the perfect edge device for running small models.

Reality, as usual, was more nuanced. Not every model that fits on disk actually runs well. Some swap to eMMC and crawl. Others load fast but generate gibberish. After testing six models across two generations of Google’s Gemma architecture, surviving a GPU driver rabbithole, and learning the hard way how memory budgets work on unified memory, I found a setup that delivers 25.5 tokens per second on GPU with zero swap.

This post covers both rounds of testing, the hardware limits, the GPU configuration that took hours to figure out, and the service architecture that keeps everything running.



Update: 19 November 2026
#

Five months on, the Jetson is still doing this job. But “still running” and “still configured the way this post describes” turned out to be two different things. This update covers what changed on the box, what the re-test showed, and what quietly rotted in between. The original June article is reproduced below, unchanged — where numbers differ, the June values stay visible as the baseline.

What changed on the box
#

  • OS fully patched. 353 pending packages → 0. L4T itself stays at R36.4.7, which is current for the r36.4 channel; JetPack 7 would be an SDK Manager reflash, not an apt upgrade.
  • Ollama 0.30.6 → 0.40.16.
  • The installer handles the JetPack CUDA libs now. install.sh detects /etc/nv_tegra_release matching R36 and downloads ollama-linux-arm64-jetpack6 on its own, before any NVIDIA GPU probing. Everything in The GPU Driver Rabbit Hole above remains valid for ≤ 0.30.x, and is no longer necessary.
  • fwupd purged. It was the single package apt upgrade held back, and firmware on a Jetson comes from flashing, not from fwupd.
  • The 8K model was rebuilt from the same QAT blob as gemma4:e2b-8k.

Re-test: the same models, same prompts, five months later
#

Ollama 0.40.16, temperature 0, 150 completion tokens, cold load measured per model, memory from MemAvailable deltas:

ModelModeTok/sCold loadMemory (idle → loaded)Vision projectorSwap
gemma4:e2b-8kGPU 100%33.357s0.7 → 6.0 GBGPU~0
gemma4:e2b-32kGPU 100%33.1110s0.6 → 6.0 GBCPU fallback~0
gemma3:4b-8kGPU 100%17.556s0.7 → 5.4 GBGPU~0
  • E2B is ~30% faster than in June: 33.3 tok/s versus 25.5 on identical weights — the 0.40.x runner improved. Gemma 3 4B finally runs on the GPU at 17.5 tok/s, identical to its old CPU figure because it is bandwidth-bound either way.
  • It uses more memory: ~6.0 GB loaded versus the 3.6 GB measured in June. The newer runner allocates larger compute buffers, which matters enormously on 8 GB.
  • Quality re-verified. Same three probes: E2B still answers the drying-shirt trap correctly and still writes the more thorough sieve (904 vs 606 characters, with edge cases and complexity notes). Gemma 3 4B also passes both, now on GPU.
  • New retrieval probe. A ~6k-token haystack of filler log entries with one secret code planted at 50% depth: all three models return 4417-ZULU. At this scale E2B’s long-context reputation holds — the thing to fear on this hardware is memory pressure, not recall.

Thinking mode silently eats the token budget
#

Gemma 4 E2B ships with thinking mode on by default, and in Ollama 0.4x the chain-of-thought lands in a separate thinking field of the API response. My first re-test round returned empty response bodies with full token counts: with num_predict capped at 120, the model spent the entire budget reasoning invisibly and never reached the answer.

  • Set "think": false for plain completions, or raise num_predict well past the expected answer length.
  • Forcing "think": true through the API on 0.40.16 crashed the runner with this model (llama-server terminated: signal: aborted (core dumped)). Rely on the model default, read the thinking field, and test before depending on it.

The 32K trap
#

At some point the production model drifted from gemma4:e2b-8k to gemma4:e2b-32k — same QAT weights, context bumped to 32768. On paper strictly better: same 33 tok/s, four times the context. In practice strictly worse on this hardware:

msg="llama-server startup failed with projector GPU offload; retrying with projector CPU offload"
error="cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 230740096
graph_reserve: failed to allocate compute buffers"

What 32K costs over 8K:

  • Cold load doubles (110s vs 57s) — KV cache and graph-reserve buffers are preallocated for the full context at startup.
  • The vision projector no longer fits. The CLIP encoder needs ~600 MiB on the GPU; after the 4.3 GB weights, the compute buffers and the 32K KV preallocation there is not enough left. Ollama silently falls back to the CPU for the projector — text stays at 100% GPU, so nobody notices until image input is suddenly slow.
  • ~1.4 GB of headroom left for the whole system after load (6.0 GB of ~7.4 GB). Any second GPU allocation is one cudaMalloc failed away from degraded service.

To rule out leftover allocator state, I rebooted and re-ran the comparison back to back on an idle box:

TestCold loadTok/sProjector OOM eventsProjector placement
A — 8K, straight after boot (preload)——0GPU (vision + audio)
B — 32K, same boot105.9s33.03CPU fallback
C — back to 8K, same boot55.7s33.10GPU again

The 32K failure reproduces from a clean boot, so it is a context-window effect, not accumulated state. Test C is the interesting one: 8K recovered immediately afterwards, projectors back on the GPU, with CmaFree reading 0 kB — the gate is the CUDA allocation check, not the CMA counter. And if you see projector OOM at 8K, you most likely have debris from repeated load/unload cycles: reboot and retest before blaming context size. I briefly misread exactly that during this re-test.

The June conclusion stands, now with numbers: 8K is the sweet spot. Overshooting does not make the model slower in tokens/second — it makes it less capable, by pushing parts of the model off the GPU.

What rotted, and the fix
#

The most useful finding was not in the benchmarks, it was the gap between this post and the running machine:

Documented (June)Reality (November)
/etc/ollama/model.conf drives model selectiondirectory and file gone — keepalive fell back to a hardcoded model name
ollama-preload.service warms gemma4 at bootdisabled, and its script still preloaded llama3.2:3b-12k, deleted months earlier
8K context as the tuned sweet spotrunning 32K, with GPU vision offload broken

Nothing crashed. No alarm fired. The keepalive dutifully pinged a name baked into a script, the preload loaded a model that no longer existed and had been disabled anyway, and the only symptom was a WARN line in the journal that nobody reads. Everything above about careful boot ordering was correct in June and had quietly stopped being true.

sudo mkdir -p /etc/ollama
sudo tee /etc/ollama/model.conf >/dev/null <<EOF
OLLAMA_MODEL=gemma4:e2b-8k
OLLAMA_KEEPALIVE_INTERVAL=240
EOF
sudo sed -i 's/llama3.2:3b-12k/gemma4:e2b-8k/' /usr/local/bin/ollama-preload.sh
sudo systemctl enable ollama-preload.service

Verified after a reboot, because a fix you never observed at boot is a hope: the preload unit fired ~100 seconds after start, loaded gemma4:e2b-8k at 100% GPU with 8192 context, and the boot journal contains no cudaMalloc failed and no projector fallback. First request was warm.

The lesson worth the price of admission: infrastructure described in a blog post is not monitored by that blog post. If a systemd unit and a config file are load-bearing, they need the same drift detection as everything else — or at minimum a scheduled grep of journalctl -u ollama for WARN lines.

Current setup
#

June 2026November 2026
Ollama0.30.6, manual JetPack 6 tarball0.40.16, JetPack libs from install.sh
Active modelgemma4:e2b-8kgemma4:e2b-8k
Inference25.5 tok/s33.3 tok/s
Memory loaded3.6 GB~6.0 GB
GPU100% CUDA0 Orin100% CUDA0 Orin
Swapzero~0
Bootmanual preload orderingpreload + keepalive + model.conf, verified after reboot

Two lessons from the second pass: the context window is a memory decision, not a capability decision, and a machine that still answers is not the same as a machine that is still configured correctly.


The original article, as published in June 2026, follows unchanged.

The Hardware Reality Check
#

The Jetson Orin Nano Developer Kit has 8GB of shared memory. That means RAM and GPU VRAM come from the same pool. There is no dedicated graphics memory to fall back on. Storage is a 56GB eMMC module, fast enough for OS duties but brutally slow when used as swap.

Key constraints from day one:

  • 8GB unified memory limits model size and context length
  • ARM64 architecture restricts which models have native support
  • eMMC swap is a performance cliff — once the system starts swapping, inference drops from 17 tokens per second to under 2

This means model selection is not about downloading the latest 8B parameter release and hoping for the best. It is about finding the sweet spot between size, speed, and quality on very specific hardware.

Round One: Gemma 3
#

I tested six models with the same prompt — “Hello, how are you?” — and measured tokens per second, memory footprint, and overall responsiveness.

ModelSizeTokens/secVerdict
llama3.1:8b4.9 GB~8-10Too large – heavy swap makes it unusable
llama3.2:3b2.0 GB~15-18Fast but mediocre reasoning quality
llama3.2:3b-12k2.0 GB~15-18Same speed with extended context, same quality limits
qwen3.5:2b2.7 GB~18-22Fastest of the bunch – but weak at reasoning tasks
gemma3:4b3.3 GB~16-20Runner-up – solid speed/quality balance, fits in RAM
gemma3:4b-8k3.3 GB~17.5Winner (GPU not yet functional) – best reasoning, zero swap

Gemma 3 4B with a custom 8K context window became the production model. 17.5 tok/s on CPU, zero swap, stable for months. But it ran on CPU only — the GPU was idle because the original Ollama binary had dropped JetPack 5 support.

Round Two: Gemma 4 Arrives
#

In early 2026, Google released Gemma 4 with two edge variants: E2B (2.3B effective) and E4B (4.5B effective). The on-paper specs were compelling: 128K native context, thinking mode, function calling, system prompt support. The default Ollama quantization was 7.2 GB — too large — but the QAT tag changed everything.

QAT (Quantization-Aware Training) quantizes during training rather than after. The result: gemma4:e2b-it-qat at 4.3 GB instead of 7.2 GB. Same architecture, 40% smaller. This is not a niche optimization; it is the difference between fitting and failing on 8 GB hardware.

FeatureGemma 3 4BGemma 4 E2B
Disk size3.3 GB4.3 GB (QAT)
Default context8K128K
Thinking modeNoYes
Function callingNoYes
System promptNoYes
MMLU Pro score~50%60%

MMLU Pro scores are approximate, sourced from published benchmarks and community results on comparable hardware. Your mileage will vary with quantization and prompt style.

Pulling the model was the easy part. Making it run on GPU was not.

The GPU Driver Rabbit Hole
#

The Jetson runs Ubuntu 22.04 with a CUDA 12.6 driver. Ollama 0.30.6 expects a cuda_jetpack6 directory — which was missing despite the system having the right driver version. The problem: the CUDA toolkit directory layout was from an older JetPack 5 installation (cuda_jetpack5), and the Ollama binary only checks for cuda_jetpack6.

Three failed approaches before finding the right one:

Attempt 1: JetPack 5 CUDA libs (CUDA 11.x). Symlinked cuda_jetpack6 to the existing cuda_jetpack5. GPU was detected but model loading failed with cudaMalloc failed: out of memory. These libs route all GPU allocations through CMA (Contiguous Memory Allocator), which defaults to 256 MB on Jetson. An LLM needs gigabytes.

Attempt 2: Generic CUDA 12 libs. Symlinked to cuda_v12 from the main ARM64 tarball. GPU skipped entirely — the libggml-cuda.so was compiled for desktop GPU architectures (SM 5.0 through 9.0) but not Orin’s CC 8.7.

Attempt 3: The CMA trap. Tried cma=4096M in the kernel boot parameters to expand the memory pool. This broke GPU detection entirely — the CUDA driver could not initialize when CMA consumed half the system RAM. Even cma=1024M had the same effect. The lesson: never touch CMA on Jetson.

The working solution: Extract the JetPack 6 CUDA tarball from the Ollama release.

curl -L https://github.com/ollama/ollama/releases/download/v0.30.6/ollama-linux-arm64-jetpack6.tar.zst -o ollama-jp6.tar.zst
sudo tar --zstd -xf ollama-jp6.tar.zst -C /usr/local
sudo systemctl restart ollama

This provides libggml-cuda.so compiled with Orin CC 8.7 support and CUDA 12.6 runtime libs that match the Jetson driver. GPU discovery confirmed:

inference compute: library=CUDA compute=8.7 name=CUDA0 description=Orin

No CMA tweaks. No symlinks. Just the right libs in the right place.

Benchmark: Gemma 4 on GPU
#

With GPU working, the numbers were decisive:

ModelModeTok/sCold LoadRAMGPU
gemma3:4b-8kCPU17.50.79s4.5 GBNo
gemma4:e2b-it-qatCPU12.470s5.0 GBNo
gemma4:e2b-4kGPU25.767s3.4 GB100%
gemma4:e2b-8kGPU25.5~30-70s3.6 GB100%

Gemma 4 on GPU is 46% faster than Gemma 3 on CPU while using 1 GB less RAM. The 8K context window has zero speed penalty versus 4K — the KV cache is negligible next to the model weights. The 128K native context support is there if needed, though I settled on 8K as the practical sweet spot.

Cold load is the only downside: 30-70 seconds versus Gemma 3’s sub-second CPU load. But a keepalive service that pings the model every 4 minutes makes this a non-issue. The model stays in GPU memory permanently.

Creating the Custom 8K Context Model
#

The default gemma4:e2b-it-qat is a raw weights download with Llama’s default context cap. To set an explicit 8K context window (matching what fits comfortably in the 8GB unified memory) and give it a friendly name, use a Modelfile:

# Modelfile for gemma4:e2b-8k
FROM gemma4:e2b-it-qat
PARAMETER num_ctx 8192

Then create the named model:

ollama create gemma4:e2b-8k -f Modelfile

The gemma4:e2b-4k variant in the benchmarks was the same base model capped at 4096 context for comparison. The 8K cap shows zero speed penalty – the KV cache overhead is negligible next to the 4.3 GB model weights. You could go higher (128K native is supported) but at some point memory pressure from the KV cache starts eating into the safety margin.

Quality: Is Gemma 4 Actually Smarter?
#

Benchmarks on paper are one thing. Real prompts are another. I tested both models on three tasks:

Logic reasoning: “If a shirt takes 4 hours to dry, how long for 3 shirts?”

Gemma 4 answered correctly (4 hours, simultaneous drying) with structured step-by-step reasoning: “Since all three shirts dry independently at the same rate, you only need to wait the time required for one shirt to finish.” Gemma 3 sometimes fell for the multiplication trap.

Code generation: “Write a Sieve of Eratosthenes in Python.”

Gemma 4 produced clean, commented code with proper edge cases (n < 2 returns empty), complexity analysis (O(N log log N)), and usage examples. Gemma 3 was adequate but less thorough.

Long-form generation: “Write a technical essay about transformer architecture.”

Gemma 4 generated 2,700+ coherent tokens with technical depth on attention mechanisms, positional encoding, and multi-head attention. Sustained 25.5 tok/s throughout with no degradation.

Despite having fewer effective parameters (2.3B vs Gemma 3’s ~4B), the QAT quantization and architectural improvements in Gemma 4 produce noticeably better output. The thinking mode — where the model outputs a chain-of-thought before the final answer — adds further quality for complex reasoning tasks.

The CMA Lesson
#

The Jetson’s Contiguous Memory Allocator defaults to 256 MB. On the old CUDA 11 libs, this was a hard bottleneck — every GPU memory allocation went through CMA, which is orders of magnitude too small. On the JetPack 6 CUDA 12 libs, GPU memory allocations bypass CMA and use system memory directly.

But CMA still matters for compute buffer allocation. When a model loads, a small compute buffer (100-200 MB) goes through CMA. If CMA is fragmented from a previous model load/unload cycle, the new load fails with cudaMalloc failed even though 6+ GB of system RAM is free. CMA fragmentation is permanent — it survives Ollama restarts and only a full reboot clears it.

The fix: never unload the model. The keepalive service is not just for cold-start latency — it prevents CMA fragmentation. If the model stays loaded, CMA is consumed once at initial load and never touched again.

The boot sequence must be careful:

  1. Ollama starts, GPU discovery runs (CMA still clean)
  2. Preload fires, loads gemma4 (uses CMA once, model stays warm forever)
  3. Keepalive takes over (pings every 4 minutes, never unloads)
  4. The preload service must point at gemma4 — if gemma3 loads at boot, it consumes CMA and gemma4’s GPU load fails later

Keeping the Model Warm
#

The service architecture uses three systemd units:

  • ollama.service — Main Ollama daemon, always running on port 11434
  • ollama-preload.service — Oneshot that loads gemma4 20 seconds after Ollama starts, warming the model at boot
  • ollama-keepalive.service — User service that pings the model every 4 minutes to prevent eviction and CMA fragmentation

The preload and keepalive scripts read the model name from /etc/ollama/model.conf, making it trivial to switch models by changing one line:

OLLAMA_MODEL=gemma4:e2b-8k

Monitoring What Matters
#

  • jtop — Jetson-specific monitoring. Watch GPU utilization (100% during inference), RAM usage (3.6 GB under load), and temperature (under 70C with the reference cooler).
  • tegrastats — Low-level telemetry for power draw, per-core CPU usage, and memory.
  • htop — General system view, mostly to confirm swap stays near zero.

If swap usage climbs during inference, something is wrong. The fix is never to add more swap — it is to use a smaller model or reduce context.

Final Setup
#

  • Device: NVIDIA Jetson Orin Nano Developer Kit (8GB)
  • OS: Ubuntu 22.04.5 LTS (JetPack 6)
  • Ollama: 0.30.6 with JetPack 6 CUDA libs
  • Active model: gemma4:e2b-8k (custom 8K context, QAT quantized)
  • Inference speed: 25.5 tokens/sec (GPU, warm)
  • Memory footprint: 3.6 GB total (weights + KV cache)
  • GPU: 100% CUDA0 Orin
  • Swap usage: Zero

Lessons Learned
#

QAT quantization matters more than you think. The -qat tag on Ollama is the difference between a model that fits (4.3 GB) and one that does not (7.2 GB). Always check for QAT variants before dismissing a model for edge hardware.

JetPack 6 CUDA libs are required for GPU on Orin. The standard ARM64 ollama tarball lacks Orin support. The JetPack 6 tarball has it. This is not documented anywhere obvious.

Never touch CMA. The cma= kernel parameter breaks GPU detection entirely. Default CMA (256 MB) is sufficient when using the correct CUDA libs.

Keepalive prevents CMA fragmentation. On Jetson, it is the difference between a working GPU inference server and a brick that needs a reboot after every model unload. CMA fragmentation is permanent and unrecoverable without a full system reboot.

Gemma 4 on edge is worth the effort. 46% faster, noticeably smarter, 1 GB lighter on RAM, with thinking mode and function calling. The hour of CUDA debugging pays for itself in every inference.

The Jetson Orin Nano is not going to replace a GPU server. But as a dedicated local LLM endpoint — handling RAG queries, chat, code generation, and light automation — it punches well above its weight class. The key is respecting the hardware limits, choosing the right model, and getting the CUDA configuration right the first time.