diff --git a/docs/adr/034-ollama-production-deployment-and-keep-alive.md b/docs/adr/034-ollama-production-deployment-and-keep-alive.md new file mode 100644 index 00000000..0ff4a790 --- /dev/null +++ b/docs/adr/034-ollama-production-deployment-and-keep-alive.md @@ -0,0 +1,125 @@ +# ADR-034: Ollama in production — deployment, keep-alive pinning, and corrected init recipe + +**Date:** 2026-06-06 +**Status:** Accepted +**Deciders:** Marcel Raddatz +**Relates to:** #758 (bug), #759 (fix), #737 (NL search infrastructure) +**Corrects:** ADR-028 §10–§11 (init recipe and readiness probe) + +--- + +## Context + +ADR-028 introduced Ollama as a Docker Compose service for NL search and documented +its topology, graceful-degradation contract, and memory budget. Two defects survived +that work and only surfaced once NL search reached staging (#758): + +1. **Ollama was added only to the dev `docker-compose.yml`.** Staging and production + deploy from the self-contained `docker-compose.prod.yml`, which had no `ollama` + service. The backend defaults to `app.ollama.base-url: http://ollama:11434`, so its + client bean was active and resolved to a non-existent host → `ResourceAccessException` + → HTTP 503 on every NL search. +2. **The init recipe documented in ADR-028 §10 never worked.** The `ollama/ollama` image + `ENTRYPOINT` is `ollama`, so a bare `command: sh -c "…"` ran as `ollama sh -c "…"` + (`unknown command "sh"`), and the image ships **no curl**, so the curl-based readiness + loop and the curl healthcheck could never pass. + +This ADR records the production deployment decision and the corrected operational +contract. It is also the durable record of *why* `OLLAMA_KEEP_ALIVE=-1` is set, so a +future maintainer does not "optimize" it away and reintroduce the cold-load 503. + +--- + +## Decisions + +### 1. Ollama is a first-class production service + +`docker-compose.prod.yml` now defines `ollama` + `ollama-model-init` + the +`ollama-models` volume, mirroring the dev stack. The graceful-degradation contract from +ADR-028 §3 is preserved: `backend` has **no** hard `depends_on` on `ollama`, so an absent +or unhealthy Ollama still yields a clean 503 rather than blocking backend startup. + +### 2. Corrected init recipe (supersedes ADR-028 §10) + +The init container overrides the image entrypoint to a shell and probes readiness with +`ollama list` (not curl, which the image lacks): + +```sh +ollama serve & until ollama list >/dev/null 2>&1; do sleep 1; done && \ + (ollama list | grep -q 'qwen2.5:7b-instruct-q4_K_M' || ollama pull qwen2.5:7b-instruct-q4_K_M) +``` + +```yaml +entrypoint: ["/bin/sh", "-c"] +``` + +The pull is **guarded by a grep on the cached model list**. A model already on the volume +exits clean without any registry round-trip. This makes re-up offline-safe: a host reboot +during a registry/network blip can no longer fail init (which, via +`condition: service_completed_successfully`, would otherwise block the `ollama` service +and take NL search down until the registry was reachable again). The same recipe is used +in dev and prod — one mental model. + +### 3. Healthcheck uses `ollama list` (supersedes ADR-028 §11 probe) + +```yaml +healthcheck: + test: ["CMD", "ollama", "list"] +``` + +`ollama list` hits the local API and exits non-zero when the server is down — the correct +probe for a curl-less image. The `start_period: 60s` rationale from ADR-028 §11 still holds. + +### 4. `OLLAMA_KEEP_ALIVE=-1` — pin the model in memory + +```yaml +environment: + OLLAMA_KEEP_ALIVE: "-1" +``` + +By default Ollama evicts an idle model after ~5 minutes. The next query then pays a +cold-load penalty that exceeds the backend read timeout, producing an NL search 503 after +any idle period. Pinning the model (`-1` = never unload) keeps warm-path latency +predictable (~18 s on CPU). **Do not remove this** without re-introducing the post-idle +cold-load 503. + +### 5. Read timeout raised 30 → 60 s + +`app.ollama.timeout-seconds` is raised from 30 to 60 (`application.yaml`, mirrored in +`DEPLOYMENT.md`). Warm CPU inference is ~18 s; the higher ceiling absorbs the one cold +model load on the first query after an Ollama (re)start, before §4's pin takes hold. + +**Implicit NFR made explicit:** NL search shall return a result or a 503 within 60 s; the +cold-start path immediately after an Ollama restart is the only path that approaches this +ceiling. + +### 6. Hard-OOM trade-off (refines ADR-028 §2) + +`memswap_limit == mem_limit` (both `${OLLAMA_MEM_LIMIT:-8g}`) disables swap for the +container. Combined with §4's pinned model, a memory-pressure event is a **hard OOM-kill, +not graceful latency degradation**. This is deliberate — swap-thrashing an LLM is worse +than a clean restart — but it means the 8 GB envelope is a real ceiling. `qwen2.5-7B-q4` +plus its KV cache under load sits close enough to 8 GB that this needs a Prometheus +memory alert on the `ollama` container before it bites in production (tracked as +observability follow-up, not in this PR). + +--- + +## Consequences + +### Positive + +- NL search works on staging/production, not just dev — the actual deploy artifact now + matches the documented architecture. +- Re-up is offline-safe: a cached model never depends on registry reachability. +- The keep-alive pin and timeout ceiling make NL search latency predictable on CPU. + +### Risks and operational implications + +- **Hard OOM under memory pressure** (§6): a Prometheus alert on `ollama` container memory + is required before this is load-bearing in prod. Tracked as an observability follow-up. +- **Unauthenticated inference** relies entirely on `archiv-net` isolation (ADR-028 §7/§12, + unchanged). Sending an `Authorization` header from `RestClientOllamaClient` is a separate + durable hardening item, tracked outside this PR. +- ADR-028 §10–§11 describe a recipe that never functioned; this ADR is the authoritative + init/healthcheck contract going forward.