marcel/familienarchiv
1
0
Fork 0
Code Issues 119 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(infra): Ollama missing from prod/staging compose + broken model-init recipe — NL search 503 on staging #758

New Issue
Closed
opened 2026-06-06 19:18:31 +02:00 by marcel · 1 comment
marcel commented 2026-06-06 19:18:31 +02:00
Owner
Copy Link

Symptom

On staging, every natural-language search returns "Intelligente Suche nicht verfügbar" (HTTP 503 SMART_SEARCH_UNAVAILABLE). Backend logs:

WARN org.raddatz.familienarchiv.search.RestClientOllamaClient — Ollama inference failed: ResourceAccessException

Root cause

Two independent defects, both downstream of #737:

  1. Ollama was never added to docker-compose.prod.yml. #737 added the ollama + ollama-model-init services and the ollama_models volume to the dev docker-compose.yml only. Staging and production deploy from docker-compose.prod.yml (a self-contained file, not an overlay), which has no Ollama service. The backend defaults to app.ollama.base-url: http://ollama:11434 (application.yaml), so the conditional client bean is active and tries to connect to a host that does not exist → ResourceAccessException → 503. Confirmed on the host: no ollama container, no process, nothing on :11434.

  2. The model-init recipe merged in #737 is broken (so even the dev compose never actually worked):

    • The ollama/ollama image's ENTRYPOINT is ollama, so command: sh -c "..." is parsed as ollama sh -c "..." →
      Error: unknown command "sh" for "ollama" → init exits 1 → ollama never starts (service_completed_successfully gate).
    • The image ships no curl, so both the init readiness loop (until curl -sf .../api/tags) and the
      service healthcheck (["CMD","curl","-f",...]) can never succeed.

Hotfix already applied to staging (2026-06-06)

To restore NL search immediately I patched the on-disk /opt/familienarchiv/docker-compose.prod.yml, pulled the model, and verified end-to-end. The corrected recipe (entrypoint override + ollama list for readiness/health, no curl) is now running:

  • ollama-model-init exits 0; model qwen2.5:7b-instruct-q4_K_M (4.7 GB) cached in the ollama-models volume
  • ollama container is healthy
  • docker exec archiv-staging-backend-1 wget -qO- http://ollama:11434/api/tags returns the model list
  • one-shot inference via ollama run succeeds within the 8 GB limit

⚠️ This on-disk patch will be overwritten by the next CI deploy (which checks out the repo docker-compose.prod.yml, currently without Ollama). The fix must land in the repo — this issue + PR.

Fix (repo)

  • docker-compose.prod.yml: add ollama-model-init + ollama services and the ollama-models volume, using the corrected recipe:
    • entrypoint: ["/bin/sh", "-c"] on the init container; command uses until ollama list >/dev/null 2>&1; do sleep 1; done for readiness (no curl)
    • service healthcheck: ["CMD", "ollama", "list"] (no curl)
    • no container_name (prod namespaces by compose project); ADR-019 hardening (read_only, cap_drop: [ALL], no-new-privileges, tmpfs /tmp)
  • docker-compose.yml (dev): fix the same broken model-init entrypoint/command and the curl healthcheck so the dev stack actually starts Ollama.

Out of scope / follow-ups

  • The merged RestClientOllamaClient does not send an Authorization: Bearer header (the OLLAMA_API_KEY plumbing described in #737 was not implemented). OLLAMA_API_KEY is therefore omitted from the prod service for now. Track separately if auth is still wanted.
  • docs/DEPLOYMENT.md NL-search hardware tier + env-var rows, Prometheus ollama scrape job, and the Grafana latency dashboard (all listed in #737) remain open.

Acceptance

  • Fresh docker compose -f docker-compose.prod.yml ... up -d starts ollama-model-init → exits 0 → ollama reaches healthy within start_period.
  • docker exec <backend> wget -qO- http://ollama:11434/api/tags returns the model list.
  • A real NL search on staging returns smart results instead of 503.
  • Dev docker compose up -d likewise brings Ollama to healthy.
## Symptom On staging, every natural-language search returns **"Intelligente Suche nicht verfügbar"** (HTTP 503 `SMART_SEARCH_UNAVAILABLE`). Backend logs: ``` WARN org.raddatz.familienarchiv.search.RestClientOllamaClient — Ollama inference failed: ResourceAccessException ``` ## Root cause Two independent defects, both downstream of #737: 1. **Ollama was never added to `docker-compose.prod.yml`.** #737 added the `ollama` + `ollama-model-init` services and the `ollama_models` volume to the **dev** `docker-compose.yml` only. Staging and production deploy from `docker-compose.prod.yml` (a self-contained file, *not* an overlay), which has no Ollama service. The backend defaults to `app.ollama.base-url: http://ollama:11434` (`application.yaml`), so the conditional client bean is active and tries to connect to a host that does not exist → `ResourceAccessException` → 503. Confirmed on the host: no `ollama` container, no process, nothing on `:11434`. 2. **The model-init recipe merged in #737 is broken** (so even the dev compose never actually worked): - The `ollama/ollama` image's `ENTRYPOINT` is `ollama`, so `command: sh -c "..."` is parsed as `ollama sh -c "..."` → `Error: unknown command "sh" for "ollama"` → init exits 1 → `ollama` never starts (`service_completed_successfully` gate). - The image ships **no `curl`**, so both the init readiness loop (`until curl -sf .../api/tags`) and the service healthcheck (`["CMD","curl","-f",...]`) can never succeed. ## Hotfix already applied to staging (2026-06-06) To restore NL search immediately I patched the on-disk `/opt/familienarchiv/docker-compose.prod.yml`, pulled the model, and verified end-to-end. The corrected recipe (entrypoint override + `ollama list` for readiness/health, no curl) is now running: - `ollama-model-init` exits **0**; model `qwen2.5:7b-instruct-q4_K_M` (4.7 GB) cached in the `ollama-models` volume - `ollama` container is **healthy** - `docker exec archiv-staging-backend-1 wget -qO- http://ollama:11434/api/tags` returns the model list - one-shot inference via `ollama run` succeeds within the 8 GB limit ⚠️ This on-disk patch will be **overwritten by the next CI deploy** (which checks out the repo `docker-compose.prod.yml`, currently without Ollama). The fix must land in the repo — this issue + PR. ## Fix (repo) - [ ] `docker-compose.prod.yml`: add `ollama-model-init` + `ollama` services and the `ollama-models` volume, using the **corrected** recipe: - `entrypoint: ["/bin/sh", "-c"]` on the init container; command uses `until ollama list >/dev/null 2>&1; do sleep 1; done` for readiness (no curl) - service `healthcheck: ["CMD", "ollama", "list"]` (no curl) - no `container_name` (prod namespaces by compose project); ADR-019 hardening (`read_only`, `cap_drop: [ALL]`, `no-new-privileges`, tmpfs `/tmp`) - [ ] `docker-compose.yml` (dev): fix the same broken `model-init` entrypoint/command and the curl healthcheck so the dev stack actually starts Ollama. ## Out of scope / follow-ups - The merged `RestClientOllamaClient` does **not** send an `Authorization: Bearer` header (the `OLLAMA_API_KEY` plumbing described in #737 was not implemented). `OLLAMA_API_KEY` is therefore omitted from the prod service for now. Track separately if auth is still wanted. - `docs/DEPLOYMENT.md` NL-search hardware tier + env-var rows, Prometheus `ollama` scrape job, and the Grafana latency dashboard (all listed in #737) remain open. ## Acceptance - Fresh `docker compose -f docker-compose.prod.yml ... up -d` starts `ollama-model-init` → exits 0 → `ollama` reaches healthy within `start_period`. - `docker exec <backend> wget -qO- http://ollama:11434/api/tags` returns the model list. - A real NL search on staging returns smart results instead of 503. - Dev `docker compose up -d` likewise brings Ollama to healthy.
marcel added this to the Archive Intelligence — NL Search milestone 2026-06-06 19:18:31 +02:00
marcel added the P1-highbugdevops labels 2026-06-06 19:18:36 +02:00
marcel commented 2026-06-06 19:27:20 +02:00
Author
Owner
Copy Link

Follow-up: second root cause — idle model unload (cold-load timeout)

After the deploy fix, NL search worked on the first query and then went 503 again within minutes. The Ollama container stayed healthy throughout (ollama list passes regardless of whether the model is in RAM), so this was a separate issue:

  • ollama ps showed UNTIL <N> minutes from now — Ollama unloads the model after its default ~5 min keep-alive.
  • Once unloaded, the next query cold-loads the 4.7 GB model, which exceeds the backend's 30 s read timeout → ResourceAccessException → SMART_SEARCH_UNAVAILABLE. Warm inference is ~18 s; the cold load after idle is what timed out. (The original 17:20 failures in the first report were the same cold-load timeout right after the container restart.)

Fix (added to #759):

  • OLLAMA_KEEP_ALIVE=-1 on the ollama service (both compose files) — model stays resident, no idle unload. Verified on staging: ollama ps now shows UNTIL Forever; host has 47 GB free so the pinned ~5 GB is comfortable.
  • app.ollama.timeout-seconds 30 → 60 in application.yaml — absorbs the one unavoidable cold load (first query after an Ollama restart, before the model is pinned).

Staging is live again (keep-alive applied to the on-disk compose + container recreated). The timeout bump takes effect for prod/staging on the next backend deploy when #759 merges.

## Follow-up: second root cause — idle model unload (cold-load timeout) After the deploy fix, NL search worked on the first query and then went 503 again within minutes. The Ollama container stayed **healthy** throughout (`ollama list` passes regardless of whether the model is in RAM), so this was a separate issue: - `ollama ps` showed `UNTIL <N> minutes from now` — Ollama unloads the model after its default **~5 min keep-alive**. - Once unloaded, the next query cold-loads the 4.7 GB model, which exceeds the backend's **30 s** read timeout → `ResourceAccessException` → `SMART_SEARCH_UNAVAILABLE`. Warm inference is ~18 s; the cold load after idle is what timed out. (The original 17:20 failures in the first report were the same cold-load timeout right after the container restart.) **Fix (added to #759):** - `OLLAMA_KEEP_ALIVE=-1` on the ollama service (both compose files) — model stays resident, no idle unload. Verified on staging: `ollama ps` now shows `UNTIL Forever`; host has 47 GB free so the pinned ~5 GB is comfortable. - `app.ollama.timeout-seconds` 30 → 60 in `application.yaml` — absorbs the one unavoidable cold load (first query after an Ollama restart, before the model is pinned). Staging is live again (keep-alive applied to the on-disk compose + container recreated). The timeout bump takes effect for prod/staging on the next backend deploy when #759 merges.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P0-critical
Blocks a core user journey, causes data loss, or is a security/accessibility barrier. Work on this before P1+.
 P1-high
Significant friction on a main user journey; workaround exists but is non-obvious. Next up after P0.
 P2-medium
Noticeable improvement; doesn't block anything; schedule opportunistically.
 P3-later
Cosmetic or parking-lot work; fine to stay open indefinitely.
 audit
Read-only audit / assessment work; produces a report rather than changing code
 bug
Something isn't working
 cleanup
Removal of dead code, vague comments, naming violations, and scratch artifacts
 collaboration
We want to extend the family archive to have more collaboration tools
 conversation
descoped
We will do that later
 devops
documentation
README, ARCHITECTURE, GLOSSARY, CONTRIBUTING, per-domain docs
 epic
Marker for epic-level issues that group multiple child issues
 feature
file-upload
legibility
Codebase Legibility Refactor — preparing the codebase for human evaluation and bus-factor reduction
 needs-discussion
Has an open decision or design question that must be resolved before implementation can start.
 notification
person
phase-1: security
Security hygiene — must be done first
 phase-2: container-images
Production-ready multi-stage Docker images
 phase-3: prod-compose
Production compose overlay + Caddy reverse proxy
 phase-4: spring-prod-profile
Spring Boot production configuration profile
 phase-5: backups
Database and object storage backup strategy
 phase-6: deployment-docs
.env.example, DEPLOYMENT.md, runbook
 phase-7: monitoring
Prometheus, Loki, Grafana, Alertmanager
 refactor
Code restructuring without behaviour change
 security
test
ui
UI/UX and accessibility issues
 user
No Label P1-high bug devops
Milestone
No items
No Milestone Archive Intelligence — NL Search
Projects
Clear projects
No project
Assignees
Clear assignees
Jens marcel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: marcel/familienarchiv#758
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?