devops: fix observability stack bind mounts failing in DooD runner #598

New Issue

marcel · 2026-05-15T18:59:31+02:00

marcel commented

2026-05-15 18:59:31 +02:00

Problem

The nightly and release CI jobs start the observability stack via docker compose -f docker-compose.observability.yml up. In Docker-out-of-Docker (DooD), the job container's workspace lives at /workspace/marcel/familienarchiv/, but only /var/run/docker.sock is volume-mapped to the host — no workspace directory is shared.

When Docker Compose resolves relative bind mounts like ./infra/observability/prometheus/prometheus.yml, it passes that absolute path to the host daemon. The host has no such path, so it auto-creates an empty directory there. The container then fails to start because the mount target is a directory, not a file:

Error response from daemon: failed to create task for container: error mounting
"/workspace/marcel/familienarchiv/infra/observability/prometheus/prometheus.yml"
to rootfs at "/etc/prometheus/prometheus.yml": not a directory

Five mounts are affected:

prometheus.yml
loki-config.yml
promtail-config.yml
tempo.yml
grafana/provisioning/

Fix

docker-compose.observability.yml: Make the config base path configurable via ${OBS_CONFIG_DIR:-./infra/observability}. Keeps working out-of-the-box for local docker compose up.
nightly.yml: Before starting the observability stack, docker cp the config tree from the job container to a stable host path (/srv/familienarchiv-staging/obs-configs). Add OBS_CONFIG_DIR=/srv/familienarchiv-staging/obs-configs to .env.staging.
release.yml: Same pattern with /srv/familienarchiv-production/obs-configs.

ADR-011 (single-tenant runner trust model)
Same DooD limitation previously solved for infra/minio/bootstrap.sh by baking into a custom image (#506)

## Problem The nightly and release CI jobs start the observability stack via `docker compose -f docker-compose.observability.yml up`. In Docker-out-of-Docker (DooD), the job container's workspace lives at `/workspace/marcel/familienarchiv/`, but only `/var/run/docker.sock` is volume-mapped to the host — **no workspace directory is shared**. When Docker Compose resolves relative bind mounts like `./infra/observability/prometheus/prometheus.yml`, it passes that absolute path to the **host** daemon. The host has no such path, so it auto-creates an empty directory there. The container then fails to start because the mount target is a directory, not a file: ``` Error response from daemon: failed to create task for container: error mounting "/workspace/marcel/familienarchiv/infra/observability/prometheus/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml": not a directory ``` Five mounts are affected: - `prometheus.yml` - `loki-config.yml` - `promtail-config.yml` - `tempo.yml` - `grafana/provisioning/` ## Fix 1. **`docker-compose.observability.yml`**: Make the config base path configurable via `${OBS_CONFIG_DIR:-./infra/observability}`. Keeps working out-of-the-box for local `docker compose up`. 2. **`nightly.yml`**: Before starting the observability stack, `docker cp` the config tree from the job container to a stable host path (`/srv/familienarchiv-staging/obs-configs`). Add `OBS_CONFIG_DIR=/srv/familienarchiv-staging/obs-configs` to `.env.staging`. 3. **`release.yml`**: Same pattern with `/srv/familienarchiv-production/obs-configs`. ## Related - ADR-011 (single-tenant runner trust model) - Same DooD limitation previously solved for `infra/minio/bootstrap.sh` by baking into a custom image (#506)

marcel added the P1-high bug devops labels 2026-05-15 18:59:34 +02:00

marcel referenced this issue from a commit

2026-05-15 19:03:22 +02:00

fix(ci): sync observability configs to host before docker compose up (#598)

marcel referenced a pull request that will close this issue

2026-05-15 19:03:45 +02:00

fix(ci): sync observability configs to host before docker compose up #599

marcel referenced this issue

2026-05-15 19:41:59 +02:00

fix(ci): sync observability configs to host before docker compose up #599

marcel closed this issue

2026-05-15 19:51:05 +02:00