fix(deploy): wire SENTRY_DSN and enable ECS JSON logging for prod (#641) #643

Merged

marcel merged 1 commits from fix/issue-641-sentry-dsn-json-logging into main 2026-05-20 09:39:20 +02:00

Conversation 7 Commits 1 Files Changed 2 +3 -1

marcel commented 2026-05-20 08:17:00 +02:00

Owner

Copy Link

Closes #641

Summary

• docker-compose.prod.yml: Add SENTRY_DSN: ${SENTRY_DSN:-} to backend environment — the variable was already written to .env.staging by nightly.yml (line 81) but never forwarded into the container, so Sentry.captureException() was silently a no-op
• docker-compose.prod.yml: Add LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs to backend environment — enables Spring Boot 4.0 ECS JSON structured logging so Loki gets single-entry JSON log lines with log.level instead of 50 separate unlinked stack trace lines
• loki-logs.json: Update Grafana Loki dashboard query from | logfmt to | json to match the new ECS format

Action required before this lands

Set the SENTRY_DSN secret in Gitea repository settings → Actions → Secrets with value:
https://686ec2daa9bb45dc8e264e1e2727c8a4@glitchtip.archiv.raddatz.cloud/2

Test plan

• Trigger a nightly deploy after setting the secret
• Confirm a known error (e.g. the document page 500 from #642) appears in GlitchTip
• In Grafana → Logs / App, select app = backend — confirm logs show as single JSON entries with detected_level visible
• Confirm stack traces appear as one log entry (not 50+ fragmented lines)

Closes #641 ## Summary - **`docker-compose.prod.yml`**: Add `SENTRY_DSN: ${SENTRY_DSN:-}` to backend environment — the variable was already written to `.env.staging` by `nightly.yml` (line 81) but never forwarded into the container, so `Sentry.captureException()` was silently a no-op - **`docker-compose.prod.yml`**: Add `LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs` to backend environment — enables Spring Boot 4.0 ECS JSON structured logging so Loki gets single-entry JSON log lines with `log.level` instead of 50 separate unlinked stack trace lines - **`loki-logs.json`**: Update Grafana Loki dashboard query from `| logfmt` to `| json` to match the new ECS format ## Action required before this lands Set the `SENTRY_DSN` secret in Gitea repository settings → Actions → Secrets with value: `https://686ec2daa9bb45dc8e264e1e2727c8a4@glitchtip.archiv.raddatz.cloud/2` ## Test plan - [ ] Trigger a nightly deploy after setting the secret - [ ] Confirm a known error (e.g. the document page 500 from #642) appears in GlitchTip - [ ] In Grafana → Logs / App, select `app = backend` — confirm logs show as single JSON entries with `detected_level` visible - [ ] Confirm stack traces appear as one log entry (not 50+ fragmented lines)

marcel added 1 commit 2026-05-20 08:17:01 +02:00

fix(deploy): wire SENTRY_DSN and enable ECS JSON logging for prod (#641) ... e89a90ff66

Pass SENTRY_DSN env var through to the backend container so the Sentry SDK
actually ships exceptions to GlitchTip — the variable was written to
.env.staging by nightly.yml but never forwarded into the container.

Enable Spring Boot 4.0 ECS structured logging (LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs)
so Loki receives single-entry JSON log lines with parsed log.level, enabling
detected_level filtering in Grafana instead of 50-line unlinked stack trace blobs.

Update Grafana Loki dashboard query from | logfmt to | json to match the new format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

marcel commented 2026-05-20 09:15:54 +02:00

Author

Owner

Copy Link

🏛️ Markus Keller — Software Architect

Verdict: ✅ Approved

Two independent gaps fixed correctly in one PR:

SENTRY_DSN passthrough (docker-compose.prod.yml)
The root cause is clear: nightly.yml writes the secret to .env.staging, but Docker Compose does not auto-inject env-file variables into containers — they must be explicitly declared under environment:. The ${SENTRY_DSN:-} empty-default pattern is correct: the SDK silently disables itself when the var is empty, so production starts cleanly even before the secret is set.

ECS structured logging (docker-compose.prod.yml + loki-logs.json)
LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs is the right lever — it's the Spring Boot 3.4+ property for ECS JSON output. Pairing it with the | json parser fix in the Grafana dashboard is necessary and correct; | logfmt was silently parsing nothing against Spring Boot's plain-text format.

No layer boundary concerns. Both fixes are strictly in the infrastructure config layer; no application logic is touched. The change is minimal and precisely scoped to the stated problem.

One note for future PRs: the Grafana dashboard JSON lives in the repo, which is good for reproducibility. If the log schema evolves (new fields added to ECS output), the dashboard queries may need co-updating — worth keeping in mind as the observability setup matures.

## 🏛️ Markus Keller — Software Architect **Verdict: ✅ Approved** Two independent gaps fixed correctly in one PR: **`SENTRY_DSN` passthrough** (`docker-compose.prod.yml`) The root cause is clear: `nightly.yml` writes the secret to `.env.staging`, but Docker Compose does not auto-inject env-file variables into containers — they must be explicitly declared under `environment:`. The `${SENTRY_DSN:-}` empty-default pattern is correct: the SDK silently disables itself when the var is empty, so production starts cleanly even before the secret is set. **ECS structured logging** (`docker-compose.prod.yml` + `loki-logs.json`) `LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs` is the right lever — it's the Spring Boot 3.4+ property for ECS JSON output. Pairing it with the `| json` parser fix in the Grafana dashboard is necessary and correct; `| logfmt` was silently parsing nothing against Spring Boot's plain-text format. No layer boundary concerns. Both fixes are strictly in the infrastructure config layer; no application logic is touched. The change is minimal and precisely scoped to the stated problem. **One note for future PRs:** the Grafana dashboard JSON lives in the repo, which is good for reproducibility. If the log schema evolves (new fields added to ECS output), the dashboard queries may need co-updating — worth keeping in mind as the observability setup matures.

marcel commented 2026-05-20 09:16:01 +02:00

Author

Owner

Copy Link

👨‍💻 Felix Brandt — Senior Fullstack Developer

Verdict: ✅ Approved

Pure infra config — no application code changes. The fixes are correct and complete.

docker-compose.prod.yml

• SENTRY_DSN: ${SENTRY_DSN:-} properly threads the Gitea secret through compose substitution into the container. The application.yaml binding sentry.dsn: ${SENTRY_DSN:} then picks it up via Spring's env-var interpolation.
• LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs is the correct Spring Boot env var. This is a first-class Spring Boot 3.4+ feature, not a workaround — it activates the bundled ECS encoder on the console appender.

loki-logs.json

• | json is required for ECS output (which is structured JSON). The old | logfmt parser was a silent no-op against Spring Boot's %d{…} %-5level … text pattern.
• The graph panel query (line 127) uses count_over_time without a parser — that's fine; it doesn't need to parse fields.

Nothing to block on. Ship it.

## 👨‍💻 Felix Brandt — Senior Fullstack Developer **Verdict: ✅ Approved** Pure infra config — no application code changes. The fixes are correct and complete. **`docker-compose.prod.yml`** - `SENTRY_DSN: ${SENTRY_DSN:-}` properly threads the Gitea secret through compose substitution into the container. The `application.yaml` binding `sentry.dsn: ${SENTRY_DSN:}` then picks it up via Spring's env-var interpolation. - `LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs` is the correct Spring Boot env var. This is a first-class Spring Boot 3.4+ feature, not a workaround — it activates the bundled ECS encoder on the console appender. **`loki-logs.json`** - `| json` is required for ECS output (which is structured JSON). The old `| logfmt` parser was a silent no-op against Spring Boot's `%d{…} %-5level …` text pattern. - The graph panel query (line 127) uses `count_over_time` without a parser — that's fine; it doesn't need to parse fields. Nothing to block on. Ship it.

marcel commented 2026-05-20 09:16:09 +02:00

Author

Owner

Copy Link

🛠️ Tobias Wendt — DevOps / Infrastructure

Verdict: ✅ Approved

This directly touches the production deployment config, so I reviewed it closely.

What's correct:

• SENTRY_DSN: ${SENTRY_DSN:-} uses an empty default (:-) — the backend starts cleanly whether or not the secret is set. No compose startup abort. Good defensive default.
• The secret never touches the image layer; it's injected at runtime via the environment: block. Safe.
• LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs is additive — it only changes the console log format, not log content or level. No regression risk.
• Dashboard | json update: necessary. Without it, Loki returns zero parsed fields, making the dashboard useless even though logs are being stored correctly.

Operational note:
The PR description correctly documents the required follow-up: set the SENTRY_DSN Gitea secret before the next nightly deploy. This is the correct workflow — the compose file just wires the variable; the actual secret value lives in Gitea. The suggested deploy order (merge #643 → nightly → verify Sentry active → merge #644 to validate error disappears) is sound.

Minor observation:
MANAGEMENT_TRACING_SAMPLING_PROBABILITY already has a ${…:-0.1} default pattern — the SENTRY_DSN addition follows the same convention consistently. Good.

## 🛠️ Tobias Wendt — DevOps / Infrastructure **Verdict: ✅ Approved** This directly touches the production deployment config, so I reviewed it closely. **What's correct:** - `SENTRY_DSN: ${SENTRY_DSN:-}` uses an empty default (`:-`) — the backend starts cleanly whether or not the secret is set. No compose startup abort. Good defensive default. - The secret never touches the image layer; it's injected at runtime via the `environment:` block. Safe. - `LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs` is additive — it only changes the console log format, not log content or level. No regression risk. - Dashboard `| json` update: necessary. Without it, Loki returns zero parsed fields, making the dashboard useless even though logs are being stored correctly. **Operational note:** The PR description correctly documents the required follow-up: set the `SENTRY_DSN` Gitea secret before the next nightly deploy. This is the correct workflow — the compose file just wires the variable; the actual secret value lives in Gitea. The suggested deploy order (merge #643 → nightly → verify Sentry active → merge #644 to validate error disappears) is sound. **Minor observation:** `MANAGEMENT_TRACING_SAMPLING_PROBABILITY` already has a `${…:-0.1}` default pattern — the `SENTRY_DSN` addition follows the same convention consistently. Good.

marcel commented 2026-05-20 09:16:15 +02:00

Author

Owner

Copy Link

📋 Elicit — Requirements Engineer

Verdict: ✅ Approved

Checking against issue #641 requirements:

Requirement	Addressed?
Sentry DSN forwarded to backend container	✅ SENTRY_DSN: ${SENTRY_DSN:-} added to backend environment:
GlitchTip receives backend errors	✅ Unblocked — DSN now reaches the Sentry SDK
Grafana Loki dashboard shows useful log entries	✅ `
No new secrets hardcoded	✅ Uses ${…:-} substitution
Staging deploy not broken when secret absent	✅ Empty fallback prevents compose abort

The fix is minimal and precisely scoped to the stated issues. No out-of-scope changes were introduced.

One open action (outside scope of this PR, documented in PR description): the SENTRY_DSN Gitea repository secret must be set to the GlitchTip project DSN before the next nightly deploy for Sentry capture to activate.

## 📋 Elicit — Requirements Engineer **Verdict: ✅ Approved** Checking against issue #641 requirements: | Requirement | Addressed? | |---|---| | Sentry DSN forwarded to backend container | ✅ `SENTRY_DSN: ${SENTRY_DSN:-}` added to backend `environment:` | | GlitchTip receives backend errors | ✅ Unblocked — DSN now reaches the Sentry SDK | | Grafana Loki dashboard shows useful log entries | ✅ `| logfmt` → `| json`, log format changed to ECS JSON | | No new secrets hardcoded | ✅ Uses `${…:-}` substitution | | Staging deploy not broken when secret absent | ✅ Empty fallback prevents compose abort | The fix is minimal and precisely scoped to the stated issues. No out-of-scope changes were introduced. **One open action** (outside scope of this PR, documented in PR description): the `SENTRY_DSN` Gitea repository secret must be set to the GlitchTip project DSN before the next nightly deploy for Sentry capture to activate.

marcel commented 2026-05-20 09:16:24 +02:00

Author

Owner

Copy Link

🔒 Nora "NullX" Steiner — Security Expert

Verdict: ✅ Approved

Reviewed with focus on secret handling and data exposure risk.

SENTRY_DSN: ${SENTRY_DSN:-}

• DSN contains an authentication key (format: https://<key>@host/<project>). It is injected via environment variable, not hardcoded — correct.
• The :- empty fallback is safe: the Sentry SDK disables itself silently, rather than crashing or leaking.
• The DSN value is not echoed to logs. Spring Boot ECS logging does not dump environment variables by default.
• In the Sentry/GlitchTip threat model: a leaked DSN allows an attacker to submit fake error reports (low impact) but not read existing data. Still correct to protect it as a secret.

LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs

• Changes log format to structured JSON. No new fields are added beyond what was logged before. ECS format does not include sensitive application data (passwords, tokens) unless they were already being logged — which would be a pre-existing issue, not introduced here.
• No concern with multi-line stack traces leaking into unexpected sinks; Promtail handles them as individual log lines.

loki-logs.json

• Dashboard change is cosmetic/query-level. No new data sources, no new access paths. The Grafana dashboard is internal-only (no public exposure in the compose config). No concern.

No security blockers.

## 🔒 Nora "NullX" Steiner — Security Expert **Verdict: ✅ Approved** Reviewed with focus on secret handling and data exposure risk. **`SENTRY_DSN: ${SENTRY_DSN:-}`** - DSN contains an authentication key (format: `https://<key>@host/<project>`). It is injected via environment variable, not hardcoded — correct. - The `:-` empty fallback is safe: the Sentry SDK disables itself silently, rather than crashing or leaking. - The DSN value is not echoed to logs. Spring Boot ECS logging does not dump environment variables by default. - In the Sentry/GlitchTip threat model: a leaked DSN allows an attacker to submit fake error reports (low impact) but not read existing data. Still correct to protect it as a secret. **`LOGGING_STRUCTURED_FORMAT_CONSOLE: ecs`** - Changes log format to structured JSON. No new fields are added beyond what was logged before. ECS format does not include sensitive application data (passwords, tokens) unless they were already being logged — which would be a pre-existing issue, not introduced here. - No concern with multi-line stack traces leaking into unexpected sinks; Promtail handles them as individual log lines. **`loki-logs.json`** - Dashboard change is cosmetic/query-level. No new data sources, no new access paths. The Grafana dashboard is internal-only (no public exposure in the compose config). No concern. No security blockers.

marcel commented 2026-05-20 09:16:32 +02:00

Author

Owner

Copy Link

🧪 Sara Holt — QA / Tester

Verdict: ✅ Approved

Infrastructure config changes — no unit/integration tests applicable. Validation is deployment-time.

Test plan for after merge + nightly deploy:

1. Verify SENTRY_DSN reaches the container

   docker inspect archiv-staging-backend-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep SENTRY
   

   Expected: SENTRY_DSN=https://...@glitchtip.archiv.raddatz.cloud/2

2. Verify ECS JSON logging is active

   docker logs archiv-staging-backend-1 --tail 5
   

   Expected: lines starting with { (JSON), not 2026-05-20 … (plain text)

3. Verify Loki ingests structured logs

   • Open Grafana → Explore → Loki datasource
   • Query: {job="archiv-staging-backend-1"} | json | level="INFO"
   • Expected: results with parsed level, message, @timestamp fields

4. Verify GlitchTip receives errors (after PR #644 is also merged and the 500 is confirmed fixed — but test the pipeline with a known error first)

   • Trigger a controlled 500 in staging
   • Check GlitchTip https://glitchtip.archiv.raddatz.cloud for the new event

Note: The SENTRY_DSN Gitea secret must be set before step 1 will pass.

## 🧪 Sara Holt — QA / Tester **Verdict: ✅ Approved** Infrastructure config changes — no unit/integration tests applicable. Validation is deployment-time. **Test plan for after merge + nightly deploy:** 1. **Verify SENTRY_DSN reaches the container** ```bash docker inspect archiv-staging-backend-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep SENTRY ``` Expected: `SENTRY_DSN=https://...@glitchtip.archiv.raddatz.cloud/2` 2. **Verify ECS JSON logging is active** ```bash docker logs archiv-staging-backend-1 --tail 5 ``` Expected: lines starting with `{` (JSON), not `2026-05-20 …` (plain text) 3. **Verify Loki ingests structured logs** - Open Grafana → Explore → Loki datasource - Query: `{job="archiv-staging-backend-1"} | json | level="INFO"` - Expected: results with parsed `level`, `message`, `@timestamp` fields 4. **Verify GlitchTip receives errors** (after PR #644 is also merged and the 500 is confirmed fixed — but test the pipeline with a known error first) - Trigger a controlled 500 in staging - Check GlitchTip `https://glitchtip.archiv.raddatz.cloud` for the new event **Note:** The `SENTRY_DSN` Gitea secret must be set before step 1 will pass.

marcel commented 2026-05-20 09:16:36 +02:00

Author

Owner

Copy Link

🎨 Leonie Voss — UI / UX Expert

Verdict: ✅ Approved

This is a pure infrastructure and observability config change — no frontend code, no UI components, no user-facing behavior is affected. Users will not see any change.

The indirect benefit is that once the Grafana dashboard is functional and GlitchTip receives errors, developers get faster feedback loops when UI-breaking backend errors occur. That's a positive for the overall development experience.

Nothing to flag from a UI perspective. LGTM.

## 🎨 Leonie Voss — UI / UX Expert **Verdict: ✅ Approved** This is a pure infrastructure and observability config change — no frontend code, no UI components, no user-facing behavior is affected. Users will not see any change. The indirect benefit is that once the Grafana dashboard is functional and GlitchTip receives errors, developers get faster feedback loops when UI-breaking backend errors occur. That's a positive for the overall development experience. Nothing to flag from a UI perspective. LGTM.

marcel referenced this pull request 2026-05-20 09:17:05 +02:00

fix(document): add trainingLabels to Document.full entity graph (#642) #644

marcel referenced this pull request 2026-05-20 09:17:18 +02:00

fix(document): add trainingLabels to Document.full entity graph (#642) #644

marcel merged commit e89a90ff66 into main 2026-05-20 09:39:20 +02:00

marcel deleted branch fix/issue-641-sentry-dsn-json-logging 2026-05-20 09:39:20 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

P0-critical

Blocks a core user journey, causes data loss, or is a security/accessibility barrier. Work on this before P1+.

P1-high

Significant friction on a main user journey; workaround exists but is non-obvious. Next up after P0.

P2-medium

Noticeable improvement; doesn't block anything; schedule opportunistically.

P3-later

Cosmetic or parking-lot work; fine to stay open indefinitely.

audit

Read-only audit / assessment work; produces a report rather than changing code

bug

Something isn't working

cleanup

Removal of dead code, vague comments, naming violations, and scratch artifacts

collaboration

We want to extend the family archive to have more collaboration tools

conversation

descoped

We will do that later

devops

documentation

README, ARCHITECTURE, GLOSSARY, CONTRIBUTING, per-domain docs

epic

Marker for epic-level issues that group multiple child issues

feature

file-upload

legibility

Codebase Legibility Refactor — preparing the codebase for human evaluation and bus-factor reduction

notification

person

phase-1: security

Security hygiene — must be done first

phase-2: container-images

Production-ready multi-stage Docker images

phase-3: prod-compose

Production compose overlay + Caddy reverse proxy

phase-4: spring-prod-profile

Spring Boot production configuration profile

phase-5: backups

Database and object storage backup strategy

phase-6: deployment-docs

.env.example, DEPLOYMENT.md, runbook

phase-7: monitoring

Prometheus, Loki, Grafana, Alertmanager

refactor

Code restructuring without behaviour change

security

test

ui

UI/UX and accessibility issues

user

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

Jens marcel

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marcel/familienarchiv#643