docs(infra): correct server specs — Hetzner Serverbörse i7-6700 64 GB, not CX32
All checks were successful
CI / Unit & Component Tests (push) Successful in 3m18s
CI / OCR Service Tests (push) Successful in 21s
CI / Backend Unit Tests (push) Successful in 3m46s
CI / fail2ban Regex (push) Successful in 48s
CI / Semgrep Security Scan (push) Successful in 23s
CI / Compose Bucket Idempotency (push) Successful in 1m6s
All checks were successful
CI / Unit & Component Tests (push) Successful in 3m18s
CI / OCR Service Tests (push) Successful in 21s
CI / Backend Unit Tests (push) Successful in 3m46s
CI / fail2ban Regex (push) Successful in 48s
CI / Semgrep Security Scan (push) Successful in 23s
CI / Compose Bucket Idempotency (push) Successful in 1m6s
Replace all references to the CX32 VPS (8 GB RAM, Hetzner Cloud) with the actual production server: a Hetzner Serverbörse dedicated server with an Intel Core i7-6700 (4C/8T, 3.4 GHz) and 64 GB RAM. Affected files: - .claude/personas/devops.md — monthly cost line + upgrade example - docs/infrastructure/production-compose.md — sizing section + cost table - docs/DEPLOYMENT.md — OCR memory table + OCR_MEM_LIMIT env var description - docs/adr/004-pdfbox-thumbnails.md — thumbnailExecutor memory ceiling note - docs/adr/021-tmpdir-persistent-volume-staging.md — OOMKill rationale in alternatives Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -154,9 +154,9 @@ Schedule monthly automated restore tests. If the restore fails, the backup is wo
|
||||
```
|
||||
Every alert needs: description, severity, likely cause, resolution steps, escalation path.
|
||||
|
||||
3. **Upgrading VPS tier before profiling**
|
||||
3. **Upgrading hardware before profiling**
|
||||
```
|
||||
# "The app feels slow" → upgrade from CX32 to CX42
|
||||
# "The app feels slow" → order more RAM / a faster CPU
|
||||
# Actual cause: unindexed query scanning 100k rows
|
||||
```
|
||||
Profile with Grafana dashboards first. Most perceived performance issues are application bugs, not resource constraints.
|
||||
@@ -404,8 +404,8 @@ Hetzner Object Storage (S3-compatible, replaces MinIO in prod)
|
||||
Prometheus + Loki + Alertmanager
|
||||
```
|
||||
|
||||
### Monthly Cost: ~23 EUR
|
||||
CX32 VPS (4 vCPU, 8GB RAM): 17 EUR · Object Storage (~200GB): 5 EUR · SMTP relay: ~1 EUR
|
||||
### Monthly Cost: ~6 EUR (excl. server)
|
||||
Hetzner dedicated server (Serverbörse, i7-6700, 64 GB RAM): see invoice · Object Storage (~200GB): 5 EUR · SMTP relay: ~1 EUR
|
||||
|
||||
### Reference Documentation
|
||||
- Full CI workflow, Gitea vs GitHub differences: `docs/infrastructure/ci-gitea.md`
|
||||
|
||||
@@ -52,11 +52,12 @@ The OCR service requires significant RAM for model loading. The dev compose sets
|
||||
|
||||
| Production target | RAM | Recommended OCR limit | Notes |
|
||||
|---|---|---|---|
|
||||
| Hetzner CX42 | 16 GB | 12 GB | Recommended for OCR-enabled production |
|
||||
| Hetzner CX32 | 8 GB | 6 GB | Accept reduced batch sizes and slower throughput |
|
||||
| Hetzner CX22 | 4 GB | — | Disable the OCR service (`profiles: [ocr]`); run OCR on demand only |
|
||||
| Current server (Hetzner Serverbörse, i7-6700) | 64 GB | 12 GB | Default `mem_limit: 12g` works comfortably |
|
||||
| ≥ 16 GB RAM | 16+ GB | 12 GB | Default works |
|
||||
| 8 GB RAM | 8 GB | 6 GB | Set `OCR_MEM_LIMIT=6g`; accept reduced batch sizes |
|
||||
| 4 GB RAM | 4 GB | — | Disable OCR service (`profiles: [ocr]`); run OCR on demand only |
|
||||
|
||||
A CX32 cannot honour the default `mem_limit: 12g` — set the `OCR_MEM_LIMIT=6g` env var (in `.env.production` / `.env.staging`, or as a Gitea secret consumed by the workflow) before deploying on a CX32. The prod compose interpolates this var with a 12g default.
|
||||
On servers with less than 16 GB RAM the default `mem_limit: 12g` cannot be honoured — set the `OCR_MEM_LIMIT` env var (in `.env.production` / `.env.staging`, or as a Gitea secret consumed by the workflow). The prod compose interpolates this var with a 12g default.
|
||||
|
||||
### Dev vs production differences
|
||||
|
||||
@@ -140,7 +141,7 @@ All vars are set in `.env` at the repo root (copy from `.env.example`). The back
|
||||
| `ALLOWED_PDF_HOSTS` | SSRF protection — comma-separated list of allowed PDF source hosts. **Do not widen to `*`** | `minio,localhost,127.0.0.1` | YES | — |
|
||||
| `KRAKEN_MODEL_PATH` | Directory containing Kraken HTR models (populated by `download-kraken-models.sh`) | `/app/models/` | — | — |
|
||||
| `BLLA_MODEL_PATH` | Kraken baseline layout analysis model path | `/app/models/blla.mlmodel` | — | — |
|
||||
| `OCR_MEM_LIMIT` | Container memory cap for ocr-service in `docker-compose.prod.yml`. Set to `6g` on CX32 hosts; leave unset on CX42+ to use the 12g default | `12g` (prod compose default) | — | — |
|
||||
| `OCR_MEM_LIMIT` | Container memory cap for ocr-service in `docker-compose.prod.yml`. Set to `6g` on servers with 8 GB RAM; leave unset (12g default) on servers with ≥ 16 GB RAM | `12g` (prod compose default) | — | — |
|
||||
| `XDG_CACHE_HOME` | XDG cache base dir — redirects Matplotlib and other XDG-aware libraries away from the read-only `HOME` (`/home/ocr`) to the writable cache volume | `/app/cache` | — | — |
|
||||
| `TORCH_HOME` | PyTorch model cache — redirects `~/.cache/torch` to the writable models volume | `/app/models/torch` | — | — |
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ Render thumbnails in-process in Spring Boot using **Apache PDFBox 3.0.4** (alrea
|
||||
|
||||
**Harder:**
|
||||
- PDFBox is a parser attack surface. Mitigated by a 30-second watchdog timeout in `ThumbnailAsyncRunner` and by the fire-and-forget contract (failures never break upload).
|
||||
- Memory ceiling: the `thumbnailExecutor` is capped at 2 threads on the CX32 (8 GB). A busy backfill alongside OCR can approach the 3 GB heap — acceptable but not comfortable. Streaming via `FileService.downloadFileStream` keeps this bounded for PDFs up to 50 MB.
|
||||
- Memory ceiling: the `thumbnailExecutor` is capped at 2 threads on memory-constrained hosts. A busy backfill alongside OCR can approach the 3 GB heap on an 8 GB server — acceptable but not comfortable. The current production server (64 GB) has ample headroom. Streaming via `FileService.downloadFileStream` keeps this bounded for PDFs up to 50 MB.
|
||||
|
||||
### Operational caveats (intentional)
|
||||
|
||||
|
||||
@@ -62,7 +62,7 @@ The `/tmp` tmpfs remains at 512 MB and continues to serve training-ZIP extractio
|
||||
## Alternatives considered
|
||||
|
||||
**Approach B — Enlarge `/tmp` to 4 GB**
|
||||
One-line change. Discarded because: (1) 4 GB tmpfs counts against the cgroup `mem_limit`; on CX32 hosts with `OCR_MEM_LIMIT=6g` the combined Surya resident set + tmpfs would trigger OOMKill on cold start; (2) staging GB-scale model files through RAM is using the wrong storage tier; (3) any future model larger than 4 GB requires another bump.
|
||||
One-line change. Discarded because: (1) 4 GB tmpfs counts against the cgroup `mem_limit`; on servers with `OCR_MEM_LIMIT=6g` the combined Surya resident set + tmpfs would trigger OOMKill on cold start; (2) staging GB-scale model files through RAM is using the wrong storage tier; (3) any future model larger than 4 GB requires another bump.
|
||||
|
||||
**Approach C — Both TMPDIR redirect and enlarged /tmp**
|
||||
Belt-and-suspenders: Approach A + 1 GB tmpfs. Discarded in favour of the cleaner Approach A. The defence-in-depth benefit does not outweigh the extra compose churn; the 512 MB cap on `/tmp` is intentional.
|
||||
|
||||
@@ -20,24 +20,19 @@ The observability stack (Prometheus, Loki, Grafana, Tempo, GlitchTip) ships as a
|
||||
|
||||
---
|
||||
|
||||
## VPS Sizing Recommendations
|
||||
## Server Sizing
|
||||
|
||||
### Recommended: Hetzner CX32
|
||||
### Current Production Server: Hetzner Dedicated (Serverbörse)
|
||||
|
||||
**Specs**: 4 vCPU, 8 GB RAM, 80 GB SSD · **Cost**: 17 EUR/mo
|
||||
**Specs**: Intel Core i7-6700 (4C/8T, 3.4 GHz), 64 GB RAM · acquired via Hetzner server auction
|
||||
|
||||
Sufficient for the application stack (Postgres, MinIO, OCR with `mem_limit: 12g`, backend, frontend, Caddy) on a CX32 today. Once the observability stack lands (Prometheus/Loki/Grafana/Alertmanager add ~2 GB) consider a CX42.
|
||||
Comfortably handles the full application stack (Postgres, MinIO, OCR with `mem_limit: 12g`, backend, frontend, Caddy, full observability stack) with headroom to spare. The 64 GB RAM means OCR, Ollama inference, and the observability stack can all run concurrently without memory pressure.
|
||||
|
||||
### When to Upgrade: Hetzner CX42
|
||||
### When to Reconsider Hardware
|
||||
|
||||
**Specs**: 8 vCPU, 16 GB RAM · **Cost**: 29 EUR/mo
|
||||
|
||||
Upgrade when:
|
||||
- Observability stack adds memory pressure (Loki + Grafana with >30 days retention)
|
||||
- OCR throughput needs scaling beyond a single-node Surya/Kraken setup
|
||||
- Real user load profiled in Grafana shows response-time degradation
|
||||
|
||||
Never upgrade the VPS tier before profiling — most perceived performance issues are application bugs, not resource constraints.
|
||||
- CPU is Skylake (2015) — single-threaded performance is the likely bottleneck before RAM
|
||||
- Profile with Grafana dashboards before concluding hardware is the constraint
|
||||
- Most perceived performance issues are application bugs (unindexed queries, N+1 loads), not resource limits
|
||||
|
||||
---
|
||||
|
||||
@@ -45,12 +40,11 @@ Never upgrade the VPS tier before profiling — most perceived performance issue
|
||||
|
||||
| Service | Cost |
|
||||
|---|---|
|
||||
| Hetzner CX32 VPS | 17.00 EUR |
|
||||
| Hetzner dedicated server (Serverbörse, i7-6700, 64 GB RAM) | see invoice |
|
||||
| Hetzner DNS | 0.00 EUR |
|
||||
| Hetzner SMTP relay | ~1.00 EUR |
|
||||
| **Total** | **~18 EUR/mo** |
|
||||
|
||||
MinIO data lives on the VPS disk (no Object Storage line item yet). The Hetzner OBS migration would add ~5 EUR/mo at ~200 GB.
|
||||
MinIO data lives on the server disk (no Object Storage line item yet). The Hetzner OBS migration would add ~5 EUR/mo at ~200 GB.
|
||||
|
||||
Equivalent SaaS stack: 200–300 EUR/mo.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user