marcel/familienarchiv
1
0
Fork 0
Code Issues 119 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(search): NL search — auto-summaries and entity extraction per document #744

New Issue
Closed
opened 2026-06-06 12:17:11 +02:00 by marcel · 0 comments
marcel commented 2026-06-06 12:17:11 +02:00
Owner
Copy Link

Part of epic #735. Post-v1. Depends on translation infrastructure (#741) — same Ollama pipeline.

Goal

Every letter gets a one-sentence gist plus structured extraction of places, dates, events, and people mentioned. This structured output feeds the map view, person-linking, and reading journeys.

Scope

  • Batch job: process all existing transcriptions once; incremental processing for new uploads
  • Per-document output stored in new columns or a document_enrichments table:
    • summary: one-sentence gist in modern German
    • mentioned_places: extracted place names
    • mentioned_events: extracted events/occasions
    • mentioned_persons_raw: names of people mentioned in the letter body (not sender/receiver)
  • Admin UI to trigger/re-trigger enrichment on a document or in bulk
  • Enrichment data surfaces in document detail view and feeds search ranking

Open questions before implementation

  • Storage: new document_enrichments table or extend Document entity?
  • Batch strategy: process all at once (background job) or lazily on first view?
  • Cost/time estimate: how long does one enrichment take on the CPU server? (~1–2s × 5000 letters = hours)
  • Who can trigger batch enrichment? ADMIN only?
Part of epic #735. Post-v1. Depends on translation infrastructure (#741) — same Ollama pipeline. ## Goal Every letter gets a one-sentence gist plus structured extraction of places, dates, events, and people mentioned. This structured output feeds the map view, person-linking, and reading journeys. ## Scope - Batch job: process all existing transcriptions once; incremental processing for new uploads - Per-document output stored in new columns or a `document_enrichments` table: - `summary`: one-sentence gist in modern German - `mentioned_places`: extracted place names - `mentioned_events`: extracted events/occasions - `mentioned_persons_raw`: names of people mentioned in the letter body (not sender/receiver) - Admin UI to trigger/re-trigger enrichment on a document or in bulk - Enrichment data surfaces in document detail view and feeds search ranking ## Open questions before implementation - Storage: new `document_enrichments` table or extend `Document` entity? - Batch strategy: process all at once (background job) or lazily on first view? - Cost/time estimate: how long does one enrichment take on the CPU server? (~1–2s × 5000 letters = hours) - Who can trigger batch enrichment? ADMIN only?
marcel added this to the Archive Intelligence — NL Search milestone 2026-06-06 12:17:11 +02:00
marcel added the P3-laterfeature labels 2026-06-06 12:19:07 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P0-critical
Blocks a core user journey, causes data loss, or is a security/accessibility barrier. Work on this before P1+.
 P1-high
Significant friction on a main user journey; workaround exists but is non-obvious. Next up after P0.
 P2-medium
Noticeable improvement; doesn't block anything; schedule opportunistically.
 P3-later
Cosmetic or parking-lot work; fine to stay open indefinitely.
 audit
Read-only audit / assessment work; produces a report rather than changing code
 bug
Something isn't working
 cleanup
Removal of dead code, vague comments, naming violations, and scratch artifacts
 collaboration
We want to extend the family archive to have more collaboration tools
 conversation
descoped
We will do that later
 devops
documentation
README, ARCHITECTURE, GLOSSARY, CONTRIBUTING, per-domain docs
 epic
Marker for epic-level issues that group multiple child issues
 feature
file-upload
legibility
Codebase Legibility Refactor — preparing the codebase for human evaluation and bus-factor reduction
 needs-discussion
Has an open decision or design question that must be resolved before implementation can start.
 notification
person
phase-1: security
Security hygiene — must be done first
 phase-2: container-images
Production-ready multi-stage Docker images
 phase-3: prod-compose
Production compose overlay + Caddy reverse proxy
 phase-4: spring-prod-profile
Spring Boot production configuration profile
 phase-5: backups
Database and object storage backup strategy
 phase-6: deployment-docs
.env.example, DEPLOYMENT.md, runbook
 phase-7: monitoring
Prometheus, Loki, Grafana, Alertmanager
 refactor
Code restructuring without behaviour change
 security
test
ui
UI/UX and accessibility issues
 user
No Label P3-later feature
Milestone
No items
No Milestone Archive Intelligence — NL Search
Projects
Clear projects
No project
Assignees
Clear assignees
Jens marcel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: marcel/familienarchiv#744
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?