docs(import): add import-migration analysis + normalizer spec
Document the raw archive spreadsheet findings (IMP-01..12) and a requirements spec for an offline normalizer that produces a clean canonical dataset before import. Local docs only; no Gitea issue yet. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
384
docs/import-migration/02-normalization-spec.md
Normal file
384
docs/import-migration/02-normalization-spec.md
Normal file
@@ -0,0 +1,384 @@
|
||||
# Spec — Import Normalizer
|
||||
|
||||
> Authored in the voice of **"Elicit"**, requirements engineer (see
|
||||
> `.claude/personas/req_engineer.md`). This is a requirements artifact: it states
|
||||
> *what* the normalizer must do and *how we'll know it's done*, in problem/behaviour
|
||||
> language. Technology choices already made during brainstorming (Python, openpyxl,
|
||||
> overrides-and-rerun) are recorded as **constraints**, not re-litigated here.
|
||||
|
||||
- **Status:** Draft for review
|
||||
- **Date:** 2026-05-25
|
||||
- **Related:** [`01-findings-spreadsheet-analysis.md`](./01-findings-spreadsheet-analysis.md) (issues `IMP-01..12`), [`README.md`](./README.md)
|
||||
- **Scope boundary:** This spec covers the **offline normalizer** that turns the raw
|
||||
spreadsheets into a clean, canonical dataset + review artifacts. Wiring the canonical
|
||||
contract into the Java `MassImportService` and the `Document`/`Person` model is **Phase 2**
|
||||
and gets its own spec. This spec only *defines the contract* Phase 2 must satisfy.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project Brief
|
||||
|
||||
**Vision.** Turn the family's human-curated, free-form archive spreadsheets into a clean,
|
||||
canonical dataset that imports deterministically — without hand-editing thousands of rows
|
||||
and without losing the historical nuance of how things were originally written.
|
||||
|
||||
**Problem.** The real archive (`…aktuell…xlsx`, 7,943 rows) and the person register
|
||||
(`Personendatei 2.xlsx`, 163 people) were authored for humans to read, not machines to
|
||||
import. Dates are written as they appeared in each letter (≈90% unparseable by the current
|
||||
importer), the column layout differs from what the importer expects, and the same person
|
||||
appears under many names. Importing as-is produces garbage (see `IMP-01..12`).
|
||||
|
||||
**Goal (measurable).**
|
||||
- G1 — After the automated pass, **≤ 5%** of dated rows remain `UNKNOWN`; after the
|
||||
overrides-iteration loop, **≤ 0.5%**.
|
||||
- G2 — **100%** of source rows are represented in the canonical output or in a review file —
|
||||
*zero silent drops*.
|
||||
- G3 — **100%** of original values (raw date string, raw name string, source row number)
|
||||
are preserved.
|
||||
- G4 — A full run over the current inputs completes in **< 60 s** on the dev laptop and is
|
||||
**byte-identical** when re-run with unchanged inputs+overrides.
|
||||
|
||||
**Primary actor.** Marcel — solo owner & data steward (tech comfort 4/5). Also: a future
|
||||
agent re-running the pipeline; and the `MassImportService` as the downstream consumer.
|
||||
|
||||
**Non-Goals (explicitly out of scope).**
|
||||
- NG1 — Changing `MassImportService` or the DB schema (that is Phase 2).
|
||||
- NG2 — Uploading/attaching the ~7,000 PDFs (they arrive later; import matches by `index`).
|
||||
- NG3 — A GUI. The interface is spreadsheets in, CSVs out, an overrides file hand-edited.
|
||||
- NG4 — Perfect genealogical reconstruction. We resolve confidently-matchable people; the
|
||||
long tail stays as provisional persons.
|
||||
- NG5 — OCR/transcription content (the new xlsx has no transcription column).
|
||||
|
||||
**Key assumptions.** (A1) Sheet `Familienarchiv` is the document source of truth.
|
||||
(A2) Archive date range is **1873–1957** (drives the 2-digit-year century rule).
|
||||
(A3) `index` is the stable document key and the basis for future PDF matching.
|
||||
(A4) `Schlagwort` is a broad tag; `Inhalt` is a short summary/topic.
|
||||
|
||||
**Risks.** (R1) 2-digit/partial dates are genuinely ambiguous → mitigated by precision flag
|
||||
+ overrides. (R2) Name matching false-positives merge distinct people → mitigated by
|
||||
conservative matching + review before merge. (R3) Source spreadsheet may be re-exported with
|
||||
layout drift → mitigated by header-name-based mapping, not fixed indices.
|
||||
|
||||
---
|
||||
|
||||
## 2. Personas
|
||||
|
||||
**Marcel — Data Steward.** Role: solo owner of Familienarchiv. Context: holds the complete
|
||||
raw archive; PDFs follow. Tech comfort: 4/5 (semi-technical, reads CSV/spreadsheets fluently,
|
||||
not keen to hand-edit 7,600 rows). Primary goal: a clean, importable dataset he trusts.
|
||||
Frustrations: dates in ~20 formats; one ancestor under 4 name variants. **JTBD:** *"When I
|
||||
have raw, human-curated archive spreadsheets, I want to transform them into a clean importable
|
||||
dataset without losing how things were originally written, so I can load the archive and keep
|
||||
correcting edge cases as they surface."*
|
||||
|
||||
**The Returning Agent.** Role: a future assistant session resuming the work. Goal: re-run the
|
||||
pipeline deterministically and understand exactly what still needs human input. **JTBD:**
|
||||
*"When I pick this up cold, I want one command and a clear residue report, so I can continue
|
||||
without re-deriving context."*
|
||||
|
||||
---
|
||||
|
||||
## 3. Constraints & Decisions Already Made
|
||||
|
||||
These were settled during brainstorming and are fixed inputs to the requirements below.
|
||||
|
||||
| # | Decision | Rationale |
|
||||
| --- | --- | --- |
|
||||
| C1 | **New canonical layout** with explicit headers (not the old positional ODS shape). | Fits the new data; importer becomes header-driven in Phase 2. |
|
||||
| C2 | Dates stored as **parsed (nullable) + raw + precision**. | Historical archive; never lose the original; enable "ca. 1916". |
|
||||
| C3 | **Include person resolution** (register + alias/marriage map → canonical persons) in this effort. | Maiden-name dedup needs the register. |
|
||||
| C4 | **Overrides-file + re-run** loop for residue. | Deterministic, diffable, repeatable. |
|
||||
| C5 | Implementation: **Python 3.12 + openpyxl**, standalone tool at `tools/import-normalizer/`. | Fast iteration; no Spring rebuild / coverage gate on transform code. |
|
||||
| C6 | Century rule for archive **1873–1957**: 2-digit `00–57`→`19YY`, `73–99`→`18YY`, `58–72`→**flag**; 3-digit `DDD`→`1DDD`; never 20xx. | Stated by Marcel. Boundaries live in config. |
|
||||
| C7 | `Schlagwort`→tag, `Inhalt`→summary. | Matches importer's existing semantics. |
|
||||
| C8 | Non-register correspondents become **provisional persons**. | ~945 distinct sender strings vs 163 register people. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Functional Requirements
|
||||
|
||||
Each requirement has a stable ID. User stories use Connextra + Given-When-Then; system rules
|
||||
use EARS. Traceability to findings in §8.
|
||||
|
||||
### 4.1 Ingest & layout (`FR-INGEST`, `FR-MAP`)
|
||||
|
||||
**US-MAP-01** — *As the data steward, I want each source column mapped to a named canonical
|
||||
field regardless of its position, so a re-exported spreadsheet with shifted columns still
|
||||
imports correctly.*
|
||||
- AC1 — Given the `Familienarchiv` sheet, when the normalizer reads the header row, then it
|
||||
maps columns by **header name** (not fixed index) to the canonical fields.
|
||||
- AC2 — Given a header the normalizer does not recognise, when it runs, then it records the
|
||||
unknown header in `review/summary.txt` and continues (does not crash).
|
||||
- AC3 — Given a required source header is **absent**, when it runs, then it aborts with a
|
||||
clear message naming the missing header (fail loud, before producing partial output).
|
||||
|
||||
- **REQ-INGEST-01** — The normalizer shall read only the `Familienarchiv` sheet of the
|
||||
document workbook and the `Tabelle1` sheet of the person workbook.
|
||||
- **REQ-MAP-01** — Header matching shall be case-insensitive and tolerant of internal
|
||||
multiple spaces (e.g. `"Datum des Briefes"`).
|
||||
|
||||
### 4.2 Row triage (`FR-TRIAGE`) — resolves IMP-06, IMP-07, IMP-08
|
||||
|
||||
**US-TRIAGE-01** — *As the data steward, I want rows that have data but no index surfaced
|
||||
rather than dropped, so I never lose a letter silently.*
|
||||
- AC1 — Given a row whose `index` is blank but which has any other non-empty cell, when the
|
||||
normalizer runs, then that row is written to `review/blank-index-rows.csv` with its source
|
||||
row number and is **not** emitted as a canonical document.
|
||||
- AC2 — Given a fully empty row, when it runs, then the row is skipped and counted (not
|
||||
reported as an anomaly).
|
||||
|
||||
- **REQ-TRIAGE-01** — If two or more rows resolve to the same `index`, then the normalizer
|
||||
shall emit all of them to `review/duplicate-index.csv` and mark each canonical row
|
||||
`needs_review = duplicate_index` (it shall **not** silently drop either).
|
||||
- **REQ-TRIAGE-02** — Where a row is identified as a section/banner row (blank index, text
|
||||
only in a name column), the normalizer shall classify it as such in the blank-index report.
|
||||
- **REQ-TRIAGE-03** — Rows whose `index` ends in `x` (a transcription/back-side of the base
|
||||
letter, not yet independently mappable) shall be **skipped** — not emitted as a canonical
|
||||
document — and written to `review/skipped-x-suffix.csv` with their source row and base index
|
||||
(`index` minus the trailing `x`), so they can be linked in a later pass. (Resolves IMP-10.)
|
||||
|
||||
### 4.3 Date normalization (`FR-DATE`) — resolves IMP-02, IMP-03
|
||||
|
||||
**US-DATE-01** — *As the data steward, I want every date interpreted as precisely as the
|
||||
source allows, with the original always kept, so I can sort the archive and still see what the
|
||||
letter actually said.*
|
||||
- AC1 — Given a parseable date, when normalized, then `date_iso` holds the best-effort ISO
|
||||
date, `date_raw` holds the verbatim source string, and `date_precision` ∈
|
||||
`{DAY, MONTH, SEASON, YEAR, RANGE, APPROX, UNKNOWN}`.
|
||||
- AC2 — Given an unparseable date, when normalized, then `date_iso` is empty,
|
||||
`date_precision = UNKNOWN`, `date_raw` is preserved, and the value appears in
|
||||
`review/unparsed-dates.csv`.
|
||||
- AC3 — Given the same `date_raw` appears in `overrides/dates.csv`, when normalized, then the
|
||||
override's `(iso, precision)` wins over the automatic parse.
|
||||
|
||||
- **REQ-DATE-01** — The parser shall accept, at minimum, these forms (see §10 examples):
|
||||
Excel/ISO; `D.M.YYYY`/`D.M.YY`; `D/M. YY[YY]` (slash treated as dot); Roman-numeral months
|
||||
`I–XII`; German + English month names, full and abbreviated, with or without a separating
|
||||
space; `Month YYYY`; season/holiday + year; bare `YYYY`; and start-anchored ranges.
|
||||
- **REQ-DATE-02** — Precision shall be assigned by what is known: full day → `DAY`; month+year
|
||||
→ `MONTH` (day = 1); a **named feast/holiday + year** → resolved to its **actual calendar
|
||||
date for that year** → `DAY`; a **season + year** → representative mid-season month (day = 1)
|
||||
→ `SEASON`; year only → `YEAR` (month = Jan, day = 1); a range → start date + `RANGE`; a
|
||||
value carrying an uncertainty marker (`?`, `um`, `ca`, `circa`) → `APPROX` with best-effort date.
|
||||
- **REQ-DATE-03** — Two-digit and three-digit years shall be expanded per **C6**; a 2-digit
|
||||
year in `58–72` shall yield `UNKNOWN` + a review entry rather than a guess.
|
||||
- **REQ-DATE-04** — Trailing editorial notes (e.g. `", 2. Brief"`) shall be stripped before
|
||||
parsing and preserved (kept within `date_raw`; not invented into the date).
|
||||
- **REQ-DATE-05** — The parser shall be pure and side-effect-free so it can be unit-tested in
|
||||
isolation (see NFR-TEST-01).
|
||||
- **REQ-DATE-06** — **Movable feasts are never mapped to a fixed month**; they shall be
|
||||
computed per year from Easter (Gauss/Butcher computus): Karfreitag = Easter−2, Ostern =
|
||||
Easter Sunday, Himmelfahrt = Easter+39, Pfingst(sonntag) = Easter+49, Pfingstmontag =
|
||||
Easter+50, Fronleichnam = Easter+60, 1.–4. Advent = the 4th…1st Sunday before 25 Dec. Fixed
|
||||
feasts use a lookup table (Neujahr=01-01, Heiligabend=12-24, Weihnachten=12-25,
|
||||
Silvester=12-31, …). Seasons map to representative months: Frühling/Frühjahr=Apr, Sommer=Jul,
|
||||
Herbst=Oct, Winter=Jan. The feast/season tables and Easter algorithm live in `config.py`
|
||||
(NFR-MAINT-01).
|
||||
|
||||
### 4.4 Person resolution & dedup (`FR-PERS`, `FR-DEDUP`) — resolves IMP-04, IMP-05, IMP-11
|
||||
|
||||
**US-PERS-01** — *As the data steward, I want the genealogical register turned into canonical
|
||||
people with all their known facts, so documents can link to real persons.*
|
||||
- AC1 — Given a register row, when parsed, then a canonical person is produced with
|
||||
`person_id`, name parts, `maiden_name`, birth/death (parsed + raw + place), spouse,
|
||||
generation, nickname, notes — applying the same date rules as §4.3 to birth/death dates.
|
||||
- AC2 — Given multi-value given names (`"Charlotte,Meta,Jacobi"`), when parsed, then the
|
||||
primary given name is the first; the remainder are retained as additional names/aliases.
|
||||
|
||||
**US-PERS-02** — *As the data steward, I want each sender/receiver string matched to a
|
||||
canonical person where possible and never dropped otherwise, so the correspondence graph is
|
||||
complete.*
|
||||
- AC1 — Given a sender/receiver string, when resolved, then it maps to a register
|
||||
`person_id` via the alias index (exact → normalized/casefold → conservative fuzzy).
|
||||
- AC2 — Given no confident match, when resolved, then a **provisional person** is created from
|
||||
the cleaned string, linked, and listed in `review/unmatched-names.csv` (occurrence count +
|
||||
example source rows).
|
||||
- AC3 — Given the string appears in `overrides/names.csv`, when resolved, then it maps to the
|
||||
specified `person_id` (override wins).
|
||||
- AC4 — Given a multi-person receiver cell (`"Eugenie u Walter de Gruyter"`, `"Herbert u
|
||||
Clara"`, `"…//…"`, `"Hedi und Tutu (Gruber)"`), when resolved, then it is split into
|
||||
individual people, each resolved independently; ambiguous space-joined pairs
|
||||
(`"Ella Anita"`) are emitted to `review/ambiguous-receivers.csv` rather than guessed.
|
||||
|
||||
- **REQ-DEDUP-01** — The alias index shall be derived from the register: canonical
|
||||
"First Last", maiden form (`geb als`), spouse-surname married form, nickname, and
|
||||
first-name-only **only when unambiguous** across the register.
|
||||
- **REQ-DEDUP-02** — The normalizer shall not merge two distinct strings into one person on
|
||||
fuzzy similarity alone above a configured threshold without the match being reported; merges
|
||||
must be auditable.
|
||||
- **REQ-PERS-01** — Sender cells shall be parsed for multi-person content using the same rules
|
||||
as receiver cells (today the importer parses only receivers — IMP-11).
|
||||
|
||||
### 4.5 Overrides & idempotency (`FR-OVR`) — supports the iteration loop
|
||||
|
||||
- **REQ-OVR-01** — When the normalizer runs, then it shall load `overrides/dates.csv` and
|
||||
`overrides/names.csv` if present and apply them; absence of either file shall not be an error.
|
||||
- **REQ-OVR-02** — While overrides are unchanged and inputs are unchanged, re-running shall
|
||||
produce **byte-identical** canonical outputs and review files (NFR-IDEM-01).
|
||||
- **REQ-OVR-03** — Each override application shall be counted in `review/summary.txt` (how many
|
||||
dates/names were resolved by override vs automatically).
|
||||
|
||||
### 4.6 Canonical output & provenance (`FR-OUT`, `FR-PROV`) — resolves IMP-01, IMP-09, IMP-12
|
||||
|
||||
- **REQ-OUT-01** — The normalizer shall write `out/canonical-documents.xlsx` and
|
||||
`out/canonical-persons.xlsx` with the headered schemas in §6.
|
||||
- **REQ-PROV-01** — Every canonical document row shall carry `source_row` (1-based row number
|
||||
in the source sheet) so any value can be traced back to the original.
|
||||
- **REQ-PROV-02** — Every canonical row shall carry a `needs_review` field listing zero or more
|
||||
flags (`duplicate_index`, `unparsed_date`, `unmatched_sender`, `unmatched_receiver`,
|
||||
`index_file_mismatch`, …) so the import and the UI can foreground uncertain data.
|
||||
- **REQ-OUT-02** — Where the source `Datei` path disagrees with the index-derived filename
|
||||
(IMP-09), the normalizer shall record the discrepancy in `review/index-file-mismatch.csv`
|
||||
and flag the row; it shall **not** alter the `index` (the stable key).
|
||||
|
||||
---
|
||||
|
||||
## 5. Non-Functional Requirements
|
||||
|
||||
| ID | Category | Requirement (measurable) |
|
||||
| --- | --- | --- |
|
||||
| NFR-DATA-01 | Data integrity | 100% of source rows are accounted for in output **or** a review file; 100% of original date/name strings preserved verbatim. |
|
||||
| NFR-IDEM-01 | Determinism | Identical inputs + overrides ⇒ byte-identical outputs across runs and machines. |
|
||||
| NFR-PERF-01 | Performance | Full run over 7,943 doc rows + 163 person rows completes in < 60 s on the dev laptop. |
|
||||
| NFR-ACCUR-01 | Date accuracy | After automated pass, `UNKNOWN` dates ≤ 5% of dated rows; after overrides iteration, ≤ 0.5%. |
|
||||
| NFR-ACCUR-02 | Name coverage | Every sender/receiver occurrence yields a linked person (register or provisional); 0 dropped. |
|
||||
| NFR-I18N-01 | Encoding | UTF-8 end-to-end; German diacritics and ß round-trip with no mojibake in any output. |
|
||||
| NFR-TEST-01 | Testability | `dates.py` and `persons.py` have pytest tests covering every format/alias category in §10 with real examples from the archive. |
|
||||
| NFR-MAINT-01 | Maintainability | Column-name map, century boundaries, season→month map, and fuzzy threshold live in `config.py`, not inline in logic. |
|
||||
| NFR-OBSERV-01 | Observability | `review/summary.txt` reports per-run stats: rows in, documents out, dates by precision, names matched vs provisional, overrides applied, anomalies by type. |
|
||||
| NFR-SAFETY-01 | Source safety | Source workbooks are opened read-only and never written. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Data Dictionary (canonical contract)
|
||||
|
||||
This is the contract Phase 2 (the importer) must consume. Field-level, format-level — not a
|
||||
DB schema.
|
||||
|
||||
### 6.1 `canonical-documents.xlsx`
|
||||
|
||||
| Field | Required | Format / values | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `index` | yes | string | Stable key; basis for PDF matching. |
|
||||
| `box` | no | string | from `Box`. |
|
||||
| `folder` | no | string | from `Mappe`. |
|
||||
| `sender_person_id` | no | person_id | resolved; empty if no sender. |
|
||||
| `sender_name` | no | string | canonical display name (or cleaned raw if provisional). |
|
||||
| `receiver_person_ids` | no | `id\|id\|…` | pipe-separated. |
|
||||
| `receiver_names` | no | `name\|name\|…` | pipe-separated, aligned with ids. |
|
||||
| `date_iso` | no | `YYYY-MM-DD` | best-effort; empty if `UNKNOWN`. |
|
||||
| `date_raw` | no | string | verbatim source date. |
|
||||
| `date_precision` | yes | enum | `DAY\|MONTH\|SEASON\|YEAR\|RANGE\|APPROX\|UNKNOWN`. |
|
||||
| `location` | no | string | from `Ort`. |
|
||||
| `tags` | no | `tag\|tag` | from `Schlagwort`. |
|
||||
| `summary` | no | string | from `Inhalt`. |
|
||||
| `source_row` | yes | int | provenance (NFR-DATA-01). |
|
||||
| `needs_review` | yes | `flag\|flag` or empty | review flags (REQ-PROV-02). |
|
||||
|
||||
### 6.2 `canonical-persons.xlsx`
|
||||
|
||||
| Field | Required | Format | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `person_id` | yes | slug | stable id (e.g. `de-gruyter-eugenie`); collisions suffixed. |
|
||||
| `last_name` | yes | string | from `Familienname`. |
|
||||
| `first_name` | no | string | primary given name. |
|
||||
| `maiden_name` | no | string | from `geb als` — drives dedup. |
|
||||
| `title` | no | string | e.g. honorifics if present. |
|
||||
| `nickname` | no | string | from quoted `Bemerkung`/spouse field. |
|
||||
| `birth_date` / `birth_date_raw` / `birth_place` | no | ISO / string / string | §4.3 rules. |
|
||||
| `death_date` / `death_date_raw` / `death_place` | no | ISO / string / string | §4.3 rules. |
|
||||
| `spouse` | no | person_id or name | from `verheiratet mit`. |
|
||||
| `generation` | no | string | `G 1`..`G 4`. |
|
||||
| `notes` | no | string | from `Bemerkung`. |
|
||||
| `aliases` | no | `a\|b\|c` | every surface form that maps here. |
|
||||
| `provisional` | yes | bool | true if created from a document string, not the register. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Prioritized Backlog (MoSCoW)
|
||||
|
||||
| ID | Item | MoSCoW | Effort | Depends on |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| B1 | Project scaffolding + read both workbooks (`FR-INGEST`, header map `FR-MAP`) | Must | S | — |
|
||||
| B2 | Row triage + blank/duplicate/empty reports (`FR-TRIAGE`) | Must | S | B1 |
|
||||
| B3 | Date parser + precision + century rule + Easter/feast computus + season map + tests (`FR-DATE`) | Must | L | B1 |
|
||||
| B4 | Person register parser → canonical persons (`FR-PERS` US-PERS-01) | Must | M | B1 |
|
||||
| B5 | Alias index + name resolution + multi-person split (`FR-DEDUP`, US-PERS-02) | Must | L | B4 |
|
||||
| B6 | Overrides load + apply + idempotency (`FR-OVR`) | Must | S | B3,B5 |
|
||||
| B7 | Canonical writers + provenance + review summary (`FR-OUT`, `FR-PROV`) | Must | M | B2,B3,B5 |
|
||||
| B8 | Index↔Datei mismatch report (`REQ-OUT-02`) | Should | XS | B1 |
|
||||
| B9 | Ambiguous-receiver review path (US-PERS-02 AC4) | Should | S | B5 |
|
||||
| B10 | Comma-split `Inhalt` into extra tags | Could | XS | B7 |
|
||||
| B11 | Phase-2 importer wiring (separate spec) | Won't (this spec) | — | B7 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Traceability — Findings → Requirements
|
||||
|
||||
| Finding | Severity | Addressed by |
|
||||
| --- | --- | --- |
|
||||
| IMP-01 layout mismatch | blocker | C1, FR-MAP, REQ-OUT-01 |
|
||||
| IMP-02 free-text dates | blocker | FR-DATE (all), C2, C6 |
|
||||
| IMP-03 no ISO/normalized cols | blocker | FR-DATE, FR-PERS |
|
||||
| IMP-04 register unimported | major | C3, US-PERS-01, §6.2 |
|
||||
| IMP-05 name variants → dupes | major | C3, FR-DEDUP |
|
||||
| IMP-06 blank-index dropped | major | US-TRIAGE-01 |
|
||||
| IMP-07 duplicate indices | minor | REQ-TRIAGE-01 |
|
||||
| IMP-08 section rows / tags vs summary | minor | REQ-TRIAGE-02, C7 |
|
||||
| IMP-09 index↔file mismatch | minor | REQ-OUT-02, B8 |
|
||||
| IMP-10 `x`-suffix rows | minor | REQ-TRIAGE-03 (skip + log this pass) |
|
||||
| IMP-11 sender not split / ` u ` sep | minor | REQ-PERS-01, US-PERS-02 AC4 |
|
||||
| IMP-12 first-sheet, no validation | minor | REQ-INGEST-01, FR-MAP AC2/AC3 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Open Questions / TBD Register
|
||||
|
||||
| ID | Question | Why it matters | Ref | Resolution |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| OQ-01 ✅ | Season/holiday → date. | Accuracy of ~70 SEASON/feast rows. | REQ-DATE-06 | **Resolved (2026-05-25):** movable feasts (Ostern, Pfingsten, Himmelfahrt, Advent, …) **computed per year from Easter — never a fixed month**; fixed feasts looked up (Weihnachten=12-25, Neujahr=01-01, …); seasons = mid-season month (Frühling=Apr, Sommer=Jul, Herbst=Oct, Winter=Jan). |
|
||||
| OQ-02 ✅ | Date ranges: start only, or start+end? | Sorting/display of ~315 range values. | REQ-DATE-02 | **Confirmed:** store **start** in `date_iso`, precision `RANGE`, full text in `date_raw`. |
|
||||
| OQ-03 ✅ | `person_id` format. | Stability across re-runs; diffability. | §6 | **Confirmed:** readable slug `lastname-firstname`, numeric suffix on collision. |
|
||||
| OQ-04 ✅ | `x`-suffix row handling. | 42 rows. | REQ-TRIAGE-03 | **Resolved (2026-05-25):** `x` rows are transcriptions of the base letter but not yet mappable → **skip this pass**, log to `review/skipped-x-suffix.csv` for later linking. |
|
||||
| OQ-05 ✅ | Importer output format. | Phase-2 reader. | B11 | **Confirmed:** `.xlsx` (openpyxl-native, headered). |
|
||||
| OQ-06 ✅ | Fuzzy-match policy. | False-positive person merges (R2). | REQ-DEDUP-02 | **Confirmed:** conservative — report all fuzzy matches; no silent merge. |
|
||||
|
||||
*All open questions resolved as of 2026-05-25. New ambiguities discovered during build go here.*
|
||||
|
||||
---
|
||||
|
||||
## 10. Glossary & Worked Examples
|
||||
|
||||
**Precision** — how exactly a date is known (`DAY` … `UNKNOWN`). **Provisional person** — a
|
||||
person created from a document name string with no register match. **Alias index** — map from
|
||||
every known surface form of a name to a canonical `person_id`. **Override** — a
|
||||
human-supplied correction applied deterministically on each run.
|
||||
|
||||
**Date examples → expected outcome:**
|
||||
|
||||
| `date_raw` | `date_iso` | `date_precision` |
|
||||
| --- | --- | --- |
|
||||
| `15.2.1888` | 1888-02-15 | DAY |
|
||||
| `6.März 1888` | 1888-03-06 | DAY |
|
||||
| `22.III.18` | 1918-03-22 | DAY |
|
||||
| `13.5.09` | 1909-05-13 | DAY |
|
||||
| `10.Oct.95` | 1895-10-10 | DAY |
|
||||
| `17/6. 1916` | 1916-06-17 | DAY |
|
||||
| `Mai 1895` | 1895-05-01 | MONTH |
|
||||
| `Pfingsten 1922` | 1922-06-04 | DAY (computed: Easter 1922 = Apr 16, +49 days) |
|
||||
| `Herbst 1913` | 1913-10-01 | SEASON |
|
||||
| `1905` | 1905-01-01 | YEAR |
|
||||
| `8.1.1916 - 15.3.1916` | 1916-01-08 | RANGE |
|
||||
| `17.Nov (?) 1887` | 1887-11-17 | APPROX |
|
||||
| `?` | *(empty)* | UNKNOWN |
|
||||
|
||||
**Name examples → expected outcome:**
|
||||
|
||||
| raw cell | resolves to |
|
||||
| --- | --- |
|
||||
| `Eugenie Müller` (+ register `geb Müller`) | `de-gruyter-eugenie` (matched via maiden alias) |
|
||||
| `Eugenie de Gruyter` | `de-gruyter-eugenie` |
|
||||
| `Herbert u Clara` | `cram-herbert` + `cram-clara` (split, surname distributed) |
|
||||
| `Hedi und Tutu (Gruber)` | `gruber-hedi` + `gruber-tutu` |
|
||||
| `Ella Anita` | → `review/ambiguous-receivers.csv` (not auto-split) |
|
||||
| `Hans Wittkopf` (not in register) | provisional `wittkopf-hans` |
|
||||
Reference in New Issue
Block a user