familienarchiv/docs/import-migration/01-findings-spreadsheet-analysis.md

# Spreadsheet Analysis — Findings (2026-05-25)

Analysis of the **real raw archive** spreadsheets against the current `MassImportService`
(`backend/.../importing/MassImportService.java`). Goal: import ~7,600 letter rows + a
163-person register, with PDFs to follow.

Every issue has an ID (`IMP-NN`), severity, evidence, and a proposed approach.

---

## 0. Context: how the importer reads a row today

`MassImportService` reads **sheet index 0** and maps columns by configurable indices
(`app.import.col.*`, defaults in the source):

| Property | Default col | Meaning |
| --- | --- | --- |
| `colIndex` | 0 | Index (→ filename `<index>.pdf`) |
| `colBox` | 1 | Box |
| `colFolder` | 2 | Mappe |
| `colSender` | 3 | Sender (raw) |
| `colReceivers` | 5 | Receivers (raw) |
| `colDate` | 7 | Date |
| `colLocation` | 9 | Location |
| `colTags` | 10 | Tag (single) |
| `colSummary` | 11 | Summary |
| `colTranscription` | 13 | Transcription |

These defaults match the **ODS** file exactly (`Index, Box, Mappe, Von, BriefeschreiberIn,
An, EmpfängerIn, Datum, Datum Originalformat, Ort, Schlagwort, Inhalt, Zeitlicher Kontext,
Transkript` = 14 cols). The ODS was the development target. The new xlsx is a different beast.

Per-row pipeline: skip if Index blank → derive filename from Index → validate filename →
look for file on disk (recursive; metadata-only if absent) → check PDF magic bytes →
`importSingleDocument` (upsert by `originalFilename`, dedupe non-placeholders as
`ALREADY_EXISTS`). Date parsing is **ISO-only** (`LocalDate.parse`).

---

## IMP-01 — New xlsx column layout ≠ importer defaults 🔴 BLOCKER

The new `…aktuell…xlsx` (sheet `Familienarchiv`, 7,943 rows × 12 cols) has a **denser,
different** layout. There is an extra `Datei` column at index 1, and the normalized
`Von`/`An`/ISO-`Datum` columns from the ODS **do not exist**.

| col | New xlsx header | Importer default expects | Result with defaults |
| --- | --- | --- | --- |
| 0 | Index | Index | ✅ ok |
| 1 | **Datei** (path) | Box | ❌ Box ← `..\__scan\W-0001.pdf` |
| 2 | Box | Mappe | ❌ Mappe ← `V` |
| 3 | Mappe | Sender | ❌ Sender ← `1` |
| 4 | BriefeschreiberIn (sender) | — (unused) | ❌ sender ignored |
| 5 | EmpfängerIn (receiver) | Receivers | ✅ coincidentally ok |
| 6 | Datum des Briefes | — (unused) | ❌ date ignored |
| 7 | Ort (location) | Date | ❌ Date ← `Rotterdam` → null |
| 8 | Schlagwort (tag) | — (unused) | ❌ tag ignored |
| 9 | Inhalt (summary) | Location | ❌ Location ← summary text |
| 10 | — | Tag | ❌ empty |
| 11 | — | Summary | ❌ empty |
| 13 | — | Transcription | ❌ column doesn't exist |

**Impact:** importing as-is produces almost entirely garbage metadata.

**Proposed approach (decide with Marcel):**
- (a) Re-map via the existing `app.import.col.*` properties — fast, no code. New mapping:
  `index=0, box=2, folder=3, sender=4, receivers=5, date=6, location=7, tags=8, summary=9`,
  and there is **no** transcription column (point it past the end or add a "missing column"
  convention). Caveat: tags land in `colTags` but the real per-letter keywords are in
  `Inhalt` (col 9) — see IMP-08 note on tags vs summary.
- (b) Make the importer **header-driven** (map by header name, not index) so it survives
  layout drift across files. More robust, needs a code change (→ Gitea issue).

Recommendation: (b) is the durable fix given we have ≥3 different layouts already.

---

## IMP-02 — 90% of dates are free-text the parser can't read 🔴 BLOCKER

The dates are written **as in the letter**. `parseDate()` only does `LocalDate.parse()`
(ISO `yyyy-MM-dd`), so anything non-ISO becomes `null`.

Of **7,319** rows with a date value (col 6):

| kind | count | parses today? |
| --- | --- | --- |
| Real Excel date cells (→ ISO via POI) | 748 | ✅ |
| Free-text date strings | 6,571 | ❌ → null |

→ **90% of dated rows lose their date.** (623 rows have no date at all.)

Observed free-text formats (counts approximate, from col 6):

| Format | Count | Examples |
| --- | --- | --- |
| `D.M.YY` | 1,338 | `11.10.08`, `13.5.09` |
| `D.RomanMonth.YY/YYYY` | ~1,527 | `22.III.18`, `19.XII.1954`, `1.III.27` |
| `D.Month YYYY` | 950 | `6.März 1888`, `9.März 1888` (note: **no space** after the dot) |
| `D.M.YYYY` | 358 | `15.2.1888`, `7.3.1888` |
| Approximate / unknown | 146 | `?`, `13.7.18?`, `17.Nov (?) 1887`, `13.Januar ? 1907` |
| `Month YYYY` / season / holiday | 41+27 | `Mai 1895`, `Herbst 1913`, `Pfingsten 1922`, `Ostern 1890` |
| `YYYY` only | 17 | `1905`, `1949` |
| `D.M.` no year | 10 | `8.9.`, `14.3.` |
| Ranges | 5+ | `8.1.1916 - 15.3.1916`, `1881/82`, `1945/46?` |
| Abbrev/English months, no space | many | `29.Sept.1891`, `10.Oct.95`, `9.December1889`, `18.Dez.1916` |
| Slash separator | ~315 | `2/2. 18`, `17/6. 1916`, `10/4. 1917` |
| English `Month D. YYYY` | several | `April 12. 1922`, `Oct.5. 1916`, `Mai 23. 1917` |
| Trailing notes | 5+ | `26.4.1888, 2. Brief`, `31.8.1888,2.Brief` |
| 3-digit year (typo) | 107 | `30.1.889` (→ 1889), `4.3.1023` (in person file → 1923) |
| Day-range within month | several | `7./8. Sept.1923` |

**Proposed approach:** build a tolerant German/historical date parser (→ Gitea issue, it's
a code change). Requirements:
- Numeric `D.M.YY[YY]` and `D/M. YY[YY]` (slash = dot).
- Roman-numeral months (`I`–`XII`).
- German + English month names, full + abbreviated, with/without separating space
  (`März`, `Sept.`, `Dez`, `December`, `Oct.`).
- 2-digit and 3-digit year normalization (`08`→1908? needs a century rule; `889`→1889).
- Partial dates → store what's known. The schema only has a single `documentDate
  LocalDate`; **decide** whether to (i) store first-of-month/year, (ii) add a
  `datePrecision` enum + `dateOriginal` text column, or (iii) keep raw text in a new
  `documentDateRaw` field and leave `documentate` null when imprecise. Recommendation:
  preserve the **original string** always (new column) + best-effort parsed date +
  precision flag, so nothing is lost and the UI can show "ca. 1916".
- Unparseable/approximate (`?`, `Herbst 1913`) → keep raw, leave parsed date null, **do
  not drop the row**.

**Cross-check:** even after IMP-01 is fixed so the date column is read, IMP-02 still bites.
Both must be solved before a real import.

---

## IMP-03 — New xlsx has no normalized/ISO date or name columns 🔴 BLOCKER

The ODS had helper columns the importer relied on: `Von`/`An` (normalized names) and
`Datum` (ISO) alongside `Datum Originalformat`. The new xlsx has **only the raw**
`BriefeschreiberIn` / `EmpfängerIn` / `Datum des Briefes`. So:
- Names must be parsed from raw strings (PersonNameParser already does receivers; **sender
  is taken raw, never split** — fine for senders, which are single, but no normalization).
- Dates must be parsed from raw (IMP-02).

This is the root reason IMP-01/02 exist: the new file is the *uncurated* source, not the
hand-normalized ODS. Tie any importer redesign to this reality — we will not get clean
helper columns in the 7k-row file.

---

## IMP-04 — Person register not imported at all 🟠 MAJOR

`Personendatei 2.xlsx` → sheet `Tabelle1`, **163 people**, columns:
`Generation, Familienname, Vorname, geb als (maiden), Geburtsdatum, Geburtsort,
Todesdatum, Sterbeort, verheiratet mit, Bemerkung`.

Today `MassImportService` has **no person-register import**. Persons are only
auto-created as bare aliases from the document sender/receiver strings
(`personService.findOrCreateByAlias`). All this rich genealogical data is unused:
- birth/death dates + places,
- maiden names (the key to dedup — see IMP-05),
- `verheiratet mit` (marriage links → `PersonRelationship` domain),
- `Bemerkung` relationship hints (`"Schwester v Marie Cram"`, `"Nichte von Herbert"`),
- `Generation` (G 1–G 4),
- nicknames in quotes (`"Tante Lolly"`).

Data-quality notes in this file too: multi-value `Vorname` (`Charlotte,Meta,Jacobi`);
mixed Excel-date vs text dates; typos (`4.3.1023`); missing-day dates (`.12.1955`);
trailing spaces (`30.8.1862 `).

**Proposed approach:** a separate **Person import** (→ Gitea issue). Order matters: import
persons *first* so documents can link to real people instead of creating alias stubs.
Use `geb als` + `verheiratet mit` to pre-build the alias/relationship graph.

---

## IMP-05 — Name variations create duplicate Persons 🟠 MAJOR

The same person appears under several surface forms across the document sheet:
- `Eugenie Müller` (151) vs `Eugenie de Gruyter` (452) — maiden vs married.
- `Clara Cram` (sender 1,284) vs `Clara de Gruyter` (455) vs `Clara de Gruyter sen.` (66).
- `Walter de Gruyter` (589) vs bare `Walter` (78).

`findOrCreateByAlias` keys on the raw string, so each variant becomes (or matches) a
distinct alias and likely a **distinct Person**. Result: fragmented person records,
broken Briefwechsel pairing, wrong stats.

**Proposed approach:** drive dedup from the register's `geb als` column (IMP-04) —
`Eugenie de Gruyter geb Müller` tells us the two strings are one person. Build an alias
map (married ↔ maiden ↔ nickname) before/while importing documents. This is partly data
(an alias mapping table/sheet) and partly code (consume it). Likely a Gitea issue once the
mapping format is decided.

945 distinct sender strings / 274 distinct receiver strings — expect a long-tail of
variants to reconcile. Don't try to be perfect on the first pass; get the high-frequency
names right.

---

## IMP-06 — 93 data rows with blank Index are silently dropped 🟠 MAJOR

`processRows` does `if (index.isBlank()) continue;`. **93 rows** have a blank Index but
carry other data (sender/receiver/date/etc.). These are silently skipped — they don't even
appear in the `skippedFiles` report (that list only covers rows that *had* an index but
failed file checks).

**Proposed approach:** before import, triage these 93 rows — are they continuation rows,
section markers, or genuine letters missing an ID? At minimum, surface a count/warning so
nothing vanishes unnoticed. Possibly a small importer change to report blank-index skips.

---

## IMP-07 — 43 duplicate Index values 🟡 MINOR

43 Index values repeat (e.g. `W-0388`, `Eu-0332`, `C-0234`, `C-0235`, `C-0236`, `J-0175`).
Since the filename is derived from Index, the importer's upsert keys both rows on the same
`originalFilename`: the second occurrence is treated as `ALREADY_EXISTS` (if the first
isn't a placeholder) and **its metadata is lost**, or it overwrites a placeholder.

**Proposed approach:** list the 43 duplicates, check whether they're true duplicates or
two distinct letters that share an ID by mistake. Fix in the source data, or extend the ID
scheme. Data task first; software only if the ID scheme must change.

---

## IMP-08 — Section/title rows interleaved with data 🟡 MINOR

Row 2 of the sheet is a section header sitting only in the sender column
(`Brautbriefe von Walter der Gruyter an Eugenie Müller`) with a blank Index — caught by the
blank-Index skip (overlaps IMP-06). There may be more such banners scattered through 7,943
rows. Also relevant: the per-letter **keywords live in `Inhalt` (col 9)** as comma-joined
values (`Tilburg,Verwandschaft`, `poetisch,Reise nach Breda`), while `Schlagwort` (col 8)
holds a single broad tag (`Brautbriefe`). The importer only takes **one** tag column —
decide which column feeds tags vs summary, and whether to split comma-lists into multiple
tags.

**Proposed approach:** scan for rows where Index is blank but other cells are set (already
have the count: relates to the 93 in IMP-06). Confirm tag vs summary column choice with
Marcel.

---

## IMP-09 — Index ↔ Datei filename mismatches 🟡 MINOR

The `Datei` column (col 1) holds explicit relative paths (`..\__scan\W-0001.pdf`) but they
don't always agree with the Index. Example: row 20 has Index `W-0010x` but Datei
`..\__scan\W-0011x.pdf`. The importer derives the filename from **Index**, so it will look
for `W-0010x.pdf` and may miss the actual scan. (Note: the `Datei` paths themselves are
Windows-style with `\` and `..` and would be **rejected** by `isValidImportFilename` if anyone
tried to use that column directly — 7,623 rows use backslashes, 7,455 contain `..`.)

**Proposed approach:** when the PDFs arrive, reconcile Index-derived names against actual
filenames; produce a mismatch report. Keep deriving from Index (stable IDs) but flag
disagreements. Mostly a data/QA task.

---

## IMP-10 — `x`-suffix rows (letter backsides / enclosures) 🟡 MINOR

**42 rows** have an `x`-suffixed Index (`W-0001x`, `W-0002x`, …). They're sparse — typically
only Index + Datei + sender + receiver, no box/folder/date. They appear to be the reverse
side or an enclosure of the preceding letter. The importer treats each as an independent
Document, and the `metadataComplete` heuristic flags them complete as soon as a sender is
present (date/box/folder all missing).

**Proposed approach:** decide whether `x` rows should be (a) separate documents, (b) extra
pages/files attached to their parent, or (c) skipped. Affects both the data model and the
`metadataComplete` heuristic. Discuss with Marcel.

---

## IMP-11 — Multi-receiver separators include bare `u` / `u.` 🟡 MINOR

`PersonNameParser.parseReceivers` already handles ` und `, ` u `, `//`, `geb.`,
parenthesised shared surnames, and `Familie` filtering — good. But the real data also uses
the abbreviation in forms the top-receivers list shows are common:
`Eugenie u Walter de Gruyter` (230), `Herbert u Clara` (94), `Juan u Marie Cram` (75),
and space-joined pairs like `Ella Anita` (79) that may be two people.
Raw separator tally on receivers: ` und ` ×70, `,` ×11, `;` ×2, `/` ×1 — plus the many ` u `
cases above. Senders are **not** parsed at all (taken raw), which is fine unless a sender
cell ever holds two names.

**Proposed approach:** add `MassImportServiceTest` cases for the real-world strings above;
extend the parser only where it actually fails. `Ella Anita`-style space-joined pairs are
ambiguous — likely leave as one person unless the register says otherwise (ties to IMP-05).

---

## IMP-12 — Importer reads only the first sheet, no validation 🟡 MINOR

`readXlsx` does `workbook.getSheetAt(0)`. For the new xlsx that's `Familienarchiv` (✅), but
the file also contains `Inhaltsverzeichnis grob`, `Inhaltsverzeichnis WdG`, `Tabelle4`.
There is no header validation: if the wrong file/sheet is dropped in `/import`, the importer
will happily map columns positionally and import nonsense. Also `findSpreadsheetFile()` picks
the **first** spreadsheet found in `/import` — with three spreadsheets present there today,
which one wins is filesystem-order-dependent.

**Proposed approach:** (a) validate the header row against expected names before importing;
(b) make the target sheet/file explicit (config or header match) rather than "first found".
Ties into the header-driven mapping in IMP-01(b).

---

## Summary of recommended sequencing

1. **Decide the importer mapping strategy** (IMP-01): positional re-config vs header-driven.
   Header-driven is the durable choice and unblocks IMP-03/12.
2. **Build the tolerant date parser** (IMP-02) with original-string preservation + precision.
3. **Import the Person register first** (IMP-04) and build the alias/marriage graph,
   which feeds person dedup (IMP-05).
4. **Then import documents**, with reporting for blank-index (IMP-06), duplicates (IMP-07),
   and section rows (IMP-08).
5. **Reconcile files** when the ~7,000 PDFs arrive (IMP-09), and decide `x`-row semantics
   (IMP-10).

Code-change items (→ Gitea issues when we get there): IMP-01(b), IMP-02, IMP-04, IMP-05
(consume side), IMP-06 reporting, IMP-12. Pure-data items stay in this folder.