# Unresolved-Name Classification Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Add a focused `review/unresolved-names.csv` that isolates sender/receiver strings whose *name itself* is problematic (unknown/illegible, single-token, relational-only, collective/group, prose-in-name-column, or a genuine two-given-name pair), and fix the ambiguous-pair heuristic so a plain `First Surname` external person (e.g. `Mieze Schefold`) is no longer falsely flagged. **Architecture:** A pure `classify_name(raw, given_names)` function in `persons.py` returns a `NameClass`. `ResolutionContext` classifies every *unmatched* name and records the non-`RESOLVABLE` ones in `self.unresolved`. A runtime-built given-name set (register first names + a small config supplement) lets the classifier distinguish a two-given-name pair (`Ella Anita` → two people) from a first+surname single person (`Mieze Schefold`). The orchestrator writes the aggregated report and per-category stats, replacing the noisy `ambiguous-receivers.csv`. **Tech Stack:** Python 3.12, openpyxl, pytest — extends the existing `tools/import-normalizer/`. **Context:** This builds on the completed normalizer (PR #663). Run all tests with CWD = the tool dir, e.g. `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_X.py -v`. Reuse the existing venv at `tools/import-normalizer/.venv` (do NOT recreate it). Commit on the current branch `docs/import-migration` (never main, never push). Each commit message ends with a trailing `Co-Authored-By: Claude Opus 4.7 ` line. --- ## File Structure ``` tools/import-normalizer/ ├── config.py # + RELATIONAL_TERMS, COLLECTIVE_TERMS, UNKNOWN_NAME_MARKERS, PROSE_MAX_LEN, EXTRA_GIVEN_NAMES ├── persons.py # + NameClass, classify_name(), build_given_names(); ResolutionContext gains given_names + self.unresolved ├── normalize.py # writes unresolved-names.csv (replaces ambiguous-receivers.csv) + per-category stats ├── README.md # + unresolved-names.csv row in the review-file table └── tests/ ├── test_config.py # + name-table presence test ├── test_persons.py # + classify_name + build_given_names tests ├── test_documents.py # ambiguous test → unresolved test (+ resolvable-pair test) └── test_normalize.py # integration asserts unresolved-names.csv ``` --- ### Task 1: Config — name-classification tables **Files:** - Modify: `tools/import-normalizer/config.py` - Modify: `tools/import-normalizer/tests/test_config.py` - [ ] **Step 1: Add the failing test** to `tests/test_config.py` ```python def test_name_classification_tables(): assert "tante" in config.RELATIONAL_TERMS assert "familie" in config.COLLECTIVE_TERMS assert "unbekannt" in config.UNKNOWN_NAME_MARKERS assert config.PROSE_MAX_LEN >= 30 assert "anita" in config.EXTRA_GIVEN_NAMES ``` - [ ] **Step 2: Run to verify it fails** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_config.py::test_name_classification_tables -v && cd -` Expected: FAIL — `AttributeError: module 'config' has no attribute 'RELATIONAL_TERMS'`. - [ ] **Step 3: Implement** — append to `config.py` (after the existing tables, before/after `KNOWN_LAST_NAMES` — anywhere at module level) ```python # --- Name classification (unresolved-name review) --- # Relational reference terms — a sender/receiver named by relation, not a proper name. RELATIONAL_TERMS = { "tante", "onkel", "mutter", "vater", "oma", "opa", "großmutter", "grossmutter", "großvater", "grossvater", "schwester", "bruder", "cousin", "cousine", "kusine", "neffe", "nichte", "tochter", "sohn", "schwager", "schwägerin", "schwiegermutter", "schwiegervater", "enkel", "enkelin", "vetter", "base", "witwe", "witwer", } # Collective/group terms — not a single person. Matched against alpha-only word tokens # (so "Fam.Cram" -> ["fam","cram"] matches "fam"), NOT as substrings/prefixes. COLLECTIVE_TERMS = { "familie", "fam", "kinder", "eltern", "geschwister", "großeltern", "grosseltern", "alle", "diverse", "div", "gebrüder", "gebr", } # Markers of an unknown/illegible name (the literal "?" is handled separately in code). # All long enough to be safe as SUBSTRING matches — do NOT add short tokens like "nn" # (it occurs inside real names: Hanni, Johanna, Anna). UNKNOWN_NAME_MARKERS = {"unbekannt", "unbek", "unleserlich", "unklar", "unsicher"} # A name-column value longer than this (chars) is treated as prose/description, not a name. PROSE_MAX_LEN = 40 # Common given names that may appear in two-given-name pairs (e.g. "Ella Anita") but are not # in the family register. Only used to detect AMBIGUOUS_PAIR — extend as review surfaces more. EXTRA_GIVEN_NAMES = { "ella", "anita", "kurt", "georg", "hanni", "mieze", "ellen", "leni", "klara", "margret", "gustava", "emmy", "minna", "sophie", "helga", "raymonde", "augusta", } ``` - [ ] **Step 4: Run to verify it passes** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_config.py -v && cd -` Expected: PASS (all config tests). - [ ] **Step 5: Commit** ```bash git add tools/import-normalizer/config.py tools/import-normalizer/tests/test_config.py git commit -m "feat(normalizer): config tables for name classification" ``` --- ### Task 2: `classify_name` + `NameClass` **Files:** - Modify: `tools/import-normalizer/persons.py` - Modify: `tools/import-normalizer/tests/test_persons.py` - [ ] **Step 1: Add failing tests** to `tests/test_persons.py` ```python from persons import NameClass GIVEN = {"ella", "anita", "kurt", "georg", "clara", "eugenie"} def test_classify_unknown(): assert persons.classify_name("?", GIVEN) is NameClass.UNKNOWN assert persons.classify_name("A. Kredell?", GIVEN) is NameClass.UNKNOWN assert persons.classify_name("unbekannt", GIVEN) is NameClass.UNKNOWN def test_classify_prose(): assert persons.classify_name("Adressenliste v Clara Cram zur Kondolenz", GIVEN) is NameClass.PROSE assert persons.classify_name("Clara de Gruyter(*1871)", GIVEN) is NameClass.PROSE # digit assert persons.classify_name('"Cramiade" Gedicht', GIVEN) is NameClass.PROSE # quote def test_classify_collective(): assert persons.classify_name("Familie", GIVEN) is NameClass.COLLECTIVE assert persons.classify_name("Fam.Cram", GIVEN) is NameClass.COLLECTIVE assert persons.classify_name("Eltern Cram", GIVEN) is NameClass.COLLECTIVE assert persons.classify_name("seine Kinder", GIVEN) is NameClass.COLLECTIVE def test_classify_relational(): assert persons.classify_name("Cousine Emmy Haniel", GIVEN) is NameClass.RELATIONAL assert persons.classify_name("Schwester Hanni", GIVEN) is NameClass.RELATIONAL def test_classify_single_token(): assert persons.classify_name("Agnes", GIVEN) is NameClass.SINGLE_TOKEN assert persons.classify_name("A.B.", GIVEN) is NameClass.SINGLE_TOKEN def test_classify_ambiguous_pair(): assert persons.classify_name("Ella Anita", GIVEN) is NameClass.AMBIGUOUS_PAIR assert persons.classify_name("Kurt Georg", GIVEN) is NameClass.AMBIGUOUS_PAIR def test_classify_resolvable_single_person(): # first + surname (surname not a given name) -> one real person, NOT ambiguous assert persons.classify_name("Mieze Schefold", GIVEN) is NameClass.RESOLVABLE assert persons.classify_name("Adolf Butenandt", GIVEN) is NameClass.RESOLVABLE ``` - [ ] **Step 2: Run to verify it fails** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_persons.py -k classify -v && cd -` Expected: FAIL — `NameClass` / `classify_name` not defined. - [ ] **Step 3: Implement** — add to `persons.py`. Add `from enum import StrEnum` to the imports if not present, then add: ```python class NameClass(StrEnum): RESOLVABLE = "resolvable" UNKNOWN = "unknown" SINGLE_TOKEN = "single_token" RELATIONAL = "relational" COLLECTIVE = "collective" PROSE = "prose" AMBIGUOUS_PAIR = "ambiguous_pair" _QUOTE_CHARS = "\"'“”„‚‘’" def classify_name(raw: str, given_names: set[str]) -> NameClass: """Classify a (post-split) sender/receiver string by why it may be unresolvable. Precedence (first match wins): UNKNOWN -> PROSE -> COLLECTIVE -> RELATIONAL -> SINGLE_TOKEN -> AMBIGUOUS_PAIR -> RESOLVABLE. """ s = raw.strip() if not s: return NameClass.RESOLVABLE low = s.lower() tokens = s.split() # alpha-only word tokens: "Fam.Cram" -> ["fam","cram"], so collective/relational terms # are matched as whole words (no substring/prefix false positives like "Allerton"). alpha_words = re.findall(r"[a-zäöüß]+", low) if "?" in s or any(m in low for m in config.UNKNOWN_NAME_MARKERS): return NameClass.UNKNOWN if (len(s) > config.PROSE_MAX_LEN or any(c.isdigit() for c in s) or any(q in s for q in _QUOTE_CHARS) or len(tokens) > 3): return NameClass.PROSE if any(w in config.COLLECTIVE_TERMS for w in alpha_words): return NameClass.COLLECTIVE if any(w in config.RELATIONAL_TERMS for w in alpha_words): return NameClass.RELATIONAL if len(tokens) == 1: return NameClass.SINGLE_TOKEN if len(tokens) == 2 and all(_norm(t) in given_names for t in tokens): return NameClass.AMBIGUOUS_PAIR return NameClass.RESOLVABLE # Known limitation: a 4+-token name with no digits/quotes (e.g. "Anna von der Heide") is # classified PROSE. Such multi-particle names are rare here and usually resolve via the # register; if they surface in review, lower-priority than the real prose entries. ``` > Note: `_norm` already exists in `persons.py` (added in the alias-index task) and strips accents + lowercases. `classify_name` uses it so given-name matching is accent-insensitive. - [ ] **Step 4: Run to verify it passes** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_persons.py -v && cd -` Expected: PASS (all persons tests, including the 7 new classify tests). - [ ] **Step 5: Commit** ```bash git add tools/import-normalizer/persons.py tools/import-normalizer/tests/test_persons.py git commit -m "feat(normalizer): classify_name + NameClass" ``` --- ### Task 3: `build_given_names` **Files:** - Modify: `tools/import-normalizer/persons.py` - Modify: `tools/import-normalizer/tests/test_persons.py` - [ ] **Step 1: Add failing test** to `tests/test_persons.py` ```python def test_build_given_names(): people = persons.parse_register([ {"last_name": "de Gruyter", "first_name": "Eugenie"}, {"last_name": "Cram", "first_name": "Charlotte,Meta"}, # comma -> primary + extra given ]) g = persons.build_given_names(people, {"Anita"}) assert "eugenie" in g assert "charlotte" in g and "meta" in g # primary + extra given names assert "anita" in g # from the extra set, normalized assert "schefold" not in g ``` - [ ] **Step 2: Run to verify it fails** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_persons.py::test_build_given_names -v && cd -` Expected: FAIL — `build_given_names` not defined. - [ ] **Step 3: Implement** — add to `persons.py` ```python def build_given_names(register: list[Person], extra: set[str]) -> set[str]: """Set of normalized given names from the register (first + extra given) plus a supplement. Used by classify_name to tell a two-given-name pair (two people) from a first+surname. """ names: set[str] = set() for p in register: if p.first_name: names.add(_norm(p.first_name)) for g in p.extra_given_names: names.add(_norm(g)) for e in extra: names.add(_norm(e)) return names ``` - [ ] **Step 4: Run to verify it passes** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_persons.py -v && cd -` Expected: PASS. - [ ] **Step 5: Commit** ```bash git add tools/import-normalizer/persons.py tools/import-normalizer/tests/test_persons.py git commit -m "feat(normalizer): build_given_names from register + supplement" ``` --- ### Task 4: Integrate — ResolutionContext records unresolved; orchestrator writes the report This task touches `persons.py`, `normalize.py`, and two test files together so the whole suite stays green in one commit (removing `ctx.ambiguous` requires updating its only consumer, `normalize.py`, in the same change). **Files:** - Modify: `tools/import-normalizer/persons.py` (ResolutionContext) - Modify: `tools/import-normalizer/normalize.py` - Modify: `tools/import-normalizer/tests/test_documents.py` - Modify: `tools/import-normalizer/tests/test_normalize.py` - [ ] **Step 1: Update the failing tests first** In `tests/test_documents.py`, **replace** the existing `test_ambiguous_space_pair_flagged_not_split` function entirely with these two functions: ```python def test_ambiguous_pair_recorded_in_unresolved(): people = persons.parse_register([{"last_name": "de Gruyter", "first_name": "Walter"}]) ctx = persons.ResolutionContext(persons.AliasIndex(people), name_overrides={}, given_names={"ella", "anita"}) raw = documents.RawRow(source_row=7, index="C-0200", sender="", receivers="Ella Anita") doc = documents.to_canonical(raw, ctx, date_overrides={}) assert len(doc.receiver_person_ids) == 1 # not split — one provisional assert any(name == "Ella Anita" and cat == "ambiguous_pair" for name, cat, _ in ctx.unresolved) def test_resolvable_first_surname_pair_not_unresolved(): ctx = persons.ResolutionContext(persons.AliasIndex([]), name_overrides={}, given_names={"ella", "anita"}) ctx.resolve_one("Mieze Schefold", source_row=1) # surname is not a given name assert ctx.unresolved == [] # RESOLVABLE -> not recorded ``` In `tests/test_normalize.py`, in the `_doc_wb` fixture, change the `C-0001` row's receiver from empty to `"?"` so the run produces an unresolved entry. Find the line that appends the `C-0001` row and set its `EmpfängerIn` cell to `"?"`. For example the row currently reads: ```python ws.append(["C-0001", "", "", "", "Hans Wittkopf", "", "Freitag 1919", "", "", ""]) ``` change the 6th cell (EmpfängerIn) from `""` to `"?"`: ```python ws.append(["C-0001", "", "", "", "Hans Wittkopf", "?", "Freitag 1919", "", "", ""]) ``` Then add these assertions inside `test_run_end_to_end`, right after the existing `assert (review_dir / "unparsed-dates.csv").exists()` line: ```python assert (out_dir / "canonical-documents.xlsx").exists() # (keep existing asserts above) assert (review_dir / "unresolved-names.csv").exists() unresolved_text = (review_dir / "unresolved-names.csv").read_text(encoding="utf-8") assert "unknown" in unresolved_text and "?" in unresolved_text # the "?" receiver assert not (review_dir / "ambiguous-receivers.csv").exists() # replaced ``` - [ ] **Step 2: Run to verify they fail** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/test_documents.py tests/test_normalize.py -v && cd -` Expected: FAIL — `ResolutionContext` has no `given_names`/`unresolved`; `unresolved-names.csv` not written. - [ ] **Step 3a: Implement — `ResolutionContext` in `persons.py`** Replace the `ResolutionContext.__init__` body's two lines (`self.ambiguous` and add `given_names`) and the relevant methods. The new `__init__`: ```python def __init__(self, alias_index: AliasIndex, name_overrides: dict[str, str], given_names: set[str] | None = None): self.index = alias_index self.name_overrides = name_overrides self.given_names = given_names or set() self.provisional: dict[str, Person] = {} self.unmatched: dict[str, list] = {} self.unresolved: list[tuple] = [] # (raw_name, category, source_row) for non-RESOLVABLE names self._raw_to_pid: dict[str, str] = {} self.override_hits = 0 ``` In `resolve_one`, the provisional branch must classify the name. Replace this existing block: ```python # provisional person (unmatched) — never reuse a register id self.unmatched.setdefault(name, []).append(source_row) if name in self._raw_to_pid: return self._raw_to_pid[name], name, False ``` with: ```python # provisional person (unmatched) — never reuse a register id self.unmatched.setdefault(name, []).append(source_row) category = classify_name(name, self.given_names) if category is not NameClass.RESOLVABLE: self.unresolved.append((name, str(category), source_row)) if name in self._raw_to_pid: return self._raw_to_pid[name], name, False ``` Replace the entire `resolve_receivers` method (the ambiguous detection now lives in `resolve_one` via `classify_name`): ```python def resolve_receivers(self, raw: str, source_row: int): return [self.resolve_one(part, source_row) for part in split_receivers(raw)] ``` - [ ] **Step 3b: Implement — `normalize.py`** Find the line that builds the context: ```python ctx = persons.ResolutionContext(alias_index, name_overrides) ``` replace it with (build the given-name set from the register + config supplement): ```python given_names = persons.build_given_names(register, config.EXTRA_GIVEN_NAMES) ctx = persons.ResolutionContext(alias_index, name_overrides, given_names=given_names) ``` Replace the `ambiguous-receivers.csv` write line: ```python writers.write_review_csv(review_dir / "ambiguous-receivers.csv", ["raw", "part", "source_row"], ctx.ambiguous) ``` with an aggregated unresolved-names report: ```python unresolved_agg: dict[tuple, list] = {} for name, category, row in ctx.unresolved: unresolved_agg.setdefault((category, name), []).append(row) unresolved_rows = sorted( ([cat, name, len(rows), " ".join(map(str, sorted(rows)[:5]))] for (cat, name), rows in unresolved_agg.items()), key=lambda r: (r[0], -r[2], r[1])) writers.write_review_csv(review_dir / "unresolved-names.csv", ["category", "raw", "count", "example_rows"], unresolved_rows) ``` In the `stats` dict, replace the `"ambiguous_receivers"` line: ```python "ambiguous_receivers": len(ctx.ambiguous), ``` with a per-category breakdown: ```python "unresolved_name_occurrences": len(ctx.unresolved), "unresolved_unknown": sum(1 for _, c, _ in ctx.unresolved if c == "unknown"), "unresolved_single_token": sum(1 for _, c, _ in ctx.unresolved if c == "single_token"), "unresolved_relational": sum(1 for _, c, _ in ctx.unresolved if c == "relational"), "unresolved_collective": sum(1 for _, c, _ in ctx.unresolved if c == "collective"), "unresolved_prose": sum(1 for _, c, _ in ctx.unresolved if c == "prose"), "unresolved_ambiguous_pair": sum(1 for _, c, _ in ctx.unresolved if c == "ambiguous_pair"), ``` - [ ] **Step 4: Run the whole suite to verify green** Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/ -q && cd -` Expected: PASS (all tests, no `ambiguous` references remain). Also grep to confirm no dangling references: Run: `grep -rn "ctx.ambiguous\|ambiguous-receivers\|ambiguous_receivers\|self.ambiguous" tools/import-normalizer/*.py` Expected: no matches. - [ ] **Step 5: Commit** ```bash git add tools/import-normalizer/persons.py tools/import-normalizer/normalize.py tools/import-normalizer/tests/test_documents.py tools/import-normalizer/tests/test_normalize.py git commit -m "feat(normalizer): unresolved-names report + fix ambiguous-pair over-flagging" ``` --- ### Task 5: README — document the new report **Files:** - Modify: `tools/import-normalizer/README.md` - [ ] **Step 1: Update the review-file table** in `README.md`. Replace the `ambiguous-receivers.csv` row with an `unresolved-names.csv` row. Find the table row referencing `ambiguous-receivers.csv` and replace it with: ```markdown | `unresolved-names.csv` | Names whose value is itself problematic, grouped by `category`: `unknown` (`?`/illegible), `single_token` (first OR last name only), `relational` (`Tante …`), `collective` (`Familie …`), `prose` (a description landed in a name column), `ambiguous_pair` (two given names → likely two people, not auto-split). Review highest-impact categories first; add decisions to `overrides/names.csv`. | ``` If the README has no such row (older version), add the row above to the review-file table. - [ ] **Step 2: Add a note** to the iteration-loop section of `README.md` (after the table): ```markdown > `unresolved-names.csv` is the focused "names that need a human" list — distinct from > `unmatched-names.csv` (which is just non-family correspondents that got provisional persons). > The given-name set that drives `ambiguous_pair` detection is the register's first names plus > `config.EXTRA_GIVEN_NAMES` — add names there if a real two-person cell isn't being flagged. ``` - [ ] **Step 3: Verify the suite is still green** (README-only change, but confirm nothing references the old file) Run: `cd tools/import-normalizer && .venv/bin/python -m pytest tests/ -q && cd -` Expected: PASS. - [ ] **Step 4: Commit** ```bash git add tools/import-normalizer/README.md git commit -m "docs(normalizer): document unresolved-names.csv review report" ``` --- ## Self-Review **Spec coverage** (against the agreed proposal): - Focused report isolating problem name classes → Task 4 writes `review/unresolved-names.csv` with a `category` column; categories defined in Task 2 `classify_name`. ✓ - Fix ambiguous over-flagging of `First Surname` → Task 2 `AMBIGUOUS_PAIR` requires *both* tokens in the given-name set; `Mieze Schefold` → `RESOLVABLE` (tested). ✓ - Distinguish "not fully known" (unknown/single-token/relational/collective/prose) from "can't split cleanly" (ambiguous_pair) → all are `NameClass` values, each its own category column value. ✓ - Per-category counts in summary → Task 4 stats. ✓ - Senders covered too (not just receivers) → classification happens in `resolve_one`, which both `resolve_sender` and `resolve_receivers` call. ✓ **Placeholder scan:** No TBD/TODO; every code step has complete code. The README replacement gives the exact row text. **Type consistency:** `NameClass` (StrEnum) defined Task 2; `classify_name(raw, given_names)` and `build_given_names(register, extra)` signatures used consistently in Task 4; `ResolutionContext(alias_index, name_overrides, given_names=…)` matches the new `__init__`; `self.unresolved` is `list[tuple]` of `(raw, category, source_row)` and read with that shape in both the report and the stats. `str(category)` yields the StrEnum value (e.g. `"ambiguous_pair"`), matching the stat comparisons and the test assertions. **Cross-task green:** Task 4 deliberately bundles the `persons.py` + `normalize.py` + test changes into one commit because removing `ctx.ambiguous` breaks its consumer otherwise — no red commit is left behind (lesson from the prior build). **Out of scope (future):** Spanish month names + `Mon DD-YYYY` date form (separate date-parser enhancement); promoting `unresolved` rows into a document-level `needs_review` flag; auto-splitting confirmed `ambiguous_pair` entries via overrides.