Commit Graph

842 Commits

Author SHA1 Message Date
Marcel
72700bd28f test(annotations): add Testcontainers integration tests for V33 chk_annotation_bounds [B1]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 14:36:37 +02:00
Marcel
40c8f548db docs(annotations): fix ANNOTATION_UPDATE_FAILED Javadoc to reflect 400 status [M3]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 14:34:55 +02:00
Marcel
a19faa3806 feat(annotations): add @Slf4j and DataIntegrityViolationException catch to updateAnnotation [M2]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 14:34:03 +02:00
Marcel
f00b470928 test(annotations): add failing test for DataIntegrityViolationException defense [M2 red]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 14:33:43 +02:00
Marcel
65d606d8bb test(annotations): add missing height and x boundary validation tests [M4]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 14:31:07 +02:00
Marcel
4d3207fc27 test(annotations): verify save() is called in updateAnnotation test [M5]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 14:30:50 +02:00
Marcel
2350b4f845 fix(annotations): make resize overlay keyboard-interactive
Some checks failed
CI / Unit & Component Tests (push) Failing after 1s
CI / Backend Unit Tests (push) Failing after 1s
CI / Unit & Component Tests (pull_request) Failing after 2s
CI / Backend Unit Tests (pull_request) Failing after 1s
- Add tabindex="0" so the SVG can receive DOM focus
- Auto-focus the SVG on mount so arrow keys work immediately after
  clicking an annotation to select it
- Show preview rect during keyboard nudging (not just pointer drag) by
  checking hasLiveChanges instead of only checking dragState
- Suppress default browser focus outline (outline: none) on the SVG

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 11:47:41 +02:00
Marcel
9fe5b32a69 feat(annotations): add N/S/E/W edge midpoint handles to resize overlay
Extends the 4-corner L-bracket handles with 4 tick-mark edge handles
(short lines along each edge), enabling single-axis resize from any edge.
Updates applyHandleDrag to route each handle to the correct axis.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 11:40:39 +02:00
Marcel
fcc0efbf02 refactor(annotations): replace 8-square handles with 4 corner L-brackets
- 4 corner-only handles (nw/ne/sw/se), no edge midpoints
- Each handle renders as two short perpendicular lines meeting at the corner
  (10px arms, navy, square linecap) — no fill, no box
- Thin dashed selection border added to SVG overlay to signal edit mode
- Simplify applyHandleDrag to remove dead n/s/e/w branches

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 11:14:30 +02:00
Marcel
e7f88a4ea1 fix(annotations): use pixel-space viewBox so handles stay square on non-square annotations
ResizeObserver binds actual SVG pixel dimensions; viewBox matches them so
16px handle squares and 44px hit areas are physically correct regardless of
the annotation's aspect ratio.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 11:03:15 +02:00
Marcel
c610a3cc37 feat(annotations): wire updateAnnotation context and error display into PdfViewer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 11:00:50 +02:00
Marcel
3fb32ea285 feat(annotations): pass isResizable to AnnotationShape based on selection + transcribeMode
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:57:13 +02:00
Marcel
3b756cd718 feat(annotations): add isResizable prop to AnnotationShape to render edit overlay
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:55:13 +02:00
Marcel
f5362a5850 feat(annotations): add AnnotationEditOverlay component with resize handles and drag
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:52:07 +02:00
Marcel
953cb2c910 feat(i18n): add ANNOTATION_UPDATE_FAILED error code and annotation_edit_mode_active translation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:43:10 +02:00
Marcel
ff231db671 feat(annotations): add PATCH endpoint for annotation resize/move
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:42:08 +02:00
Marcel
1558881c01 feat(annotations): add updateAnnotation service method with partial-update DTO
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:39:50 +02:00
Marcel
26c7181ba4 feat(annotations): add ANNOTATION_UPDATE_FAILED error code
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:38:33 +02:00
Marcel
f76a6c0ee5 migration(annotations): add chk_annotation_bounds CHECK constraint (V33)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:38:11 +02:00
Marcel
ca10e8a6a9 fix(test): update TranscriptionEditView empty-state assertion after text change
Some checks failed
CI / Unit & Component Tests (pull_request) Failing after 1s
CI / Backend Unit Tests (pull_request) Failing after 2s
CI / Unit & Component Tests (push) Failing after 3s
CI / Backend Unit Tests (push) Failing after 2s
Commit 5afdc37 changed the empty state from transcription_empty_cta
('Markiere einen Bereich…') to transcription_empty_draw_hint
('Zeichnen Sie Bereiche…') but left the spec asserting the old text.
Updated the locator to match the current component output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:11:57 +02:00
Marcel
22ee3dce68 fix(api): remove duplicate import and align patchTrainingLabel OpenAPI response to 204
Removed duplicate import of org.mockito.ArgumentMatchers.eq from
DocumentControllerTest (lines 32+35). Added @ApiResponse(responseCode="204")
to patchTrainingLabel so the generated OpenAPI spec matches the actual
NoContent response the controller returns.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:07:41 +02:00
Marcel
99847980d2 fix(a11y): replace unicode glyphs with SVG icons in TrainingHistory status badges
WCAG 1.4.1 (Use of Color) requires non-color redundant cues for status.
The unicode ✓/✗ characters had inconsistent screen-reader support.
Replaced with explicit aria-hidden SVG icons (checkmark / x-circle)
alongside the translated status text labels.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:06:11 +02:00
Marcel
8f6e398af7 fix(i18n): replace hardcoded German training label chip strings with Paraglide keys
TranscriptionEditView rendered 'Kurrent-Erkennung' and 'Segmentierung'
as hardcoded German strings, breaking the en/es locales. Added
training_chip_kurrent and training_chip_segmentation keys to all three
message files and wired them up via m.training_chip_kurrent() /
m.training_chip_segmentation().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:04:52 +02:00
Marcel
30a17c97e8 fix(ocr): fail closed when TRAINING_TOKEN is not configured
_check_training_token previously skipped auth when TRAINING_TOKEN was
empty, allowing unauthenticated requests to reach /train and /segtrain.
Now returns 503 ("Training not configured on this node") when the token
is absent, so missing configuration fails closed rather than open.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:02:13 +02:00
Marcel
dc283ba271 fix(training): remove @Transactional from triggerTraining to avoid holding DB connection during OCR HTTP call
OcrTrainingService.triggerTraining() and triggerSegTraining() held a DB
connection open for the entire ketos training run (potentially minutes),
risking connection pool exhaustion. Replaced class-level @Transactional
with TransactionTemplate for narrow DB writes: guard+create and
result-record each run in their own short transaction; the HTTP call to
the OCR service runs between them with no open connection.

Also replaces blockRepository.findAll().size() with blockRepository.count()
in getTrainingInfo() to avoid loading every block into heap on each poll.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 09:59:12 +02:00
Marcel
62be895b9e fix(ocr): drop uvicorn workers from 2 to 1
Two workers × ~5 GB Surya model load = ~10 GB required, exceeding the
8 GB memory cap and causing OOM on the first /train call. Two OS
processes also cause model-state divergence after training, contradicting
the single-node constraint documented in ADR-001.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 09:55:55 +02:00
Marcel
7b79dc105b test(migrations): add Testcontainers integration tests for V23 + V30 constraints
Some checks failed
CI / Unit & Component Tests (push) Failing after 1s
CI / Backend Unit Tests (push) Failing after 1s
CI / Unit & Component Tests (pull_request) Failing after 1s
CI / Backend Unit Tests (pull_request) Failing after 0s
V23 introduced a JSONB check constraint (chk_annotation_polygon_quad)
requiring polygon arrays to have exactly 4 points. V30 introduced a
partial unique index preventing two concurrent RUNNING training runs.
These are DB-level invariants that unit tests cannot verify — five
Testcontainers tests now assert they are correctly applied by Flyway
and enforced by PostgreSQL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 23:07:17 +02:00
Marcel
e933aacc92 docs(infra): add .env.example with OCR_TRAINING_TOKEN
Fresh cloners had no tracked reference for required env vars.
.env is gitignored (contains real credentials). .env.example
documents all variables including the new OCR_TRAINING_TOKEN
for the Python OCR microservice training endpoints.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 23:03:10 +02:00
Marcel
fdba3211aa fix(a11y): add aria-live to OcrProgress page counter
Screen readers did not announce page-by-page OCR progress updates.
Wrapping the counter text in a span with aria-live=polite ensures
assistive technology announces each page completion without
interrupting the user.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 23:02:25 +02:00
Marcel
287920a982 docs(ocr): document single-node constraint for OCR training
Training reloads the Kraken model in-process on the Python service.
The DB-level RUNNING constraint prevents concurrent API calls but
cannot protect against multi-replica deployments. Added explicit
comments in docker-compose.yml and OcrTrainingService to prevent
accidental horizontal scaling. See ADR-001.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 23:01:45 +02:00
Marcel
2b355e748e fix(ocr): increase presigned URL TTL from 15 min to 1 hour
A 100-page document at ~10 s/page takes ~17 min on CPU-only hardware,
which could cause the presigned URL to expire mid-OCR job. 1 hour gives
ample headroom for any realistic document size in this archive.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 23:00:52 +02:00
Marcel
2181fe0b50 test(annotations): fix AnnotationServiceTest — add missing TranscriptionBlockRepository mock
The cascade-delete commit (5a5a8b6) added blockRepository.deleteByAnnotationId()
to AnnotationService.deleteAnnotation(), but the test class was not updated to
mock TranscriptionBlockRepository. Mockito injected null, causing deleteAnnotation_succeeds_whenOwner
to throw NPE. Adds the mock, verifies the cascade call, and adds an inOrder test
asserting the block is deleted before the annotation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 23:00:09 +02:00
Marcel
5a5a8b6e5c fix(annotations): cascade-delete transcription block when annotation is deleted
Some checks failed
CI / Unit & Component Tests (push) Failing after 1s
CI / Backend Unit Tests (push) Failing after 1s
CI / Unit & Component Tests (pull_request) Failing after 1s
CI / Backend Unit Tests (pull_request) Failing after 1s
The DELETE endpoint was returning 500 due to a FK constraint violation.
`deleteAnnotation` now calls `blockRepository.deleteByAnnotationId()`
before removing the annotation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 22:31:02 +02:00
Marcel
5afdc37653 feat(ui): manual-first OCR workflow — remove full-page auto-segmentation
Drawing annotations is now the primary workflow. OCR only runs on
manually drawn regions (guided mode always). Full-page layout detection
and the useExistingAnnotations checkbox are removed entirely.

- OcrTrigger: guided-only, disabled with hint when no annotations exist
- TranscriptionEditView: empty state shows draw-regions instruction,
  OCR trigger moved out of collapsible and shown inline after block list
- i18n: add ocr_trigger_no_annotations, ocr_section_heading,
  transcription_empty_draw_hint; remove ocr_use_existing_annotations keys

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 22:24:50 +02:00
Marcel
669f2f8b98 fix(training): output CoreML format and fix best-model finder
ketos 7 defaults to safetensors output, but kraken's load_any() only
handles CoreML (.mlmodel). Adding --weights-format coreml ensures the
hot-swap after training produces a file that load_any() can parse.

Also fixed _find_best_model to look for best_<score>.mlmodel (produced
by --weights-format coreml) in addition to the previous checkpoint_*
pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 21:57:42 +02:00
Marcel
49c9022285 fix(training): switch to PAGE XML format for kurrent recognition training
Kraken 7 removed support for the legacy `path` format (image + .gt.txt
pairs) in VGSLRecognitionDataModule despite the CLI still advertising it.
Switching to PAGE XML (-f page) format which is the supported standard.

- Java export now writes .xml alongside .png (PAGE XML with TextLine,
  Baseline at 75% height, and Unicode transcription)
- XML special characters in transcription text are escaped (&amp; &lt; &gt;)
- Python trainer globs *.xml and passes -f page to ketos train
- Regenerated frontend API types to include cer/loss/accuracy/epochs on
  OcrTrainingRun (were missing, causing empty CER column in history)
- Updated and extended TrainingDataExportServiceTest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 21:45:08 +02:00
Marcel
94b9c56527 fix(segtrain): reduce input height to 800px on first run to avoid OOM
ketos segtrain has no batch-size flag (-B), so with the default 1800px
input height the intermediate CNN feature maps consume ~500 MB+ per
image, causing the kernel OOM-killer (exit -9) to terminate the process.

On first run (no existing blla.mlmodel), override the VGSL spec to use
800px height instead. Subsequent runs load the saved model with
--resize both, preserving incremental fine-tuning.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 21:37:24 +02:00
Marcel
89a18c430e fix(training): limit CPU threads and epochs to prevent RAM exhaustion
Force CPU-only training (--device cpu), cap OpenMP/BLAS thread pool at 2
(--threads 2), and reduce epochs from 50 to 10 (-N 10). 50 epochs on a
laptop OOM-killed the container. 10 epochs is sufficient for incremental
fine-tuning runs; more data is added over time and training re-run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 21:09:13 +02:00
Marcel
8dec5b5976 fix(training): disable DataLoader workers in subprocess training
DataLoader worker subprocesses crash inside Docker due to multiprocessing
fork restrictions. Pass --workers 0 to both ketos train and ketos segtrain
so data loading runs in the main process.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 20:58:32 +02:00
Marcel
e33164c4aa fix(training): use ketos CLI subprocess instead of missing Python API
kraken.ketos has no .train or .segtrain attributes in Kraken 7 — both are
only exposed as CLI commands. Rewrites both training functions to invoke
`ketos train` / `ketos segtrain` via subprocess and parse the best
val_metric from checkpoint filenames.

Also fixes the OcrTrainingCard history so it only shows non-blla runs
(recognition model), matching SegmentationTrainingCard which already
filtered to blla-only.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 20:50:21 +02:00
Marcel
22954f348a feat(training): track and display CER per training run
After each training run, the Character Error Rate (CER = 1 - accuracy),
loss, accuracy, and epoch count are now stored on the OcrTrainingRun
record and shown in the training history table.

Also adds the missing POST /api/ocr/segtrain endpoint and the
triggerSegTraining service method so the segmentation training card
can actually trigger training.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 19:01:10 +02:00
Marcel
a99afef319 fix(training): only count reviewed blocks as checked text for recognition
Previously all MANUAL blocks counted as eligible training data, even ones
where text was filled in by guided OCR but never explicitly reviewed. This
caused segmentation and recognition counts to always match.

Now only reviewed=true blocks qualify for recognition training, so the
counts properly reflect: segments = all drawn annotation boxes,
checked text = only boxes where the user has verified the transcription.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 18:00:59 +02:00
Marcel
1fd5c31fd1 fix(training): pass trainingInfo directly to SegmentationTrainingCard
The parent was manually remapping availableSegBlocks → availableBlocks
before passing props, which broke after the card was updated to read
availableSegBlocks directly. Pass the full trainingInfo object instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 17:55:16 +02:00
Marcel
a514cbca18 fix(training): segmentation card reads availableSegBlocks not availableBlocks
Both cards were reading the same availableBlocks field, so the segmentation
box always showed the kurrent recognition count. Use the correct
availableSegBlocks field from the training info response.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 17:54:20 +02:00
Marcel
063095f58c fix(training): count segmentation blocks regardless of text content
The findSegmentationBlocks query was filtering out blocks with non-empty
text. Segmentation training only needs annotation geometry (polygon/bbox),
not transcription text — so any MANUAL block on a KURRENT_SEGMENTATION
document should count, regardless of whether it has text.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 17:14:40 +02:00
Marcel
b6f74fd6fc refactor(annotations): remove overlap check to allow intersecting regions
Historical letter lines often intersect, so the system must support
overlapping annotation regions. Removed the overlap guard from
createAnnotation(), deleted ErrorCode.ANNOTATION_OVERLAP, and cleaned
up all tests and frontend error mappings that referenced it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 16:48:18 +02:00
Marcel
8618e520b5 fix(ocr): fill empty MANUAL blocks in guided OCR mode
When a user draws annotation boxes to mark OCR regions, the blocks are
created with source=MANUAL and empty text. upsertGuidedBlock was
protecting all MANUAL blocks unconditionally, so guided OCR silently
produced no output for these drawn-but-empty blocks.

Changed the guard to only protect non-empty MANUAL blocks — empty ones
are treated like OCR blocks and get their text filled in.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 16:25:23 +02:00
Marcel
3e34366702 fix(ocr): use cw-1/ch-1 for synthetic baseline bounds to pass Kraken's >= check
Kraken's segmentation bounds check rejects coordinates where any point
satisfies x >= im.width or y >= im.height (strictly >=, not >). Using
(cw, ch) as the boundary corner was triggering this for every crop.
Changed to (cw-1, ch-1) so all coordinates are strictly inside the image.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 16:21:00 +02:00
Marcel
051c43f088 fix(ocr): use synthetic baseline in guided OCR to avoid blla crash on small crops
blla.segment() is a full-page layout detection model that kills the worker
process when called on tiny annotation crops (e.g. 597x89 px). For guided
OCR the annotation region IS already the text line, so segmentation is
unnecessary. Replace the blla call with a single synthetic BaselineLine that
spans the full crop width — rpred then runs recognition on the whole crop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 16:09:35 +02:00
Marcel
ee58b63517 feat(ocr): add guided OCR mode using existing annotation regions
When a document has manually drawn annotation boxes, the user can now
enable "Nur annotierte Bereiche" in the OCR trigger panel. The engine
skips layout detection entirely and runs recognition only within the
pre-drawn bounding boxes, preserving manual transcription blocks.

- Python: adds OcrRegion model, extend OcrRequest/OcrBlock; guided
  branch in /ocr/stream groups by page and crops each region
- Engines: add extract_region_text() to both Kraken and Surya
- Java: adds OcrBlockResult.annotationId, OcrClient.OcrRegion,
  TriggerOcrDTO.useExistingAnnotations; OcrAsyncRunner dispatches to
  upsertGuidedBlock when annotationId is present; OcrService threads
  the flag through to runSingleDocument
- TranscriptionService: adds upsertGuidedBlock (creates, updates OCR,
  or preserves MANUAL blocks)
- Frontend: guided OCR toggle in OcrTrigger shown when blocks exist;
  skips destructive-replace confirmation in guided mode

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 15:57:54 +02:00