Compare commits

...

10 Commits

Author SHA1 Message Date
Marcel
18e675a5b2 fix(import): address non-blocking review feedback — touch target, glossary, edge-case test
All checks were successful
CI / Unit & Component Tests (push) Successful in 3m18s
CI / OCR Service Tests (push) Successful in 19s
CI / Backend Unit Tests (push) Successful in 3m22s
CI / fail2ban Regex (push) Successful in 41s
CI / Semgrep Security Scan (push) Successful in 18s
CI / Compose Bucket Idempotency (push) Successful in 1m0s
- Add min-h-[44px] py-2 to <summary> in ImportStatusCard for 44 px touch target
- Add SkippedFile and skipped count entries to docs/GLOSSARY.md
- Add MassImportServiceTest case: ALREADY_EXISTS fires before file I/O when doc is UPLOADED and file is present on disk

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
a3fc838855 fix(import): surface S3 failures + already-exists in skippedFiles, a11y + max-height
- Change importSingleDocument return type from boolean to Optional<String>
  so callers in processRows receive the skip reason on every non-success path.
  S3 upload failures now surface as "S3_UPLOAD_FAILED" and already-imported
  documents as "ALREADY_EXISTS" in the skippedFiles list shown in the admin UI.
- Add two new tests: runImportAsync_addsS3UploadFailed_toSkippedFiles and
  runImportAsync_addsAlreadyExists_toSkippedFiles; update
  importSingleDocument_skips_whenDocumentAlreadyUploadedNotPlaceholder and
  the S3-failure test to assert on the Optional return value.
- Add i18n keys for S3_UPLOAD_FAILED and ALREADY_EXISTS in de/en/es messages.
- Svelte ImportStatusCard: add aria-hidden="true" to SVG chevron, wrap
  conditional warning section in aria-live="polite" div, add max-h-64
  overflow-y-auto to skipped-files <ul> to cap height on large batches.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
d5043053e0 fix(import): address round-3 review concerns
- Add comment to openFileStream() explaining package-private visibility
  is intentional (Mockito spy seam for IOException test)
- Key {#each} skippedFiles by filename instead of array index
- Add test: skipped section hidden when state is FAILED
- Add test: reasonLabel returns raw code for unknown reason strings

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
c932dd19d9 fix(admin): address round-2 review concerns on ImportStatusCard
- Use loop index as each key (handles duplicate filenames)
- Increase skipped filename font from text-xs to text-sm
- Add motion-safe guard to details chevron transition
- Replace text-warning with text-amber-900 to meet WCAG AA contrast

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
c532ad21bf test(admin): add regression test for skipped section hidden during RUNNING
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
0e95bd9160 fix(import): add @Schema annotations and fix IOException test coverage
- Add @Schema(requiredMode = REQUIRED) to SkippedFile and ImportStatus
  record components so TypeScript codegen produces non-optional fields
  when generate:api is next run
- Extract openFileStream(File) as package-private method so the
  IOException path can be tested deterministically without relying on
  OS-level file permissions (which are bypassed when running as root)
- Replace assumeTrue-based IOException test with Mockito spy that stubs
  openFileStream — test now runs in CI unconditionally (45 tests, 0 skipped)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
e312cce4e1 fix(test): skip IOException test when running as root
setReadable(false) silently no-ops as root; check canRead() to guard
the assumption correctly so the test is skipped in Docker CI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
5587722800 fix(import): address PR review concerns
- remove duplicate List import in AdminControllerTest
- derive skipped() from skippedFiles.size() — drop redundant int field
- use machine codes for SkippedFile.reason (INVALID_PDF_SIGNATURE, FILE_READ_ERROR)
- map reason codes to i18n strings in ImportStatusCard (de/en/es)
- replace raw amber Tailwind classes with warning semantic token
- fix <summary> accessibility: replace list-none with rotating chevron SVG
- replace <p> with <span> inside <summary> (phrasing content rule)
- extract setupOneValidOneFakeImport() helper — remove 3x copy-paste
- add lenient mock to short-file test for defensive coverage
- add IOException path test for isPdfMagicBytes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
0451b6630c feat(admin): surface skipped file count in ImportStatusCard
Adds SkippedFile to the local ImportStatus type and updates
ImportStatusCard to show an amber skipped-count section with a
collapsible filename list in the DONE state. Only rendered when
skipped > 0. i18n keys added for de/en/es.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
Marcel
f77fb79cd2 feat(import): validate PDF magic bytes before S3 upload
Reads first 4 bytes of each candidate file before upload; rejects any
file whose header does not match %PDF (0x25 0x50 0x44 0x46). Skipped
files are counted and collected in ImportStatus.skippedFiles so
operators can see what was rejected without querying Loki.

Breaking: ImportStatus record gains skipped + skippedFiles fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-19 09:45:03 +02:00
10 changed files with 432 additions and 30 deletions

View File

@@ -1,6 +1,8 @@
package org.raddatz.familienarchiv.importing;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.poi.ss.usermodel.*;
@@ -31,6 +33,7 @@ import javax.xml.parsers.DocumentBuilderFactory;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
@@ -53,9 +56,33 @@ public class MassImportService {
public enum State { IDLE, RUNNING, DONE, FAILED }
public record ImportStatus(State state, String statusCode, @JsonIgnore String message, int processed, LocalDateTime startedAt) {}
public record SkippedFile(
@Schema(requiredMode = Schema.RequiredMode.REQUIRED) String filename,
@Schema(requiredMode = Schema.RequiredMode.REQUIRED) String reason
) {}
private volatile ImportStatus currentStatus = new ImportStatus(State.IDLE, "IMPORT_IDLE", "Kein Import gestartet.", 0, null);
public record ImportStatus(
@Schema(requiredMode = Schema.RequiredMode.REQUIRED) State state,
@Schema(requiredMode = Schema.RequiredMode.REQUIRED) String statusCode,
@JsonIgnore String message,
@Schema(requiredMode = Schema.RequiredMode.REQUIRED) int processed,
@Schema(requiredMode = Schema.RequiredMode.REQUIRED) List<SkippedFile> skippedFiles,
LocalDateTime startedAt
) {
// Note: @Schema on a record accessor method is not picked up by SpringDoc; the
// "skipped" count is a computed convenience field derived from skippedFiles.size().
@JsonProperty("skipped")
public int skipped() { return skippedFiles.size(); }
/** Defensive-copy constructor — callers cannot mutate the stored list after construction. */
public ImportStatus {
skippedFiles = List.copyOf(skippedFiles);
}
}
record ProcessResult(int processed, List<SkippedFile> skippedFiles) {}
private volatile ImportStatus currentStatus = new ImportStatus(State.IDLE, "IMPORT_IDLE", "Kein Import gestartet.", 0, List.of(), null);
public ImportStatus getStatus() {
return currentStatus;
@@ -117,22 +144,22 @@ public class MassImportService {
if (currentStatus.state() == State.RUNNING) {
throw DomainException.conflict(ErrorCode.IMPORT_ALREADY_RUNNING, "A mass import is already in progress");
}
currentStatus = new ImportStatus(State.RUNNING, "IMPORT_RUNNING", "Import läuft...", 0, LocalDateTime.now());
currentStatus = new ImportStatus(State.RUNNING, "IMPORT_RUNNING", "Import läuft...", 0, List.of(), LocalDateTime.now());
try {
File spreadsheet = findSpreadsheetFile();
log.info("Starte Massenimport aus: {}", spreadsheet.getAbsolutePath());
int processed = processRows(readSpreadsheet(spreadsheet));
ProcessResult result = processRows(readSpreadsheet(spreadsheet));
currentStatus = new ImportStatus(State.DONE, "IMPORT_DONE",
"Import abgeschlossen. " + processed + " Dokumente verarbeitet.",
processed, currentStatus.startedAt());
"Import abgeschlossen. " + result.processed() + " Dokumente verarbeitet.",
result.processed(), result.skippedFiles(), currentStatus.startedAt());
} catch (NoSpreadsheetException e) {
log.error("Massenimport fehlgeschlagen: keine Tabellendatei", e);
currentStatus = new ImportStatus(State.FAILED, "IMPORT_FAILED_NO_SPREADSHEET",
"Fehler: " + e.getMessage(), 0, currentStatus.startedAt());
"Fehler: " + e.getMessage(), 0, List.of(), currentStatus.startedAt());
} catch (Exception e) {
log.error("Massenimport fehlgeschlagen", e);
currentStatus = new ImportStatus(State.FAILED, "IMPORT_FAILED_INTERNAL",
"Fehler: " + e.getMessage(), 0, currentStatus.startedAt());
"Fehler: " + e.getMessage(), 0, List.of(), currentStatus.startedAt());
}
}
@@ -254,8 +281,10 @@ public class MassImportService {
// --- Import logic (works on neutral List<String> rows) ---
private int processRows(List<List<String>> rows) {
int count = 0;
private ProcessResult processRows(List<List<String>> rows) {
int processed = 0;
List<SkippedFile> skippedFiles = new ArrayList<>();
for (int i = 1; i < rows.size(); i++) { // skip header row
List<String> cells = rows.get(i);
String index = getCell(cells, colIndex);
@@ -266,18 +295,58 @@ public class MassImportService {
if (fileOnDisk.isEmpty()) {
log.warn("Datei nicht gefunden, importiere nur Metadaten: {}", filename);
}
importSingleDocument(cells, fileOnDisk, filename, index);
count++;
if (fileOnDisk.isPresent()) {
try {
if (!isPdfMagicBytes(fileOnDisk.get())) {
log.warn("Überspringe {}: Datei beginnt nicht mit %PDF-Signatur", filename);
skippedFiles.add(new SkippedFile(filename, "INVALID_PDF_SIGNATURE"));
continue;
}
} catch (IOException e) {
log.error("Fehler beim Prüfen der Magic-Bytes für {}", filename, e);
skippedFiles.add(new SkippedFile(filename, "FILE_READ_ERROR"));
continue;
}
}
Optional<String> skipReason = importSingleDocument(cells, fileOnDisk, filename, index);
if (skipReason.isPresent()) {
skippedFiles.add(new SkippedFile(filename, skipReason.get()));
} else {
processed++;
}
}
return count;
return new ProcessResult(processed, skippedFiles);
}
// package-private: Mockito spy in tests can override to inject IOException
InputStream openFileStream(File file) throws IOException {
return new FileInputStream(file);
}
private boolean isPdfMagicBytes(File file) throws IOException {
try (InputStream is = openFileStream(file)) {
byte[] header = is.readNBytes(4);
return header.length == 4
&& header[0] == 0x25 // %
&& header[1] == 0x50 // P
&& header[2] == 0x44 // D
&& header[3] == 0x46; // F
}
}
/**
* Imports a single document row.
*
* @return empty Optional on success; an Optional containing the skip reason on failure/skip.
*/
@Transactional
protected void importSingleDocument(List<String> cells, Optional<File> file, String originalFilename, String index) {
protected Optional<String> importSingleDocument(List<String> cells, Optional<File> file, String originalFilename, String index) {
Optional<Document> existing = documentService.findByOriginalFilename(originalFilename);
if (existing.isPresent() && existing.get().getStatus() != DocumentStatus.PLACEHOLDER) {
log.info("Dokument {} existiert bereits, überspringe.", originalFilename);
return;
return Optional.of("ALREADY_EXISTS");
}
String archiveBox = getCell(cells, colBox);
@@ -313,7 +382,7 @@ public class MassImportService {
status = DocumentStatus.UPLOADED;
} catch (Exception e) {
log.error("S3 Upload Fehler für {}", file.get().getName(), e);
return;
return Optional.of("S3_UPLOAD_FAILED");
}
}
@@ -355,6 +424,7 @@ public class MassImportService {
thumbnailAsyncRunner.dispatchAfterCommit(saved.getId());
}
log.info("Importiert{}: {}", file.isEmpty() ? " (nur Metadaten)" : "", originalFilename);
return Optional.empty();
}
// --- Helpers ---

View File

@@ -135,7 +135,7 @@ class MassImportServiceTest {
@Test
void runImportAsync_throwsConflict_whenAlreadyRunning() {
MassImportService.ImportStatus running = new MassImportService.ImportStatus(
MassImportService.State.RUNNING, "IMPORT_RUNNING", "Running...", 0, LocalDateTime.now());
MassImportService.State.RUNNING, "IMPORT_RUNNING", "Running...", 0, List.of(), LocalDateTime.now());
ReflectionTestUtils.setField(service, "currentStatus", running);
assertThatThrownBy(() -> service.runImportAsync())
@@ -154,9 +154,76 @@ class MassImportServiceTest {
.build();
when(documentService.findByOriginalFilename("doc001.pdf")).thenReturn(Optional.of(existing));
service.importSingleDocument(minimalCells("doc001.pdf"), Optional.empty(), "doc001.pdf", "doc001");
Optional<String> result = service.importSingleDocument(minimalCells("doc001.pdf"), Optional.empty(), "doc001.pdf", "doc001");
verify(documentService, never()).save(any());
assertThat(result).isPresent().contains("ALREADY_EXISTS");
}
// ─── importSingleDocument — already-exists guard fires before file I/O ─────
@Test
void importSingleDocument_skipsWithAlreadyExists_whenDocumentUploadedAndFileIsPresent(@TempDir Path tempDir) throws Exception {
// Document already exists with status UPLOADED (not PLACEHOLDER).
// A physical PDF file is also present on disk (valid magic bytes).
// Expected: ALREADY_EXISTS is returned and no S3 upload is attempted —
// the guard fires before any file I/O, so no partial processing occurs.
Document existing = Document.builder()
.id(UUID.randomUUID())
.originalFilename("present.pdf")
.status(DocumentStatus.UPLOADED)
.build();
when(documentService.findByOriginalFilename("present.pdf")).thenReturn(Optional.of(existing));
Path physicalFile = tempDir.resolve("present.pdf");
byte[] pdfHeader = {0x25, 0x50, 0x44, 0x46, 0x2D}; // %PDF-
Files.write(physicalFile, pdfHeader);
Optional<String> result = service.importSingleDocument(
minimalCells("present.pdf"), Optional.of(physicalFile.toFile()), "present.pdf", "present");
assertThat(result).isPresent().contains("ALREADY_EXISTS");
verify(s3Client, never()).putObject(any(PutObjectRequest.class), any(RequestBody.class));
verify(documentService, never()).save(any());
}
// ─── importSingleDocument — S3 failure surfaced in skippedFiles ──────────
@Test
void runImportAsync_addsS3UploadFailed_toSkippedFiles_whenS3Throws(@TempDir Path tempDir) throws Exception {
byte[] pdfHeader = {0x25, 0x50, 0x44, 0x46, 0x2D}; // %PDF-
Files.write(tempDir.resolve("upload_fail.pdf"), pdfHeader);
buildMinimalImportXlsx(tempDir, "upload_fail.pdf");
ReflectionTestUtils.setField(service, "importDir", tempDir.toString());
when(documentService.findByOriginalFilename("upload_fail.pdf")).thenReturn(Optional.empty());
doThrow(new RuntimeException("S3 unavailable"))
.when(s3Client).putObject(any(PutObjectRequest.class), any(RequestBody.class));
service.runImportAsync();
assertThat(service.getStatus().skipped()).isEqualTo(1);
assertThat(service.getStatus().skippedFiles())
.extracting(MassImportService.SkippedFile::filename, MassImportService.SkippedFile::reason)
.containsExactly(org.assertj.core.groups.Tuple.tuple("upload_fail.pdf", "S3_UPLOAD_FAILED"));
}
@Test
void runImportAsync_addsAlreadyExists_toSkippedFiles_whenDocumentAlreadyUploaded(@TempDir Path tempDir) throws Exception {
buildMinimalImportXlsx(tempDir, "existing.pdf");
ReflectionTestUtils.setField(service, "importDir", tempDir.toString());
Document existing = Document.builder()
.id(UUID.randomUUID())
.originalFilename("existing.pdf")
.status(DocumentStatus.UPLOADED)
.build();
when(documentService.findByOriginalFilename("existing.pdf")).thenReturn(Optional.of(existing));
service.runImportAsync();
assertThat(service.getStatus().skipped()).isEqualTo(1);
assertThat(service.getStatus().skippedFiles())
.extracting(MassImportService.SkippedFile::reason)
.containsExactly("ALREADY_EXISTS");
}
// ─── importSingleDocument — create new document (metadata only) ───────────
@@ -208,7 +275,7 @@ class MassImportServiceTest {
}
@Test
void importSingleDocument_returnsEarly_whenS3UploadFails(@TempDir Path tempDir) throws Exception {
void importSingleDocument_returnsS3UploadFailed_whenS3UploadFails(@TempDir Path tempDir) throws Exception {
Path tempFile = tempDir.resolve("fail.pdf");
Files.write(tempFile, "data".getBytes());
@@ -216,10 +283,11 @@ class MassImportServiceTest {
doThrow(new RuntimeException("S3 error"))
.when(s3Client).putObject(any(PutObjectRequest.class), any(RequestBody.class));
service.importSingleDocument(
Optional<String> result = service.importSingleDocument(
minimalCells("fail.pdf"), Optional.of(tempFile.toFile()), "fail.pdf", "fail");
verify(documentService, never()).save(any());
assertThat(result).isPresent().contains("S3_UPLOAD_FAILED");
}
// ─── importSingleDocument — sender handling ───────────────────────────────
@@ -325,8 +393,8 @@ class MassImportServiceTest {
@Test
void processRows_returnsZero_whenOnlyHeaderRow() {
List<List<String>> rows = List.of(List.of("header", "col1"));
Integer result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
assertThat(result).isEqualTo(0);
MassImportService.ProcessResult result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
assertThat(result.processed()).isEqualTo(0);
}
@Test
@@ -335,8 +403,8 @@ class MassImportServiceTest {
List.of("header"),
minimalCells("") // blank index
);
Integer result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
assertThat(result).isEqualTo(0);
MassImportService.ProcessResult result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
assertThat(result.processed()).isEqualTo(0);
verify(documentService, never()).findByOriginalFilename(any());
}
@@ -349,9 +417,9 @@ class MassImportServiceTest {
List.of("header"),
minimalCells("doc001") // no dot → appends ".pdf"
);
Integer result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
MassImportService.ProcessResult result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
assertThat(result).isEqualTo(1);
assertThat(result.processed()).isEqualTo(1);
verify(documentService).findByOriginalFilename("doc001.pdf");
}
@@ -364,9 +432,9 @@ class MassImportServiceTest {
List.of("header"),
minimalCells("doc002.pdf") // has dot → used as-is
);
Integer result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
MassImportService.ProcessResult result = ReflectionTestUtils.invokeMethod(service, "processRows", rows);
assertThat(result).isEqualTo(1);
assertThat(result.processed()).isEqualTo(1);
verify(documentService).findByOriginalFilename("doc002.pdf");
}
@@ -525,6 +593,67 @@ class MassImportServiceTest {
assertThat(result).isEqualTo("hello");
}
// ─── PDF magic byte validation regression ─────────────────────────────────
@Test
void runImportAsync_uploadsValidPdf_andSkipsFakeOne(@TempDir Path tempDir) throws Exception {
setupOneValidOneFakeImport(tempDir);
service.runImportAsync();
verify(s3Client, times(1)).putObject(any(PutObjectRequest.class), any(RequestBody.class));
}
@Test
void runImportAsync_setsSkippedCount_toOne_whenOneFakeFile(@TempDir Path tempDir) throws Exception {
setupOneValidOneFakeImport(tempDir);
service.runImportAsync();
assertThat(service.getStatus().skipped()).isEqualTo(1);
}
@Test
void runImportAsync_includesRejectedFilename_inSkippedFiles(@TempDir Path tempDir) throws Exception {
setupOneValidOneFakeImport(tempDir);
service.runImportAsync();
assertThat(service.getStatus().skippedFiles())
.extracting(MassImportService.SkippedFile::filename)
.contains("fake.pdf");
}
@Test
void runImportAsync_skipsFile_whenShorterThanFourBytes(@TempDir Path tempDir) throws Exception {
Files.write(tempDir.resolve("tiny.pdf"), new byte[]{0x25, 0x50, 0x44}); // only 3 bytes
buildMinimalImportXlsx(tempDir, "tiny.pdf");
ReflectionTestUtils.setField(service, "importDir", tempDir.toString());
lenient().when(documentService.findByOriginalFilename(any())).thenReturn(Optional.empty());
service.runImportAsync();
assertThat(service.getStatus().skipped()).isEqualTo(1);
}
@Test
void runImportAsync_skipsFile_whenMagicBytesCheckThrowsIOException(@TempDir Path tempDir) throws Exception {
Files.writeString(tempDir.resolve("unreadable.pdf"), "some content");
buildMinimalImportXlsx(tempDir, "unreadable.pdf");
ReflectionTestUtils.setField(service, "importDir", tempDir.toString());
lenient().when(documentService.findByOriginalFilename(any())).thenReturn(Optional.empty());
MassImportService spyService = spy(service);
doThrow(new java.io.IOException("simulated read error")).when(spyService).openFileStream(any(File.class));
spyService.runImportAsync();
assertThat(spyService.getStatus().skipped()).isEqualTo(1);
assertThat(spyService.getStatus().skippedFiles())
.extracting(MassImportService.SkippedFile::reason)
.containsExactly("FILE_READ_ERROR");
}
// ─── readOds — XXE security regression ───────────────────────────────────
// Security regression — do not remove.
@@ -621,4 +750,28 @@ class MassImportServiceTest {
}
return destination.toFile();
}
private void setupOneValidOneFakeImport(Path tempDir) throws Exception {
byte[] pdfHeader = {0x25, 0x50, 0x44, 0x46, 0x2D}; // %PDF-
Files.write(tempDir.resolve("real.pdf"), pdfHeader);
Files.writeString(tempDir.resolve("fake.pdf"), "not a pdf");
buildMinimalImportXlsx(tempDir, "real.pdf", "fake.pdf");
ReflectionTestUtils.setField(service, "importDir", tempDir.toString());
when(documentService.findByOriginalFilename(any())).thenReturn(Optional.empty());
when(documentService.save(any())).thenAnswer(inv -> inv.getArgument(0));
}
private void buildMinimalImportXlsx(Path dir, String... filenames) throws Exception {
Path xlsx = dir.resolve("import.xlsx");
try (XSSFWorkbook wb = new XSSFWorkbook()) {
org.apache.poi.ss.usermodel.Sheet sheet = wb.createSheet("Sheet1");
sheet.createRow(0).createCell(0).setCellValue("Index");
for (int i = 0; i < filenames.length; i++) {
sheet.createRow(i + 1).createCell(0).setCellValue(filenames[i]);
}
try (OutputStream out = Files.newOutputStream(xlsx)) {
wb.write(out);
}
}
}
}

View File

@@ -47,7 +47,7 @@ class AdminControllerTest {
@WithMockUser(authorities = "ADMIN")
void importStatus_returns200_withStatusCode_whenAdmin() throws Exception {
MassImportService.ImportStatus status = new MassImportService.ImportStatus(
MassImportService.State.IDLE, "IMPORT_IDLE", "Kein Import gestartet.", 0, null);
MassImportService.State.IDLE, "IMPORT_IDLE", "Kein Import gestartet.", 0, List.of(), null);
when(massImportService.getStatus()).thenReturn(status);
mockMvc.perform(get("/api/admin/import-status"))
@@ -61,7 +61,7 @@ class AdminControllerTest {
@WithMockUser(authorities = "ADMIN")
void importStatus_messageField_notPresentInApiResponse() throws Exception {
MassImportService.ImportStatus status = new MassImportService.ImportStatus(
MassImportService.State.IDLE, "IMPORT_IDLE", "Kein Import gestartet.", 0, null);
MassImportService.State.IDLE, "IMPORT_IDLE", "Kein Import gestartet.", 0, List.of(), null);
when(massImportService.getStatus()).thenReturn(status);
mockMvc.perform(get("/api/admin/import-status"))

View File

@@ -57,6 +57,10 @@ _See also [Annotation](#annotation-documentannotation)._
**Mass import** — an asynchronous batch process (`MassImportService`) that reads an Excel or ODS file and creates `Person`s, `Tag`s, and `PLACEHOLDER` `Document`s in one shot. Only one import can run at a time (`IMPORT_ALREADY_RUNNING` error if attempted concurrently).
**SkippedFile** (`MassImportService.SkippedFile`) — a file that was presented for import but not processed, recorded with a `filename` and a `reason` code. Possible reasons: `INVALID_PDF_SIGNATURE` (magic-byte validation failed), `S3_UPLOAD_FAILED` (file upload to MinIO/S3 threw an exception), `FILE_READ_ERROR` (the file could not be opened for reading), or `ALREADY_EXISTS` (a document with the same filename already exists in the archive with a status other than `PLACEHOLDER`).
**skipped count** — the total number of `SkippedFile` entries accumulated during a single import run (`ImportStatus.skipped()`). Shown in the amber warning section of the Import Status Card in the admin UI; a value of zero suppresses the section entirely.
**Transcription queue** — the set of `Document`s and `TranscriptionBlock`s awaiting work, computed on-the-fly from `Document`/`Block` status. Three views: segmentation queue, transcription queue, ready-to-read queue. NOT a persistent entity — no `transcription_queues` table exists.
_See also [DocumentStatus lifecycle](#documentstatus-lifecycle)._

View File

@@ -352,6 +352,11 @@
"admin_system_import_status_running": "Import läuft…",
"admin_system_import_status_done": "Import abgeschlossen",
"admin_system_import_status_done_label": "Dokumente verarbeitet",
"admin_system_import_skipped_label": "übersprungen",
"import_reason_invalid_pdf_signature": "Keine gültige PDF-Signatur",
"import_reason_file_read_error": "Fehler beim Lesen der Datei",
"import_reason_s3_upload_failed": "Upload-Fehler (S3)",
"import_reason_already_exists": "Bereits importiert",
"admin_system_import_status_failed": "Import fehlgeschlagen",
"admin_system_import_failed_no_spreadsheet": "Keine Tabellendatei gefunden.",
"admin_system_import_failed_internal": "Interner Fehler beim Import.",

View File

@@ -352,6 +352,11 @@
"admin_system_import_status_running": "Import running…",
"admin_system_import_status_done": "Import complete",
"admin_system_import_status_done_label": "Documents processed",
"admin_system_import_skipped_label": "skipped",
"import_reason_invalid_pdf_signature": "Invalid PDF signature",
"import_reason_file_read_error": "File read error",
"import_reason_s3_upload_failed": "Upload error (S3)",
"import_reason_already_exists": "Already imported",
"admin_system_import_status_failed": "Import failed",
"admin_system_import_failed_no_spreadsheet": "No spreadsheet file found.",
"admin_system_import_failed_internal": "Import failed due to an internal error.",

View File

@@ -352,6 +352,11 @@
"admin_system_import_status_running": "Importación en curso…",
"admin_system_import_status_done": "Importación completada",
"admin_system_import_status_done_label": "Documentos procesados",
"admin_system_import_skipped_label": "omitidos",
"import_reason_invalid_pdf_signature": "Firma PDF no válida",
"import_reason_file_read_error": "Error al leer el archivo",
"import_reason_s3_upload_failed": "Error de carga (S3)",
"import_reason_already_exists": "Ya importado",
"admin_system_import_status_failed": "Importación fallida",
"admin_system_import_failed_no_spreadsheet": "No se encontró ninguna hoja de cálculo.",
"admin_system_import_failed_internal": "Error interno durante la importación.",

View File

@@ -15,6 +15,14 @@ const failureMessage = $derived(
? m.admin_system_import_failed_no_spreadsheet()
: m.admin_system_import_failed_internal()
);
function reasonLabel(code: string): string {
if (code === 'INVALID_PDF_SIGNATURE') return m.import_reason_invalid_pdf_signature();
if (code === 'FILE_READ_ERROR') return m.import_reason_file_read_error();
if (code === 'S3_UPLOAD_FAILED') return m.import_reason_s3_upload_failed();
if (code === 'ALREADY_EXISTS') return m.import_reason_already_exists();
return code;
}
</script>
<div class="rounded-sm border border-line bg-surface p-6 shadow-sm">
@@ -48,6 +56,41 @@ const failureMessage = $derived(
</p>
<p class="mt-1 text-xs text-green-800">{m.admin_system_import_status_done()}</p>
</div>
<div aria-live="polite">
{#if importStatus.skipped > 0}
<details class="mb-4 rounded-sm border border-warning/40 bg-warning/10 p-4 text-amber-900">
<summary class="flex min-h-[44px] cursor-pointer list-none items-center gap-2 py-2">
<svg
aria-hidden="true"
class="details-chevron h-4 w-4 shrink-0 motion-safe:transition-transform"
viewBox="0 0 16 16"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="M6 4l4 4-4 4" />
</svg>
<div>
<span data-testid="skipped-count" class="block text-base font-bold"
>{importStatus.skipped}</span
>
<span class="block font-sans text-xs font-bold tracking-widest uppercase">
{m.admin_system_import_skipped_label()}
</span>
</div>
</summary>
<ul class="mt-3 max-h-64 space-y-1 overflow-y-auto">
{#each importStatus.skippedFiles as skipped (skipped.filename)}
<li class="font-mono text-sm text-ink-2">
{skipped.filename}{reasonLabel(skipped.reason)}
</li>
{/each}
</ul>
</details>
{/if}
</div>
<button
data-import-trigger
onclick={ontrigger}
@@ -79,3 +122,9 @@ const failureMessage = $derived(
</button>
{/if}
</div>
<style>
details[open] .details-chevron {
transform: rotate(90deg);
}
</style>

View File

@@ -8,6 +8,8 @@ const makeStatus = (overrides: Partial<ImportStatus> = {}): ImportStatus => ({
state: 'IDLE',
statusCode: 'IMPORT_IDLE',
processed: 0,
skipped: 0,
skippedFiles: [],
startedAt: null,
...overrides
});
@@ -128,4 +130,106 @@ describe('ImportStatusCard', () => {
await getByRole('button').click();
expect(ontrigger).toHaveBeenCalledOnce();
});
it('shows skipped count when DONE and skipped > 0', async () => {
const { getByTestId } = render(ImportStatusCard, {
props: {
importStatus: makeStatus({
state: 'DONE',
statusCode: 'IMPORT_DONE',
processed: 10,
skipped: 3,
skippedFiles: [
{ filename: 'fake.pdf', reason: 'INVALID_PDF_SIGNATURE' },
{ filename: 'other.pdf', reason: 'INVALID_PDF_SIGNATURE' },
{ filename: 'tiny.pdf', reason: 'INVALID_PDF_SIGNATURE' }
]
}),
ontrigger: () => {}
}
});
await expect.element(getByTestId('skipped-count')).toHaveTextContent('3');
});
it('shows skipped filenames in collapsible list when DONE and skipped > 0', async () => {
const { getByText } = render(ImportStatusCard, {
props: {
importStatus: makeStatus({
state: 'DONE',
statusCode: 'IMPORT_DONE',
processed: 5,
skipped: 1,
skippedFiles: [{ filename: 'fake.pdf', reason: 'INVALID_PDF_SIGNATURE' }]
}),
ontrigger: () => {}
}
});
await expect.element(getByText('fake.pdf')).toBeInTheDocument();
});
it('does not show skipped section when DONE and skipped is 0', async () => {
const { getByTestId } = render(ImportStatusCard, {
props: {
importStatus: makeStatus({ state: 'DONE', statusCode: 'IMPORT_DONE', processed: 5 }),
ontrigger: () => {}
}
});
await expect.element(getByTestId('skipped-count')).not.toBeInTheDocument();
});
it('does not show skipped section when RUNNING even with skipped > 0', async () => {
const { getByTestId } = render(ImportStatusCard, {
props: {
importStatus: makeStatus({
state: 'RUNNING',
statusCode: 'IMPORT_RUNNING',
processed: 5,
skipped: 2,
skippedFiles: [
{ filename: 'a.pdf', reason: 'INVALID_PDF_SIGNATURE' },
{ filename: 'b.pdf', reason: 'INVALID_PDF_SIGNATURE' }
]
}),
ontrigger: () => {}
}
});
await expect.element(getByTestId('skipped-count')).not.toBeInTheDocument();
});
it('does not show skipped section when FAILED even with skipped > 0', async () => {
const { getByTestId } = render(ImportStatusCard, {
props: {
importStatus: makeStatus({
state: 'FAILED',
statusCode: 'IMPORT_FAILED_INTERNAL',
skipped: 1,
skippedFiles: [{ filename: 'bad.pdf', reason: 'INVALID_PDF_SIGNATURE' }]
}),
ontrigger: () => {}
}
});
await expect.element(getByTestId('skipped-count')).not.toBeInTheDocument();
});
it('shows raw reason code for unknown skip reasons', async () => {
const { getByText } = render(ImportStatusCard, {
props: {
importStatus: makeStatus({
state: 'DONE',
statusCode: 'IMPORT_DONE',
processed: 1,
skipped: 1,
skippedFiles: [{ filename: 'odd.pdf', reason: 'SOME_FUTURE_CODE' }]
}),
ontrigger: () => {}
}
});
await expect.element(getByText('SOME_FUTURE_CODE', { exact: false })).toBeInTheDocument();
});
});

View File

@@ -1,6 +1,13 @@
export type SkippedFile = {
filename: string;
reason: string;
};
export type ImportStatus = {
state: 'IDLE' | 'RUNNING' | 'DONE' | 'FAILED';
statusCode: string;
processed: number;
skipped: number;
skippedFiles: SkippedFile[];
startedAt: string | null;
};