Fix structured dtype v2 round-trip and silent loss of field offsets/padding

[d-v-b]

Summary

Fixes two verified HIGH-severity bugs in structured-dtype handling. Addresses B9–B10 of #181.

Bug 1 — Nested structured dtypes failed the Zarr V2 JSON round-trip

check_structured_dtype_v2_inner (src/zarr/core/dtype/common.py) is a recursive type guard for the inner elements of a V2 structured-dtype name. For a nested field such as ["nested", [["x","<f4"],["y","<f8"]]], the last element is a list of [name, dtype] field pairs, but the function recursed into itself — which expects a single [name, dtype] pair — and returned False.

Consequence: Struct.to_json(zarr_format=2) emits exactly this nested form, then get_data_type_from_json(j, zarr_format=2) raised ValueError: No Zarr data type found. Zarr wrote V2 metadata it could not read back.

Fix: when the last element is a non-string Sequence, validate it with check_structured_dtype_name_v2 (the list-of-fields checker) instead of recursing as a single pair. Verified the full round-trip from_native_dtype → to_json(v2) → from_json → to_native_dtype for nested dtypes 2 and 3 levels deep, plus an end-to-end V2 array write/reopen/read.

Bug 2 — Structured dtypes silently lost field offsets / padding / alignment

Structured.from_native_dtype (src/zarr/core/dtype/npy/structured.py) records only (name, dtype) pairs, and to_native_dtype rebuilds with np.dtype([(name, dtype), ...]), which re-packs fields contiguously. So np.dtype([('a','i1'),('b','i8')], align=True) (itemsize 16) round-tripped to itemsize 9 — silently changing field offsets and misinterpreting stored chunk bytes (data corruption).

Decision — loud failure rather than layout preservation. numpy's only faithfully round-trippable representation of an aligned/padded structured dtype is dtype.descr, which inserts synthetic empty-named void padding fields (e.g. ('', '|V7')). The Zarr V2 structured name form is a list of [name, dtype] pairs and could carry those padding entries shape-wise, but doing so would require materially changing the public Structured.fields model (adding synthetic void fields), and would ripple through the V3 form, scalar handling, and all field-iteration logic — a large, risky change to field semantics that is not what the zarr-v2 / numcodecs ecosystem historically wrote. Given the hard constraints "reading existing packed data must keep working unchanged" and "no drive-by refactors," the surgical and safe choice is to fail loudly rather than silently corrupt.

Fix: from_native_dtype now detects a non-default-packed layout (a recursive check that compares the dtype's itemsize and per-field offsets against a contiguously-repacked rebuild, so padding inside a nested field is caught even when the outer dtype is packed) and raises a clear ValueError explaining that aligned/padded structured dtypes are not serializable. The error is a ValueError (not DataTypeValidationError) on purpose: the data type registry suppresses DataTypeValidationError as "does not match", but a padded layout does match this dtype class and must surface to the caller.

Reading existing stored data is unaffected — the from_json paths reconstruct packed dtypes exactly as before; only from_native_dtype of a non-packed numpy dtype now raises.

Tests

Added regression tests in tests/test_dtype/test_npy/test_structured.py: the exact repro cases plus a 2-level-nested case and align=True cases (top-level padding and nested-field padding), and a packed round-trip guard against false positives.

$ uv run pytest tests/test_dtype/ -q
719 passed, 5 skipped

$ uv run pytest tests/test_dtype/test_npy/test_structured.py -q
41 passed

$ uv run pytest tests/test_v2.py -q
27 passed

$ uv run mypy src/zarr/core/dtype/common.py src/zarr/core/dtype/npy/structured.py
Success: no issues found in 2 source files

Notes

• A towncrier changelog entry is still needed before this PR is taken out of draft.

🤖 Generated with Claude Code