feat(zarr-metadata): add metadata model layer

[d-v-b]

🤖 AI text below 🤖

Adds zarr_metadata.model: frozen-dataclass models that are canonical, lossless representations of Zarr v2/v3 metadata documents, plus structural validators. This is part 1 of the thin-metadata refactor (design spec): the models ported here (from the zng_metadata prototype) become the metadata layer zarr-python consumes after the next zarr-metadata release.

What's included

• model._validation — ValidationProblem / MetadataValidationError diagnostics and validate_* / is_* / parse_* trios for JSON values, v3 metadata fields, and v2/v3 array + group documents. Structural checks only (key presence, JSON shapes); no domain interpretation.
• model._array — ZarrMetadataV3 (name + configuration, the in-memory form of MetadataV3), ArrayMetadataModelV3, ArrayMetadataModelV2 (+*Partial TypedDicts with drift-guard tests). Every v3 extension point is held as a ZarrMetadataV3; fill_value is verbatim JSON (NaN as "NaN"); JSON arrays are tuples. from_key_value/to_key_value handle the zarr.json / .zarray+.zattrs store layouts.
• model._group — GroupMetadataModelV3 (with the consolidated_metadata convention as a typed field), GroupMetadataModelV2, ConsolidatedMetadataModelV3 (entries parsed into thin child models), ConsolidatedMetadataModelV2 (flat .zmetadata map held verbatim — merging file fragments into node models would lose which nodes had a .zattrs entry and break byte-faithful round-tripping; see the design-spec refinement).
• Front-door exports of the model classes from zarr_metadata; validators and key/store constants stay under zarr_metadata.model.
• 188 new tests (161 ported from the prototype suite, 27 new for groups/consolidated). pyright --strict clean.

API-hardening pass (added after an ergonomics review)

A fresh agent with no design context consumed defective documents through this API and reported back; its findings drove five changes (see the harden model validation commit):

1. Validators now enforce what the types declare. The v2 validator checks dtype / order / compressor / filters / dimension_separator shapes, and all four document validators check the fixed zarr_format / node_type literals. (Resolves the previously-flagged "validators don't check literal values" question.)
2. ValidationProblem carries a machine-readable kind (missing_key / invalid_type / invalid_value / invalid_json) — consumers dispatch on it instead of string-matching messages.
3. One exception type for all ingestion failures: from_key_value no longer leaks KeyError / JSONDecodeError, and constructor invariants raise MetadataValidationError (still a ValueError) instead of bare ValueError.
4. ZarrMetadataV3 → NamedConfigModelV3, plus a role-named alias MetadataFieldModelV3 used in all annotation positions: annotations convey the logical meaning of the field, not its serialized form. Today the alias is exactly NamedConfigModelV3; if a future spec revision adds a field form that cannot normalize to name + configuration, the alias widens to a union and annotation sites do not move (mirrors the raw-layer NamedConfigV3 / MetadataV3 split).
5. Discoverability: the validate_*/is_*/parse_* contract is documented on zarr_metadata.model itself; update() documents that it does not re-validate; the v2 to_json vs to_key_value attributes split is documented on both.

An adversarial pass (constructing invalid documents that pass validation) closed four more holes (close validation holes commit): JSON booleans and negative values in shape/chunks; dimension_names length not matching shape; non-JSON attributes/configuration values (previously either silently rewritten by json.dumps on round-trip or escaping as a raw TypeError); and the consolidated-metadata envelope, now deep-validated by the same validate_consolidated_metadata_v3 the model constructor uses, so the is_group_metadata_v3 type guard never vouches for a document from_json would reject.

Must-understand extension fields (decision made after reading the spec text): the spec requires an implementation to fail to open groups/arrays carrying fields it does not recognize that are not explicitly must_understand: false — and fields are implicitly must-understand. Since recognition is reader-specific (consolidated_metadata is itself such a field) and a document carrying a must-understand extension is still a valid document, the refusal duty sits at open/resolve time, not in document validation. The v3 models now expose must_understand_fields — the obligation partition of extra_fields — so a reader discharges the duty with meta.must_understand_fields.keys() - recognized; the design spec pins this as a part-2 resolve-layer requirement (core zarr-python recognizes no extra fields, matching today's parse_extra_fields behavior). Still unchecked, as domain territory: empty v2 dtype record lists and empty codec names.

Still flagged: always-emitted .zattrs Resolved: the .zattrs file's presence is part of the store. v2 attributes is dict | UNSET: UNSET (no file) emits nothing, any dict — including an explicit empty {} — emits the file, and the two stores stay distinct through round-trips.

Optional pydantic integration

zarr_metadata.pydantic ships one Annotated field type per model (metadata: zmp.ArrayMetadataV3 on any BaseModel). Design was gamed out with prototypes against two alternatives: __get_pydantic_core_schema__ dunders on the core classes (verified working across pydantic 2.0–2.13, rejected to keep the dependency-free layer framework-free) and pydantic-aware subclasses in the namespace (rejected on empirical failures — identity split breaks equality, core instances are rejected by subclass-typed fields, nested construction escapes the subclass hierarchy). The chosen shape keeps instances as the core classes (free interop with non-pydantic code), imports pydantic eagerly at the module (loud failure when absent), and quarantines protocol risk to one module. Validation delegates to from_json, so pydantic coercion can never bypass the structural validators; tests/model/test_pydantic.py additionally documents the hand-rolled recipes and the engine-backed BaseModel pattern for pydantic-zarr-style consumers.

Follow-up

After merge: cut zarr_metadata-v0.4.0, bump zarr-python's floor pin, then part 2 (resolve layer + thin metadata swap in zarr-python) proceeds per the design spec.

🤖 Generated with Claude Code

[d-v-b]

🤖 AI text below 🤖

The design decisions taken in this PR (where the upstreamed models diverge from the zng_metadata prototype) are now documented as issues on the prototype repo:

• zng#19 — tracking: models upstreamed, port adaptations, action after zarr_metadata-v0.4.0
• zng#20 — ZarrMetadataV3 → NamedConfigModelV3 + role alias MetadataFieldModelV3
• zng#21 — ValidationProblem.kind + one exception type; Ok/Err result type rejected
• zng#22 — validators enforce declared types (v2 fields, literals, adversarial holes)
• zng#23 — must_understand_fields and the fail-to-open duty
• zng#24 — group/consolidated models; v2 .zmetadata verbatim

🤖 Generated with Claude Code