refactor: use lower-level metadata models in zarr-python#4129
Draft
d-v-b wants to merge 59 commits into
Draft
Conversation
…#176) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@57e3a13...e79a696) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](j178/prek-action@6ad8027...bdca6f1) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v7...043fb46) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v7...3e5f45b) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Findings from an API-ergonomics exercise (a fresh agent consuming defective metadata documents): - ValidationProblem gains a machine-readable kind (missing_key / invalid_type / invalid_value / invalid_json), ending message string-matching in consumers. - The v2 array validator now enforces what its types declare (dtype, order, compressor, filters, dimension_separator), and all four document validators check the fixed zarr_format / node_type literals. - All ingestion failures surface as MetadataValidationError: missing store keys and undecodable bytes in from_key_value (previously KeyError / JSONDecodeError) and constructor invariants (previously bare ValueError). - ZarrMetadataV3 is renamed NamedConfigModelV3: it models a name + configuration pair, and the old name read as a whole-document type. - Discoverability: the validate_*/is_*/parse_* contract is documented on zarr_metadata.model itself; update() documents that it does not re-validate; the v2 to_json/to_key_value attributes split is documented on both. Assisted-by: ClaudeCode:claude-fable-5
…aFieldModelV3 Model fields and consumer signatures should convey the logical meaning of the type (a metadata-document field), not the form it takes when JSON-serialized (a named configuration). MetadataFieldModelV3 is today exactly NamedConfigModelV3; if a future spec revision adds a field form that cannot normalize to name + configuration, the alias widens to a union and annotation sites do not move. Mirrors the raw-layer split between NamedConfigV3 (shape) and MetadataV3 (field union). Assisted-by: ClaudeCode:claude-fable-5
test_v3_to_json_includes_required_fields hand-enumerated keys with chained asserts, restating what ARRAY_METADATA_REQUIRED_KEYS_V3 already defines. Now: one coverage assert driven by the constant (tracks the TypedDict automatically) and one whole-document equality for the values. Assisted-by: ClaudeCode:claude-fable-5
The subset assert against ARRAY_METADATA_REQUIRED_KEYS_V3 was redundant: equality with a literal that spells out the full document already covers every required key. One dict, one assert. Assisted-by: ClaudeCode:claude-fable-5
Invalid documents that previously passed validation:
- shape/chunks containing JSON booleans (bool is an int subclass in
Python but not an integer in a metadata document) or negative values
- dimension_names whose length does not match shape
- attributes and configuration values that are not JSON-serializable —
now checked recursively like fill_value, so an int-keyed dict cannot
be silently rewritten by json.dumps on round-trip and a set() cannot
escape as a TypeError from to_key_value
- consolidated_metadata envelopes: the group validator now deep-validates
the envelope and its entries via the shared
validate_consolidated_metadata_v3, which ConsolidatedMetadataModelV3
.from_json also uses, so is_group_metadata_v3 never vouches for a
document the model constructor would reject
Three pre-existing test fixtures paired dimension_names=('x',) with the
default scalar shape () and were themselves spec-invalid; they now use a
matching 1-d shape.
Deliberately unchanged, pending a design decision: unknown extension
fields with must_understand: true still pass (which layer owns the
spec's refusal duty), and empty v2 dtype records / empty codec names
still pass (domain territory).
Assisted-by: ClaudeCode:claude-fable-5
The v3 core spec: 'An implementation MUST fail to open Zarr groups or arrays if any metadata fields are present which (a) the implementation does not recognize and (b) are not explicitly set to "must_understand": false' — and fields are implicitly must-understand unless waived. The model layer cannot discharge this itself: recognition is reader-specific (consolidated_metadata is itself an extension field one reader understands and another does not), and a document carrying a must-understand extension is still a valid document. So the models partition by obligation: must_understand_fields is the subset of extra_fields not explicitly waived, and a compliant reader fails to open when must_understand_fields.keys() - recognized is non-empty. The design spec pins that duty on the part-2 resolve layer, matching what zarr-python's parse_extra_fields enforces today. Assisted-by: ClaudeCode:claude-fable-5
Delegate wholesale rather than letting pydantic introspect the dataclass: InstanceOf (is-instance core schema) + BeforeValidator(from_json) + PlainSerializer(to_json, return_type=dict). Field-by-field validation is impossible anyway (the models' annotation-only imports live behind TYPE_CHECKING, so pydantic raises class-not-fully-defined) and would diverge from the library's structural validation via coercion if it weren't. MetadataValidationError subclasses ValueError, so failed parses surface as pydantic ValidationError with the loc-annotated messages. pydantic is already in the package's test dependency group. Assisted-by: ClaudeCode:claude-fable-5
…y not Correcting the previous commit's too-strong claim: pydantic CAN introspect the model dataclass — TypeAdapter(...).rebuild() with the TYPE_CHECKING-only names supplied as _types_namespace resolves the schema, and __post_init__ invariants still run. A new test exercises that path and pins why it is not the recommended integration: it validates the model shape, not the document (bare-string data_type rejected — no from_json normalization), and pydantic's lax coercion silently re-opens holes the library validators close (shape=[True, -5] coerces to (1, -5); a wrong dimension_names count passes). Assisted-by: ClaudeCode:claude-fable-5
…ic-zarr pattern) For consumers that want a first-class BaseModel — JSON schema generation and generics for typed attributes, as in pydantic-zarr's ArraySpec — the example adds a third pattern: pydantic-native fields as the user-facing surface, with the library as the engine. A mode='before' validator canonicalizes every input via from_json(...).to_json(), so structural validation and normalization run before pydantic parses fields (the [True, -5] coercion divergence cannot occur), and to_metadata_model / to_document bridge both ways through the document form. One translation noted at the bridge: the document spells 'no dimension names' as key absence, the pydantic side as None. Assisted-by: ClaudeCode:claude-fable-5
Spec: 'If specified, must be an array of strings or null objects... If dimension_names is not specified, all dimensions are unnamed.' The null object is a permitted element (an unnamed dimension), never the field value; key absence is the only spelling of 'not specified'. Pins the validator's existing rejection so it is not later 'fixed' to accept null-as-absence, and documents that in-memory None maps to key absence on serialization. Assisted-by: ClaudeCode:claude-fable-5
…ydantic Gamed out three shapes with prototypes before choosing: - dunders on the core classes (works, verified pydantic 2.0-2.13, but puts a framework protocol in the dependency-free layer); - pydantic-aware SUBCLASSES in a namespace (rejected on empirical failures: identity split breaks equality, core instances are rejected by subclass-typed fields, and nested construction produces core-class children unless every cross-reference is overridden); - Annotated field types over the CORE classes in an opt-in module (chosen): instances are the core classes so interop is free, pydantic imports eagerly at the module (loud failure when absent), core stays framework-free, and pydantic-protocol risk is quarantined to one clearly-labeled module. The module exports one field type per model. Validation delegates to from_json (structural validation and normalization cannot be bypassed by pydantic coercion), instances pass through, serialization emits the canonical document, and WithJsonSchema describes the accepted document form so model_json_schema works. Tests cover all seven field types, core-instance interop, error quality, JSON schema, roundtrip, and that importing zarr_metadata does not import pydantic. Assisted-by: ClaudeCode:claude-fable-5
create_default(shape=(100, 100)) silently kept the scalar default's 0-d chunk grid (chunk_shape: ()), producing a structurally-valid but semantically inconsistent document — a footgun for every test fixture built on it. When shape is overridden and the grid is not, the default is now one regular chunk covering the array (v3 chunk_shape == shape, v2 chunks == shape); an explicit chunk_grid/chunks override still wins. update() stays a dumb dataclasses.replace, per its documented contract. One existing whole-document test literal carried exactly this inconsistency (shape (10,) with chunk_shape ()) and was updated. Assisted-by: ClaudeCode:claude-fable-5
…unk grid
The spec's constraint is conditional ('non-zero when the corresponding
dimensions of the arrays have non-zero length'), so chunk_shape == shape
is sound for every shape, including empty dimensions.
Assisted-by: ClaudeCode:claude-fable-5
…-way Overriding shape without a grid derives the grid; the reverse does not hold. A user-supplied chunk_grid is an extension point taken verbatim — deriving shape from it would require interpreting grid configurations, which the model layer never does and cannot do for unrecognized grid names. Pinned by test so the asymmetry reads as a decision, not an oversight; the v2 model documents the same one-way rule for chunks for cross-version consistency. Assisted-by: ClaudeCode:claude-fable-5
Audited all 24 (15 src, 9 tests); each was either obsolete, replaceable by a sound cast, or avoidable by better-typed code: - Two fill_value arg-type ignores were factually obsolete: their justifying comment said 'fill_value: object in upstream TypedDict', but 0.3.0 narrowed it to JSONValue. - Eight pre-existing call-arg/reportInvalidTypeForm ignores on the PEP 728 TypedDicts and the recursive JSONValue alias were mypy-dialect suppressions that the checker of record (pyright strict with enableExperimentalFeatures) never needed; mypy has never checked this package. - The two extra_fields comprehensions are a genuine checker limitation (a key filter cannot narrow a PEP 728 TypedDict's item-value union), now expressed as casts whose comments state the soundness claim instead of suppressing the diagnostic. - pydantic.py's generic coercer factory takes the parse callable explicitly instead of calling from_json through type[_M]. - NamedConfigModelV3.from_json casts the validated configuration (sound since configuration values are now deep-validated as JSON). - Tests: _build_v2/_build_v3 gained real Unpack[...Partial] signatures; raw-document pydantic inputs go through model_validate (the idiomatic entry point for untyped data) instead of ignoring constructor signatures; the frozen-dataclass test uses setattr for its intentional runtime error. src and tests/model now carry zero type-ignore comments. Assisted-by: ClaudeCode:claude-fable-5
roborev job 426 (branch review) found that ArrayMetadataModelV2 normalized an ABSENT dimension_separator key to '/', inherited verbatim from the zng prototype. The v2 convention's default is '.': a consumer deriving chunk keys from the model against a real-world v2 array written with the default separator would have looked for '0/0' instead of '0.0'. No test caught it because every fixture started from create_default(), which always carries an explicit separator. Absence is normalized to an explicit '.' -- a semantics-preserving spelling normalization consistent with the model's existing canonical forms (bare-string metadata fields, missing configuration). The field is deliberately NOT modeled as Optional: the document grammar has no null spelling for this key, and a None in the model invites writing 'dimension_separator': null into documents. Pinned by three tests, including explicit-null rejection. Assisted-by: ClaudeCode:claude-fable-5
Establishes the models' None/absence invariant: None in a model always
corresponds to a JSON null in the document (a v2 compressor/filters
value, an unnamed dimension inside dimension_names), and UNSET always
means the key is absent. The two are never interchangeable.
Applied to the two fields that used None as an absence marker:
dimension_names (ArrayMetadataModelV3) and consolidated_metadata
(GroupMetadataModelV3). For dimension_names this also preserves a
semantic distinction d-v-b identified: an absent field ("there are no
dimension names") and an explicit all-null array ("every dimension has
a name, which is null") are different documents; both spellings now
round-trip faithfully and compare unequal.
Normalizing absence to the all-null form was considered and rejected:
the spellings' interpretations coincide but interpretation-equivalence
is the resolve layer's business, and collapsing document-level
distinctions on that basis is the layer violation this package exists
to avoid. Verified that current zarr-python never writes
"consolidated_metadata": null (GroupMetadata.to_dict pops the key), so
None there was purely an absence marker, not a document spelling.
UnsetType is a single-member enum (identity-checkable, repr "UNSET",
deliberately truthy so `if not x` cannot silently treat it as absent);
UNSET and UnsetType are exported from zarr_metadata.model and the
package front door.
Assisted-by: ClaudeCode:claude-fable-5
…a null
Historical zarr-python versions wrote "consolidated_metadata": null into
group documents for groups without consolidated metadata, so real stores
contain the spelling; the validator was rejecting those documents
("expected a mapping"). Per the None/UNSET invariant, the field is now
honestly three-state: UNSET (key absent), None (the document's literal
null, preserved on round-trip), or a ConsolidatedMetadataModelV3.
Interpreting null as absence is the consumer's call, not a document
rewrite by this layer.
Also records an implementation constraint on the sentinel itself:
typing_extensions.Sentinel (PEP 661) is the intended spelling, but
pyright 1.1.411 degrades a Sentinel to Unknown in dataclass FIELD
annotations (function signatures work), verified by probe both with and
without enableExperimentalFeatures. Using it would reintroduce
suppressions at every use site under the strict gate, so UNSET stays a
single-member enum, with the Sentinel switch documented in
_sentinel.py for when pyright catches up.
Assisted-by: ClaudeCode:claude-fable-5
… preserve it d-v-b: the bugged spelling should not be preserved or honored. The three-state field reverts to two states (model | UNSET): a document carrying "consolidated_metadata": null — written by a historical zarr-python bug — remains readable (the validator accepts it so real stores open), but the spelling gets no model representation: it is read as absence and never written back. This is the one deliberate exception to faithful round-tripping, pinned as such: from_json(null_doc) equals from_json(absent_doc), and to_json omits the key. Assisted-by: ClaudeCode:claude-fable-5
…11115 Investigated: the Unknown-degradation of typing_extensions.Sentinel is a confirmed upstream pyright regression, not by-design. Introduced in 1.1.405 (verified: 1.1.404 is clean on the same probe, 1.1.411 fails), affects reads of any class-body attribute annotation (dataclass or plain class), does not affect function signatures or module variables, and Final on the sentinel does not help. Tracked as microsoft/pyright#11115 (open, bug+regression); #11467 closed as its duplicate. The enum sentinel stays until the fix lands. Assisted-by: ClaudeCode:claude-fable-5
… pyright Pinning a working pyright (<= 1.1.404) in CI was considered and does not suffice: the pin controls one of four checker surfaces. Contributor IDEs (Pylance bundles current pyright) and downstream consumers' pyright read the py.typed inline annotations with their own versions, and decisively, mypy 2.1.0 has no PEP 661 support at all — a sentinel in type position is a hard [valid-type] error, which would degrade these fields to Any for mypy consumers, including zarr-python itself. The enum is currently the only spelling with exact types on every surface; switch when pyright#11115 is fixed AND mypy implements PEP 661. Assisted-by: ClaudeCode:claude-fable-5
Corrects the sentinel implementation note: PEP 661 was accepted 2026-04-23 and ships as stdlib sentinel in Python 3.15. The two checker gaps blocking the Sentinel spelling (pyright regression #11115, mypy not yet implementing the PEP) are therefore temporary gaps against a Final standard, and the enum is a stopgap with a defined end state. Assisted-by: ClaudeCode:claude-fable-5
…are the laggards ty 0.0.56 types the Sentinel spelling perfectly in dataclass fields: exact T | UNSET unions, both-direction is/is-not narrowing, and wrong-typed constructor arguments rejected (verified with reveal_type, so it is real inference, not silent Any). The checker matrix for sentinel-in-type-position is therefore ty full / pyright regressed (#11115) / mypy not implemented — recorded so the switch decision has current calibration. Assisted-by: ClaudeCode:claude-fable-5
d-v-b's call: PEP 661 is Final, ty already types the sentinel spelling exactly, mypy support is in review (python/mypy#21647) and treated as imminent, and pyright has a known-good version — so use the standard sentinel today rather than carrying the enum stopgap. - UNSET is now typing_extensions.Sentinel("UNSET"), used directly in type expressions (tuple[str | None, ...] | UNSET); the UnsetType companion enum is gone from the API. - typing_extensions floor bumped to 4.14 (where Sentinel arrived). - CI pins pyright==1.1.404, the last version before the class-attribute sentinel regression (microsoft/pyright#11115); pyproject documents the same pin for local runs. 0 errors on the pin; ty checks the sentinel fields clean (its 2 remaining diagnostics are its incomplete PEP 728 extra_items write support, unrelated). - Known short-term cost, accepted deliberately: mypy-checked consumers need cast/type-ignore at narrowing sites until mypy#21647 merges, and contributors' Pylance may show phantom Unknowns until the pyright fix ships. Recorded in _sentinel.py and the changelog. - The pydantic native-introspection test reverts to documenting that introspection is unsupported (pydantic 2.13 cannot schema a Sentinel); the delegation patterns are unaffected. Assisted-by: ClaudeCode:claude-fable-5
…ifact
Resolves the last flagged round-trip question from the initial port:
to_key_value on the v2 models always emitted a .zattrs key, so a store
that never had one gained a file on round-trip. Per d-v-b's ruling,
attributes on ArrayMetadataModelV2/GroupMetadataModelV2 is now
`dict[str, JSONValue] | UNSET`: UNSET means no .zattrs file (and no
attributes key in the merged document form) and emits nothing, while
any dict — including an explicit empty {} — means the file exists and
is emitted. The two spellings stay distinct through round-trips, per
the None/UNSET invariant; create_default defaults to UNSET (a fresh
minimal node has no .zattrs).
Assisted-by: ClaudeCode:claude-fable-5
…by reference Models hold the UNSET sentinel as field values (dimension_names, attributes), so any object graph containing a model must survive pickling and deep-copying. A sentinel's contract is identity — state-based pickling would produce impostor objects that fail every `is UNSET` check — which is why typing_extensions <= 4.15 refused to pickle sentinels at all. typing_extensions 4.16 implements Sentinel.__reduce__ as pickling by reference (a lookup of the sentinel's name on its defining module), the same mechanism enum members use, so the singleton identity survives the round trip. Bump the floor and pin the behavior with tests: identity across pickle/copy/deepcopy, models holding UNSET round-tripping, and a guard that a non-importable sentinel still fails loudly rather than pickling by state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mary, JSON suffix for documents Applies the naming decisions from the PR discussion: ZarrV2/ZarrV3 moves to the front of every type name so a format version cannot be misread as a class revision, and the model dataclasses take the bare entity names (ZarrV3ArrayMetadata, ZarrV3GroupMetadata, ZarrV3ConsolidatedMetadata, ZarrV3NamedConfig, role alias ZarrV3MetadataField) while the TypedDict document forms carry a JSON suffix (ZarrV3ArrayMetadataJSON, ..., ZarrV3MetadataFieldJSON, ZarrV3NamedConfigJSON). The zarr_metadata.pydantic field types take the bare entity names, matching the model classes they validate into; the module now references the model module qualified to keep those names free. Raw-layer names released in 0.3.0 are renamed without aliases (pre-1.0), documented in changes/4119.removal.md. Validation problem messages name documents in plain English instead of type names. snake_case function names (validate_array_metadata_v3, ...) and SCREAMING_SNAKE constants are deliberately untouched: the revision ambiguity the rename fixes does not arise for them, and renaming them is a separate decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zarr now declares a runtime dependency on zarr-metadata (>=0.4.0, the first release that will contain the metadata model layer). Within a checkout the dependency resolves to packages/zarr-metadata via a uv workspace source, so CI and dev environments always test against the in-tree package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data models
Adds zarr.core.metadata.model with array_metadata_to_model/from_model and
group_metadata_to_model/from_model, bridging zarr-python's interpreted
metadata classes (ZDType, codec instances, cast fill values) and the pure
JSON-shaped model dataclasses in zarr_metadata.model.
runtime -> model round-trips through the JSON document form and is total;
model -> runtime behaves exactly like reading the equivalent document from a
store, including rejection of uninterpretable extension points.
Also teaches DataTypeRegistry.match_json to accept the v3-spec-equivalent
{"name": X, "configuration": {}} spelling of configuration-free data types:
the model layer (and other zarr implementations) normalize bare names to that
form, which the per-dtype from_json methods previously rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AsyncArray, Array, AsyncGroup, and Group gain a _future_metadata property exposing the node's metadata as a zarr_metadata document model (ArrayMetadataModelV2 | ArrayMetadataModelV3 for arrays, GroupMetadataModelV2 | GroupMetadataModelV3 for groups). This is the planned future type of the public .metadata attribute; per-format group metadata classes arrive with it. The model is derived lazily from the runtime metadata and cached, keyed on the identity of the metadata object. To make that sound, the two remaining in-place metadata mutations (array _update_attributes and AsyncGroup.update_attributes) now swap in a new metadata object instead of mutating attributes in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migrates the reads where the document model already carries exactly the value internals need: array shape/ndim/size, indexer shapes, resize/append shape checks, and group zarr_format dispatches. Interpreted reads (dtype, codecs, fill value, chunk key encoding) stay on the runtime metadata until the model becomes the stored source of truth. The zarr_format == 2 dispatches inside AsyncArray property bodies deliberately stay on self.metadata: they type-check via Literal narrowing of the constrained metadata TypeVar, and they gate access to interpreted fields that have no model equivalent yet. Arrays and groups also drop the _future_metadata cache from pickled state: the model holds PEP 661 sentinels (zarr_metadata.model.UNSET), which are process-local by design and refuse to pickle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reading Array.metadata or Group.metadata now emits ZarrPendingDeprecationWarning (a PendingDeprecationWarning, so ignored by default) announcing that the attribute's type will change to the zarr-metadata document model classes. _future_metadata previews the new interface. Internal consumers of the sync properties (the Attributes wrapper, migrate_v3, the testing strategies) move to a private no-warning _metadata accessor so zarr never warns on its own operation. The test suite ignores the warning class globally for now (asserted explicitly in tests/test_future_metadata.py); that filter comes out when the tests themselves migrate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zarr-metadata now requires typing_extensions>=4.16, whose Sentinel pickles by reference with singleton identity preserved, so the _future_metadata cache no longer *has* to be excluded from pickled node state — it still is, because it is derived state and the wire format should not couple to the cache layout. Comments updated accordingly, the lockfile picks up 4.16, and the pickle tests now round-trip whole nodes parametrized over the states sensitive to serialization (absent vs present optional keys, both formats). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follows the naming decision applied in the zarr-metadata model layer: format version at the front, dataclasses take the bare entity names. All references (conversion layer, _future_metadata annotations and docstrings, warning messages, tests, changelog) move from ArrayMetadataModelV2/V3, GroupMetadataModelV2/V3, ConsolidatedMetadataModelV3, NamedConfigModelV3 to ZarrV2ArrayMetadata / ZarrV3ArrayMetadata, ZarrV2GroupMetadata / ZarrV3GroupMetadata, ZarrV3ConsolidatedMetadata, ZarrV3NamedConfig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The zarr-metadata model layer that zarr now depends on is unreleased, and
the previous wiring only covered uv project flows, breaking everything else:
- The floor 'zarr-metadata>=0.4.0' is unresolvable from PyPI (only <0.4.0
exists), failing every hatch env (test matrix, doctests, benchmarks,
hypothesis, min_deps, upstream) and the readthedocs build. The floor now
names the newest release (>=0.3) with a TODO to bump at release time, and
each non-uv flow installs the in-tree package explicitly: hatch envs get a
'{root:uri}/packages/zarr-metadata' direct reference (repeated per env,
since hatch replaces extra-dependencies on inheritance), readthedocs gets
an extra pip install step.
- uv workspace membership made 'uv sync' inside packages/zarr-metadata
resolve the whole workspace, dragging zarr's requires-python (>=3.12) into
the subpackage's own CI, which tests down to 3.11. The workspace is now a
plain editable path source, restoring the subpackage's independence.
- min_deps pins typing_extensions==4.16.* (was 4.14.*): 4.16 is the
effective minimum because zarr-metadata requires it for sentinel pickling.
Also fixes the Lint (mypy) findings in the new tests — a bare generic Array
annotation, stale type-ignores, and two sentinel comparisons mypy cannot
narrow (python/mypy#21647) — and renames the changelog entries to the PR
number (4129) required by the changelog check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI builds from shallow clones, where hatch-vcs cannot see the zarr_metadata-v* tags and derives 0.1.devN for the in-tree package — so the '>=0.3' floor rejected the very package CI installs (uv reported the URL-pinned candidate as 'only zarr-metadata<0.3 is available'). No floor can be satisfied by both PyPI and tag-less in-tree builds until 0.4.0 ships, so the requirement carries no version until then; the TODO(release) comment marks where the real floor lands. Verified against a shallow clone: both the hatch env and the frozen root uv sync now build and import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4129 +/- ##
==========================================
+ Coverage 93.62% 93.69% +0.06%
==========================================
Files 90 91 +1
Lines 11936 12019 +83
==========================================
+ Hits 11175 11261 +86
+ Misses 761 758 -3
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR builds on the changes outlined in #4119 by adding the low-level metadata classes defined in zarr-metadata as load-bearing members of zarr-python's array and group classes. This doesn't replace the
.metadataattribute. That would take place over a few more transitional PRs. It is the first of several PRs that would refactor our metadata classes to be more future-proof.Author attestation
AI coding assistance is welcome, but a human must be the author and is responsible for the contents of the PR. The description and any review responses must be in your own words. Please read AI-assisted contributions before opening.
TODO
docs/user-guide/*.mdchanges/