Hoist per-chunk config lookups and parallelize deep group traversal

[d-v-b]

Addresses P4–P6 of #181.

Three independent performance findings were investigated. Two are implemented; one was found to be invalid on inspection and is documented below.

1. Hoist repeated async.concurrency config lookups off the per-chunk codec path (P4)

With the default codec_pipeline.batch_size=1, the read/write/encode/decode paths re-read config.get("async.concurrency") once per chunk (in read_batch/write_batch), once per batch in read/write, and once per codec per chunk via _batching_helper and the partial decode/encode mixins. Each lookup walks Donfig's nested config.

This adds an operation-scoped cache in zarr.core.config:

• async_concurrency_scope() — a context manager that captures async.concurrency once.
• get_async_concurrency() — returns the captured value inside a scope, else falls back to a live config.get.

BatchedCodecPipeline.read/write/decode/encode now wrap their bodies in the scope, and all eight per-chunk/per-codec lookups (5 in codec_pipeline.py, 3 in abc/codec.py) read through get_async_concurrency(). No public signatures changed. Config changes between operations are still honored (each operation re-captures on entry); only mid-operation changes are ignored, which is the intended hoisting behavior. Verified manually.

2. Parallelize deep group traversal across sibling subgroups (P5)

_iter_members_deep previously drained each subgroup's recursive iterator in sequence (for prefix, subgroup_iter in to_recurse.items(): async for ...), so sibling subtrees did not overlap latency. It now drives sibling iterators concurrently via per-subgroup drain tasks feeding an asyncio.Queue, yielding members as they arrive. The shared asyncio.Semaphore (already threaded into every recursive call) still bounds total concurrent getitem operations. Member ordering remains unspecified, as documented on Group.members — existing tests already sort before comparing, and the full tests/test_group.py suite passes.

3. (SKIPPED) StorePath.__truediv__ re-normalizing per chunk (P6)

Skipped: the premise does not hold against the current code. StorePath.__truediv__ calls _join_paths([self.path, other]), which is a plain "/".join(filter(...)) with no regex, no split, and no dot-segment scan. It does not call normalize_path. The per-chunk join is already a cheap fast path, so there is nothing to optimize here without a risky micro-optimization of _join_paths for negligible benefit. No change made.

Test results (uv run)

• uv run pytest tests/test_codecs/ tests/test_group.py tests/test_store/ -q → 2000 passed, 207 skipped, 2 xfailed
• uv run pytest tests/test_indexing.py -q → 424 passed, 1 skipped, 5 xfailed
• uv run pytest tests/test_config.py -q → 37 passed
• uv run mypy src/zarr/core/config.py src/zarr/core/codec_pipeline.py src/zarr/abc/codec.py src/zarr/core/group.py → Success: no issues found in 4 source files

Note

No towncrier changelog entry was added — one should be added before this is un-drafted.

🤖 Generated with Claude Code