[rhoadesScholar]

Prototype for ZEP 12 (zarr-developers/zeps#72, spec text zarr-developers/zarr-specs#400) and the offset chunk grid of zarr-developers/zarr-extensions#81.

Array.prepend(data, axis) adds data before index 0 of an axis, the way append adds it after the end. It grows by any number of elements and never touches an existing chunk:

z = zarr.create_array(store, shape=(6, 4), chunks=(4, 4), dtype="int32")
z[:] = 1
z.prepend(np.full((3, 4), 2, dtype="int32"))
z.shape   # (9, 4)
z.domain  # ((-3, 6), (0, 4))
# keys: c/-1/0 is new; c/0/0 and c/1/0 are unchanged, byte for byte

The first prepend switches the written metadata from shape to bounds ({"inclusive_min": [-3, 0], "exclusive_max": [6, 4]}); the chunk grid stays regular, and chunk grid indices below 0 come out by floor division. Readers that do not implement bounds refuse the array, since shape is absent, rather than misread it. Indexing stays zero-based: index i is coordinate inclusive_min + i, and Array.domain reports the coordinate range. The offset chunk grid (offset + base_chunk_grid, a regular base only) is read and written; it places chunk 0 at a chosen coordinate and is not needed to prepend.

To try it: pip install "zarr @ git+https://github.com/zarr-developers/zarr-python@refs/pull/4516/head"

What growing in front costs. Two chunk rows in front of an array of 64³ uint8 chunks, one machine (NVMe), wall seconds:

chunks	rename every chunk file	store get/set/delete	copy to a new array	prepend's metadata step
512	0.03	0.76	0.32	0.001
4 096	0.13	5.9	3.1	0.002
32 768 (8 GiB)	1.0	44	26	0.002

Script: https://github.com/rhoadesScholar/zarr-chunk-offset/tree/main/bench

How it works

• ArrayV3Metadata reads exactly one of shape and bounds and keeps inclusive_min beside the derived shape; to_dict writes bounds when some lower bound is non-zero or the stored document already used bounds, so a resize or attribute update keeps the form it found. Entries and lengths are checked against the signed 64-bit range.
• OffsetChunkGridMetadata wraps a RegularChunkGridMetadata; the metadata derives the grid index of the first chunk and the first element's position inside it from inclusive_min, the grid origin and chunk_shape.
• A regular dimension gains a shift, the position of the first element inside the first chunk, and a buffer_offset(chunk) that is 0 except for a partly filled first chunk. The indexers add it at the six places they convert array positions to chunk-local ones, so a partly filled first chunk is handled like a partly filled last chunk, write path included.
• Chunk keys add the grid index of the first chunk, so they can be negative. Decoding rejects -0.

Tests. A bounds store and an offset-grid store written by tensorstore (google/tensorstore#296) read back here with the same domain, grid origin, keys and values, and a write through this branch keeps their bounds form. Every selection type is read and written against NumPy on an array whose first chunk is partly filled, and on the registry README's offset-grid example (keys c/-1/0, c/-1/1, c/0/0, c/0/1). Also covered: prepending by part of a chunk, whole chunks, part of a shard, and on an array already on an offset grid; repeated prepends followed by resize; reopening; bounds and offset validation and refusals. mypy and ruff are clean.

Not covered here: rectilinear grids (a non-zero lower bound is refused for them), Zarr v2 arrays, and switching an array to bounds without prepending.

History: the first version of this PR used an offset chunk key encoding, the second a regular_bounded chunk grid; both were replaced on review in zarr-developers/zarr-extensions#79 and zarr-developers/zeps#72.

Refs zarr-developers/zarr-specs#9, zarr-developers/zarr-extensions#30.

[codecov]

Codecov Report

❌ Patch coverage is 96.05911% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.92%. Comparing base (27f034e) to head (5d88716).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/zarr/core/metadata/v3.py	96.82%	4 Missing ⚠️
src/zarr/core/chunk_grids.py	90.47%	2 Missing ⚠️
src/zarr/codecs/sharding.py	50.00%	1 Missing ⚠️
src/zarr/core/array.py	97.56%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4516      +/-   ##
==========================================
+ Coverage   94.90%   94.92%   +0.02%     
==========================================
  Files          96       96              
  Lines       14048    14226     +178     
==========================================
+ Hits        13332    13504     +172     
- Misses        716      722       +6     

Files with missing lines	Coverage Δ
src/zarr/core/chunk_key_encodings.py	98.66% <100.00%> (+0.07%)	⬆️
src/zarr/core/indexing.py	96.58% <100.00%> (+<0.01%)	⬆️
src/zarr/codecs/sharding.py	95.65% <50.00%> (-0.14%)	⬇️
src/zarr/core/array.py	98.56% <97.56%> (-0.04%)	⬇️
src/zarr/core/chunk_grids.py	96.18% <90.47%> (-0.18%)	⬇️
src/zarr/core/metadata/v3.py	97.07% <96.82%> (+0.14%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rhoadesScholar]

Ready for review; the registry entry is zarr-developers/zarr-extensions#80. cc @d-v-b

[d-v-b]

thanks for this PR! I really want to see this feature added to zarr! But I think we want to be sure that we support negative chunk indices consistently across the ecosystem, so I'll hold off on reviewing until we get alignment from the other implementers on the zarr-extensions PR.

[rhoadesScholar]

Converting to draft while the design is worked out in zarr-developers/zarr-extensions#79. This branch is a prototype and will be reworked to match.

[rhoadesScholar]

The metadata form is being settled in ZEP 12 (zarr-developers/zeps#72, spec text zarr-developers/zarr-specs#400). This PR stays on the regular_bounded grid until that lands, then moves its metadata read to the ZEP's fields; the rest of the change is unaffected.