ngff_zarr._zarrista_utils

Compatibility layer between ngff-zarr and zarrista (Rust/zarrs-backed Zarr).

zarrista is v3-focused and its API surface differs from zarr-python in ways that matter for OME-Zarr writing. The empirically verified constraints this module works around (see the Phase 01 capability spike):

• No group-create API for either zarr format: group metadata (zarr.json for format 3, .zgroup/.zattrs for format 2) is hand-written JSON.

• ArrayBuilder emits only zarr format 3 metadata: format 2 arrays are created via Array.from_metadata with a .zarray-style dict, which writes real v2 stores that zarr-python reads back correctly.

• Consolidated .zmetadata cannot be written by zarrista for format 2: it is hand-written by aggregating the per-node JSON documents.

• Every write requires C-contiguous input, and arrays expose shape as a list and dtype as zarrista’s own DataType: dask interop goes through a thin adapter (:class:_ZarristaArrayAdapter).

• Supported stores are FilesystemStore/MemoryStore/read-only ZipStore only: :func:normalize_store restricts targets to local directory paths.

Sharding vocabulary differs from zarr-python: the builder chunk grid is the shard (outer chunk) grid and subchunk_shape is the read chunk, so ngff-zarr’s chunks_per_shard multiplies chunks into the builder grid shape while chunks itself becomes the subchunk shape.

zarrista is imported lazily inside each function to keep module import cheap.

Module Contents

Classes

_ZarristaArrayAdapter	Minimal numpy-protocol surface over a zarrista Array for dask.
LocalZarrArray	Handle for an array node within a local directory store.
LocalZarrGroup	Read-only handle for a group node within a local directory store.

Functions

_native_contiguous	Return value as a C-contiguous, native-byte-order ndarray.
_normalize_selection	selection as one unit-step slice per axis, the axes an integer index drops, and the residual striding to apply to the fetched block.
_canonical_compressor	Normalize any accepted compressor form to a (name, params) pair.
_evolve_blosc_params	Resolve constructor-defaulted blosc typesize/shuffle from the dtype.
_blosc_shuffle_int
_compressor_to_v2_config	Map a compressor to the numcodecs config dict stored in .zarray.
_compressor_to_zarrista_codec	Map a compressor to a zarrista bytes-to-bytes codec (zarr format 3).
_is_local_path	Whether store is a local directory path zarrista can target directly.
_zarrista_filesystem_store_path	Root path of a zarrista FilesystemStore, or None otherwise.
resolve_store_path	Resolve store to the local directory path backing it, or None.
normalize_store	Normalize store to a local directory path zarrista can target.
_retry_on_windows_sharing_violation	Run operation, retrying the transient Windows sharing violation.
_read_doc_text	Read a zarr metadata document, tolerating a concurrent replace.
read_group_attributes	Attributes of the group at group_path within store.
has_consolidated_metadata	Whether the store at store carries consolidated metadata.
_existing_group_attributes	Attributes of the group already at path, or {} when absent.
_write_group_doc	Serialize doc to path atomically (temp file + rename).
create_zarrista_group	Create group metadata at path and return the opened zarrista group.
create_zarrista_subgroup	Create the group at group_path within the store at store.
_normalize_array_metadata_doc	Rewrite a zarrista-written array metadata document for engine parity.
create_zarrista_array	Create a zarr array under the store at path and return it.
write_dask_array	Write dask array darr into an existing zarrista array.
open_zarrista_array	Open the existing array at component within the store at path.
is_zarrista_array	Whether obj is a raw zarrista.Array handle.
zarrista_array_to_dask	Wrap zarrista Array arr as a lazy dask array.
open_zarrista_lazy	Open the array at component within path as a lazy dask array.
open_ozx_store	Store handle for reading the zip archive (.ozx/.zip) at path.
open_zip_lazy	Open the array at component within zip-archive store lazily.
_is_bytes_mapping	Whether store is a key-to-bytes mapping for the pure-Python reader.
open_lazy_array	Open the array at component within store as a lazy dask array.
_load_local_doc
_local_node_document	(node_type, metadata document) for the node at dirpath.
_local_node_attrs	User attributes for the node at dirpath (.zattrs or inline).
open_local_node	Open the node at node_path within the local directory store store.
_remote_handle	A handle on store’s remote backend, or None when it has none.
open_array	Open the array at path within store.
consolidate_metadata	Write consolidated metadata for the store at path.

Data

__all__
_BLOSC_SHUFFLE_TO_INT
_BLOSC_SHUFFLE_TO_STR
OME_ROOT_KEYS
_FILESYSTEM_STORE_REPR

API

ngff_zarr._zarrista_utils.__all__

[‘LocalZarrArray’, ‘LocalZarrGroup’, ‘OME_ROOT_KEYS’, ‘consolidate_metadata’, ‘create_zarrista_array…

ngff_zarr._zarrista_utils._BLOSC_SHUFFLE_TO_INT

None

ngff_zarr._zarrista_utils._BLOSC_SHUFFLE_TO_STR

None

ngff_zarr._zarrista_utils.OME_ROOT_KEYS

‘frozenset(…)’

ngff_zarr._zarrista_utils._native_contiguous(value) → numpy.ndarray

Return value as a C-contiguous, native-byte-order ndarray.

zarrista ingests write input through DLPack, which rejects both non-contiguous buffers and non-native byte order (e.g. big-endian >u2 input); the target array’s stored byte order comes from its metadata, so byteswapping the in-memory values is safe.

ngff_zarr._zarrista_utils._normalize_selection(

shape,

selection,

) → tuple[tuple[slice, ...], list[int], tuple[slice, ...] | None]

selection as one unit-step slice per axis, the axes an integer index drops, and the residual striding to apply to the fetched block.

An ellipsis and missing trailing axes expand to full slices. An integer becomes the length-1 slice zarrista reads it as, and its axis is reported so numpy semantics, where the axis is dropped, can be restored. A stepped (or negative-step) slice becomes the ascending unit-step span covering the selected indices, and the residual (None when every step is 1) restores the requested stride and direction on the fetched block: zarrista serves unit steps only, and the covering span is what its chunks decode anyway.

class ngff_zarr._zarrista_utils._ZarristaArrayAdapter(

arr,

store_path: pathlib.Path | None = None,

node_path: str | None = None,

)

Minimal numpy-protocol surface over a zarrista Array for dask.

zarrista exposes shape as a list, dtype as its own DataType, has no .chunks attribute, and rejects non-C-contiguous or non-native-byte-order write input. dask needs a tuple shape, a numpy dtype, ndarray __getitem__ results, and a tolerant __setitem__ (see :func:_native_contiguous).

store_path and node_path record where an array of a local store lives, so the writer can tell that an image still reads from the store it is about to replace. They are None for arrays without a local directory.

Initialization

_fetch(selection) → numpy.ndarray

Materialize selection; the async remote adapter overrides this.

__getitem__(selection)

__setitem__(selection, value)

ngff_zarr._zarrista_utils._canonical_compressor(compressor) → tuple[str, dict] | None

Normalize any accepted compressor form to a (name, params) pair.

Accepts a numcodecs codec object, a zarr v3 codec object, a bare codec name string, a numcodecs-style config dict ({"id": ...}), or a zarr v3 codec dict ({"name": ..., "configuration": ...}). Returns None for None.

ngff_zarr._zarrista_utils._evolve_blosc_params(compressor, params: dict, itemsize: int) → dict

Resolve constructor-defaulted blosc typesize/shuffle from the dtype.

zarr-python’s BloscCodec serializes unset attrs as typesize=1 / shuffle="bitshuffle", records which were defaulted in _tunable_attrs, and evolves those from the dtype at write time. Mirror that so the stored metadata matches what zarr-python would write; explicitly chosen values are kept.

ngff_zarr._zarrista_utils._blosc_shuffle_int(value, itemsize: int = 4) → int

ngff_zarr._zarrista_utils._compressor_to_v2_config(compressor, dtype: numpy.dtype) → dict | None

Map a compressor to the numcodecs config dict stored in .zarray.

ngff_zarr._zarrista_utils._compressor_to_zarrista_codec(compressor, dtype: numpy.dtype)

Map a compressor to a zarrista bytes-to-bytes codec (zarr format 3).

ngff_zarr._zarrista_utils._is_local_path(store) → bool

Whether store is a local directory path zarrista can target directly.

True for str/pathlib.Path/os.PathLike targets that are neither remote URL strings nor .zip/.ozx archive paths (zarrista can only read zip archives); MutableMappings and store objects are excluded.

ngff_zarr._zarrista_utils._FILESYSTEM_STORE_REPR

‘compile(…)’

ngff_zarr._zarrista_utils._zarrista_filesystem_store_path(store) → pathlib.Path | None

Root path of a zarrista FilesystemStore, or None otherwise.

zarrista’s stores are pyo3 classes with no Python-visible attributes; the only channel for the root path is the round-trippable repr (FilesystemStore(path='...'), quoted per Python’s string repr), parsed with ast.literal_eval. Tests pin the format so upstream drift surfaces as a failure rather than a silently unresolvable store.

ngff_zarr._zarrista_utils.resolve_store_path(store) → pathlib.Path | None

Resolve store to the local directory path backing it, or None.

Accepts plain str/pathlib.Path/os.PathLike targets (e.g. the default config.cache_store) and zarrista FilesystemStore handles. Returns None for anything with no local directory (in-memory mappings, remote URL strings, store objects, …).

ngff_zarr._zarrista_utils.normalize_store(store) → pathlib.Path

Normalize store to a local directory path zarrista can target.

Accepts str/pathlib.Path/os.PathLike directory paths and zarrista FilesystemStore handles. Raises TypeError for anything else (in-memory mappings, remote URL strings, zarr store objects, …) and ValueError for zip targets, which zarrista can only read.

ngff_zarr._zarrista_utils._retry_on_windows_sharing_violation(operation, cleanup=None)

Run operation, retrying the transient Windows sharing violation.

os.replace swaps group documents in place. Windows denies access to the destination for the instant the swap takes, so a concurrent reader or writer of that path sees PermissionError where POSIX sees the old or new file. Retry briefly, then give up and let the error surface.

ngff_zarr._zarrista_utils._read_doc_text(doc: pathlib.Path) → str

Read a zarr metadata document, tolerating a concurrent replace.

ngff_zarr._zarrista_utils.read_group_attributes(

store,

group_path=None,

*,

zarr_format: int,

) → dict | None

Attributes of the group at group_path within store.

Returns None when no group node exists there and {} for a group with no attributes, so callers can distinguish “absent” from “empty” (the compat-layer equivalent of a zarr-python membership test plus attrs.asdict()). Raises ValueError when the path holds an array node, mirroring zarr-python’s refusal to open a group over an array.

ngff_zarr._zarrista_utils.has_consolidated_metadata(store, zarr_format: int) → bool

Whether the store at store carries consolidated metadata.

Checks the root zarr.json for an inline consolidated_metadata block (zarr format 3) or for a .zmetadata sidecar (format 2), so in-place metadata rewrites can refresh consolidation exactly when zarr-python would.

ngff_zarr._zarrista_utils._existing_group_attributes(path: pathlib.Path, zarr_format: int) → dict

Attributes of the group already at path, or {} when absent.

ngff_zarr._zarrista_utils._write_group_doc(path: pathlib.Path, doc: dict) → None

Serialize doc to path atomically (temp file + rename).

Group documents on shared ancestors (e.g. an HCS row group) can be read and rewritten by concurrent well/field writers; the atomic replace keeps readers from ever observing a truncated document.

On Windows os.replace raises PermissionError when another writer or reader momentarily holds the destination open, so the replace is retried briefly. POSIX renames onto an open file without complaint.

ngff_zarr._zarrista_utils.create_zarrista_group(

path,

attributes: dict | None,

zarr_format: int,

*,

overwrite: bool = False,

)

Create group metadata at path and return the opened zarrista group.

zarrista has no group-create API, so the metadata documents are written directly: .zgroup + .zattrs for zarr format 2, zarr.json for format 3. With overwrite any existing content below path is removed first, matching zarr.open_group(mode="w"); otherwise an existing group’s attributes are preserved and updated with attributes, matching mode="a" followed by per-key attribute assignment. The returned zarrista.Group reads the documents back, hiding the hand-written JSON from callers.

ngff_zarr._zarrista_utils.create_zarrista_subgroup(

store,

group_path,

attributes: dict | None,

zarr_format,

)

Create the group at group_path within the store at store.

Missing ancestor groups along group_path are created with empty attributes, matching zarr-python’s implicit-parent creation; an existing leaf group keeps its other attributes (see :func:create_zarrista_group). Returns the zarrista group for the leaf.

ngff_zarr._zarrista_utils._normalize_array_metadata_doc(

doc_path: pathlib.Path,

zarr_format: int,

itemsize: int,

)

Rewrite a zarrista-written array metadata document for engine parity.

zarrs’ serialization differs from zarr-python’s in ways that are spec-legal but break byte-level store equivalence: it stamps node_type into v2 .zarray docs and a _zarrs provenance attribute into v3 docs, omits the always-materialized storage_transformers/attributes fields, and writes an explicit endian for the bytes codec even for single-byte dtypes (where zarr-python omits the configuration). Only the document is rewritten; the already-open zarrista array keeps its in-memory metadata.

ngff_zarr._zarrista_utils.create_zarrista_array(

path,

name: str,

shape: tuple[int, ...],

dtype,

chunks: int | tuple[int, ...],

zarr_format: int,

compressor=None,

chunks_per_shard: int | tuple[int, ...] | None = None,

dimension_names: tuple[str, ...] | None = None,

dimension_separator: str = '/',

compressors=None,

shards: tuple[int, ...] | None = None,

fill_value=None,

attributes: dict | None = None,

chunk_key_encoding: dict | None = None,

)

Create a zarr array under the store at path and return it.

name is the array’s path within the store (e.g. "0"). compressor accepts the same forms as ngff-zarr’s writer: a numcodecs codec, a zarr v3 codec, or their config dicts; None means no compression. compressors (format 3 only) is an ordered sequence of bytes-to-bytes codecs written after the bytes codec; it overrides compressor, and an empty sequence means no compression. Sharding (format 3 only) is requested either via chunks_per_shard — an int or per-dimension tuple of chunks per shard, matching to_ngff_zarr — or via shards, the explicit outer shard shape whose subchunks are chunks. fill_value None means the dtype’s zero, matching zarr-python. attributes are the array’s user attributes. chunk_key_encoding (format 3 only) is the zarr v3 metadata form (e.g. {"name": "v2", "configuration": {"separator": "/"}}), used by in-place v2->v3 upgrades to keep existing chunk keys resolvable. The metadata is stored immediately; the returned zarrista.Array is ready for chunk writes.

ngff_zarr._zarrista_utils.write_dask_array(darr, zarrista_array, region=None) → None

Write dask array darr into an existing zarrista array.

Writes are chunk-aligned: darr is rechunked to the target’s write granularity (the shard shape for sharded arrays) so concurrent block writes never touch the same stored chunk, then streamed with dask.array.store. A region (tuple of slices) selects the target subset and should be aligned to the same granularity. Empty arrays (a zero-length dimension) are a no-op.

ngff_zarr._zarrista_utils.open_zarrista_array(path, component: str | None = None)

Open the existing array at component within the store at path.

Returns the raw zarrista.Array, ready for region __setitem__ writes into an already-created array (either zarr format). Write input must be C-contiguous; :class:_ZarristaArrayAdapter or

Func:

write_dask_array handle that for adapter-mediated writes.

ngff_zarr._zarrista_utils.is_zarrista_array(obj) → bool

Whether obj is a raw zarrista.Array handle.

ngff_zarr._zarrista_utils.zarrista_array_to_dask(

arr,

store_path: pathlib.Path | None = None,

node_path: str | None = None,

) → dask.array.Array

Wrap zarrista Array arr as a lazy dask array.

Chunks follow the stored chunk grid, using the subchunk (inner chunk) shape for sharded arrays so reads stay at the efficient granularity. store_path and node_path name the array’s place in a local store (see :class:_ZarristaArrayAdapter).

ngff_zarr._zarrista_utils.open_zarrista_lazy(path, component: str | None = None) → dask.array.Array

Open the array at component within path as a lazy dask array.

ngff_zarr._zarrista_utils.open_ozx_store(path)

Store handle for reading the zip archive (.ozx/.zip) at path.

A :class:~.rfc9_zip.ZipReadStore: metadata and group navigation go through the pure-Python mapping reader while pixel data reads through zarrista’s native zip store (see :func:open_zip_lazy).

ngff_zarr._zarrista_utils.open_zip_lazy(

store: ngff_zarr.rfc9_zip.ZipReadStore,

component: str | None = None,

)

Open the array at component within zip-archive store lazily.

Routes through zarrista’s read-only zip store rather than the mapping reader because .ozx archives hold sharded zarr v3 arrays by default, which only zarrista can decode. The node path combines the store’s prefix (HCS well/field views) with component.

ngff_zarr._zarrista_utils._is_bytes_mapping(store) → bool

Whether store is a key-to-bytes mapping for the pure-Python reader.

str/Path are excluded even though paths are iterable: only real mapping objects (plain dicts, zip-archive views, tifffile’s aszarr store) go through :mod:._v2_store_reader.

ngff_zarr._zarrista_utils.open_lazy_array(store, component: str | None = None) → dask.array.Array

Open the array at component within store as a lazy dask array.

The single read-dispatch point: zip-archive views read pixels through zarrista’s zip store, remote store handles through zarrista’s async API over obstore, other bytes mappings go through the pure-Python store reader (either zarr format), and local paths through zarrista. Any other store object raises TypeError.

ngff_zarr._zarrista_utils._load_local_doc(doc_path: pathlib.Path) → dict

ngff_zarr._zarrista_utils._local_node_document(

dirpath: pathlib.Path,

zarr_format: int,

) → tuple[str, dict] | None

(node_type, metadata document) for the node at dirpath.

Returns None when no node of zarr_format exists there. For format 2 an array wins when both .zarray and .zgroup are present, matching zarr-python.

ngff_zarr._zarrista_utils._local_node_attrs(

dirpath: pathlib.Path,

doc: dict,

zarr_format: int,

) → ngff_zarr._v2_store_reader.AttrsDict

User attributes for the node at dirpath (.zattrs or inline).

class ngff_zarr._zarrista_utils.LocalZarrArray(store_root: pathlib.Path, path: str, zarr_format: int, doc: dict)

Handle for an array node within a local directory store.

Exposes the node’s shape, dtype, chunks and attrs, numpy style [] reads and writes through zarrista, and :meth:to_dask for lazy pixel access. A write lands on the stored chunks it covers, so a producer filling a store created with metadata_only=True needs no lock while its regions follow the chunk grid.

Initialization

_array() → ngff_zarr._zarrista_utils._ZarristaArrayAdapter

property dtype: numpy.dtype

property chunks: tuple[int, ...]

The stored chunk grid, the inner chunk shape of a sharded array.

__getitem__(selection) → numpy.ndarray

__setitem__(selection, value) → None

to_dask() → dask.array.Array

class ngff_zarr._zarrista_utils.LocalZarrGroup(store_root: pathlib.Path, path: str, zarr_format: int, doc: dict)

Read-only handle for a group node within a local directory store.

Provides the slice of the zarr-python Group surface the read path uses – attrs (a dict with asdict()), __contains__ / __getitem__ path navigation, and keys() – backed by direct metadata-document reads (.zgroup/.zattrs for zarr format 2, zarr.json for format 3). Child arrays open their pixel data lazily through zarrista.

Initialization

_child_path(name) → str

__getitem__(name)

__contains__(name) → bool

keys() → list[str]

Names of all immediate child nodes (arrays and groups), sorted.

ngff_zarr._zarrista_utils.open_local_node(store, node_path=None, *, zarr_format: int)

Open the node at node_path within the local directory store store.

The compat-layer read equivalent of zarr.open: metadata documents are read directly from the filesystem, no zarr-python involved. Returns a

Class:

LocalZarrGroup or :class:LocalZarrArray, or None when no node of zarr_format exists at that path.

ngff_zarr._zarrista_utils._remote_handle(store, storage_options: dict | None)

A handle on store’s remote backend, or None when it has none.

The reader’s own rule (from_ome_zarr): a URL string, or a handle already built around one. A zarr-python FsspecStore wrapping a URL is neither – its str() is a repr rather than the URL obstore would be given – and the reader does not route one here either, so it keeps the local path it has today.

ngff_zarr._zarrista_utils.open_array(store, path: str | None = None, storage_options: dict | None = None)

Open the array at path within store.

The handle reads and writes regions through zarrista and needs no other Zarr library: to_ome_zarr(..., metadata_only=True) creates the arrays of a store, and this opens one of them for a producer to fill. The node is looked up as zarr format 3 first, then format 2.

A URL (http(s), S3, GCS, Azure) is opened through the same async engine from_ome_zarr reads it with, and comes back read-only: writing a region needs a local directory store. storage_options is passed to that engine, fsspec-style names included.

ngff_zarr._zarrista_utils.consolidate_metadata(path, zarr_format: int) → None

Write consolidated metadata for the store at path.

zarrista cannot write zarr format 2 consolidated metadata, so .zmetadata is hand-written by aggregating the per-node JSON documents (what OME-Zarr 0.4 stores carry). Format 3 consolidation is likewise hand-inlined into the root zarr.json: routing it through zarrista’s consolidation API would re-serialize the child documents with zarrs conventions (e.g. the bare-string shorthand for configuration-less codecs) that zarr-python’s reader rejects, so the raw documents are embedded verbatim instead, matching zarr.consolidate_metadata.