According to a recent LinkedIn post from Earthmover, the open-source Zarr-Python 3.2 release now includes an experimental feature called Rectilinear Chunk Grid, enabling variable-length chunking for large array data. The post suggests this approach can better align physical storage layouts with the logical structure of complex and irregular datasets.
Claim 55% Off TipRanks
- Unlock hedge fund-level data and powerful investing tools for smarter, sharper decisions
- Discover top-performing stock ideas and upgrade to a portfolio of market leaders with Smart Investor Picks
The post highlights three main technical use cases: daily appends to coarse time-series archives such as ERA5, virtualizing legacy archives like NetCDF and GRIB with non-uniform file extents, and handling data partitioned by external groupings such as chromosomes or forecast segments. For investors, this indicates Earthmover’s ecosystem is targeting high-value scientific and geospatial workloads where efficient data access and storage are critical.
According to the post, variable-length chunking is supported only in Zarr v3, with the feature still experimental in version 3.2 and expected to stabilize in 3.3, suggesting near-term ongoing development and refinement. Run-length encoding is described as part of the specification to keep metadata compact and compatible with the existing codec pipeline, which could enhance performance and cost efficiency in storage-heavy deployments.
The LinkedIn post also references collaboration with contributors from Development Seed and community developers, as well as support from The Navigation Fund and work done at the Zarr Summit in Rome in October 2025. This collaborative development model may reinforce Earthmover’s position within the open-source data infrastructure community, potentially improving adoption and integration prospects across climate, genomics, and other data-intensive sectors.