Skip to content

Support fill_null and zip for more dtypes - #11415

Draft
robert3005 wants to merge 1 commit into
developfrom
rk/fill-null-zip-dtype-coverage
Draft

robert3005 wants to merge 1 commit into
developfrom
rk/fill-null-zip-dtype-coverage

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Summary

fill_null and zip did not have uniform support across dtypes.

  • fill_null accepted all dtypes at bind time, but failed at execution with No FillNullKernel for canonical array for all canonical arrays except Bool, Primitive, Decimal and VarBinView when the input had valid rows and null rows.
  • zip had no native kernel for Decimal, Extension, FixedSizeList, Map and Null. These used the builder fallback, which does one slice and one append for each mask run. zip panicked for Union and Variant, because these dtypes have no builder.

Changes

fill_null:

  • Extension: fills the storage array and wraps the result in the extension dtype. This makes fill_null work for temporal columns.
  • Struct: fills each field in the null rows.
  • List, FixedSizeList, Map: a lazy zip of the input with a constant fill over the input validity.
  • Null, Union, Variant: a clear error when the input has valid rows and null rows.
  • FillNull::supports_dtype now accepts all dtypes except Null and dtypes that contain a Union or a Variant, so the CASE WHEN rewrite does not make a fill_null that fails later.

zip:

  • Decimal: a new zip_fixed_width in arrays/fixed_width, shared with Primitive. The blend code moved out of the Primitive kernel without a change in behavior. Decimals with different storage widths use the builder.
  • Extension: zips the storage arrays and wraps the result.
  • FixedSizeList: expands the row mask by the list size and zips the elements.
  • Map: uses the ListView kernel on the entries.
  • Null: returns the first branch.
  • Union, Variant: a mixed mask returns an error and does not panic.

Not in this PR:

  • The new fill_null kernels for Struct and Extension are called only from the canonical dispatch. They are not registered as execute-parent kernels.
  • Union and Variant kernels.

Tests: new tests next to each kernel, and the fill_null test that recorded the old failure is now an rstest over all supported dtypes with a nullable and a non-nullable fill value.

Checks run:

  • cargo clippy -p vortex-array --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-09-10 fmt -p vortex-array --check
  • cargo nextest run -p vortex-array --all-features (5920 passed, 1 skipped)

Checks not run: doctests, benchmarks, and the checks of other crates.

🤖 Generated with Claude Code

fill_null failed at execution for all canonical arrays except Bool,
Primitive, Decimal and VarBinView when the input had some nulls. zip had
no native kernel for Decimal, Extension, FixedSizeList, Map and Null, and
it panicked for Union and Variant.

fill_null now fills Extension through its storage, Struct through its
fields, and List, FixedSizeList and Map through a zip with a constant
fill. zip has kernels for Decimal (shared with Primitive through the
fixed-width code), Extension, FixedSizeList, Map and Null. Union and
Variant return an error instead of a panic.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@connortsui20 connortsui20 mentioned this pull request Nov 12, 2026
12 of 49 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant