[ErikBPF]

Which issue does this PR close?

Refs #7573. This draft does not close the issue.

Rationale for this change

Delta's InterleaveBits is the Z-order key OPTIMIZE ... ZORDER BY sorts on. It falls back to Spark today, and the issue records the observed fallback on Parquet input. This branch takes the expression through the Comet pipeline for the one- and two-input INT shapes that the issue scopes.

The core hook is a CometExpressionContrib ServiceLoader interface next to CometScanContrib: the Delta contrib registers a serde for InterleaveBits, so core names no Delta class. This is a draft until maintainers agree on that seam in #7573; the other option discussed there is matching the Delta class name in core.

Hilbert ordering, range-partition expressions and broader clustering operations remain out of scope. OPTIMIZE ... ZORDER BY still falls back because its inputs are Delta's range-partition expressions, which Comet does not convert.

What changes are included in this PR?

• Core CometExpressionContrib hook: ServiceLoader discovery, core-wins and duplicate-claim handling, a serdeFor lookup used by QueryPlanSerde and the ScalaUDF guard check, and focused tests.
• Delta contrib serde for InterleaveBits, with fallback for other arities and non-INT children.
• Native delta_interleave_bits kernel: one or two Int32 inputs, 4 * n big-endian bytes, null-as-zero, never null.
• Delta contrib suite covering both Delta fast/default paths, nulls, signed extrema, seeded random pairs, column/literal mixes, empty tables, constant-folding control, fallback reasons, config disable, and OPTIMIZE still matching Spark.
• User-guide and contrib README documentation; the core suite runs in the Linux and macOS PR tiers.

How are these changes tested?

• cargo test -p datafusion-comet-spark-expr --lib delta_funcs: 9 passed. The goldens include 130 pairs generated from both of Delta's interleave implementations.
• CometExpressionContribSuite: 7 passed (Spark 4.1.3).
• CometDeltaInterleaveBitsSuite: 9 passed (Spark 4.1.3 / Delta 4.3.1), including both Delta interleave paths and the OPTIMIZE regression.
• Spark 4.2.0 main compilation passes. The Delta suite runs 8 of 9 tests on the spark-4.2 profile; the OPTIMIZE case fails with NoSuchMethodError in QueryExecution because Delta 4.3.1 targets Spark 4.1. The Spark 4.2 / Delta 4.4 pairing is separate work in feat(delta): pair Spark 4.2 with Delta 4.4.1 #7593.

These are local/Apollo runs, not a published CI run. The PR stays a draft until the expression-admission design in #7573 is settled.

[dwsmith1983]

@ErikBPF thanks for opening this. Could you add me as a collaborator on ErikBPF/datafusion-comet so I can push to feat/7573-interleave-handoff? "Allow edits by maintainers" doesn't cover me, since I don't have write access here. I'll bring over only the Morton interleave for InterleaveBits; Hilbert will get its own issue and PR.

[dwsmith1983]

Here's how I'd split the work so each piece stays reviewable:

1. A core PR adding a CometExpressionContrib hook, so the Delta contrib can register a serde for InterleaveBits. It waits on the maintainers' answer in Support Delta InterleaveBits for one or two INT inputs #7573.
2. A Rust-only PR with the delta_interleave_bits kernel, tested against vectors generated from both of Delta's code paths.
3. This PR on top of both, with the Delta serde and an expression suite that takes over the checks from your probe.

I'll open the first two from my fork, link them here, and add you as a collaborator on it so you can push to those branches. Once you've added me on yours, I'll move the serde and suite into this PR and hand it back to you.

[dwsmith1983]

The native kernel is up as #7600 (draft). The core hook PR follows once the design question in #7573 is settled.

[ErikBPF]

@dwsmith1983 Thanks. I sent a collaborator invitation for ErikBPF/datafusion-comet with push access; accept it and you can push directly to feat/7573-interleave-handoff. No separate branch is needed.

The proposed split works for us:

1. Core CometExpressionContrib hook once Support Delta InterleaveBits for one or two INT inputs #7573 gets a maintainer answer.
2. Rust-only delta_interleave_bits kernel checked against both Delta reference paths.
3. This PR on top with the Delta serde and the expression suite.

I will keep the manual probe in this PR as the reference baseline until the suite supersedes it. The only open dependency is the expression-admission design in #7573; we will not add core Delta class matching on our side unless that is the maintainers' preference.

[dwsmith1983]

@ErikBPF pushed the Delta serde and the expression suite (dcbbdec). Until the two pieces underneath merge, this diff also carries them:

• feat: add a native kernel for Delta's InterleaveBits #7600: the native kernel.
• The core CometExpressionContrib hook, which lets a contrib register an expression serde. I'll open it as its own PR once the design question in Support Delta InterleaveBits for one or two INT inputs #7573 is answered.

After those merge and this branch merges main, what's left here is the serde, the suite and a "Z-Order Keys" section in the Delta user guide. I removed the probe under dev/repros/delta because each of its checks now has a test in CometDeltaInterleaveBitsSuite, under both Delta paths; bring it back if you'd rather keep it as a baseline.

It's yours from here for cleanup and extending. Open items I left:

• The OPTIMIZE test checks zOrderBy by substring; an exact match on ["a","b"] would be tighter.
• delta.md says the OPTIMIZE path "does not benefit yet"; the docs style guide keeps plans out of the user guide.
• The empty-table test writes no batches, so the kernel's zero-length-batch path isn't exercised.
• No benchmark against Spark yet. It would also tell us whether a faster bit-spread interleave is worth adding to the kernel.