[danking]

DataFusion fragments that Vortex cannot execute natively can now run inside a Vortex expression tree through a typed, process-local df_expression ScalarFn. The wrapper preserves DF result types, nulls, Arrow field metadata, runtime join-map state, and evaluation errors; it exports referenced inputs and cannot be serialized. Callbacks containing higher-order functions retain the full original input layout to preserve parameter indices.

The public API accepts synchronous, non-volatile, static physical expressions. Live filters are handled through snapshots in the stacked integration. Unsupported Vortex result types, including decimal outputs whose Arrow coefficients need not fit the declared precision, remain in DF; decimal intermediates inside a representable result are allowed. An internal Boolean-only advisory mode retains rows on optional-filter evaluation failure; exact evaluation always returns errors.

Focused tests cover nullable single/composite hash lookups, a real DF-created dense ArrayMap, invalid column bindings, typed projections, scalar row counts, wrapping arithmetic, rejected volatile/live expressions, and a registered Boolean extension's inferred dtype and executed values. These eight shim cases pass within the stacked expression check: cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib df_expression --no-default-features (16 cases total, 0.06s). Additional stacked tests check UDF-visible fields/scalar shape, lambda parameter bindings, and DF-owned CASE projection rewrites. Formatting uses CI-pinned nightly-2026-09-10.

Supersedes the membership-only #11426. @danking please review.

AI assistance: prepared with Codex and reviewed in the designated Codex review chat.

Stacked converter integration: #11432.

[codspeed]

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 1921 untouched benchmarks
⏩ 50 skipped benchmarks¹

---

_{Comparing dk/df-expression-shim (6e97fda) with develop (d81bd4c)}

Footnotes

1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩