[Sunninsky]

Which issue does this PR close?

• Part of Release DataFusion 54.2.0 (minor/patch) Release #23805 (the 54.2.0 patch release tracking issue).
• Backport of fix: ensure deferred-filtered outer joins preserve streamed output order #24573 to branch-54.

Rationale for this change

LEFT, RIGHT and FULL sort-merge joins with a join filter can return rows out of order. These join types advertise that they preserve the order of one input (maintains_input_order is [true, false] for LEFT), so an operator above is allowed to rely on it. Deferred-filtered joins stage their output in a second BatchCoalescer (self.output), but in the Exhausted state poll_next returned the final filtered batch directly, so rows still buffered in output from an earlier flush came out after it.

The three streamed-order tests from #24573 fail on branch-54 as they do on main: for the LEFT JOIN shape with mixed group sizes the streamed keys come back as [5, 6, 0, 1, 2, 3, 4] instead of [0, 1, 2, 3, 4, 5, 6]. This is a wrong result, which makes it eligible for backport.

#24573 fixed it on main (labelled v56.0.0); it is on branch-56 but not on branch-54. It was requested for branch-54 on #26868 (#26868 (comment)), "to prevent regressions on upgrade to 54".

What changes are included in this PR?

This is a partial port, not a plain cherry-pick. The sort-merge join on this branch predates the generator refactor (#23761, #23976), and a cherry-pick of #24573 conflicts in nine places in materializing_stream.rs (and once in tests.rs) because most of that commit refactors code that is different here. Only the ordering fix and its tests are ported:

• datafusion/physical-plan/src/joins/sort_merge_join/materializing_stream.rs: in the Exhausted state the final filtered batch is pushed into self.output instead of being returned directly. The existing drain of self.output below it then emits it, in order, after the rows that were already staged there.
• datafusion/physical-plan/src/joins/sort_merge_join/tests.rs: left_join_with_filter_preserves_streamed_order, right_join_with_filter_preserves_streamed_order and left_join_with_partial_filter_preserves_streamed_order, with their helpers, taken from fix: ensure deferred-filtered outer joins preserve streamed output order #24573.

Not ported, because they are refactors or performance work on code that is different here and the fix does not need them: join_arrays returning Result, the direct-addressed source lookup in materialize_right_columns and its two tests, new_output_coalescer, and the benchmark change.

Are these changes tested?

Yes. On this branch (Rust 1.95.0) I ran:

• With the three tests added and no fix, cargo test -p datafusion-physical-plan --lib sort_merge_join fails exactly those three.
• With the fix, cargo test -p datafusion-physical-plan --lib: 1442 passed, including the three.
• cargo test -p datafusion-sqllogictest --test sqllogictests -- join: all 14 matching files pass.
• cargo fmt --all -- --check and cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings pass.

Are there any user-facing changes?

Outer sort-merge joins with a join filter return rows in the order they advertise. No public API change.

🤖 Generated with Claude Code