[yujun777]

What problem does this PR solve?

Problem Summary:

Incremental outer-join refresh currently checks the null-producing side's pre- and post-refresh snapshots to determine whether to retract or insert a NULL-extended row. It builds these two existence checks even when the complete join key is proven unique.

For a unique matching key, the pre/post row counts are each either zero or one. Summing the complete signed delta therefore gives net_count = post_count - pre_count, and the NULL repair factor is exactly -net_count. This change retains the original detail events and derives the repair directly from the delta, eliminating both repair ANTI JOINs and their snapshot subtrees.

The simplified plan below shows only the contribution from changes to B in A LEFT JOIN B. The retained A snapshot and other refresh contributions still require their normal snapshot reads.

 INNER JOIN (complete join key)
 |-- A_post snapshot
 `-- UNION ALL
     |-- delta_B detail events
-    |-- PROJECT (NULL payload, factor = -1)
-    |   `-- LEFT ANTI JOIN
-    |       |-- FILTER (has_positive)
-    |       |   `-- AGGREGATE (key, MAX positive/negative flags)
-    |       |       `-- delta_B
-    |       `-- B_pre snapshot
-    `-- PROJECT (NULL payload, factor = +1)
-        `-- LEFT ANTI JOIN
-            |-- FILTER (has_negative)
-            |   `-- AGGREGATE (key, MAX positive/negative flags)
-            |       `-- delta_B
-            `-- B_post snapshot
+    `-- PROJECT (NULL payload, factor = -net_count)
+        `-- FILTER (net_count <> 0)
+            `-- AGGREGATE (key, SUM(dml_factor) AS net_count)
+                `-- FILTER (join keys IS NOT NULL)
+                    `-- delta_B

The specialization uses the existing uniqueness trait of the original relation, accepts direct slot equality keys, and supports LEFT and RIGHT OUTER JOIN. Ordinary value updates keep their -old/+new detail events. Nonunique keys, unproven expression keys, residual predicates, and FULL OUTER JOIN keep their existing rewrite paths. NULL keys are excluded only from repair candidates because ordinary equality cannot match them. The internal aggregate is generated during delta rewriting, after MV normalization.

Release note

Reduce snapshot existence checks during incremental outer-join refresh when the null-producing side's complete join key is proven unique.

Check List (For Author)

• Test:
  • FE and ASAN BE builds, plus separate Checkstyle validation.
  • 194 focused FE tests: IvmJoinDeltaHandlerTest, IvmDeltaRewriterTest, IvmNormalizeUniqKeysMTMVJoinTest, and CreateMTMVCommandTest.
  • Regression comparison: test_ivm_unique_key_null_repair and test_ivm_outer_join_1 through test_ivm_outer_join_6 (7 suites passed).
  • New golden output generated with the runner and inspected: 8 change stages, 32 empty signed bag differences against full base-table queries.
  • Actual chained aggregate-MV analyzed plans: LEFT/RIGHT use SUM repair without repair ANTI JOINs; incomplete-key and residual cases retain their existence checks.
• Behavior changed: Yes. Eligible refresh plans derive NULL repair from the signed delta instead of snapshot ANTI JOINs; query results are preserved.
• Does this need documentation: No. This is an internal refresh-plan optimization.

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Reviewed all five changed files at head 1308f418e7898693e36e85cbd4eb173594ff4f8d and traced normalization, recursive delta contributions, uniqueness traits, snapshot binding, sequence propagation and the final MV sink. No new reportable defect was substantiated. No existing inline findings or additional user focus were supplied, and the live comment refresh found none.

Review converged in round 1: both complete-scope reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS; every risk was independently checked and dismissed with evidence, with no unresolved source-coverage gaps.

Critical checkpoint conclusions:

1. Goal and correctness: for non-NULL complete keys proven unique on the original null-side relation, pre/post counts are each 0 or 1, so NULL-row repair is exactly -SUM(dml_factor). The detail branch preserves ordinary old/new payload events. New structural FE tests and chained-MV regressions support the intended behavior by inspection.
2. Scope and reuse: the change is focused on one production handler and its tests. It reuses existing traits, copying, union and snapshot helpers and shares repair-payload projection.
3. Concurrency and locks: rewrite state is local to one visitor/statement. No threads, locks, shared mutable state or lock-order changes are added.
4. Lifecycle: lazy snapshots consult fixed stream/window mappings and fresh-copy plans; they do not advance stream offsets or allocate delta indices. Output/ExprId mapping and fallback source identity are preserved. No new resource or static-initialization lifecycle is introduced.
5. Configuration: no configuration or dynamic-setting changes.
6. Compatibility: no persisted metadata, storage format, public symbol or rolling-upgrade protocol changes.
7. Thrift/protobuf: no changed definitions; field-ID comparison is inapplicable.
8. Parallel paths: LEFT and RIGHT share the specialization. Nonunique/incomplete/expression keys retain existence checks; residual/volatile join predicates and FULL retain general repairs. Preserved-side, INNER/CROSS and nested contribution mechanisms remain intact.
9. Conditions: direct-slot uniqueness is checked on the original relation rather than a multi-version stream. Comments explain the guard and repair-only non-NULL filtering. Net-zero repairs are removed after summing the complete delta.
10. Tests: six added FE cases inspect both directions, composite keys and fallback gates, including union/ExprId/type/NULL/snapshot assertions. Eight regression stages cover first/last matches, repeated payload updates, NULL keys/payloads, duplicate retained keys, key migration, both-side changes and cancellation. The oracle disables SELECT/DML MV rewrites and compares signed bags against original base tables.
11. Expected output: inspected all eight value stages and 32 empty difference blocks; totals, NULL SUM/count distinctions and migrated matches agree with the test SQL. Generated-output provenance was not independently rerun.
12. Observability: existing EXPLAIN/dry-run plans expose the specialization; existing exception and task audit/log boundaries remain sufficient for this plan-only change.
13. Transactions and persistence: no EditLog, offset-commit, failover or refresh-state mutation changes. Existing INSERT refresh and committed snapshot/epoch recording paths are retained.
14. Data writes and atomicity: the existing MV sink is reused. BIGINT SUM and the bounded TINYINT repair factor align with union outputs; NULL match markers, padding sequence and negative-factor delete signs preserve the existing write contract. No new crash/cleanup path is introduced.
15. FE/BE propagation: no new wire fields, operator types or function signatures; existing SUM/filter/project/cast/join/union operators are used.
16. Performance: eligible plans remove two snapshot subtrees and repair ANTI JOINs, using one signed grouping aggregate while retaining detail events. Unneeded eager snapshot construction is removed. This is a structural conclusion; performance was not measured.
17. Additional checks and core invariants: uniqueness propagation through scans/projects/joins and chained MVs, duplicate-producing scan guards, analyzer stage order, nullable keys, complete recursive deltas and sink schema were checked. Visibility/stream boundaries and storage/delete-bitmap/native-memory behavior are unchanged; existing invariant failures still propagate.

Validation limit: static source and plan-algebra review only. No builds, tests, benchmarks, source modifications or submodule initialization were performed by this review. Author-reported execution results were not treated as reviewer-run validation. No remaining source question or uncovered changed scope was identified.

Review converged after 1 round(s).

[yujun777]

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70561
commit: 1308f418e7898693e36e85cbd4eb173594ff4f8d
base: ee73acc6131e6c6c752eb59e252404590be6ad77
reviewed_at: 2026-11-13T16:44+08:00
reviewer: yujun777
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

• fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java:557 — Specialization uses original-relation uniqueness over direct equality slots; nonunique, expression, residual and FULL OUTER paths retain their existing repair logic.
• fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmJoinDeltaHandler.java:590 — Internal SUM is created after MV normalization; business aggregate classification, normalized signature, original detail events and repair sequence contracts remain unchanged.
• Existing validation logs inspected: FE/ASAN BE builds, Checkstyle, 194 focused FE tests and 7 regression suites passed; real chained LEFT/RIGHT plans use SUM without repair ANTI joins. Formal review itself was read-only.

_{Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.}

[yujun777]

run external