[CALCITE-7924] RelMdExpressionLineage Cartesian expansion over UNION ALL causes excessive memory use and GC by Mrhs121 · Pull Request #5396 · apache/calcite

Mrhs121 · 2026-11-06T18:35:49Z

Jira Link

Changes Proposed

Problem

This issue was discovered while compiling SQL in Feldera and reproduced using Calcite alone. Expression-lineage inference over UNION ALL can construct the Cartesian product of the possible origins of each referenced column. With N branches and K referenced columns, this can produce N^K candidate expressions and cause excessive allocation and GC during planning. Both scalar expressions and window expressions in Project/Calc are affected.

Proposed solution

Bound candidate construction in the shared RelMdExpressionLineage.createAllPossibleExpressions helper, before it copies any candidate RexNode:

Check the product of replacement-set sizes for referenced input columns against a configurable work budget. Repeated occurrences of a reference share a substitution, and unused mapping keys do not add a dimension.
Use division to check the product without integer overflow. Count construction attempts before deduplication, rather than the size of the final result set.
If the budget would be exceeded, return null (unknown metadata), never a truncated set presented as complete lineage. Keep the existing expansion and results within the budget, including small window-expression cases; this is not a blanket rejection of window lineage.
Introduce calcite.metadata.expression.lineage.max.expansions, defaulting to 10000. -1 restores unlimited expansion, and 0 disables expansions involving input references. Constants require no expansion and remain unaffected. Values below -1 fall back to the default.
Document the bounded behavior in the metadata contract. The shared helper also covers all-predicates inference.

The budget is per expansion, not query-wide. It does not bound metadata traversal or the size of an individual expression. The default threshold and configuration mechanism are open for discussion.

Reproducer and regression coverage

Add a parameterized, storage-free program generator with adjustable UNION ALL branch count, referenced-column count, and scalar/window expression type. The tests exercise both Project and Calc, budget boundaries, unlimited/zero budgets, duplicate candidates, repeated and unused references, overflow prevention, metadata cache invalidation, and all-predicates inference.

For example, ExpressionLineageProgram.rel(16, 4, true) produces a plan whose computed-column lineage expands to 65,536 candidates without the limit. With the proposed default it returns unknown before enumerating those candidates. Smaller cases still return complete lineage, and querying a passthrough column remains supported after a budget rejection.

Observed impact

Pure Calcite benchmark: In the 32-branch, four-column window case, median lineage-call time decreased from 2,026.93 ms to 10.97 ms, with peak process RSS decreasing from 879.31 MiB to 169.55 MiB. The 8-branch case remains within budget and retains all 4,096 candidates. This benchmark runs Calcite alone, without Feldera.

Feldera SQL compilation test: For the full anonymized SQL from feldera/feldera#7870, changing only the Calcite budget implementation reduced SQL-to-Rust generation from 402.32 s to 4.83 s and peak process RSS from 8.35 GiB to 522.48 MiB. This test uses the Feldera SQL compiler; the comparison does not include native Rust compilation or pipeline execution.

The improvement comes from bounding optional metadata inference, not from computing the same large candidate set faster. The reproducer and detailed measurements are in the linked Jira issue.

…ALL causes excessive memory use and GC Bound Cartesian candidate construction before copying expressions, returning unknown metadata rather than a partial result when the configurable per-expansion budget is exceeded. Preserve within-budget lineage and add a parameterized SQL generator with Project, Calc, all-predicates, and budget-boundary regression coverage.

Mrhs121 · 2026-11-06T18:38:29Z

@mihaibudiu Could you please review this proposed per-expansion work budget? The PR includes a parameterized Calcite reproducer and returns unknown lineage before enumerating an over-budget Cartesian product, while preserving within-budget behavior. I would appreciate your feedback on the budget scope, default threshold, and configuration mechanism. The original Feldera report is feldera/feldera#7870. Thank you!

mihaibudiu

This looks fine in essence, but I think it can be improved.

mihaibudiu · 2026-11-06T23:10:26Z


+    if (maxExpansions >= 0) {
+      // Every referenced column adds a dimension to the Cartesian product.
+      // Count before copying RexNodes, even if equal candidates would later


estimate the count before...

the comment about division should be on the division operator.
it should say that instead of computing the product of counts and comparing with remaining you rather divide remaining to avoid overflow in the product

mihaibudiu · 2026-11-06T23:12:09Z

+
+import static com.google.common.base.Preconditions.checkArgument;
+
+/** Generates storage-free SQL programs whose expression lineage has


what is a storage-free SQL program, and why is it important for this test?
I would make this class a static class in RelMdExpressionLineageTest

mihaibudiu · 2026-11-06T23:13:05Z

+
+  /** Generates a SQL program with passthrough columns and a final computed
+   * column, either an arithmetic expression or a window expression. */
+  static String sql(int branches, int columns, boolean window) {


can you show in a comment an example program generated for a small value of branches?

mihaibudiu · 2026-11-06T23:16:21Z

+    return "select " + fields + ", " + expression + " as computed_value from (" + inputs + ") u";
+  }
+
+  /** Parses and validates the generated SQL using only Calcite. */


reading this javadoc I don't know what this function does.
Either refer to sql() above or describe what this function returns: the representation of a large SQL program which requires exponential cost to analyze lineage

mihaibudiu · 2026-11-06T23:18:47Z

+        RelMetadataQuery mq = rel.getCluster().getMetadataQuery();
+        RexNode result = RexInputRef.of(columns, rel.getRowType().getFieldList());
+        Set<RexNode> lineage = mq.getExpressionLineage(rel, result);
+        if (Math.pow(branches, columns) > 10_000) {


can you use the value of the property instead of 10K?

mihaibudiu · 2026-11-06T23:19:30Z

+          assertNotNull(lineage);
+          assertEquals((int) Math.pow(branches, columns), lineage.size());
+        }
+        // An expensive result must not poison other columns or later queries.


I don't know how results poison columns or queries.

mihaibudiu · 2026-11-06T23:20:24Z

+    }
+  }
+
+  @Test void testBudgetBoundaryAndMappingUnchanged() {


is this test and the next one needed? Doesn't the previous one cover them?

mihaibudiu · 2026-11-06T23:24:15Z

+    assertNull(expand(expression, mapping, 3));
+  }
+
+  @Test void testRepeatedAndUnusedReferences() {


how is this test related to the current change?

Only distinct, referenced columns contribute to the estimate. That was the reason for the test, but it doesn't need a separate case here. Removed.

mihaibudiu · 2026-11-06T23:24:56Z

+    assertNull(expand(expression, mapping, 1));
+  }
+
+  @Test void testZeroBudgetAndConstants() {


this is also a low-value test

mihaibudiu · 2026-11-06T23:26:16Z

+    assertThrows(IllegalArgumentException.class, () -> ExpressionLineageProgram.sql(2, 0, false));
+  }
+
+  @Test void testAllPredicatesUsesTheSameBudget() throws Exception {


I am also not sure how this test is relevant for this change. Too many tests are not good either.

AllPredicates uses the same expansion helper. Since this change adds no predicate-specific logic, I removed the separate test.

mihaibudiu · 2026-11-13T19:27:29Z

what is the status of this PR?

mihaibudiu

please squash the commits so we can merge this

Mrhs121 mentioned this pull request Nov 6, 2026

[SQL] SQL compiler: combinatorial expression-lineage expansion for ROW_NUMBER over UNION ALL causes excessive memory usage and GC stalls feldera/feldera#7870

Open

mihaibudiu reviewed Nov 6, 2026

View reviewed changes

mihaibudiu mentioned this pull request Nov 6, 2026

fix(sql): prevent combinatorial window lineage expansion feldera/feldera#7871

Closed

12 tasks

[CALCITE-7924] Simplify expression lineage budget tests

f1a0f89

mihaibudiu approved these changes Nov 14, 2026

View reviewed changes


		import static com.google.common.base.Preconditions.checkArgument;

		/** Generates storage-free SQL programs whose expression lineage has

Conversation

Mrhs121 commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Link

Changes Proposed

Problem

Proposed solution

Reproducer and regression coverage

Observed impact

Uh oh!

Mrhs121 commented Nov 6, 2026

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Nov 13, 2026

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mrhs121 commented Nov 6, 2026 •

edited

Loading