[iwanttobepowerful]

CALCITE-7882

1. Symptom

With expand=false, a JOIN ... ON condition containing multiple sub-queries can fail with "Unable to implement":

SELECT e.ename
FROM emp e JOIN dept d
  ON e.deptno = d.deptno
  AND e.sal < (SELECT max(s.sal) FROM emp s)
  AND EXISTS (SELECT 1 FROM emp s WHERE s.mgr = e.empno)

The scalar sub-query is uncorrelated, while the EXISTS references the current join scope. Both remain in the ON condition as RexSubQuery, and the Join declares the correlation variable used by the EXISTS, for example variablesSet=[$cor0].

2. Root cause: rewriting one sub-query loses the scope of the remaining ones

SubQueryRemoveRule.matchJoin rewrites one sub-query per invocation and relies on repeated rule application. Sub-queries equal to the selected one are replaced together.

In the original implementation:

1. The first selected sub-query is the uncorrelated scalar query. It uses no correlation variable, so the rule takes the right-input branch.
2. That branch rebuilds the Join with the two-argument RelBuilder.join overload, which supplies an empty variablesSet.
3. The rebuilt Join no longer declares $cor0, although its ON condition still contains the correlated EXISTS referencing $cor0.EMPNO.
4. On the next invocation, the rule intersects the variables used by the EXISTS with the rebuilt Join's now-empty variablesSet. The result is empty, so the sub-query is wrongly treated as uncorrelated.
5. The rewrite creates a plain Join instead of a Correlate. The correlation reference survives into the executable plan without a binding, and implementation fails.

The underlying problem is the loss of correlation-variable scope across incremental sub-query rewrites, not an incorrectly computed scalar expression.

3. Fix: preserve the scope and remap correlation references

(a) Keep declaring the variables that the remaining sub-queries use

matchJoin splits the Join's variables into two sets:

• subQueryVariables: used by the selected sub-query, bound by its rewrite;
• remainingVariables: used by the other sub-queries still in the ON condition, which the replacement relational expression must keep declaring.

The two single-side branches rebuild the Join through a new helper, rebuildJoin. When remainingVariables is non-empty, it explicitly creates a LogicalJoin that declares them. It does not use:

• RelBuilder.join with a non-empty variable set, because that may create a Correlate, which cannot represent an ON condition referencing both inputs;
• Join.copy, because that would keep the original, now stale, variable set.

When remainingVariables is empty, the ordinary RelBuilder.join is used as before. In the cross-product branch (INNER joins whose selected sub-query depends on both inputs), the remaining variables are declared on the replacement Filter.

(b) Update both the correlation row type and the field indexes

A correlation variable declared by the Join references the combined input row. Rewriting a sub-query appends auxiliary fields, which changes the shape of that row. remapCorrelationVariables builds a Mapping and applies RelOptUtil.RexCorrelVariableMapShuttle to update the row type and field-access indexes, including references inside nested sub-queries.

Branch	Auxiliary fields appended	Remapping of the remaining variables
Cross-product + Filter	After both original inputs	Retype only
Left-input rewrite	Between the original left and right fields	Retype, and shift right-side indexes
Right-input rewrite	After the original right fields	Retype only

The selected sub-query's own correlation references are also adapted to the input it is rewritten against:

• Left-side rewrite: retyped to the left input; indexes unchanged.
• Right-side rewrite: retyped to the right input, and indexes shifted by -nFieldsLeft.

Indexes that a negative shift pushes outside the target row type are left unmapped, so an invalid reference fails instead of silently resolving to the wrong field.

In rebuildJoin, the correlation row type is built from both inputs with SqlValidatorUtil.createJoinType, not from the Join's output type. This matters for SEMI and ANTI joins, whose ON condition can reference the right input even though their output does not contain it.

(c) Separate correlation ids shared by the selected and remaining sub-queries

Preserving remainingVariables is not enough when several correlated sub-queries share an id. For example, two EXISTS sub-queries may both use $cor0. Rewriting one moves its binding to an auxiliary Correlate over a single input, while the other still needs $cor0 to be defined by the Join over the joint row. Reusing $cor0 would define the same id twice with different row types.

renameSharedVariables gives the selected sub-query fresh ids only for the variables it shares with the remaining sub-queries. Non-shared variables and variables bound by an outer scope are left unchanged.

The uncorrelated scalar query in the original example shares no id with the EXISTS: it demonstrates scope preservation, while shared-id handling covers an additional case.

4. Rename correlation variables over the complete expression

renameSharedVariables takes a RexNode and returns a Pair<RexNode, Set<CorrelationId>> holding the rewritten expression and its updated variable set. It delegates to a new DeduplicateCorrelateVariables.go(RexBuilder, CorrelationId, Iterable<CorrelationId>, RexNode) overload, which reuses the existing traversal and covers:

• ordinary expression operands;
• sub-queries nested inside another sub-query's operands;
• each sub-query's relational expression, and the sub-queries nested within it.

Renaming only subQuery.rel would be insufficient. If an operand of an IN holds a correlated scalar sub-query and keeps the old id while the binder declares only the new one, that reference would be undefined.

5. Tests

• sub-query.iq: the example above, and a variant that combines a scalar sub-query, a correlated EXISTS and a NOT IN in one ON condition.
• RelOptUtilTest.testDeduplicateCorrelationsInRexExpression: checks that the new RexNode overload renames nested operands and relational expressions, leaves outer-scope ids alone, and is idempotent.

6. Scope and limitations

This change preserves correlation scope, binding identity and field geometry across incremental rewrites. It does not change which cases are rewritten:

• A sub-query that depends on both inputs of a non-INNER join is still not rewritten, because flattening to a cross-product plus Filter is valid only for INNER joins.
• The selection of the Logic for single-side ANTI and LEFT_MARK joins is not changed.

[mihaibudiu]

I see this is still a draft. Should I wait with the review?

[iwanttobepowerful]

I see this is still a draft. Should I wait with the review?

I'd like more time to consider the most appropriate implementation approach.

[mihaibudiu]

The high-level plan makes sense, but this won't be easy to review.
I hope that unsupported constructs will provide a clear error message.

[mihaibudiu]

Please describe how these results were validated in a comment.

[zzwqqq]

Should we also consider non-repeatable Sample inputs? Reading them twice can produce different samples and incorrect results.

[iwanttobepowerful]

Thx. I'll look into it later.

[zzwqqq]

Should we use the configured JoinFactory here to support custom join types?

[zzwqqq]

Should we use the configured JoinFactory here as well?

[iwanttobepowerful]

I'll look into it later.

[mihaibudiu]

While looking at [CALCITE-7894] I came up with this test which seems to be fixed by this PR:

select (select count(*) from (values (0, 1), (1, 2), (2, 3)) as l(k, a)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = l.k)) as c1,
       (select count(*) from (values (0, 1), (1, 2), (5, 3)) as r(k, b)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = r.k)) as c2;

[iwanttobepowerful]

While looking at [CALCITE-7894] I came up with this test which seems to be fixed by this PR:

select (select count(*) from (values (0, 1), (1, 2), (2, 3)) as l(k, a)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = l.k)) as c1,
       (select count(*) from (values (0, 1), (1, 2), (5, 3)) as r(k, b)
        where exists (select 1 from (values (0), (1), (2)) as s(v) where s.v = r.k)) as c2;

This case was already working before; it is not affected by this change.

[iwanttobepowerful]

The high-level plan makes sense, but this won't be easy to review. I hope that unsupported constructs will provide a clear error message.

Sorry, I've bundled several fixes into one commit. I'll split them into multiple smaller PRs later so they're easier to review.

[mihaibudiu]

checkstyle is unhappy

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
83.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mihaibudiu]

what is the status of this PR?