[kasakrisz]

What changes were proposed in this pull request?

• Add a new Tez physical optimization: PartialGroupByJoinTransposeOptimizer. It pushes a copy of the map-side (hash) GROUP BY below an inner join. The copy goes to the join input that holds every aggregation argument.

TS - SEL - RS \
               JOIN - SEL - GBY(hash) - RS - GBY(mergepartial)
TS - SEL - RS /

=>

TS - SEL - GBY(partialhash) - RS \
                                  JOIN - GBY(hash) - RS - GBY(mergepartial)
TS - SEL -------------------- RS /

• The pushed GROUP BY runs in a new mode: partialhash, in the vertex of the join input, before its existing shuffle.
• The GROUP BY above the join keeps its mode and output, but it merges the partial results (PARTIAL2) instead of reading the rows. The reduce-side GROUP BY does not change.
• When a pushed GROUP BY that is matched again can be pushed further down a chain of joins and a Select operator that projects its keys and partial results replaces it above the lower join.
• The pushed GROUB BY keys are the original keys available on that input, the input's join keys, and its columns used by residual (non-equi) join filters.
• The new optimization is run by TezCompiler after annotates the statistics. It is run before the TopNKey optimization and dynamic partition pruning. Otherwise, TopNKey operators and semijoin branches would block the push-down.
• Synthetic join predicates of residual filters are re-pointed to the output of the pushed GROUP BY.
• In StatsRulesProcFactory a GROUP BY with partial hash mode is estimated at its parent's row count. This is pessimistic, so map join conversion decisions don't change.
• Synthetic join predicates doesn't change filter statistics. This was necessary because stats annotation run is moved before dynamic partition pruning which removes Synthetic join predicates.

Why are the changes needed?

To improve performance of queries with joins and aggregates.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

mvn test -Dtest.output.overwrite -Dtest=TestMiniLlapLocalCliDriver -Dqfile=partial_aggr_join_transpose.q -pl itests/qtest -Pitests

[sonarqubecloud]

Quality Gate passed

Issues
38 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud