[WIP] HIVE-29641: Upgrade Calcite to 1.42.0#6523
Draft
rubenada wants to merge 36 commits into
Draft
Conversation
asf-ci-hive
added
tests pending
tests failed
and removed
tests pending
tests failed
labels
…n on Jenkins: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.6.0:shade (default) on project hive-druid-handler: Error creating shaded jar: Problem shading JAR /home/jenkins/agent/workspace/hive-precommit_PR-6523/druid-handler/target/hive-druid-handler-4.3.0-SNAPSHOT.jar entry org/apache/hive/druid/org/apache/calcite/runtime/SqlFunctions.class: org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class org/apache/hive/druid/org/apache/calcite/runtime/SqlFunctions.class: Index 65536 out of bounds for length 334
As a consequence of defining Calcite default charset system property in a previous commit, now in RexLiteral#appendAsJava (which already contained the literal charset vs default charset verification, added to SqlImplementor in CALCITE-6006 1.36) literals that used to print the charset don't do it anymore, e.g. _UTF-16LE'ten' => 'ten'
Due to "[CALCITE-5717] RelBuilder.project of literals on a single-row Aggregate should create a Values" (introduced 1.35), cases like TestMiniLlapLocalCliDriver with file explainuser_1.q led to: "java.lang.UnsupportedOperationException: Values with non-empty tuples are not supported. at org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:264)". The reason is that RelBuilder#project_ contains the simplification (only applicable if config.simplifyValues is true): "If the expressions are all literals, and the input is a Values with N rows [...], replace with a Values with same tuple N times"; and with CALCITE-5717 that simplification was extended to not only "Values with N rows" but also "Aggregates with 1 row", and this case would lead to creating a non-empty HiveValues, which is not supported in our case. The easiest solution to prevent that from happening is disabling simplifyValues in HiveRelBuilder config.
Another attempt to fix druid-handler asm javadoc error seen on Jenkins: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.6.0:shade (default) on project hive-druid-handler: Error creating shaded jar: Problem shading JAR /home/jenkins/agent/workspace/hive-precommit_PR-6523/druid-handler/target/hive-druid-handler-4.3.0-SNAPSHOT.jar entry org/apache/hive/druid/org/apache/calcite/runtime/SqlFunctions.class: org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class org/apache/hive/druid/org/apache/calcite/runtime/SqlFunctions.class: Index 65536 out of bounds for length 334
[CALCITE-5798] Improve simplification of '(x < y) IS NOT TRUE' when x and y are not nullable (1.35) [CALCITE-6317] Incorrect constant replacement when group keys are NULL (1.37) Adjust test plans due to change in representation of floating point values in RexLiteral: [CALCITE-2067] RexLiteral cannot represent accurately floating point values, including NaN, Infinity (1.38) Adjust test plans simplifications around IS_NOT_NULL: [CALCITE-5639] RexSimplify should remove IS NOT NULL check when LIKE comparison is present (1.35) (actually more generic than just LIKE) [CALCITE-5769] Optimizing 'CAST(e AS t) IS NOT NULL' to 'e IS NOT NULL' (1.35) (actually more generic than just CAST)
1.40 [CALCITE-6961] Support LogicalRepeatUnion in RelShuttle + [CALCITE-6959] Support LogicalAsofJoin in RelShuttle Add visit(LogicalAsofJoin) and visit(LogicalRepeatUnion) to HiveRelShuttleImpl [CALCITE-5387] Type-mismatch on nullability in JoinPushTransitivePredicatesRule RelRule Adjust RexPermuteInputsShuttle constructor in HiveJoinConstraintsRule 1.41 [CALCITE-7029] Support DPhyp to handle various join types Add visitNodeAndFieldIndex in RexVisitor implementation (HiveCalciteUtil#ConstantFinder) 1.42 [CALCITE-7351] Make getMaxNumericScale() and getMaxNumericPrecision() final Adjust HiveTypeSystemImpl
…alues in RexLiteral: [CALCITE-2067] RexLiteral cannot represent accurately floating point values, including NaN, Infinity (1.38)
Adjust test plans due to change in representation of floating point values in RexLiteral: [CALCITE-2067] RexLiteral cannot represent accurately floating point values, including NaN, Infinity (1.38) Adjust test plans simplifications around IS_NOT_NULL: [CALCITE-5639] RexSimplify should remove IS NOT NULL check when LIKE comparison is present (1.35) (actually more generic than just LIKE) [CALCITE-5769] Optimizing 'CAST(e AS t) IS NOT NULL' to 'e IS NOT NULL' (1.35) (actually more generic than just CAST) Adjust test plans: add missing padding on fixed-length char literals, due to [CALCITE-4590] Incorrect query result with fixed-length string (1.39) Note: this Calcite fix only impacts predicates with OR/IN/SEARCH, and aligns these plans with single value condition ones (which already contained the padding, e.g. ql/src/test/results/clientpositive/perf/tpcds30tb/tez/cbo_query33.q.out)
[CALCITE-6417] Map value constructor and Array value constructor unparsed incorrectly for HiveSqlDialect (1.38)
… due to [CALCITE-4590] Incorrect query result with fixed-length string (1.39) Adjust tests (condition predicates in different order) Adjust test plans due to change in representation of floating point values in RexLiteral: [CALCITE-2067] RexLiteral cannot represent accurately floating point values, including NaN, Infinity (1.38) Adjust test plans simplifications around IS_NOT_NULL: [CALCITE-5639] RexSimplify should remove IS NOT NULL check when LIKE comparison is present (1.35) (actually more generic than just LIKE) [CALCITE-5769] Optimizing 'CAST(e AS t) IS NOT NULL' to 'e IS NOT NULL' (1.35) (actually more generic than just CAST) Adjust test plans: add missing padding on fixed-length char literals, due to [CALCITE-4590] Incorrect query result with fixed-length string (1.39) Adjust test files: MINUS type added to Json due to [CALCITE-5607] Serialize return type during RelJson.toJson(RexNode node) for SqlKind.MINUS (1.37)
…c issue on Jenkins (still tbd)
[CALCITE-7160] Simplify AND/OR with DISTINCT predicates to SEARCH (1.41) Adjust test plans simplifications around IS_NOT_NULL: [CALCITE-5639] RexSimplify should remove IS NOT NULL check when LIKE comparison is present (1.35) (actually more generic than just LIKE) [CALCITE-5769] Optimizing 'CAST(e AS t) IS NOT NULL' to 'e IS NOT NULL' (1.35) Adjust test plans on array construction due to: [CALCITE-6417] Map value constructor and Array value constructor unparsed incorrectly for HiveSqlDialect (1.38)
…creating the filter RelNode. This is required to avoid this exception (new assertion introduced in 1.41 via [CALCITE-7070] FILTER_REDUCE_EXPRESSIONS crashes on expression BETWEEN ( NULL) AND X ) seen in TestMiniLlapLocalCliDriver subquery_null_agg when trying to create a Filter with NULL = NULL condition: java.lang.AssertionError: Comparison with NULL in pulledUpPredicates at org.apache.calcite.util.Litmus.lambda$static$0(Litmus.java:31) at org.apache.calcite.util.Litmus.check(Litmus.java:76) at org.apache.calcite.plan.RelOptPredicateList.<init>(RelOptPredicateList.java:128) ...
…y comparison RexCall create in HiveFunctionHelper), and avoid RexSimplify solution on CalcitePlanner (too many side effects)
…ntSnapshotMaterializationRule, use snapshotId -1 instead, and then switch it back in HivePushdownSnapshotFilterRule; this is required because an incorrect NULL comparison predicate would trigger a new assertion introduced in 1.41 (via [CALCITE-7070] FILTER_REDUCE_EXPRESSIONS crashes on expression BETWEEN ( NULL) AND X ): java.lang.AssertionError: Comparison with NULL in pulledUpPredicates at org.apache.calcite.util.Litmus.lambda$static$0(Litmus.java:31) at org.apache.calcite.util.Litmus.check(Litmus.java:76) at org.apache.calcite.plan.RelOptPredicateList.<init>(RelOptPredicateList.java:128)
…ge on HiveAugmentSnapshotMaterializationRule: generate $snapshotId <= -1 instead of $snapshotId <= null (which can lead to a Calcite error)
…doc generation: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.6.0:shade (default) on project hive-druid-handler: Error creating shaded jar: Problem shading JAR /../hive/druid-handler/target/hive-druid-handler-4.3.0-SNAPSHOT.jar entry org/apache/hive/druid/org/apache/calcite/runtime/SqlFunctions.class: org.apache.maven.plugin.MojoExecutionException: Error in ASM processing class org/apache/hive/druid/org/apache/calcite/runtime/SqlFunctions.class: Index 65536 out of bounds for length 334
…ing the expression SUBSTR(`value`, 5)
…d from RexSimplify for "non-safe" operations, e.g. division (if it could "hide" a runtime division by zero) or operations involving cast (only "lossless" casts can be simplified), via: [CALCITE-7145] RexSimplify should not simplify IS NULL(10/0) (1.42) [CALCITE-7296] RexSimplify should not simplify IS (NOT) NULL(CAST(10/0 AS BIGINT)) (1.42) [CALCITE-7032] Simplify 'NULL>ALL (ARRAY[1,2,NULL])' to 'NULL' (1.41)
[CALCITE-5639] RexSimplify should remove IS NOT NULL check when LIKE comparison is present (1.35) (actually more generic than just LIKE) [CALCITE-5769] Optimizing 'CAST(e AS t) IS NOT NULL' to 'e IS NOT NULL' (1.35)
[CALCITE-7005] Invalid unparse for IS TRUE,IS FALSE,IS NOT TRUE and IS NOT FALSE in Hive/Presto Dialect (1.40)
…ange in udf_between.q The plan difference comes from: 1.33 BETWEEN -> RexSimplify: 100 <= x AND x <= 200 -> HivePointLookupOptimizerRule: x BETWEEN 100, 200 -> SearchTransformer: no-op NOT BETWEEN -> RexSimplify: x < 100 OR x > 200 -> HivePointLookupOptimizerRule: x NOT BETWEEN 100, 200 -> SearchTransformer: no-op vs 1.42 BETWEEN -> RexSimpliy: Sarg x [100..200] -> HivePointLookupOptimizerRule: no-op -> SearchTransformer: x BETWEEN 100, 200 NOT BETWEEN -> RexSimpliy: Sarg x (-inf..100, 200..+inf) -> HivePointLookupOptimizerRule: no-op -> SearchTransformer: x < 100 OR x > 200 Thus, we need to adjust SearchTransformer to detect the last case and generate: x NOT BETWEEN 100, 200
|
… to avoid plan change in udf_between.q) solved 1 plan change but "broke" 8 other tests (see below *); so instead, simply revert the change in SearchTransformer and adjust udf_between.q.out. The "identify NOT BETWEEN expression in SearchTransformer" (to deal with "regression" caused by ""[CALCITE-7194] Simplify comparisons between function calls and literals to SEARCH (1.41)") can be handled in a separate, dedicated PR. * testCliDriver[vector_between_columns] – org.apache.hadoop.hive.cli.split28.TestMiniLlapLocalCliDriver4s testCliDriver[external_jdbc_table_perf] – org.apache.hadoop.hive.cli.split8.TestMiniLlapLocalCliDriver48s testCliDriver[filter_numeric] – org.apache.hadoop.hive.cli.split8.TestMiniLlapLocalCliDriver3s testCliDriver[correlationoptimizer8] – org.apache.hadoop.hive.cli.split5.TestMiniLlapLocalCliDriver5s testCliDriver[vector_between_in] – org.apache.hadoop.hive.cli.split29.TestMiniLlapLocalCliDriver36s testCliDriver[stats_histogram_null] – org.apache.hadoop.hive.cli.split22.TestMiniLlapLocalCliDriver28s testCliDriver[join34] – org.apache.hadoop.hive.cli.split11.TestMiniLlapLocalCliDriver2s testCliDriver[join35] – org.apache.hadoop.hive.cli.split11.TestMiniLlapLocalCliDriver
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Upgrade Calcite to 1.42.0.
Why are the changes needed?
Upgrade to latest Calcite version.
Does this PR introduce any user-facing change?
tbd
How was this patch tested?
tbd
Details
pom changes
Code adjustments due to Calcite breaking changes
Use non-deprecated version of RexBuilder#makeAbstractCast in HiveSqlSumAggFunction, RexNodeConverter, ASTConverter
AggCall construction adjustments in HiveAggregate and HiveSqlSumAggFunction
Add visitLambda and visitLambdaRef in RexVisitor implementation (HiveCalciteUtil#ConstantFinder)
Add visit(LogicalAsofJoin) and visit(LogicalRepeatUnion) to HiveRelShuttleImpl
Adjust RexPermuteInputsShuttle constructor in HiveJoinConstraintsRule
Add visitNodeAndFieldIndex in RexVisitor implementation (HiveCalciteUtil#ConstantFinder)
Adjust HiveTypeSystemImpl
Other code changes
_UTF-16LE'1'instead of simply'1') due to the modification inSqlImplemento#toSql(RexLiteral)via "[CALCITE-6006] RelToSqlConverter loses charset information" (introduced in 1.36)As a consequence of defining Calcite default charset system property, now
RexLiteral#appendAsJava(which already contained the literal charset vs default charset verification, added to SqlImplementor in CALCITE-6006 1.36), literals printed via this method (i.e. used to print the charset) don't do it any more, e.g._UTF-16LE'ten'=>'ten', which aligns them with the ones printed viaSqlImplemento#toSql(RexLiteral)This was also violated by materialization rules in HiveMaterializedViewRule, which used its own private static class HiveHepExtractRelNodeRule in a HepProgram as unionRewritingPullProgram. As a workaround, it is proposed to extend the MaterializedView rules (as HiveMaterializedViewRule), override the rewriteQuery
(where the UnionRewritingPullProgram is called), and unwrap the HepRelVertex tree ourselves (using (
HiveCalciteUtil.stripHepVertices) before calling the super.rewriteQuery.fieldsUsed.contains(aggregate.getGroupSet())but ratherfieldsUsed.intersects(aggregate.getGroupSet()).Also took the opportunity here to get rid of the unnecessary check aggregate.getIndicatorCount() > 0 which is always false (this method is deprecated and always returns zero)
java.lang.UnsupportedOperationException: Values with non-empty tuples are not supported. at org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:264). The reason is that RelBuilder#project_ contains the simplification (only applicable if config.simplifyValues is true): "If the expressions are all literals, and the input is a Values with N rows [...], replace with a Values with same tuple N times"; and with CALCITE-5717 that simplification was extended to not only "Values with N rows" but also "Aggregates with 1 row", and this case would lead to creating a non-empty HiveValues, which is not supported in our case. The easiest solution to prevent that from happening is disabling simplifyValues in HiveRelBuilder config.java.lang.AssertionError: Comparison with NULL in pulledUpPredicates, so in order to avoid that, a few classes needed to be adjusted. Firstly HiveFunctionHelper#getExpression must not create such expressions (seenNULL = NULLinTestMiniLlapLocalCliDriverwithsubquery_null_agg.q); secondly avoid incorrect comparison ($snapshotIdInputRef <= NULLseen inTestIcebergCliDriverwithmv_iceberg_orc.q) in HiveAugmentSnapshotMaterializationRule, use snapshotId -1 instead, and then switch it back in HivePushdownSnapshotFilterRule.