Skip to content

fix: surface SparkArithmeticException(DIVIDE_BY_ZERO) for divide-by-zero in dispatched ScalaUDF path#4653

Open
0lai0 wants to merge 3 commits into
apache:mainfrom
0lai0:fix-4517-udf-divide-zero
Open

fix: surface SparkArithmeticException(DIVIDE_BY_ZERO) for divide-by-zero in dispatched ScalaUDF path#4653
0lai0 wants to merge 3 commits into
apache:mainfrom
0lai0:fix-4517-udf-divide-zero

Conversation

@0lai0

@0lai0 0lai0 commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #4517 .

Rationale for this change

DataFusion 53+ wraps errors with .context(...), producing DataFusionError::Context(_, Box<inner>).

Previous handling mostly matched top-level DataFusionError::External(...), so Context-wrapped Spark errors could bypass typed conversion and leak as CometNativeException instead of Spark’s SparkArithmeticException(DIVIDE_BY_ZERO).

What changes are included in this PR?

  • native/core/src/execution/expressions/arithmetic.rs

    • CheckedBinaryExpr::evaluate now matches generic Err(e) (when query context exists), then recursively extracts SparkError via extract_spark_error.
    • Added recursive unwrapping for DataFusionError::Context and nested External.
    • If no Spark-typed error is found, returns Err(e) as-is (instead of re-wrapping as External).
  • native/jni-bridge/src/errors.rs

    • Added SparkPayload and extract_spark_payload to recursively unwrap Context / nested External.
    • throw_exception now handles all CometError::DataFusion { source } through extract_spark_payload, covering:
      • JavaException passthrough
      • SparkErrorWithContext JSON throw path
      • bare SparkError JSON throw path
      • fallback generic exception path when no Spark payload is found
  • spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala

    • Added regression test: divide-by-zero through dispatched ScalaUDF surfaces SparkArithmeticException (#4517).
    • Enables ANSI mode, runs SELECT 1 / identity_int(a) over data containing zero, and validates exception propagation semantics.

How are these changes tested?

./mvnw test -Dtest=none -Dsuites=org.apache.comet.CometCodegenSuite

@andygrove

Copy link
Copy Markdown
Member

Thanks @0lai0. Can you revert the Spark diff changes made in #4514 as part of this PR?

@0lai0

0lai0 commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove. Just to confirm, should I revert both the #4517 workaround (SQLQueryTestSuite DIVIDE_BY_ZERO) and the #4516 workaround (SQLQuerySuite UDF call-count) from dev/diffs?

The #4517 one is clearly no longer needed with this fix, but wanted to check on #4516 since that issue is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native divide-by-zero in a dispatched ScalaUDF surfaces CometNativeException instead of SparkArithmeticException

2 participants