Skip to content

[1.11] Fix Spark test compatibility - #7798

Closed
pan3793 wants to merge 2 commits into
apache:branch-1.11from
pan3793:branch-1.11-ci-fix
Closed

pan3793 wants to merge 2 commits into
apache:branch-1.11from
pan3793:branch-1.11-ci-fix

Conversation

@pan3793

@pan3793 pan3793 commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Why are the changes needed?

Fix two compatibility failures in the branch-1.11 CI run:

  1. The nondeterministic CTE test expects RepartitionByExpression, but Spark 3.3 uses Repartition. Check their shared RepartitionOperation superclass while retaining the value-equality and exchange-reuse assertions.
  2. The JDBC lineage test fails with No suitable driver on Spark 3.3/3.4 with JDK 8. Initialize Derby explicitly in the test classloader before opening the JDBC connection.

No production code or runtime dependencies change.

How was this patch tested?

Existing suites cover both fixes.

  1. Reproduced the JDBC lineage failure on the base branch with Spark 3.3.4/JDK 8.
  2. Spark 3.3.4/JDK 8: 79 lineage tests passed, 2 version-inapplicable tests canceled, and all 52 HiveCatalogRangerSparkExtensionSuite tests passed.
  3. dev/reformat and git diff --check passed.

With JDK 8 selected:

build/mvn -B -ntp test \
  -pl extensions/spark/kyuubi-spark-lineage,extensions/spark/kyuubi-spark-authz -am \
  -Pspark-3.3,spark-provided,flink-provided,hive-provided \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.plugin.lineage,org.apache.kyuubi.plugin.spark.authz.ranger.HiveCatalogRangerSparkExtensionSuite \
  -Drat.skip=true

dev/reformat
git diff --check

The full CI matrix was not run locally.

Was this patch assisted by generative AI tooling?

Assisted-by: DeepSeek Harness:gpt-6.1-sol

@pan3793 pan3793 changed the title [1.11] Fix test compatibility and engine deletion timing [1.11] Fix Spark test compatibility Oct 27, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (5c7e0d8) to head (b69f8f4).
⚠️ Report is 1 commits behind head on branch-1.11.

Additional details and impacted files
@@             Coverage Diff             @@
##           branch-1.11   #7798   +/-   ##
===========================================
  Coverage         0.00%   0.00%           
===========================================
  Files              698     698           
  Lines            43682   43682           
  Branches          5899    5899           
===========================================
  Misses           43682   43682           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pan3793 pan3793 self-assigned this Oct 28, 2026
@pan3793 pan3793 added this to the v1.11.2 milestone Oct 28, 2026
pan3793 added a commit that referenced this pull request Oct 28, 2026
### Why are the changes needed?

Fix two compatibility failures in the [branch-1.11 CI run](https://github.com/apache/kyuubi/actions/runs/40306781716):

1. The nondeterministic CTE test expects `RepartitionByExpression`, but Spark 3.3 uses `Repartition`. Check their shared `RepartitionOperation` superclass while retaining the value-equality and exchange-reuse assertions.
2. The JDBC lineage test fails with `No suitable driver` on Spark 3.3/3.4 with JDK 8. Initialize Derby explicitly in the test classloader before opening the JDBC connection.

No production code or runtime dependencies change.

### How was this patch tested?

Existing suites cover both fixes.

1. Reproduced the JDBC lineage failure on the base branch with Spark 3.3.4/JDK 8.
2. Spark 3.3.4/JDK 8: 79 lineage tests passed, 2 version-inapplicable tests canceled, and all 52 `HiveCatalogRangerSparkExtensionSuite` tests passed.
3. `dev/reformat` and `git diff --check` passed.

With JDK 8 selected:

```bash
build/mvn -B -ntp test \
  -pl extensions/spark/kyuubi-spark-lineage,extensions/spark/kyuubi-spark-authz -am \
  -Pspark-3.3,spark-provided,flink-provided,hive-provided \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.plugin.lineage,org.apache.kyuubi.plugin.spark.authz.ranger.HiveCatalogRangerSparkExtensionSuite \
  -Drat.skip=true

dev/reformat
git diff --check
```

The full CI matrix was not run locally.

### Was this patch assisted by generative AI tooling?

Assisted-by: DeepSeek Harness:gpt-6.1-sol

Closes #7798 from pan3793/branch-1.11-ci-fix.

Closes #7798

b69f8f4 [Cheng Pan] [1.11] Keep CI fix scoped to Spark test compatibility
476b7ec [Cheng Pan] [1.11] Fix test compatibility and engine deletion timing

Authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793

pan3793 commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

Merge Summary:

Posted by merge_kyuubi_pr.py

@pan3793 pan3793 closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants