Skip to content

fix(loader): preserve native date conversions - #53

Closed
imbajin wants to merge 11 commits into
masterfrom
fix/integration-loader-dates
Closed

imbajin wants to merge 11 commits into
masterfrom
fix/integration-loader-dates

Conversation

@imbajin

@imbajin imbajin commented Oct 9, 2026 •

Copy link
Copy Markdown

Purpose of the PR

Restore native JDBC DATETIME and Parquet logical date/timestamp conversion. MySQL returns LocalDateTime for DATETIME, while Parquet DATE/TIMESTAMP values were treated as ordinary integers; supported DATE mappings failed or produced incorrect dates.

Found in the Toolchain integration report.

loader dates: before and after

Main Changes

  • Let the JDBC driver convert LocalDateTime through ResultSet.getTimestamp(), preserving its timezone and precision.
  • Decode Parquet DATE and TIMESTAMP logical annotations with their declared units and UTC/local meaning. Resolve DATE and local TIMESTAMP in the configured file time_zone, including accepted aliases, while UTC timestamps retain their epoch.
  • Preserve TIME, unannotated integers, null and existing INT96 behavior; add native MySQL and real Parquet file regressions.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests.
  • Need tests and can be verified as follows:
    • Real Parquet writer/file/reader tests and a JDBC driver-value regression fail against the old implementations and pass after the fix, including negative timestamps and preserved unrelated types. Real fetcher regressions use a UTC JVM with GMT+8/PST sources to verify timezone propagation.
    • Three independent read-only reviews passed.
    • Organization core CI passes, including the native MySQL DATETIME functional test. The final combined distribution builds against a fixed Server SDK. Its actual native launcher imports a real gzip Parquet fixture into a disposable Server; DATE and all UTC/local timestamp units, including negative boundaries, match backend epochs exactly, with zero failures and verified persisted progress.

Does this PR potentially affect the following parts?

  • Nope
  • Dependencies
  • Modify configurations
  • The public API
  • Other affects

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need: corrects existing DATE mappings without adding types or configuration.

The ordinary Loader runtime dependency support is already in upstream master through the merged runtime fix. The JDBC conversion itself is independent. The native date fixture uses matching Loader/Server GMT+8 under the existing date serialization contract.

INT96 retains the existing JVM-default timezone interpretation; the configured source timezone applies to DATE and local logical TIMESTAMP. All date and upstream runtime suite registrations are retained after syncing master.

- convert JDBC DATETIME through driver timestamps
- decode Parquet logical date and timestamp units
- preserve unrelated primitive and TIME values
- cover native date precision and negative epochs
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T09:01:34.276442Z f6bd598 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the loader label Oct 9, 2026
@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7b66c049-c78d-459c-837b-6bc72f1fd92c




📥 Commits

Reviewing files that changed from the base of the PR and between dfecd23 and 283466a.





📒 Files selected for processing (4)
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/file/ParquetFileLineFetcher.java
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/util/ParquetUtil.java
  • hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/functional/JDBCLoadTest.java
  • hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/ParquetUtilTest.java




Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.






Walkthrough

本次改动调整 JDBC 对 LocalDateTime 的读取方式,并增加 Parquet 日期和时间戳的逻辑类型转换。Parquet 转换支持调用方指定时区。新增单元测试和集成测试覆盖这些转换。

Changes

Loader 日期时间处理

Layer / File(s) Summary
JDBC 日期时间读取
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/jdbc/JDBCFetcher.java、hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/JDBCFetcherTest.java、hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/UnitTestSuite.java、hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/functional/JDBCLoadTest.java
JDBCFetcher.nextBatch() 对 LocalDateTime 使用 getTimestamp(i) 读取。新增单元测试和集成测试,覆盖时间值、空值及整数处理。
Parquet 日期时间转换
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/util/ParquetUtil.java、hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/ParquetUtilTest.java
ParquetUtil 根据逻辑类型将日期和时间戳转换为 Date。时间戳转换支持毫秒、微秒和纳秒;测试覆盖时区、负值、缺失字段及普通整数。
Parquet 来源时区传递
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/reader/file/ParquetFileLineFetcher.java、hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/ParquetUtilTest.java
ParquetFileLineFetcher.fetch() 将数据源时区传给字段转换。测试覆盖 GMT+8 和 PST 时区下的日期与时间戳结果。

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 28346

No actionable merge-blocking issue is identified; the change is mergeable after normal checks.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 3.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 28 functions across 7 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed 标题准确概括了主要变更:修复 loader 中 JDBC 和 Parquet 的原生日期转换,并保持原生日期语义。标题简洁、明确,且与变更内容直接相关。

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR










🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR







  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

小兔捧起时间戳,
日期落进时区旁。
微秒纳秒排成队,
JDBC 读取稳当当。
Parquet 轻轻翻一页,
测试照亮新时光。

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfecd23e9d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.70%. Comparing base (e669922) to head (f6bd598).

Files with missing lines Patch % Lines
.../org/apache/hugegraph/loader/util/ParquetUtil.java 92.59% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master      #53      +/-   ##
============================================
+ Coverage     46.62%   46.70%   +0.07%     
- Complexity     4447     4461      +14     
============================================
  Files           602      602              
  Lines         29235    29263      +28     
  Branches       3422     3426       +4     
============================================
+ Hits          13630    13666      +36     
+ Misses        14329    14318      -11     
- Partials       1276     1279       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- parse the fixture through Loader input sources
- normalize the existing MySQL vendor contract
- query a connection-owned temporal table
- retain native date and precision assertions
- avoid the failing Docker Hub authentication endpoint
- retain the exact previously successful official image
- preserve service settings and all regression assertions
- fetch existing Maven and Java images from official ECR
- keep Flink probe and runtime on the same verified image
- preserve the distinct Loader and CDC MySQL versions
- retain SDK builder overrides and all test assertions
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@github-actions github-actions Bot added the hubble label Oct 9, 2026
- Restore original Docker Hub references in CI and Docker builds
- Restore original Flink and MySQL verification image tags
- Preserve the branch-specific functional fixes and runtime tests
@github-actions github-actions Bot removed the hubble label Oct 10, 2026
- Pass configured file timezone into Parquet conversion
- Interpret DATE and local timestamps in the source zone
- Preserve UTC epochs and legacy INT96 behavior
- Cover differing JVM zones and accepted timezone aliases
- drain asynchronous tasks before reporting failures
- preserve suppressed errors and interrupt state
- retain backup input when restore fails
- cover corrupt input and valid empty backups
- use configured graph space in existing schema APIs
- preserve legacy routes for empty graph spaces
- retain request and HTTP error semantics
- cover route escaping and request bodies with race
The ordinary Loader distribution cannot start existing GRAPH or Parquet sources: its selected protobuf runtime lacks the PD discovery classes, and Parquet classes are inherited only through Spark's provided dependencies. The ordinary package now carries the dependencies these existing readers require.
- Incorporate the merged Toolchain runtime and launcher fixes
- Preserve JDBC, Protobuf and Parquet suite registrations
- Retain configured source timezone conversion and regressions
@imbajin imbajin closed this Oct 10, 2026
@imbajin
imbajin deleted the fix/integration-loader-dates branch October 10, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant