Skip to content

fix(loader): propagate loading failures - #56

Closed
imbajin wants to merge 11 commits into
masterfrom
fix/integration-loader-failures
Closed

imbajin wants to merge 11 commits into
masterfrom
fix/integration-loader-failures

Conversation

@imbajin

@imbajin imbajin commented Oct 9, 2026 •

Copy link
Copy Markdown

Purpose of the PR

Loader can report success after parse, insert, or checkpoint failures. With batch fallback disabled, rejected batches can also disappear from both the failure files and the already advanced progress. This change makes unsuccessful loads fail visibly and retains the information needed for replay.

Reported during master integration testing.

loader failures: before and after

Main Changes

  • Write failed or error-stopped batches to failure-data when compensation remains safe, and propagate asynchronous errors. Treat --max-read-lines as a successful intentional stop; later real errors still take precedence.
  • Preserve the existing fully parsed progress plus failure-data recovery contract; reject unreliable progress after write, flush, cancellation, or checkpoint failures.
  • Close task admission and drain submitters, running handlers, and cancelled queued tasks before closing their clients and logs.
  • Attempt every failure-log close, retain suppressed errors and the original runtime exception type/message, and return an unsuccessful process status after cleanup.

The shutdown and recovery changes are delivered together as one independent fix: splitting them would leave unsafe intermediate resource and progress behavior. This does not add per-record acknowledgements or a new replay protocol.

flowchart LR
    A[Batch fails without fallback] --> B[Retain the batch in failure-data]
    B --> C[Drain work and flush logs]
    C --> D{Progress reliable?}
    D -- Yes --> E[Save parsed progress]
    D -- No --> F[Keep old progress]
    E --> G[Return failure]
    F --> G
Loading

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests.
  • Need tests and can be verified as follows:
    • Targeted task and context regressions pass with failing negative controls for lost compensation, checkpoint refusal, active callbacks, admission races, and cancelled queued tasks. Deterministic queued-batch regressions distinguish the read limit from generic stops and subsequent errors.
    • Real file streams and controlled in-flight work exercise log flushing, cancellation, timeout, and interruption boundaries.
    • Checkstyle and three independent final reviews pass.
    • Parse-error compatibility and first-failure cleanup regressions pass; the CLI fixture explicitly fails integer conversion and still checks nonzero exit, valid-row insertion, and retained failure-data.
    • Original-versus-fixed native package acceptance passes against the same real Server and SDK: parse failures return 0 before the fix and 1 after it; a rejected batch with fallback disabled is retained in full, and standard failure-mode replay restores the records with verified persisted progress.

Does this PR potentially affect the following parts?

  • Nope
  • Dependencies (add/update license info)
  • Modify configurations
  • The public API
  • Other affects: Loader failure status, shutdown, and existing recovery data

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need: the existing progress and failure-data contract is preserved.

Native package recovery acceptance passes. The previous upstream head passed all core profiles after one bounded retry of an external SDK download/cache failure; the read-limit correction is covered by focused regressions and is being checked on the updated head. No dependency or image changes were needed.

Synced with upstream master while retaining both upstream and PR regressions. Focused tests and style checks pass; current-head core CI is running.

@github-actions github-actions Bot added the loader label Oct 9, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-10T09:01:35.745492Z d410c4c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a84b7669-3cee-48f2-9326-6f945484e9fd











📥 Commits

Reviewing files that changed from the base of the PR and between e669922 and a74a103.












📒 Files selected for processing (9)
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadContext.java
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/failure/FailWriter.java
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/task/BatchInsertTask.java
  • hugegraph-loader/src/main/java/org/apache/hugegraph/loader/task/TaskManager.java
  • hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/functional/FileLoadTest.java
  • hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/functional/LoadTest.java
  • hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/LoadContextTest.java
  • hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/TaskManagerTest.java











Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.













Walkthrough

本次变更调整加载失败的记录和传播方式,更新批量及单条插入任务的异常处理与关闭协调,并使加载器根据加载结果返回状态。新增测试覆盖失败日志、并发关闭和 CLI 失败退出。

Changes

加载失败处理与关闭流程

层 / 文件 摘要
失败状态与日志收尾
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/executor/LoadContext.java, hugegraph-loader/src/main/java/org/apache/hugegraph/loader/failure/FailWriter.java, hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/LoadContextTest.java
LoadContext 保存首个失败原因,并在关闭时处理失败日志器、进度写入和已记录的加载失败。FailWriter 的写入方法增加同步保护,关闭失败时抛出 LoadException。单元测试覆盖关闭异常和进度写入失败。
插入任务与关闭协调
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/task/TaskManager.java, hugegraph-loader/src/main/java/org/apache/hugegraph/loader/task/BatchInsertTask.java, hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/unit/TaskManagerTest.java
TaskManager 跟踪提交状态,并在关闭时等待任务、提交者和处理器;超时或中断会登记加载失败。批次和单条插入异常会进入失败处理,按配置回退或记录失败。测试覆盖失败写入、任务取消、重试及关闭等待。
加载器结果与清理
hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java, hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/functional/LoadTest.java, hugegraph-loader/src/test/java/org/apache/hugegraph/loader/test/functional/FileLoadTest.java
HugeGraphLoader 根据上下文错误状态返回加载结果,并在任务停止后关闭上下文。main 在加载失败时以状态码 1 退出。功能测试检查返回值及 CLI 退出码。

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant HugeGraphLoader
  participant TaskManager
  participant InsertFuture
  participant LoadContext
  participant FailWriter
  HugeGraphLoader->>TaskManager: 提交插入任务
  TaskManager->>InsertFuture: 执行任务
  InsertFuture-->>TaskManager: 返回任务异常
  TaskManager->>LoadContext: 登记加载失败
  TaskManager->>FailWriter: 写入失败记录
  HugeGraphLoader->>LoadContext: 关闭上下文并检查失败状态
Loading

Merge Risk: ⚪ Minimal · up to a74a1

This change propagates loader failures to the exit status and tightens the shutdown order. The review found no concrete defect blocking merge. Run the reported pending functional tests and CI before merging.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 2.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 80 functions across 9 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed 标题“fix(loader): propagate loading failures”准确概括了主要变更。变更内容包括传播加载失败、设置失败状态、返回失败进程状态,以及在清理后保留失败信息。标题简洁且具体。

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR











⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch fix/integration-loader-failures











🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR





















  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

兔子捧着日志跑,
失败原因记得牢。
任务收尾等一等,
进度写好再回巢。
月光下,加载顺利报。

Comment @coderabbitai help to get the list of available commands.

- preserve failed batch records when fallback is disabled
- propagate worker and checkpoint errors to the exit status
- drain submissions and cancelled tasks before closing resources
- flush all failure logs and reject unreliable progress
@imbajin
imbajin force-pushed the fix/integration-loader-failures branch from a74a103 to a1ab617 Compare October 9, 2026 21:46
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.35593% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.89%. Comparing base (e669922) to head (d410c4c).

Files with missing lines Patch % Lines
.../org/apache/hugegraph/loader/task/TaskManager.java 78.85% 30 Missing and 7 partials ⚠️
...a/org/apache/hugegraph/loader/HugeGraphLoader.java 65.00% 6 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master      #56      +/-   ##
============================================
+ Coverage     46.62%   46.89%   +0.27%     
- Complexity     4447     4495      +48     
============================================
  Files           602      602              
  Lines         29235    29404     +169     
  Branches       3422     3450      +28     
============================================
+ Hits          13630    13789     +159     
- Misses        14329    14332       +3     
- Partials       1276     1283       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- avoid the failing Docker Hub authentication endpoint
- retain the exact previously successful official image
- preserve service settings and all regression assertions
- fetch existing Maven and Java images from official ECR
- keep Flink probe and runtime on the same verified image
- preserve the distinct Loader and CDC MySQL versions
- retain SDK builder overrides and all test assertions
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

- retain the first runtime exception type and message
- avoid self suppression during test cleanup
- use a deterministic CLI parse failure fixture
- verify client closure and rejected progress after failure

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f57a84774

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- retain the existing Docker Hub image sources
- remove transient network repairs from the loading fix
- preserve all failure and recovery behavior changes
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@github-actions github-actions Bot removed the hubble label Oct 10, 2026
- Distinguish configured read limits from error stops
- Keep real errors and generic stops authoritative
- Avoid failure data for batches skipped at the read limit
- Cover queued submissions and stop reason precedence
- drain asynchronous tasks before reporting failures
- preserve suppressed errors and interrupt state
- retain backup input when restore fails
- cover corrupt input and valid empty backups
- use configured graph space in existing schema APIs
- preserve legacy routes for empty graph spaces
- retain request and HTTP error semantics
- cover route escaping and request bodies with race
The ordinary Loader distribution cannot start existing GRAPH or Parquet sources: its selected protobuf runtime lacks the PD discovery classes, and Parquet classes are inherited only through Spark's provided dependencies. The ordinary package now carries the dependencies these existing readers require.
- Incorporate merged Toolchain runtime and consumer fixes
- Preserve Loader failure and read limit behavior
- Retain all upstream and Loader suite registrations
@imbajin imbajin closed this Oct 10, 2026
@imbajin
imbajin deleted the fix/integration-loader-failures branch October 10, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant