Skip to content

Fix empty missing-label losses and valid-energy MAE weighting - #255

Open
davrens101 wants to merge 2 commits into
CederGroupHub:mainfrom
davrens101:fix/missing-label-loss-metrics
Open

davrens101 wants to merge 2 commits into
CederGroupHub:mainfrom
davrens101:fix/missing-label-loss-metrics

Conversation

@davrens101

Copy link
Copy Markdown

Problem

With allow_missing_labels=True, CombinedLoss masks NaN energy/force/stress labels, but still mean-reduces an empty tensor when a whole batch lacks one target. That produces NaN loss and MAE. A real Trainer.train run exits with Exit due to NaN when its first batch has no energy label, even when force labels are available. A zero metric count alone does not help: AverageMeter still accumulates NaN * 0.

Energy MAE also uses the original batch size instead of the number of valid energies. For batches with errors [2, missing] and [4, 6], this reports 3.5 rather than the valid-label mean 4.0.

Changes

  • For an entirely unlabelled energy, force or stress target, use an empty prediction sum: a device/dtype-correct zero that retains autograd without reading masked-out NaN predictions.
  • Return zero MAE and zero count for that target, so metric aggregation remains finite.
  • Count valid energy labels when aggregating energy MAE.
  • Preserve the nonempty loss formulas, loss ratios and allow_missing_labels=False behavior.
  • Document the empty-target/metric contract and add deterministic numerical and gradient regressions.

Verification

CPU, Python 3.12, PyTorch 2.15.0+cpu:

  • tests/test_trainer.py: 18 passed (including existing training/checkpoint/W&B tests; W&B offline).
  • Actual two-structure NaCl training, batch size 1, one missing energy label: 0 completed epochs before; 1 completed epoch after, with finite train/validation metrics.
  • Independent aggregation example: 3.5 before → 4.0 after.
  • Entirely missing e/f/s targets: zero loss, zero MAE/count and zero prediction gradients after backward.

No GPU validation is claimed. Investigation, implementation and verification were AI-assisted.

@codacy-production

codacy-production Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 12 complexity

Metric Results
Complexity 12

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant