Skip to content

JDFTx IO module overhaul - #317

Draft
benrich37 wants to merge 105 commits into
materialsproject:mainfrom
benrich37:misc_fixes
Draft

benrich37 wants to merge 105 commits into
materialsproject:mainfrom
benrich37:misc_fixes

Conversation

@benrich37

@benrich37 benrich37 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

[breaking] JDFTx IO - feature additions, parsing fixes, input tag fixes, restart fidelity, and round-trip serialization

This PR has been split into 6 parts, to be reviewed and merged in their indexed order

  • pt 1: #319 - independent input tag fixes
    -- Small fixes for individual input commands that stand on their own: subtags with the wrong data type (ie electric-field components getting truncated to ints), commands that couldn't be read at all (Vexternal, charged-defect-correction, a bare van-der-waals), and a few missing options
  • pt 2: #320 - input tags, JDFTXInfile and JDFTXStructure (only this part)
    -- The bulk of the inputs side: addition of commands which have been added to JDFTx since the last PR, reworking the tag classes so commands are read and written the way JDFTx handles them, filling in JDFTx's default values when comparing inputs, and getting species names, velocities, constraints and moveScale through JDFTXInfile/JDFTXStructure correctly
  • pt 3: #321 - remove jminsettings (only this part)
    -- Removes the jminsettings module and the jsettings_* attributes, which were never set by anything (deletions only)
  • pt 4: #322 - out file parsing fixes, faster parsing and restart fidelity (only this part)
    -- The bulk of the outputs side: fixes for out files that failed to parse or parsed incorrectly, newly parsed data (SCF runs, AIMD, vector-spin, ionic Gaussian potentials), skim_levels/skip_props for quickly parsing large out files, and to_jdftxinfile restarts that actually reproduce the calculation they came from
  • pt 5: #323 - round-trip serialization of parsed outputs (only this part)
    -- Makes the parsed output classes MSONable, so they round-trip through JSON (dumpfn/loadfn, jobflow, pydantic) without losing anything
  • pt 6: #324 - repeatable input tags (only this part)
    -- Allows six commands (ie fluid-cation/fluid-anion) to appear on more than one line like they can in JDFTx, which changes their values to lists

Note for reviewers:

- Each part is stacked on the part before it. Until the earlier parts are merged, a part's "Files changed" tab also shows their changes (ie pt 2 shows pt 1 + pt 2). The "only this part" links above show what a part adds on its own.

- pts 1-4 only need pymatgen-core. pts 5 and 6 need the atomate2 change (materialsproject/atomate2#1659) to merge before them. The atomate2 change works with both old and new pymatgen-core, so it can merge at any time.

- The details live in the parts. Each part's description lists its own changes, breaking changes and todos. The diff of this PR is just all six parts together, so there is nothing to review here that isn't in one of them.

benrich37 and others added 30 commits April 9, 2026 11:31
Transfers 62 commits of work from the benrich37 fork that were built
on top of the pre-reorganization pymatgen jdftx module. History of
individual changes is preserved in benrich37/pymatgen master branch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Import all JDFTx changes from benrich37/pymatgen fork master branch
…he first place instead of initializing as a list of None (causes a lot of annoying UserWarning's down the line)
… almost-good-enough version of "add-U" parameter to the jdftxinfile master format file
… to allow functionality, tests to ensure tag works as expected, deleting old "deleteme" file
…n order-sensitive values, enforced ordered equality in AbstractTag.is_equal_to. Added AbstractTag.read_from_tokens which by default behaves the same as `read` but also returns the number of tokens read. For TagContainer, read_from_tokens now performs a read on provided tokens until an unreadable token is encountered. Changed `TagContainer.read` behavior within subtag loop for positional subtags (subtags without written tagnames) to now call `_read_positional_subtag` (which relies on `read_from_tokens` instead of regular `read` to allow for possibly reading and discarding more than one token for a subtag. Moved initial magnetic moments validations to the JDFTXInfile class. Added a `get_species_values_tag_container` convenience function for initializing "initial-magnetic-moments" and "initial-oxidation-state" as a regular TagContainer instead of the old two-purpose InitMagMomTag. Removed `InitMagMomTag` as it is no longer needed. Added docustrings to helper functions in io/jdftx/inputs.py missing docustrings.
…lues are caught. Structure of "add-U" tag changed to contain a single "U-channels" subtag, which contains the expected structure of the "add-U" tag - this is done to allow JDFTx's behavior of not allowing multiple add-U lines, but obviously accepting multiple species to have add-U values described in that one line Changes made to `TagContainer.read_from_tokens` to prevent possible subtag keys from being read as values.
…e removal_path with _get_removal_path, and call _remove_at_path for each removal_path to remove that bad tag/subtag - prevents attempting to remove duplicate paths, and prioritizes deeper subtags and later pop indices to protect from removing tag paths with parents already removed and to protect from index shifting.
…att-scale subtags being specified as IntTag's instead of FloatTag's, added tests for converting named lattices to pymatgen Lattice objects.
…uations

Lines ending in "\" are now joined with the following line by a single re.sub in JDFTXInfile.from_str. The previous
_gather_tags implementation is kept commented out in case the regex approach is not accepted. Added
test_line_continuation to check whitespace/comment handling around continuations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… function, adding testing for selective_dynamics_site_prop_to_jdftx_interpretable
…behavior can be reproduced in a single line and is only called once. Added test to test_jdftxinfile/test_jdftxstructure_infile_site_properties_conversion to ensure behavior is kept the same
…anmaps` which cleaned the values well enough itself such that the function could be replaced by np.nan. Removed `io.jdftx.inputs.get_popmaps`, since JDFTXInfile's dict inheritance allows `get_nanmaps` to operate directly on a JDFTXInfile.
…r `next` iteration. Added test to `io.jdftx.test_jdftxinfile` to ensure label fallback system performs as expected. Removed _multi_has/getattr functions and testing as they are no longer being used.
…ure.get_str` in `io.jdftx.inputs`, adding test to `test_jdftxinfile` to ensure changes to unit-converting code still works as intended.
…ded testing to ensure specie names of pmg Structure objects are left intact in JDFTXStructure.structure
…ding test to ensure hyperplane data can be safely passed back and forth between JDFTXStructure and JDFTXInfile tag reading/writing
…ystems for constraint vectors, fixed constraints being lost when a JDFTXInfile was initialized from a structure with constraints encoded into site properties, protect. Added error raising for some constraints which JDFTx would reject. General fixes to old misconception that ion prints will always be homogeneous shaping (this assumption breaks for some special case constraint types).
… from standard JDFTx units to A / fs + testing
…m on top of regular parsing to allow `JDFTXOutfile.to_jdftxinfile` to exactly reproduce the ion coordinates without risk of rounding. On top of this, structural tags are now stored to 15 decimal precision (previously 12) to match JDFTx's output. `to_jdftxinfile` now also retains the coords-type of the originating output file. Fixed bug where JDFTXInfile.strip_structure_tags stripped a non-existent "lattice-scale" instead of "latt-scale". Fixed `_infile_special_format_to_pmg_lattice` which previously used pymatgen's built in Lattice construction - there is unfortunately a mismatch between pymatgen and JDFTx, where pymatgen begins by orienting the third lattice vector along the z axis, while JDFTx begins by orienting the first lattice vector along the x axis. Fixed bug in `condense_jdftxinputs` where any vestigial structure tags in the jdftxinput argument would remain in the condensed JDFTXInfile object.
@benrich37

Copy link
Copy Markdown
Contributor Author

@shyuep Thank you for the review notes! I will address these today, and then split these diffs to be reviewed across multiple new bite-size PRs

…_parse_emin_lines` into a ValueError to allow the higher up error catchers to catch it instead of aborting the whole parse
…r slices that failed to parse and raise a ValueError if ALL slices failed to parse
…t options to avoid issues with exact-case matching checks down the line
… to set up ASE in testing environment I moved this all to pymatgen-core, so when the homogeneity protections got removed the tests that should have failed were not being ran). Adding a `_sync_movescale_and_selective_dynamics` to be called in the `JDFTXStructure` post-init to ensure movescales and selective dynamics agree with each other (if a movescale > 0, sd should be [1,1,1], else [0,0,0]). Remove unnecessary conversions of movescales to selective dynamics. Make "movescale" just as easy to set as a "selective_dynamics" (since they both encode overlapping but also unique information)
…t very complicated, so to err on the side simplicity fixing an atom in place with either movescale or selective dynamics os absolute
…moving the now-unused JDFTXOutfileSlice._get_initial_structure, deleting commented out code
@shyuep

shyuep commented Nov 19, 2026

Copy link
Copy Markdown
Member

Automated PR review generated by Claude (posted from @shyuep's account). Based on code and CI status only, nothing was run. Please sanity-check before acting.

Follow-up on the 10 commits df3591d..9500390 (~110 kB of diff; reviewed per commit).

CI: lint and 3 of 4 test jobs green on 9500390. test (macos-latest, 3.13, lowest-direct, optional) was cancelled after running 6 h (06:32 to 12:32 UTC), i.e. it hit the job time limit in pytest. The five preceding runs on this branch (up to c1b7e9a) passed on macOS, and 9500390 is mostly renames to private names, so this looks like a stall rather than a regression, consistent with the intermittent hangs in #316. Please re-run that job before merging.

Earlier items:

  1. Merge order with atomate2: still open. The Breaking Changes section still reads "atomate2 change (LINK TBD)" (the summary links Compatibility with incoming update to JDFTx pymatgen-core IO module atomate2#1659); fill it in, and hold the merge until #1659 is in an atomate2 release.
  2. Dead _gather_tags fallback: resolved; the commented-out method and note are gone and _get_lines keeps the regex.
  3. validate_value_boundaries raising on read: still open; no change to the default or the error message in these commits.
  4. Splitting the PR: acknowledged in your comment, not done yet; this push grows the diff further.

Good:

  • _sync_movescale_and_selective_dynamics (inputs.py) makes moveScale the single source of truth, rejects negative values as JDFTx does, warns on disagreement, and copies the structure instead of mutating the caller's.
  • none_slice_on_error now warns with the exception repr instead of silently dropping slices, and JOutStructures warns for failed slices too.
  • Keeping partial posns constraints off site_properties (with tests for serialization round-trip and ASE conversion) avoids ragged site properties breaking AseAtomsAdaptor.

Substantive:

  1. _sync_movescale_and_selective_dynamics: a site with partial selective dynamics (e.g. [True, True, False]) and no movescale gets float(all(sd)) = 0.0, i.e. it is fully fixed, and its selective_dynamics site property is overwritten to [False]*3. That is a warned but lossy change to a user-supplied structure (a POSCAR-derived slab with z-only relaxation becomes fully frozen). Until Linear/Planar constraints are written (the TODO), raising, or at least leaving the structure's original selective_dynamics untouched, would be safer than silently freezing the site.

Nits:

  • stacklevel=4 in the two new warnings depends on the __post_init__ call depth; it will point at the wrong frame if _sync_movescale_and_selective_dynamics is called from elsewhere.

…. To block bad in files with errors, run JDFTXInfile.validate_boundaries() directly without the warn_only=True kwarg
@benrich37

Copy link
Copy Markdown
Contributor Author

Beginning the process of splitting up this PR - keeping this PR open as a draft until all those have been merged

@benrich37
benrich37 marked this pull request as draft November 19, 2026 19:23
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
test_JDFTXOutfile_skimming failed on Windows CI with `assert 0.046875 < 0.046875`: time.process_time advances in
steps of 1/64 s there, about the CPU time of one parse of these files, so a skimmed and a full parse could read the
same. Each CPU time can be off by one step, so a speedup is now only tested where it is at least two steps:
- Only the skims that leave out most of the file are timed: whole out file slices if there are several, otherwise
  geometry steps. The two geometry-only combinations of example_latmin.out are no longer timed.
- Each timing is of enough parses for the full parse to take 6 clock steps.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@shyuep

shyuep commented Nov 20, 2026

Copy link
Copy Markdown
Member

Automated PR review generated by Claude (posted from @shyuep's account). Based on code and CI status only, nothing was run. Please sanity-check before acting.

Follow-up on the commits since the last review: 92fb9c4, 9c622ef, 73f2d02, the upstream merge 99473b9, and 2fef90f.

CI: all 5 checks green on 2fef90f (lint; ubuntu 3.12/3.14, macOS 3.13, Windows 3.13). The macOS job that timed out on 9500390 now passes.

Earlier items:

  1. Merge order with atomate2 (#1659): still open; it now applies to [breaking] JDFTx IO Module Overhaul pt 5 - Round-trip serialization of parsed outputs #323 (pt 5), which carries the @module/@class change.
  2. Dead _gather_tags fallback: resolved earlier.
  3. validate_value_boundaries raising on read: resolved. from_dict/from_str/read_line now call validate_boundaries(warn_only=True), the message names the flag, and validate_boundaries() still raises when called directly.
  4. Splitting: done as [breaking] JDFTx IO Module Overhaul pt 1 - independent input tag fixes #319-[breaking] JDFTx IO Module Overhaul pt 6 - repeatable input tags #324.
  5. Partial selective dynamics silently freezing sites: resolved. _sync_movescale_and_selective_dynamics now raises NotImplementedError, and the test covers both the argument and the site-property paths.
  6. stacklevel nit: addressed in 73f2d02.

Good:

  • Warning on read but raising on an explicit validate_boundaries() call is the right split. It also covers from_dict on stored documents, which cannot take the flag.
  • The skimming speed test now only times the skim levels that remove most of the file and scales the repeat count to the clock resolution. That is a sound fix for the coarse process_time on Windows.

Substantive:

  1. Branch bookkeeping. This branch and the six split branches have drifted apart. misc_fixes is 1 commit ahead of and 18 behind misc_fixes_D. More importantly, the misc_fixes_B0 head that [breaking] JDFTx IO Module Overhaul pt 3 - remove jminsettings #321 points to (e072a65) predates "Move the repeatable-tags fix from misc_fixes_A to misc_fixes_D" (da188aa) and the upstream merge. The local merge 6f23b2f of A into B0 is in D's history but was never pushed to misc_fixes_B0. As a result, [breaking] JDFTx IO Module Overhaul pt 3 - remove jminsettings #321 still contains the repeatable-tags change that is supposed to land in [breaking] JDFTx IO Module Overhaul pt 6 - repeatable input tags #324. Push misc_fixes_B0 (and check B/C), and consider closing this umbrella PR or converting it to a tracking issue so that reviewers look at one source of truth. [breaking] JDFTx IO Module Overhaul pt 6 - repeatable input tags #324 has the same +17299/-2027 footprint.
  2. The NotImplementedError for partial selective dynamics is a new hard failure in JDFTXInfile.from_structure. Any POSCAR-derived slab with z-only relaxation, which is common in atomate2 surface workflows, now fails at input generation. [breaking] JDFTx IO Module Overhaul pt 2 - input tags, JDFTXInfile and JDFTXStructure #320's Breaking Changes documents it. Writing these sites as Linear/Planar constraints (the listed Todo) should be the next follow-up.

Nits:

  • test_JDFTXOutfile_skimming calibrates number with the default Timer (wall-clock perf_counter), but then measures with process_time. Under pytest-xdist load, wall time exceeds CPU time, so number comes out smaller than needed for the 6-step margin the comment promises. Calibrating with timer=process_time, guarded with max(single_parse_time, CPU_CLOCK_STEP) against a zero reading, would keep the two consistent.

benrich37 and others added 4 commits November 20, 2026 15:01
… time

The number of parses per timing was estimated from the wall time of one parse, while the times compared are CPU
times. Wall time can exceed CPU time under load, which leaves the full parse spanning fewer than the intended 6
steps of a coarse CPU clock (1 step in a run with a simulated 1/64 s clock and the cores oversubscribed 4x).
The estimate is now only the starting count: it is doubled until the measured CPU time of the full parse reaches
6 steps.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…th a warning

A dict stored before velocities and constraints were kept in Cartesian angstrom (and fs) holds them as its ion
lines had them, in coordinates it doesn't record. Such a dict has no movescale key, so from_dict now reads only
what it read then (the structure and selective_dynamics) and warns about what it leaves out, instead of reading
the old values as Cartesian angstrom/fs.

Also adds a TODO for decoding arrays stored with dumpfn/MontyEncoder.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants