Skip to content

[breaking] JDFTx IO Module Overhaul pt 5 - Round-trip serialization of parsed outputs - #323

Open
benrich37 wants to merge 125 commits into
materialsproject:mainfrom
benrich37:misc_fixes_C
Open

benrich37 wants to merge 125 commits into
materialsproject:mainfrom
benrich37:misc_fixes_C

Conversation

@benrich37

@benrich37 benrich37 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

[breaking] JDFTx IO Module Overhaul pt 5 - Round-trip serialization of parsed outputs

[breaking] JDFTx IO (5/6) - round-trip serialization of parsed outputs

Summary

Merge order: this is the fifth of the six PRs that #317 is split into (misc_fixes_C). misc_fixes_A1, misc_fixes_A, misc_fixes_B0 and misc_fixes_B need to be merged first, in that order, and misc_fixes_D should be merged after it.

This PR makes parsed JDFTx outputs serializable: every output class can be dumped to JSON and loaded back unchanged. Until the PRs before it are merged, its diff also shows their changes.

Note for reviewers:

- The atomate2 change (materialsproject/atomate2#1659) needs to merge first. atomate2's JDFTx CalculationOutput.parameters stores JDFTXOutfile.as_dict() in a plain dict, and now that the dict has @module/@class, jobflow decodes it back into a JDFTXOutfile, which pydantic rejects for that field. That breaks every flow that chains atomate2 JDFTx jobs. The atomate2 change works with both old and new pymatgen-core, so it can merge first. There's a comment above the keys in JDFTXOutfile.as_dict to delete once it's in.

Major changes:

Outputs - serialization

  • feature 1: Parsed outputs round-trip (bit-exact) through dumpfn/loadfn, jobflow's jsanitize and pydantic models, and pass MatSciTest.assert_msonable
    -- Decoding used to raise KeyError: 'lattice', and vibrations runs with imaginary frequencies couldn't be dumped to JSON at all
    -- JElStep, JElSteps, JOutStructures, JDFTXOutfileSlice and JDFTXOutfile are now subclasses of MSONable, each with a from_dict
    -- Every as_dict has @module/@class and returns plain JSON types, with numpy arrays as monty's {"@module": "numpy", "@class": "array", ...} dicts (as ComputedEntry.as_dict does), so they load back as arrays
    -- JOutStructure.as_dict() is a Structure dict (lattice, sites, ...) plus the step data, and takes Structure.as_dict's verbosity/fmt arguments
    -- JDFTXOutfileSlice gained dataclass fields infile, geom_opt_label, ecomponents and the AIMD fields (pe, ke, t_k, p_bar, tmd_fs, thermostat_velocity), so as_dict keeps them
    -- JOutStructure gained annotated attributes pe, ke, t_k, p_bar, tmd_fs, igp_forces (default None), so non-MD steps give None instead of raising AttributeError
    -- Complex vibrational frequencies are stored as [real, imag], and lattices read from out-file steps are C-ordered arrays
    -- With pydantic's default ser_json_inf_nan="null", nan data values (p_bar, diverged electronic steps) come back as None; use ser_json_inf_nan="constants" for exact round trips

Tests

  • 571 -> 603 JDFTx tests
  • Every out file in test_jdftx_out_files is checked for an MSONable round trip, and an MD out file for the jsanitize and pydantic round trips

Checklist

  • Tests for the affected code pass locally (e.g. uv run pytest tests/core -k lattice
    for a lattice change — the full suite runs in CI)
    • pytest tests/io/jdftx: 603 passed
  • Lint passes: uv run ruff check . && uv run ruff format --check .
    • pyright on src/pymatgen/io/jdftx also passes, and mypy reports no error in a JDFTx file

Breaking changes?

  • This PR contains breaking changes (removals/renames, signature or
    behavior changes, changed defaults/output, dropped Python versions, ...).
    If so, prefix the title with [breaking] and describe the change and
    migration steps under ## Breaking Changes below.

Breaking Changes

All of these are in pymatgen.io.jdftx.

atomate2: atomate2's JDFTx workflows need the atomate2 change (materialsproject/atomate2#1659), which drops the top-level @module/@class keys from CalculationOutput.parameters. Without it, any flow that passes an atomate2 JDFTx job's output to a later job fails in pydantic validation. Reading stored task documents without decoding them still works.

Changed serialized outputs (as_dict):

  • JElStep, JElSteps, JOutStructures, JDFTXOutfileSlice and JDFTXOutfile dicts have @module/@class and decode back to their classes with MontyDecoder/loadfn. Numpy arrays are monty array dicts.
  • JOutStructure.as_dict() is a Structure dict (lattice, sites, ...) plus the step data, instead of its internal attributes (_lattice, _sites, ...).
  • Complex vibrational frequencies are stored as [real, imag].

Changed outputs:

  • Lattices read from out-file steps are C-ordered, so Cartesian coordinates can differ from before in the last digit.

Todos

  • Merge the atomate2 change first, then delete the comment above the @module/@class keys in JDFTXOutfile.as_dict
  • as_dict encodes nested output objects once per nesting level (_json_dict runs in each class's as_dict), which makes it about 2x slower than needed; encode once in the as_dict that was called instead (see the TODO in _json_dict)

benrich37 and others added 30 commits April 9, 2026 11:31
Transfers 62 commits of work from the benrich37 fork that were built
on top of the pre-reorganization pymatgen jdftx module. History of
individual changes is preserved in benrich37/pymatgen master branch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Import all JDFTx changes from benrich37/pymatgen fork master branch
…he first place instead of initializing as a list of None (causes a lot of annoying UserWarning's down the line)
… almost-good-enough version of "add-U" parameter to the jdftxinfile master format file
… to allow functionality, tests to ensure tag works as expected, deleting old "deleteme" file
…n order-sensitive values, enforced ordered equality in AbstractTag.is_equal_to. Added AbstractTag.read_from_tokens which by default behaves the same as `read` but also returns the number of tokens read. For TagContainer, read_from_tokens now performs a read on provided tokens until an unreadable token is encountered. Changed `TagContainer.read` behavior within subtag loop for positional subtags (subtags without written tagnames) to now call `_read_positional_subtag` (which relies on `read_from_tokens` instead of regular `read` to allow for possibly reading and discarding more than one token for a subtag. Moved initial magnetic moments validations to the JDFTXInfile class. Added a `get_species_values_tag_container` convenience function for initializing "initial-magnetic-moments" and "initial-oxidation-state" as a regular TagContainer instead of the old two-purpose InitMagMomTag. Removed `InitMagMomTag` as it is no longer needed. Added docustrings to helper functions in io/jdftx/inputs.py missing docustrings.
…lues are caught. Structure of "add-U" tag changed to contain a single "U-channels" subtag, which contains the expected structure of the "add-U" tag - this is done to allow JDFTx's behavior of not allowing multiple add-U lines, but obviously accepting multiple species to have add-U values described in that one line Changes made to `TagContainer.read_from_tokens` to prevent possible subtag keys from being read as values.
…e removal_path with _get_removal_path, and call _remove_at_path for each removal_path to remove that bad tag/subtag - prevents attempting to remove duplicate paths, and prioritizes deeper subtags and later pop indices to protect from removing tag paths with parents already removed and to protect from index shifting.
…att-scale subtags being specified as IntTag's instead of FloatTag's, added tests for converting named lattices to pymatgen Lattice objects.
…uations

Lines ending in "\" are now joined with the following line by a single re.sub in JDFTXInfile.from_str. The previous
_gather_tags implementation is kept commented out in case the regex approach is not accepted. Added
test_line_continuation to check whitespace/comment handling around continuations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… function, adding testing for selective_dynamics_site_prop_to_jdftx_interpretable
…behavior can be reproduced in a single line and is only called once. Added test to test_jdftxinfile/test_jdftxstructure_infile_site_properties_conversion to ensure behavior is kept the same
…anmaps` which cleaned the values well enough itself such that the function could be replaced by np.nan. Removed `io.jdftx.inputs.get_popmaps`, since JDFTXInfile's dict inheritance allows `get_nanmaps` to operate directly on a JDFTXInfile.
…r `next` iteration. Added test to `io.jdftx.test_jdftxinfile` to ensure label fallback system performs as expected. Removed _multi_has/getattr functions and testing as they are no longer being used.
…ure.get_str` in `io.jdftx.inputs`, adding test to `test_jdftxinfile` to ensure changes to unit-converting code still works as intended.
…ded testing to ensure specie names of pmg Structure objects are left intact in JDFTXStructure.structure
…ding test to ensure hyperplane data can be safely passed back and forth between JDFTXStructure and JDFTXInfile tag reading/writing
…ystems for constraint vectors, fixed constraints being lost when a JDFTXInfile was initialized from a structure with constraints encoded into site properties, protect. Added error raising for some constraints which JDFTx would reject. General fixes to old misconception that ion prints will always be homogeneous shaping (this assumption breaks for some special case constraint types).
… from standard JDFTx units to A / fs + testing
…m on top of regular parsing to allow `JDFTXOutfile.to_jdftxinfile` to exactly reproduce the ion coordinates without risk of rounding. On top of this, structural tags are now stored to 15 decimal precision (previously 12) to match JDFTx's output. `to_jdftxinfile` now also retains the coords-type of the originating output file. Fixed bug where JDFTXInfile.strip_structure_tags stripped a non-existent "lattice-scale" instead of "latt-scale". Fixed `_infile_special_format_to_pmg_lattice` which previously used pymatgen's built in Lattice construction - there is unfortunately a mismatch between pymatgen and JDFTx, where pymatgen begins by orienting the third lattice vector along the z axis, while JDFTx begins by orienting the first lattice vector along the x axis. Fixed bug in `condense_jdftxinputs` where any vestigial structure tags in the jdftxinput argument would remain in the condensed JDFTXInfile object.
benrich37 and others added 9 commits November 19, 2026 13:37
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The fix needs an atomate2 update (two atomate2 tests index ["fluid-cation"] as a dict), so it goes to the end of the
split, after the branches that do not: misc_fixes <- A1 <- A <- B0 <- B <- C <- D.

Taken out here, and put back unchanged by misc_fixes_D:
- can_repeat=True on tau-core, fluid-cation, fluid-anion, dump-Eresolved-density, elec-ex-corr-compare and
  dump-fermi-density (jdftxinfile_master_format.py)
- the fluid ions in _fill_default_values (inputs.py), which returns their value as a list; here it handles
  fluid-solvent only
- tests: test_repeatable_commands, test_repeated_commands_in_jdftx_echo_are_all_read,
  test_repeatable_tags_match_jdftx, test_adding_infiles_appends_repeatable_commands,
  test_fluid_ion_echo_is_read_and_written_back, the elec-ex-corr-compare case of
  test_hse06_is_accepted_with_a_warning, and the list form of two expected values in inputs_test_utils.py

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Conflict in tests/io/jdftx/test_jdftxinfile.py (adjacent import lines): kept this branch's JDFTXOutfile import and
took misc_fixes_A's import line without MASTER_TAG_LIST.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
benrich37 and others added 2 commits November 19, 2026 14:59
test_JDFTXOutfile_skimming failed on Windows CI with `assert 0.046875 < 0.046875`: time.process_time advances in
steps of 1/64 s there, about the CPU time of one parse of these files, so a skimmed and a full parse could read the
same. Each CPU time can be off by one step, so a speedup is now only tested where it is at least two steps:
- Only the skims that leave out most of the file are timed: whole out file slices if there are several, otherwise
  geometry steps. The two geometry-only combinations of example_latmin.out are no longer timed.
- Each timing is of enough parses for the full parse to take 6 clock steps.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@shyuep

shyuep commented Nov 20, 2026

Copy link
Copy Markdown
Member

Automated PR review generated by Claude (posted from @shyuep's account). Based on code and CI status only, nothing was run. Please sanity-check before acting.

CI: all 5 checks green on 4002007. Reviewed this PR's own commit (cc52f97: _output_utils.py, jdftxoutfileslice.py, jelstep.py, joutstructure(s).py, outputs.py, ~200 source lines) in full; the rest of the diff is #319-#322.

Good:

  • _OutputMSONable overriding the pydantic serialization schema so that embedded dataclasses dump through as_dict fixes the "pydantic dumps field by field and can't load it back" problem at the root.
  • from_dict on JDFTXOutfileSlice loads the echoed input with validate_value_boundaries=False and suppresses JDFTXOptionWarning. That is the right behaviour for JDFTx's placeholders.
  • Complex vibrational frequencies are stored as [real, imag] and restored. Imaginary modes would otherwise be lost in JSON.

Substantive:

  1. Merge blocker: the atomate2 change is still "LINK TBD". As the description says, @module/@class on JDFTXOutfile.as_dict breaks chained atomate2 JDFTx flows. This must not merge until that PR is merged and released, and the link should be filled in.
  2. _none_to_nan (_output_utils.py) turns every None in the stored infile dict into nan, on the premise that parsing never stores None. Any tag whose stored value is legitimately None, now or later (an optional StrTag, a None inside a list such as an absent constraint), would become nan and then fail type validation or be written as nan. Restrict the conversion to numeric tags, e.g. by walking the tag tree with get_tag_object and converting only where the tag is a FloatTag.
  3. as_dict returns _json_dict(...) (json.loads(json.dumps(..., cls=MontyEncoder))) at every level: JElStep -> JElSteps -> JOutStructure -> JOutStructures -> slice -> outfile. Each level re-serializes the already-plain dicts below it, so a long AIMD file is encoded and decoded about five times. Doing the JSON conversion once, at the outermost as_dict (or encoding only the arrays at each level), would avoid this.

Nits:

  • _OutputMSONable.__get_pydantic_core_schema__ imports pydantic_core lazily, which is correct since pydantic is optional. A one-line comment saying so would stop someone from hoisting the import.

benrich37 and others added 9 commits November 20, 2026 15:00
… time

The number of parses per timing was estimated from the wall time of one parse, while the times compared are CPU
times. Wall time can exceed CPU time under load, which leaves the full parse spanning fewer than the intended 6
steps of a coarse CPU clock (1 step in a run with a simulated 1/64 s clock and the cores oversubscribed 4x).
The estimate is now only the starting count: it is doubled until the measured CPU time of the full parse reaches
6 steps.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…th a warning

A dict stored before velocities and constraints were kept in Cartesian angstrom (and fs) holds them as its ion
lines had them, in coordinates it doesn't record. Such a dict has no movescale key, so from_dict now reads only
what it read then (the structure and selective_dynamics) and warns about what it leaves out, instead of reading
the old values as Cartesian angstrom/fs.

Also adds a TODO for decoding arrays stored with dumpfn/MontyEncoder.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@benrich37

Copy link
Copy Markdown
Contributor Author
  • Substantive 1 (atomate2): Link filled in (Compatibility with incoming update to JDFTx pymatgen-core IO module atomate2#1659), and agreed on the order. The atomate2 change works with both old and new pymatgen-core, so it can merge first. It is open, and its CI run is waiting for a maintainer's approval.
  • Substantive 2 (_none_to_nan): Leaving as is for now. It only runs on the echoed input of a stored slice, where parsing never stores None (a subtag that isn't given is left out instead), and none of the 30 out-file fixtures' echoed inputs holds one. If a tag ever does store None there, restricting the conversion to float tags as you describe is the fix.
  • Substantive 3 (as_dict re-encoding): Agreed, I measured as_dict at about 2x slower than it needs to be. Deferring to a follow-up to keep this PR from growing: there is a TODO in _json_dict (2fd2b5c), and it is listed under Todos in the description.
  • Nit (lazy pydantic_core import): Leaving as is.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants