[banjtheman]

Related issue

Closes #843

Refs #112 (open decision on how DeltaTrack reaches staffers). This ships one of the channels listed there, a fully in-browser page with no server, but doesn't make the decision.

What does this change?

DeltaTrack can now be downloaded as one HTML file (about 9 MB) that compares two versions of a bill's XML on the user's own computer. They double-click it and pick the older and newer versions. It needs no install and no internet, and nothing is uploaded. The report is the same one the command line writes, byte for byte. This gives people who can't install Python a way to compare confidential drafts, which the hosted upload app can't take (ADR 0011: user-provided bill content never leaves the user's machine).

• The build: scripts/build_offline_html.py writes deltatrack-offline.html from web/offline/template.html. It inlines Pyodide 314.0.7 (Python compiled to WebAssembly; the npm tarball is hash-pinned) plus the deltatrack package and the upload pages' stylesheet.
  • A page opened from file:// may not fetch neighbouring files. So assets are gzip+base64 inlined and handed to Pyodide through its own options (createPyodideModule, lockFileContents).
  • The wasm binary and the stdlib are answered from a fetch wrapper.
  • The browser enforces "no network". A Content-Security-Policy meta sets default-src 'none' and connect-src 'none', with only the inline-script, blob: and 'wasm-unsafe-eval' allowances Pyodide needs. The report frame is sandboxed without allow-same-origin, so its script can't reach the page.
  • Any encoding the XML parser reads. Only a PDF is refused up front, by name or by its %PDF- bytes. UTF-16 bill XML compares like UTF-8.
• XML only. pypdfium2 has no Pyodide build, so the bundle carries stand-ins (the XML path still imports it, The XML comparison cannot load in a browser because it imports a PDF library it never uses #751) and the page refuses a PDF up front.
• Publishing: new workflow .github/workflows/publish-offline.yml. On push to main it attaches the file to a rolling offline-latest release. That gives a stable one-click link: releases/download/offline-latest/deltatrack-offline.html.
  • A build job with contents: read builds the file once, tests that exact file (--offline-page) and passes it on as an artifact. A publish job holds contents: write and runs nothing but gh.
  • The new file uploads under a temporary name (up to three attempts). Only once it has landed is the old one replaced and the tag moved, so a failed upload leaves the old file in place.
  • Not the Pages job, which must stay publish-only (Remove vestigial example reports page and old-style reports #42, The release runbook can go stale against the workflows it describes, with nothing to catch it #547).
  • Not a committed file, which would be 9 MB and rewritten on every engine change.
  • The publish job has contents: write. That's the outward-facing part, and I'd appreciate a security look per AGENTS.md (I can't add labels).
• PR CI runs the engine under Pyodide: ci.yml's browser job caches the hash-pinned Pyodide tarball and runs the network-marked browser tests too, so a develop change that breaks the page fails before promotion.
• Home page: the "Browser-only — Coming soon" card becomes "Download and run it yourself", linking to that release asset. The upload page now points people with drafts at it. The three CSS rules only the disabled card used are removed, and the regenerated token block drops --muted.
• Found along the way: perf: call the glyph loop's two PDFium functions without ctypes argument checks #794 (call two PDFium functions without ctypes argument checks) wraps PDFium bindings with ctypes when parsers/pdf_text.py is imported. That broke loading the engine under Pyodide, and it broke scripts/pyodide_parity.py too, which isn't in CI. The first commit gives the parity check a pypdfium2.raw stub that ctypes can wrap. It's a separate commit, so it can come out into its own PR if you'd rather.
• Docs: docs/release.md now lists three public surfaces and five post-merge runs, and describes the two-job publish. docs/web-compare.md has a section on the offline page. scripts/README.md and TESTING.md are updated too.

Deliberately left out:

• PDF support in the offline page.
• Making the XML path stop importing PDFium (The XML comparison cannot load in a browser because it imports a PDF library it never uses #751).
• Which channel staffers get (Open decision: how DeltaTrack reaches staffers (delivery channel), waiting on Hill IT, workflow and packaging signals #112).
• The README's entry point for non-developers (The README sends readers who only want to see output to the command line, not to the example reports or the hosted app #608).

How to test

uv run python scripts/build_offline_html.py          # -> dist/deltatrack-offline.html (8.8 MB)
open dist/deltatrack-offline.html                    # pick tests/corpus/118-hr-4366/{3,4}_*.xml
uv run pytest tests/test_offline_html.py --run-browser --run-network -v
uv run pytest tests/test_offline_html.py --run-browser --run-network --offline-page dist/deltatrack-offline.html

What I ran and looked at (macOS, Chromium via Playwright; updated after review):

• Opened from file:// (origin null), under the Content-Security-Policy: ready in about 1.4 s.
  • 118-hr-4366 v1→v2 compared in 0.1 s, and v3→v4 (the Senate rewrite, 703 changes) in 0.6–0.7 s.
  • Both reports are byte-identical to compare_xml_html with the filename-derived labels, from UTF-8 and from UTF-16 input.
  • Zero requests other than file:/blob:. With a local server counting hits, fetch, XMLHttpRequest and an Image load from the page and from the report frame all reach it zero times, and the report can't read parent.document.
  • Full bill, find, the diff.json download and Copy work inside the sandboxed frame.
  • A PDF is refused with "…is a PDF. This offline version compares bill XML only."
• Home page: checked at uvicorn web.app:app --port 8077. The left card reads "Download and run it yourself".
• Parity check: scripts/pyodide_parity.py --node-dir <dir> on develop fails with "Maximum call stack size exceeded" in ctypes.cast from pdf_text.py:575. On this branch: 3 fixtures × 2 artifacts IDENTICAL, and --mutate reports MISMATCH on every HTML row.
• New tests fail when broken:
  • Reverting the raw stub to the old tripwire turns test_the_engine_bundle_imports_the_xml_path_with_pdfium_stubbed red.
  • Renaming a key in PYODIDE_ASSETS turns test_every_asset_the_page_unpacks_is_built_into_it red.
  • The browser test went red while the page couldn't start.
  • Restoring the ASCII-only XML check turns the UTF-16 case red. Removing the policy, or the frame's sandbox, turns the blocking test red.
• CI gates, all locally:
  • ruff check, ruff format --check: clean.
  • Fast: 2373 passed.
  • Browser (--run-browser --run-network, as CI now runs it): 33 passed. With --run-browser alone: 30 passed and 3 skipped, the network tests declared in ALLOWED_DEFAULT_SKIPS.
  • Slow (-m slow --deselect tests/test_govinfo_corpus_parity.py): 2231 passed.

Not verified:

• The publish workflow itself, which only runs on main.
• Firefox, Safari, and the Windows "downloaded from the internet" warning.

Known follow-ups, also raised in review:

• The home page link 404s until publish-offline.yml has run once on main.
• engine_zip globs src/deltatrack rather than unpacking the built wheel, so a local build can pick up untracked files, and the bundle has no dist-info (low priority in review; CI builds from a clean checkout).
• If a future XML-path change ever called PDFium, the ctypes-callback stub would return 0 silently rather than raise. The browser parity test would catch that before a publish.

I can't attach images from the CLI; I'll drag before/after screenshots of the home page and a short demo video into a comment.

Checklist

• Linked the issue above (Closes #...)
• Ran the CI gates locally and they pass (see What CI checks)
• New or changed behavior has tests
• For a bug fix: the test fails without the fix, and I ran it both ways to check
• Disclosed AI assistance below, if any

AI assistance

Written with Claude Code (Claude Opus 5.5) at Civic Tech DC night; I reviewed the change and ran the checks above.

🤖 Generated with Claude Code

[willhea]

@banjtheman, thanks for getting the offline delivery path working! I tested with Chromium 138: all five offline-page tests passed on the PR branch and its merge with current develop. Three committed bill pairs produced byte-identical browser/native reports, and report controls and saving worked.

I found one small compatibility issue before approval: looksLikeXml() rejects valid UTF-16 XML because it checks for an ASCII < byte and only recognizes a UTF-8 BOM. The same bill, saved as UTF-16 with the appropriate declaration, compares successfully through native compare_xml_html but gets “doesn’t look like an XML file” in the page.

Please let the XML parser handle supported encodings, while preserving the clear PDF rejection.

The behavior to preserve is that supported bill XML compares identically through the offline and native paths. A focused browser regression using UTF-16 input should verify that equality; restoring the current ASCII-only check should make it fail.

Once that correction passes the focused checks, this looks ready for approval. PDF support can remain separate.

*Comment/evaluation from GPT Sol 6.1 Effort High

[willhea]

I built this branch locally and ran it in a browser. It does what the description says. The report matched compare_xml_html byte for byte (SHA-256) on 118-hr-4366 v1 to v2 (0.1 s) and v3 to v4 (0.7 s), and startup took 1.5 s. Bad input gets a clear message every time and the page recovers afterwards: a PDF, a PDF renamed to .xml, non-bill XML and malformed XML all behave. The only things the page itself loaded were its HTML and two blob: module imports.

The browser pane I used can't open file://, so I served dist/ from http://localhost for the manual run. The from-disk case is covered by test_the_page_opened_from_disk_matches_the_command_line, which ran and passed here with --run-browser --run-network, and was not skipped. The fast tier gives 2291 passed, the same count as in the description, and ruff check and ruff format --check are clean.

The inline comments below are ordered by severity. Three are marked Confirmed locally: I reproduced those in the browser. The rest come from reading the diff. The first two are the ones I'd want settled before this merges, since the whole point of the page is that confidential drafts stay on the machine.

🤖 Generated with Claude Code

[willhea]

engine_zip decides what goes in the bundle by globbing the package directory. Hatchling already defines what the package ships. Globbing src/deltatrack means a local build picks up whatever untracked files are sitting there, such as scratch files, .DS_Store or local data, and puts them into a file meant for distribution. CI builds from a clean checkout, so this mostly affects local builds.

The bundle also has no dist-info. That means importlib.metadata.version("deltatrack") (used in pdf_observations.py:215) would raise PackageNotFoundError under Pyodide if any XML-path code ever reached it. Building with uv build --wheel and unpacking the wheel fixes both problems. Low priority.

[banjtheman]

Agreed, but left for a follow-up since you marked it low priority. CI builds from a clean checkout, so the published page isn't affected today. I can fold it in here if you'd rather.

[banjtheman]

@willhea, thank you for building it, running it in a browser and reproducing three of these. All three commits are on the branch now (b3b2813, 792221f, f31d026), on top of a merge of current develop:

• Network (must-fix). The page now declares a Content-Security-Policy, so the browser enforces it: default-src 'none' and connect-src 'none', with only the inline script, blob: and 'wasm-unsafe-eval' allowances Pyodide needs. A new browser test opens the page from disk with a local HTTP server counting hits. It runs fetch, XMLHttpRequest and an Image load from the page and from inside the report frame, and asserts zero hits. Removing the policy turns it red.
• Write token (must-fix). publish-offline.yml is now two jobs. build has contents: read: it syncs, builds once, tests that file and uploads it as an artifact. publish has contents: write, downloads the artifact and runs nothing but gh.
• UTF-16 (your earlier comment). looksLikeXml is gone. The page rejects only a PDF (by name or by its %PDF- bytes) and leaves every other encoding to the XML parser. The from-disk browser test now runs on the same bill as UTF-8 and as UTF-16 and checks both reports are byte-identical to compare_xml_html. Restoring the ASCII check turns the UTF-16 case red.

The other items are answered inline. One is deferred: engine_zip via the wheel.

Gates on the final tree:

• ruff check and ruff format --check: clean.
• Fast: 2373 passed.
• Browser with --run-browser --run-network: 33 passed.
• Browser with --run-browser only: 30 passed, and the 3 network tests are declared skips.
• Slow: 2231 passed.

Opened from file:// in Chromium, the page starts in 1.4 s under the policy. Full bill, find, the diff.json download and copy to clipboard all work inside the sandboxed frame.

🤖 Generated with Claude Code