Skip to content

fix(watch): bound each watch rebuild with GRAPHIFY_REBUILD_TIMEOUT, with a Timer fallback where SIGALRM is missing (#2173) - #4724

Closed
danieldutu wants to merge 1 commit into
Graphify-Labs:v8from
danieldutu:fix/watch-rebuild-timeout
Closed

danieldutu wants to merge 1 commit into
Graphify-Labs:v8from
danieldutu:fix/watch-rebuild-timeout

Conversation

@danieldutu

Copy link
Copy Markdown
Contributor

Defect
On v8 (92741d1), graphify/watch.py contains 0 references to SIGALRM, threading.Timer or GRAPHIFY_REBUILD_TIMEOUT. graphify/hooks.py contains 2 of each, one per embedded rebuild body. A rebuild that hangs inside graphify watch therefore blocks the watch loop with no upper bound. This is the remaining half of #2173.

Root cause
graphify/watch.py:2850 calls _rebuild_code(watch_path) outside any timeout guard. The hooks guard the same call at graphify/hooks.py:200-228: signal.alarm where SIGALRM exists, a threading.Timer where it does not (#2148). watch.py has neither path.

Fix
I added _rebuild_timeout() to graphify/watch.py and wrapped the loop's _rebuild_code call in it. It has the two paths the hooks have:

  • SIGALRM available: the handler kills multiprocessing.active_children(), then raises TimeoutError. The loop prints [graphify watch] <message> and keeps watching. The previous handler is restored on exit.
  • SIGALRM unavailable (Windows): a daemon threading.Timer kills the children and calls os._exit(1). The watcher process exits.
  • GRAPHIFY_REBUILD_TIMEOUT=0 disables the guard. The default is 600, as in the hooks.

Evidence

pytest tests/test_hooks.py -q                              178 passed   (after rebase onto 92741d1)
pytest tests/test_hooks.py -q -k watch_rebuild_timeout     1 failed, 172 deselected   (watch.py reverted to v8)
pytest tests/ -q                                           8265 passed, 200 skipped, 7 failed   (before rebase; not rerun on 92741d1)
ruff check .                                               All checks passed!
  • The new test, test_watch_rebuild_timeout_arms_a_timer_without_sigalrm, parses watch.py with ast. It asserts that the Timer fallback sits in the else-branch of the SIGALRM check, that _bail kills children before os._exit, and that watch() references _rebuild_timeout.
  • The 7 failures are all in tests/test_ext_tagged_id_clash.py. The same file gives 7 failed, 7 passed on v8 with this change stashed.
  • pyright reports errors in my environment. watchdog is not installed, so graphify/watch.py reports 3 unresolved watchdog.* imports. Other files report errors unrelated to this change. No error falls on a changed line. I did not compare the total against v8.

Behavior change
POSIX graphify watch now has a 600 s default rebuild timeout where it had none. This matches the 600 default in the hooks. On timeout the POSIX watcher logs and continues, and the Windows watcher exits. GRAPHIFY_REBUILD_TIMEOUT=0 restores the previous behavior.

Boundaries and trade-offs

  • I did not run the Windows path on a Windows host. The test is structural, like the hook tests it follows.
  • I did not reproduce a hung rebuild, so I have no timing for the timeout firing.
  • I left the hook bodies untouched. They are string literals inside a shell -c argument, and a shared import would change that carrier. A second copy of the 25 lines in watch.py was the alternative; a context manager was shorter and restores the handler.
  • The Timer path uses os._exit(1) because no other call interrupts a main thread blocked in a rebuild when SIGALRM is missing.

Question
I kept the 600 s default to match the hooks. Do you want watch to default to 0 instead, so POSIX behavior stays unchanged? I can change the default or split the Windows fallback into its own PR.

…ith a Timer fallback where SIGALRM is missing (Graphify-Labs#2173)

graphify/watch.py:2845 calls _rebuild_code(watch_path) with no timeout. A hung
rebuild blocks the watch loop with no upper bound. The hooks bound their
rebuild by GRAPHIFY_REBUILD_TIMEOUT (graphify/hooks.py:200-228), and their
SIGALRM path does not exist on Windows.

I added _rebuild_timeout() to graphify/watch.py and wrapped the watch loop's
_rebuild_code call in it, with the same two paths as the hooks. With SIGALRM,
the handler kills multiprocessing.active_children() and raises TimeoutError;
the loop logs it and continues. Without SIGALRM, a daemon threading.Timer
kills the children and calls os._exit(1). A timeout of 0 disables the guard.

tests/test_hooks.py gains one AST test of the fallback shape. It fails when
graphify/watch.py is reverted to v8.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Thanks for the pull request, @danieldutu. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 4 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; formal verification proved 0 of 1 changed function(s) (1 not verified).

Formal verification. PR-changed functions: 0/1 verified (0 proven, 0 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).

Not verified on this run: watch (vacuous: never exercised).


Graphify review — findings

Adds a GRAPHIFY_REBUILD_TIMEOUT bound (default 600s, 0 disables) around each graphify watch rebuild, so a hung rebuild no longer stalls the loop indefinitely. On POSIX a SIGALRM handler kills the pool workers and raises TimeoutError, which the loop catches and logs before carrying on. Where SIGALRM doesn't exist (Windows), a daemon timer kills the workers and calls os._exit(1), so a timeout there ends the whole watcher rather than skipping one rebuild. The timeout value is parsed with a bare int() on the env var, and the alarm can interrupt _rebuild_code at any point, including mid-write. A new AST-based test checks that the timer fallback sits in the no-SIGALRM branch, that workers are killed before os._exit, and that watch() wraps its rebuild in _rebuild_timeout.

Worth a look

  • Malformed GRAPHIFY_REBUILD_TIMEOUT crashes the watch loop — graphify/watch.py:2762 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
  • Windows fallback terminates the long-running watcher via os._exit — graphify/watch.py:2783 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
  • Async TimeoutError can interrupt _rebuild_code mid-write — graphify/watch.py:2770 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
  • Rebuild timeout raises outside the main thread on POSIX — graphify/watch.py:2767 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 818 functions depend on the 343 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: watch() — 6 callers, 11 callees

Verification — 818 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 589 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

27 of 453 test file(s) selected (6%) via static blast radius.

  • tests/test_cache_stale_import_target.py — impact
  • tests/test_compact_graph_json.py — impact
  • tests/test_elixir_import_resolution.py — impact
  • tests/test_elixir_qualified_calls.py — impact
  • tests/test_elixir_unqualified_call_scope.py — impact
  • tests/test_export_direction.py — impact
  • tests/test_external_stub_endpoints.py — impact
  • tests/test_hooks.py — impact, changed-test
  • tests/test_incremental_unscanned_placeholders.py — impact
  • tests/test_java_member_calls.py — impact
  • tests/test_kotlin_grammar.py — impact
  • tests/test_labeling.py — impact
  • tests/test_language_override_lifecycle.py — impact
  • tests/test_lost_saved_semantic_nodes.py — impact
  • tests/test_markdown_code_spans.py — impact
  • tests/test_no_cluster_undeclared_endpoints.py — impact
  • tests/test_pr10_watch_overrides.py — impact
  • tests/test_python_inherited_method_calls.py — impact
  • tests/test_python_property_reads.py — impact
  • tests/test_python_pytest_fixture_receivers.py — impact
  • tests/test_rust_use_trees.py — impact
  • tests/test_skill_persisted_scan_options.py — impact
  • tests/test_solidity_inherited_calls.py — impact
  • tests/test_terraform_modules.py — impact
  • tests/test_watch.py — impact
  • tests/test_watch_identity_cache.py — impact
  • tests/test_watch_manifest_location.py — impact

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Risky patterns (advisory)

  • medium no-stderr-to-stdout in graphify/watch.py:2898: error printed to stdout instead of stderr: print(f"[graphify watch] {exc}")

Formal verification

Could not verify: Could not verify watch.

The verifier did not have enough to check watch, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 142 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly ImportError — names the real obstacle, not a sampling gap)

· 1 grounded finding(s) anchored inline below.

Comment thread graphify/watch.py
watchdog.cancel()


def watch(watch_path: Path, debounce: float = 3.0) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — watch()

fans out to 11 callees (efferent coupling); 6 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

@nishal21

Copy link
Copy Markdown

/firstpr explain

safishamsi added a commit that referenced this pull request Oct 26, 2026
… thread

#4724 wrapped each watch-loop rebuild in _rebuild_timeout(), which arms
SIGALRM. signal.signal() only works on the main thread and raises
ValueError('signal only works in main thread') anywhere else. watch()'s
dispatch loop is run in a worker thread both under test
(test_watch_handler_honors_graphifyignore, gh-928) and when embedded, so
that ValueError escaped the loop's 'except TimeoutError' guard and tore
the watcher down before _rebuild_code ran: a non-ignored .py write never
triggered a rebuild.

Gate the SIGALRM path on threading.current_thread() is main_thread();
off the main thread, use the existing signal-free Timer fallback, which
bounds the rebuild just the same. Adds a behavioral regression test that
runs _rebuild_timeout() off the main thread and asserts it yields without
raising.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@safishamsi

Copy link
Copy Markdown
Member

Landed in v0.9.96, now on PyPI (pip install graphifyy==0.9.96). Each watch rebuild is bounded by GRAPHIFY_REBUILD_TIMEOUT with a Timer fallback. Thanks for the fix, @danieldutu.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants