Repository navigation
feat(pipeline): wire Ruby cross-file LSP resolution - #1712
benrfairless wants to merge 20 commits into
Conversation
In-process, type-aware call resolution for single Ruby files, mirroring the perl_lsp.c / php_lsp.c shape and dispatched from cbm_extract_file: - PASS 1 collects classes and modules with lexical nesting, superclasses, and include / prepend / extend mixins, plus method tables with instance and singleton methods classified from the AST. Singleton methods key on "<class_qn>.self", which cannot collide with a constant path. - PASS 1.5 infers instance-variable types from `@x = Const.new`; conflicting assignments latch to no type rather than guessing. - PASS 2 resolves constructors (`Const.new` targets the class node, the way Python's lsp_constructor does), implicit and explicit self dispatch, singleton methods, superclass chains, mixin lookup in Ruby's method-resolution order, super, and ivar / local / chained receivers. The stdlib seed covers Ruby core plus a curated Rails surface, including ActiveRecord query typing: `User.find(1)` is a User, and relation chains keep the model type. Dynamic dispatch (send, method_missing, define_method) and unresolved receivers emit no edge, and only project-defined targets are emitted. Cross-file resolution is a separate tier and is not part of this change; the 18 rows here drive the resolver through cbm_extract_file exactly as the extraction pipeline calls it. PROVENANCE: this resolver was authored clean-room from the tree-sitter-ruby grammar and Ruby language semantics. No Ruby language server source was consulted. scripts/check-lsp-originality.sh gains Shopify's ruby-lsp (MIT) as the Ruby reference so the boundary is auditable alongside the generated data it guards; `check-lsp-originality.sh --lang ruby` reports CLEAN — no verbatim string or comment overlap and no structural clones. Refs DeusData#1701 Assisted-by: OpenCode:anthropic/claude-opus-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
Adds the cross-file tier on top of the per-file resolver. cbm_run_ruby_lsp_cross builds its class table from the project-wide CBMLSPDef[], deriving each constant path by stripping the def's module QN prefix, and registers methods on both the instance and singleton receiver keys because CBMLSPDef does not carry the `def self.m` split. Ruby is registered on the fallback cbm_pxc_run_one path, like Java and Kotlin — there is no prebuilt tier-2 registry yet. Ruby also joins Rust in bypassing the own+imports module-def filter, for a different reason: Ruby constants live in one global namespace and Rails/Zeitwerk autoloads them with no require statement, so a filter keyed on requires would starve essentially all cross-file resolution in a real Ruby app. That exemption is the one deliberate design decision here. The two new rows drive cbm_run_ruby_lsp_cross with a hand-built CBMLSPDef[] in the shape pass_lsp_cross.c passes it, the second mirroring the Rails-shaped e2e fixture (User < ApplicationRecord < ActiveRecord::Base, a service object, and a caller in a third file). Refs DeusData#1701 Assisted-by: OpenCode:anthropic/claude-opus-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
|
Reviewed slice 4. The wiring is right and the Zeitwerk argument is correct. There is one consequence of it I want measured before this lands, and you have already named half of it yourself. The filter exemption is justified. Ruby constants live in one global namespace and Rails/Zeitwerk autoloads them with no The consequence: Ruby has the Rust exemption without the Rust mitigationRead the branch you edited alongside its own comment:
Rust survives the exemption because the registry is its normal path and the full per-file build is its fallback. Your PR description says the quiet part plainly: "there is no prebuilt tier-2 registry yet." So for Ruby the fallback is the only path. Following it through:
So every Ruby file rebuilds a class table over the entire project's defs. That is F × D, and D grows with F — the corpus-proportional shape, and precisely what "a language without a Tier-2 shared registry is automatically quadratic" describes. A Rails app is exactly the kind of repo where F is large. I want to be honest about the limits of this: I am reasoning from the call structure, not from a measurement. It may be perfectly acceptable at real Ruby project sizes — the constant factor is small and the class table is cheap per def. But that is a claim to check rather than assume, and right now nothing checks it. What I would likeEither a measurement, or — better, because it is permanent — a Ruby fixture under The complexity suite auto-discovers fixture directories at that path ( If the ratio comes back at ~2, this merges and the fixture stops the next person reintroducing the coupling. If it comes back at ~4, better to know now than after a Rails user reports a slow index. Smaller notesThe split is clean and I checked the part that is easy to get wrong — Registering methods on both instance and singleton receiver keys because Clearance: the two flagged files here ( Baseline note: the |
DeusData
added
enhancement
The structural extractor names a class by appending the declaration's raw name text to the enclosing class QN, so `class Admin::User` yields `<module_qn>.Admin::User` while `module Admin; class User` yields `<module_qn>.Admin.User`. The resolver normalised both to the dotted form, and because the pipeline joins resolved rows to defs by exact caller QN, every row from or to a compact-declared class was silently dropped — the most common Rails style. All three passes now enter a class through one shared frame helper that computes the extractor-shaped QN from the raw name and keeps the dotted path only for constant lookup. The class table dedupes on QN. Nesting is now one raw frame per declaration joined by '.', and the constant probe strips one frame at a time: inside `class A::B` only `A.B.x` and `x` are tried, never `A.x`, matching Ruby's lexical scoping (`class A::B` does not open A). Tests pin exact QN equality on both ends of the row for compact, nested-compact, and the outer-scope-not-opened case. Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
…ignment A `def` body inherited the enclosing scope, so a typed local from the surrounding body (or a preceding sibling method) typed a same-named parameter or local inside it. Parameters and block parameters were never bound, so they could not shadow an outer local either. And an assignment whose right-hand side had no proven type was ignored, so `a = Foo.new; a = x; a.bar` still resolved to Foo#bar. Ruby scoping is now modeled explicitly: `def`, `class` and `module` bodies start from a fresh, unparented scope; blocks and lambdas get a chained child scope; method, block and lambda parameters (including destructured groups and block-local names) bind to UNKNOWN; and any write the resolver cannot type — plain reassignment, multiple assignment, `||=` on an already-typed name, other compound operators, or an assignment inside a block — rebinds the nearest existing binding to UNKNOWN. `a ||= Foo.new` on a fresh name still types as Foo. Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
PASS 1.5 only recorded `@x = Const.new` and ignored every other write, so `@x = compute` (or a parameter, a compound operator, a multiple assignment, or an unresolved constant's .new) left the ivar typed by whichever constructor it saw elsewhere. Any such write now latches the conflict, exactly as two different constructors already did; `nil` is neutral. `@x ||= Const.new` — the memoised-constructor idiom — counts as a constructor assignment. Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
instance_eval/instance_exec, class_eval/class_exec, module_eval/ module_exec, define_method and Class.new/Module.new run their block with a different `self`, but bare and `self.` calls inside them were dispatched on the enclosing class. The eval call itself was already typed unknown; the walk into its block was not. The block of such a call is now walked with self_unknown set: bare calls, `self.` calls and `super` emit nothing there, `self` has no type, and PASS 1.5 does not attribute ivar writes inside it to the enclosing class. Receiver and argument expressions of the same call still resolve, and the suppression ends with the block. The *_exec forms join the dynamic-dispatch list. The existing negative test passed only because Meta had no `hidden`; it now has one, and the calls are written with `()` because tree-sitter-ruby parses a lone identifier as a variable reference, which the extractor never emits as a call. Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
The lookup walks skipped a mixin whose module did not resolve and carried on to the class's own methods and superclass chain, and a bare call whose class ancestry did not resolve fell through to a same-named top-level def. An unseen ancestor may define or shadow the method, so none of those answers is trustworthy. Every lookup now carries a `blocked` flag set the moment the walk meets an unresolved prepend, include, extend or superclass reference, in Ruby's lookup order: an unresolved prepend blocks even an own-method hit; an unresolved include (or superclass) stops the walk before the superclass chain; an unresolved extend blocks the singleton walk. When blocked, nothing is emitted — no Object typing fallback, no top-level fallthrough, no `super` edge. Hits found before the unseen ancestor are still returned (an own method ahead of an unresolved include; a resolved include listed after an unresolved one). Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
PASS 1 recursed through blocks and registered any `def` it met on the enclosing class or module, so ActiveSupport's `class_methods do … end` and `included do … end` bodies became instance methods of the concern and were then found through `include Concern` — edges to defs the structural extractor does not even emit. `include X` written inside such a block was likewise recorded on the concern instead of the includer. PASS 1 now tracks the block depth since the nearest class/module body and, inside a block, registers no methods and collects no mixins. In PASS 2 a class-body block def is skipped outright (no def exists for it to join), and a top-level block def (an RSpec group) is walked with self unknown: typed-receiver calls in its body may still resolve, bare and self. calls emit nothing, and other top-level code cannot reach it. Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
The nested-modules row matched caller and callee by substring, which would have let a wrongly-spelled class path pass; it now pins the full extractor QNs on both ends with strcmp, as the compact-class rows already do. Refresh the file headers to describe the scope gates, fail-closed ancestry and self-rebinding rules the resolver now follows. Assisted by OpenCode using anthropic/claude-fable-5-1. Signed-off-by: Ben Fairless <ben@oaf.org.au>
Bring the seven review fixes from DeusData#1711 into the cross-file slice. The only conflict was tests/test_ruby_lsp.c, where both sides added tests; both are kept. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
The class table was a per-file array scanned linearly. Add a read-only project-wide table the resolver consults after the file's own, hashed by QN and by dotted constant path, and let method lookup search every reopening of a class declared in more than one file. The table is only populated by the builder in the next commit, so this changes nothing on its own. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
Every Ruby file rebuilt a class table over the whole project's definitions, so cross-file resolution cost grew with files x definitions. On a replicated fixture the complexity suite's per-file registry work rose by 3.96x when the corpus doubled, against a linear ~2. Build one sealed, read-only registry (a finalized method registry plus a hashed class table keyed by QN and by dotted constant path) before the resolve workers start, and resolve each file against it with O(1) lookups, as the Go, Python and TypeScript cross registries do. The resolver reads the shared table after the file's own declarations, so a class declared in the file still wins. Two cross-file gaps close with it: - a compact `class A::B` in another file is now found: its constant path is the dotted `A.B`, where the raw `A::B` spelling never matched; - a class reopened in another file groups with its other declarations, so a method defined in any of them resolves, and a superclass named in only one carries over. The constructor edge lands on the first. The per-file fallback entry point is kept, now a thin wrapper that builds a registry for one call. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
Add the Ruby complexity fixture the maintainer asked for. The suite replicates it k and 2k times and gates on the ratio of per-file registry work: ~2 is linear, ~4 is files x corpus. It measured 3.96 before the shared registry and 2.00 after. Add unit rows for a class reopened across files, a compact A::B class in another file, and one registry serving several files unchanged, and a pipeline-level row that indexes three Ruby files and checks the cross-file and reopened-class edges come from the type-aware pass. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ben Fairless <ben@oaf.org.au>
This branch has not been deployed
2 participants
What does this PR do?
Slice 4 of 5 splitting #1706. Stacked on #1711 — see the review note below.
Adds the cross-file tier on top of the per-file resolver.
cbm_run_ruby_lsp_crossbuilds its class table from the project-wideCBMLSPDef[], deriving each constant path by stripping the def's module QN prefix, and registers methods on both the instance and singleton receiver keys becauseCBMLSPDefdoes not carry thedef self.msplit. Ruby is registered on the fallbackcbm_pxc_run_onepath, like Java and Kotlin — there is no prebuilt tier-2 registry yet.The one deliberate design decision here, and the thing most worth arguing with: Ruby joins Rust in bypassing the own+imports module-def filter, for a different reason. Ruby constants live in one global namespace and Rails/Zeitwerk autoloads them with no
requirestatement, so a filter keyed on requires would starve essentially all cross-file resolution in a real Ruby app.The two new rows drive
cbm_run_ruby_lsp_crosswith a hand-builtCBMLSPDef[]in the shapepass_lsp_cross.cpasses it, the second mirroring the Rails-shaped e2e fixture (User < ApplicationRecord < ActiveRecord::Base, a service object, and a caller in a third file).Reviewing just this slice
A PR from a fork can only target a branch in this repo, so the base here is
mainand GitHub shows #1711's commit as well as this one. Only the second commit is this slice:11418b45— this slice (5 files, +349/−7)e9f87efe— belongs to feat(lsp): add the Ruby per-file resolver #1711Merge #1711 first and this PR's diff collapses to that single commit.
Note that
internal/cbm/lsp/ruby_lsp.{c,h}appear in both PRs. That is the split the review asked for: the cross-file entry point moved out of the resolver slice so each slice's tests exercise everything it adds. I verified the two halves reassemble byte-identically to the original single-commit file.Where this sits in the series
docs/ruby-hybrid-lspCI baseline
mainat 4d7d9f1 is red for reasons unrelated to this change (#1181 × #1359 composition; fixed by #1704). Until that merges,ci-okwill be red here for that reason.Refs #1701
Checklist
git commit -s)scripts/test.sh) — see fix(extraction): keep Ruby superclass names bare #1709 for the full-leg numbers; all failures are pre-existing on unmodifiedorigin/main.make -f Makefile.cbm lint-ci)Assisted by OpenCode using anthropic/claude-opus-5.