Skip to content

GH#293: Allocate Python wrapper slots from a strong table, not from weak PYREFST - #299

Open
sprakes wants to merge 1 commit into
scoder:masterfrom
sprakes:gh293-wrapper-slots
Open

sprakes wants to merge 1 commit into
scoder:masterfrom
sprakes:gh293-wrapper-slots

Conversation

@sprakes

@sprakes sprakes commented Oct 15, 2026 •

Copy link
Copy Markdown

Found this while debugging something at my startup. We run long-running trading simulations that push Python objects into Lua, and every 100k pushes or so a stock's price came back as a different stock's. Turns out the slot numbers for PYREFST come from luaL_ref on that same weak table, and this happens:

  • py_to_lua_custom wraps object X in userdata A, gets slot r from luaL_ref on PYREFST, and records r in _pyrefs_in_lua[refkey].
  • Lua drops A. In the next GC cycle Lua clears A out of PYREFST, but A's __gc has not run yet.
  • X (or any other object) is pushed again. luaL_ref only keeps numbers unique among entries still in the table, sees r empty, and gives it to the new wrapper B.
  • A's __gc runs: luaL_unref(r) puts r on the free list and overwrites B's slot, and _pyrefs_in_lua.pop(refkey) removes the entry, which for the same object is now B's.
  • The next push of X misses B and makes a third wrapper (references to python objects can be randomly lost in lua 5.2 and 5.3 #293). If B was for the same object, its _PyReference is gone, so X can be freed while B is still live in Lua.

The patch takes the numbers from a strong PYREFSLOTS table instead and deletes the _pyrefs_in_lua entry only if pyref._ref == py_obj.ref. PYREFSLOTS[0] = 0 on Lua < 5.4 keeps luaL_unref inside __gc from allocating; teardown with max_memory exhausted crashed without it. The lookup path is unchanged.

This fixes #293, but we did some before/after microbenchmarks and they were slightly worse: over 3M pushes, lua51 1.10s to 1.13s, luajit21 1.00s to 1.02s.

Note: The full suite passes on 3.15.0. Not tested on previous Python versions. Would like help here, or I can get around to it later.

Full disclosure: I used Claude Fable 5.1 to help dig into this. Feel free to close this if you guys have a no-AI policy. We are using this build in prod though.

…eak PYREFST.

Python objects passed into Lua could stop being equal to themselves inside of
Lua, or get deallocated while Lua still used them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

references to python objects can be randomly lost in lua 5.2 and 5.3

1 participant