[danjedwards123]

1. Issues seen in Final Fantasy XIV

Running Final Fantasy XIV (via XIVLauncher) in nested gamescope on Arch with
--force-grab-cursor -f -W 2560 -H 1440 -r 240 -s 1.7:

1. Clicks don't line up with the cursor. Every click lands up and to the left of
   where the cursor is drawn, so menu buttons have to be clicked below and to the right
   of where they visually appear. Present from 3.16.29 onwards, including 3.17.0.
   3.16.28 and earlier are unaffected.

   Screenshot below demonstrating this issue on 3.17.0. Note how the cursor has to be quite far down and to the right to select the 'Start' button:

2. The cursor disappears near the top and left edges. Within a small band along the
   top and left of the screen, the cursor is not drawn at all, although clicks there still
   register. Present in 3.16.28 and earlier. From 3.16.29 it is hidden by issue 1, and it
   comes back once issue 1 is fixed.

   The band across the top and down the left where the cursor no longer becomes visible matches the size of the offset to mouse clicks introduced in 3.17.0.

   Screenshot below demonstrating this issue on 3.16.28. Here my mouse is hovered over the 'System Configuration' menu item. But when the cursor is close to the left, it isn't visible. But when moved to the right, the cursor becomes visible again, as shown in the second image.

2. Logic errors that caused them

1. Hotspot skipped for every nested backend (regression from 12706c3, first in 3.16.29).
   MouseCursor::paint() and MouseCursor::getTexture() now skip the cursor hotspot offset
   and --cursor-scale-height sizing whenever the mouse connector has INestedHints. That's
   correct for OpenVR, whose cursor overlay takes the hotspot from CursorInfo via
   SetOverlayTransformCursor. But the Wayland and SDL backends also expose INestedHints,
   and they draw the composited cursor layer as-is. That happens whenever gamescope
   composites the cursor itself, which --force-grab-cursor forces through
   g_bForceRelativeMouse in ShouldDrawCursor(). On those backends the cursor image is
   placed with its top-left corner at the pointer position instead of its hotspot.
2. Negative plane positions wrap around (Wayland backend, long-standing).
   CWaylandPlane::Present() converts the subsurface position with WaylandScaleToLogical(),
   which takes and returns uint32_t. When the cursor is offset by its hotspot, its position
   goes negative near the top and left edges. A position of -1 at scale 120 becomes
   35791394, so the subsurface is placed far off screen. 12706c3 hid this by no longer
   applying the hotspot, which is why it only shows on 3.16.28 and earlier, or once fix 1
   is applied.

3. The fix

1. steamcompmgr: only skip the cursor hotspot for connectors that apply it
   Adds INestedHints::CursorLayerIsHotspotPoint(), defaulting to false and overridden to
   true in COpenVRConnector. A helper, cursor_layer_is_hotspot_point(), replaces both
   "has nested hints" checks in MouseCursor.

   The original check asked the wrong question: "is this a nested backend?" when what
   matters is "does this backend apply the hotspot itself?". Only OpenVR does. The new
   method asks that directly, so each backend declares its own behaviour. OpenVR keeps
   what 12706c3 introduced, and Wayland and SDL return to the pre-3.16.29 behaviour,
   where gamescope moves the image up and left by the hotspot so its tip sits on the
   click point. Any future backend defaults to gamescope applying the hotspot, matching
   DRM.

2. WaylandBackend: keep negative plane positions negative
   Adds WaylandScalePositionToLogical() and uses it for wl_subsurface_set_position().
   Sizes still use the unsigned WaylandScaleToLogical(), since they can't be negative.

   The new helper multiplies in signed 64-bit, so a negative position stays negative and
   can't overflow. It rounds towards +∞, giving identical results to
   WaylandScaleToLogical() for every positive value, so nothing changes away from the
   edges. Near the top and left edges, the cursor subsurface now sits just past the edge,
   as intended, and the host compositor shows the part that's on screen.

4. Testing

Bisect

Build	Click offset	Edge vanishing
3.16.28	No	Yes
12706c3^ + f3b792a	No	Yes
12706c3 + f3b792a	Yes	Hidden by offset
3.16.30	Yes	Hidden by offset
3.17.0	Yes	Hidden by offset
3.17.0 + this PR	No	No

3.16.29, and any commit between bd52d6e and its revert f3b792a, can't launch the game under
Proton/umu: setting WAYLAND_DISPLAY to an empty string makes pressure-vessel warn and the
launch chain crash on a D-Bus assertion. To test 12706c3 and its parent directly, I
applied f3b792a (main.cpp only, unrelated to the cursor) on top of each. The parent has
no offset and 12706c3 does, so 12706c3 introduces the offset.

With 3.17.0 + this PR, it can be seen in the screenshots below that both the click offset is eliminated and the cursor no longer disappears when close to the top or left of the screen:

Steps to reproduce and verify

Both issues need gamescope to draw the cursor itself (nested Wayland backend with
--force-grab-cursor) and a client cursor whose hotspot isn't at its top-left corner.
Final Fantasy XIV's system cursor meets both.

1. Build gamescope from this branch.
2. Launch FFXIV through XIVLauncher with
   --force-grab-cursor -f -W 2560 -H 1440 -r 240 -s 1.7.
3. Offset (fix 1): Open the main menu and click small buttons and list items with
   the cursor's tip.
   • Before: clicks land up and to the left of the tip, so buttons only respond when
     clicked below and to their right.
   • After: buttons respond exactly where the tip is.
4. Edge (fix 2): Move the cursor slowly along the left edge, then the top edge.
   • Before (3.16.28, or with fix 1 only): the cursor disappears a few pixels before
     reaching the edge, though clicks there still register.
   • After: the cursor stays drawn right up to the edge.

Tested on: Arch, Niri, NVIDIA RTX 4070, GE-Proton11-6 via
umu-launcher, Final Fantasy XIV.

---

AI-assisted: This PR was prepared with help from an AI assistant (Claude), which helped trace the regression through the source, write both patches and draft this description. I reviewed the changes, built them, and did all of the testing and bisecting described above on my own hardware.