fix(vk): a submit's capture no longer misses the pages the replay's collection is marking again #1083

Merged
HumanBean merged 5 commits from nyannoying/Longshot:fix/vk-capture-collect-window into main 2026-11-16 09:29:06 +00:00
Contributor

Type: fix · Risk: low · Tested on: Windows 11, Arc B580 · Needs: Linux (any GPU); Windows on AMD or NVIDIA

Why

For one frame at a time, part of the picture is drawn from data the game had already replaced: in Kokiri Forest the
ground mist is gone for a frame. Before each batch of work the renderer copies the memory the game wrote since the last
copy (a submit's capture). The render thread, meanwhile, sorts the game's writes for its own records and puts back the
marks of the pages a capture should take. A capture made on the game's thread at that moment found those pages
unmarked and left them out, so the work read an older copy.

With the defaults this happens rarely, mostly while an area loads (26 and 36 of ~650 captures after a teleport to
Kokiri Forest, 0 or 1 of ~700 standing there). With the render thread replaying jobs as they come
(RX_VK_FRAME_GATE=0) a third of the captures land in that window, and the mist went missing in 16 to 89 of 3,600
frames. The whole-frames gate (#588) was made for a dark frame like this one in the Market and could not name the read
behind it; this may be that read. With the race fixed, turning the gate off is worth about 8% here (Kokiri Forest and the Market), which a separate pull
request will propose with its own measurements.

Whether the one-frame black or stale surfaces reported on other GPUs (#1023, #1045, #740) come from this race was not
checked; none is linked as fixed.

How to reproduce

Seen on: Windows 11 Pro 10.0.29704, Intel Arc B580 12 GB, driver 32.0.101.9154
Settings and switches: RX_VK_FRAME_GATE=0 and RX_VK_PRESENT_AFTER_SIGNAL=1 (the race every few frames); with
the defaults it shows mostly while an area loads
Steps: start with RX_VK_FRAMELOG=<file>, teleport to Kokiri Forest (child) and stand at the arrival spot;
create <file>.go (the log writes while it exists) and remove it a minute later; then python -I scripts/framelog.py <file>
Expected: no frame where part of the picture is darker than in the frames before and after it
Actual: framelog.py finds 57 events in 3,715 frames (40 s), each 4 or 5 blocks darker than in both neighbouring
frames by 24 to 28 levels: the ground mist gone for a frame

What changed

memory_collect_written() (the render thread) and capture_scan() (a submit's capture, the game's thread) now share a
mutex, collect_mu: the collection holds it from taking the write marks to marking the captured pages again, and the
capture's scan takes it, so a capture made in that window waits for the marks instead of missing them.

  • vk: capture_scan() split out of capture_make() in gpu/vk/vk_memory.cpp, unchanged, in a commit of its own, so
    the check can run it.
  • vk: collect_mu in memory_collect_written() and capture_scan(); the comment above it gives the lock order:
    collect_mu, then cap_ranges_mu and the host's map lock under it; capture_scan() takes it under its capture's order lock (submit_order_mu, sync_order_mu with RX_VK_ASYNC=0, or replay_mu from resume_queue), none of which is taken under collect_mu, and takes no lock under it.
  • tests: tests/test_vk_collect_window.py (new) runs the two functions on two threads with stand-ins for the host's
    marks and lets the capture in at each end of the collection's window; tests/test_met_wait_capture.py finds the scan
    in capture_scan().
  • docs: the 14 measured runs in docs/tested/ (2026-11-16-collect-window-nyannoying-run01 to run14).
  • docs: (review) the lock-order comment names the lock capture_scan() is taken under, and the 14 runs say XeSS was on (95c0a7a5).

Measurements

Verdict: Fixed: with the render thread replaying jobs as they come, the frames with part of the picture dark went from 17, 16 and 27 of 3,600 to none (framelog.py: 57 events to 0); with the defaults there were none before or after, and the frame rate is the same (86.5 and 86.5 fps before, 86.4 and 86.6 after).

Hardware: Windows 11 Pro 10.0.29704, Intel Arc B580 12 GB (driver 32.0.101.9154), Core i5-13600K (20 threads), 64 GB
Scene: Kokiri Forest (child) at the debug teleport's arrival spot, standing; 1280x720 window, rendered at 960x540 and
upscaled with Intel XeSS 2.0.2 (the Upscaler row's Intel XeSS, Ultra Quality: #459), no frame generation,
HOS_FPS_MAX=240 (corrected: this said "no upscaler" before; every run here had XeSS on, both sides alike)
Method: each run a new start of the game, the teleport, 40 s to settle, 40 s measured: the frame rate from the log's
5 s vi: windows (the median of 8); dark frames from a region test in the renderer (a local patch, not in this PR:
16x9 regions of every presented frame, a frame counted when 2+ regions are more than 10 levels darker than in both
neighbouring frames while those agree within 4), and in two runs from scripts/framelog.py on the frame log. The frame
log on in every run, the same on both sides. Ten runs, interleaved: A1 B1 A0 B0 B1 A1 B0 A0 AK BK.
Baseline: main's renderer (MKWVK_API_VERSION 24) does not load in this game folder, whose host is 690d8446's (22).
Before is the renderer the folder runs: 690d8446 with the render-thread changes since merged to main (#761, #838,
64ed6a2f), and the region test: 5d3d6ed0, a local commit; After is the same with this PR's code (its first three
commits), the same diff as here.
How to reproduce: the steps above, once with each renderer; RX_VK_FRAMELOG_DROP=1 drops the pictures when only
the region test's count is wanted (the region test is not in this PR).

Before: 690d8446 + #761, #838 @ 5d3d6ed0 After: this PR @ fd5238f7 on the same
fps, defaults (two runs) 86.5, 86.5 86.4, 86.6
dark frames of 3,000, defaults 0, 0 0, 0
fps, RX_VK_FRAME_GATE=0, RX_VK_PRESENT_AFTER_SIGNAL=1 (three runs) 92.6, 93.1, 92.0 93.4, 92.5, 94.0
dark frames of 3,600, the same 17, 16, 27 0, 0, 0
scripts/framelog.py events, the same (the third run) 57 in 3,715 frames 0 in 3,785 frames
the game's thread waiting for collect_mu, four area loads a session (runs 11-14: After with a local counter) n/a defaults: 831 and 932 waits, 248 and 323 ms a session, the longest 1.0 and 11.9 ms (the second at an area load); RX_VK_FRAME_GATE=0: 35,165 and 33,267 waits, 2.65 and 2.51 s, the longest 20.8 and 20.7 ms (at area loads)
Before, RX_VK_FRAME_GATE=0 (run 3), the region test:
vk: region test: 3600 frames compared, 17 with 2+ regions darker than both neighbours by more than 10, 1 brighter; largest darkening 31.0
After, the same (run 4):
vk: region test: 3600 frames compared, 0 with 2+ regions darker than both neighbours by more than 10, 0 brighter; largest darkening 0.9
Before (run 9), scripts/framelog.py:
3715 frames
events: 57 on the game's frames, 0 on made frames
frame 203 (game) darker 5 blocks, peak 28, at 2.20 s
After (run 10), scripts/framelog.py:
3785 frames
events: 0 on the game's frames, 0 on made frames
After with the counter, the 600 collections over an area load (run 13, the defaults):
vk: collect lock (600 collections): the capture waited 81 times, 61.274 ms in all, longest 11.943 ms; the collection waited 16 times, 0.652 ms in all, longest 0.169 ms; the window 212.286 ms in all, longest 20.726 ms (8201 pages at most)

The check, before the fix (696531d8) and with it (fd5238f7):

$ python -I -B tests/test_vk_collect_window.py --cxx clang++
At 696531d8, the check before the fix (the fixture exits 1):
a capture just after the collection took the marks scanned at once; it took 0 pages, not page 9, page 9's mark set (for the next capture)
FAIL:177 has9
At fd5238f7, the fix:
a capture just after the collection took the marks waited for the collection; it took 1 pages, page 9 among them, page 9's mark cleared
a capture as the collection marked the captured page again waited for the collection; it took 1 pages, page 9 among them, page 9's mark cleared
PASS: the capture holds the page the collection was marking again, at both ends of the window

No screenshots: what changes is a one-frame dropout, which a still picture before and after doesn't show; the counts
above stand for it.

Flags and settings

None.

Testing

Tested: Windows 11 Pro 10.0.29704, Arc B580 (driver 32.0.101.9154), the renderer built alone with llvm-mingw (clang 23.1.5): 14 runs in docs/tested/ (2026-11-16-collect-window-nyannoying-run01 to run14): Kokiri Forest standing before and after, with the defaults and with RX_VK_FRAME_GATE=0, then four area loads a session with a local counter of the waits; no device loss in any. tests/test_vk_collect_window.py (clang 23.1.5): fails at 696531d8 for the bug's reason, passes at fd5238f7 (11 runs of 11), and fails with each of three half fixes: the lock over taking the marks alone, over marking the pages again alone, and the capture's scan without it. python -I -B tests/run_all.py: see the checklist.

Not tested: Linux, and any GPU but the B580; g++ (no g++ on this PC, the check compiled with clang 23.1.5 from
llvm-mingw only).

Must be tested:

  • Linux, any GPU (Test Farm: @Beanstalk linux): build the renderer and run python -I -B tests/run_all.py test_vk_collect_window test_met_wait_capture, expecting both to pass with g++; then Kokiri Forest and the Market,
    expecting no new hitch and no frame with part of the picture dark (by hand: the FPS counter's Dev level, a look).
  • Windows, AMD or NVIDIA (Test Farm: @Beanstalk windows amd / nvidia): Kokiri Forest, then the Market, with the
    defaults, expecting the frame rate within a run's noise of main's and no new hitch.

Risks

What could break: a new mutex between the render thread and the game's thread. A submit's capture made while the
collection holds it waits for it (on the B580 with the defaults, about 900 times in a session with four area loads, 0.25 to 0.32 s in all, the longest 1 and 12 ms in two sessions (the 12 ms at an area load); with RX_VK_FRAME_GATE=0 about 34,000 times, 2.5 to 2.7 s, the longest about 21 ms at each load); a collection that comes during a capture's scan waits for the scan. A
deadlock would need a thread holding cap_ranges_mu or the host's map lock to take collect_mu; the only other taker,
capture_scan(), takes it under its capture's order lock (submit_order_mu, sync_order_mu or replay_mu), none of which is taken under collect_mu, and takes none under it.

Turning it off: revert the commit.

Compatibility: Nothing changes.

Issues

None fits for certain; see Why for #588 and the reports that may be related.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). On Windows with clang 23.1.5 (CXX=clang++): 553 passed, 19 failed, 93 skipped (no Vulkan SDK headers or game folder here). All 19 fail on main (0691f81c) on this PC too: test_auto_handheld, test_bug_report_defaults (since #773, diagnosed there), test_coverage_run (its tasklist output, in this PC's code page, doesn't decode as cp1252), test_crash_dialog, test_crt_options, test_failed_compile_keeps_objects, test_fastfp_fma, test_fpx_sticky, test_guest_imported and test_pending_guest_import (import_reach undeclared), test_linux_preflight, test_make_game_cache, test_missing_nup, test_pads_after_menu, test_run_all_changed, test_spawn_emit, test_tested_configurations (a CRLF check of the command line's pipe), test_vk_occlusion_queries (its cut of vk_replay.cpp no longer finds the recording's end barrier), test_walk_key. --strict --changed origin/main at the move and at the fix: the same two import checks, and the four that need the Vulkan headers.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md).
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Windows (llvm-mingw): yes; Linux:
    not built here, the Test Farm's Linux run under Must be tested builds it.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, Arc B580 · **Needs:** Linux (any GPU); Windows on AMD or NVIDIA ## Why For one frame at a time, part of the picture is drawn from data the game had already replaced: in Kokiri Forest the ground mist is gone for a frame. Before each batch of work the renderer copies the memory the game wrote since the last copy (a submit's capture). The render thread, meanwhile, sorts the game's writes for its own records and puts back the marks of the pages a capture should take. A capture made on the game's thread at that moment found those pages unmarked and left them out, so the work read an older copy. With the defaults this happens rarely, mostly while an area loads (26 and 36 of ~650 captures after a teleport to Kokiri Forest, 0 or 1 of ~700 standing there). With the render thread replaying jobs as they come (`RX_VK_FRAME_GATE=0`) a third of the captures land in that window, and the mist went missing in 16 to 89 of 3,600 frames. The whole-frames gate (#588) was made for a dark frame like this one in the Market and could not name the read behind it; this may be that read. With the race fixed, turning the gate off is worth about 8% here (Kokiri Forest and the Market), which a separate pull request will propose with its own measurements. Whether the one-frame black or stale surfaces reported on other GPUs (#1023, #1045, #740) come from this race was not checked; none is linked as fixed. ## How to reproduce **Seen on:** Windows 11 Pro 10.0.29704, Intel Arc B580 12 GB, driver 32.0.101.9154 **Settings and switches:** `RX_VK_FRAME_GATE=0` and `RX_VK_PRESENT_AFTER_SIGNAL=1` (the race every few frames); with the defaults it shows mostly while an area loads **Steps:** start with `RX_VK_FRAMELOG=<file>`, teleport to Kokiri Forest (child) and stand at the arrival spot; create `<file>.go` (the log writes while it exists) and remove it a minute later; then `python -I scripts/framelog.py <file>` **Expected:** no frame where part of the picture is darker than in the frames before and after it **Actual:** `framelog.py` finds 57 events in 3,715 frames (40 s), each 4 or 5 blocks darker than in both neighbouring frames by 24 to 28 levels: the ground mist gone for a frame ## What changed `memory_collect_written()` (the render thread) and `capture_scan()` (a submit's capture, the game's thread) now share a mutex, `collect_mu`: the collection holds it from taking the write marks to marking the captured pages again, and the capture's scan takes it, so a capture made in that window waits for the marks instead of missing them. - vk: `capture_scan()` split out of `capture_make()` in `gpu/vk/vk_memory.cpp`, unchanged, in a commit of its own, so the check can run it. - vk: `collect_mu` in `memory_collect_written()` and `capture_scan()`; the comment above it gives the lock order: `collect_mu`, then `cap_ranges_mu` and the host's map lock under it; `capture_scan()` takes it under its capture's order lock (`submit_order_mu`, `sync_order_mu` with `RX_VK_ASYNC=0`, or `replay_mu` from `resume_queue`), none of which is taken under `collect_mu`, and takes no lock under it. - tests: `tests/test_vk_collect_window.py` (new) runs the two functions on two threads with stand-ins for the host's marks and lets the capture in at each end of the collection's window; `tests/test_met_wait_capture.py` finds the scan in `capture_scan()`. - docs: the 14 measured runs in `docs/tested/` (`2026-11-16-collect-window-nyannoying-run01` to `run14`). - docs: (review) the lock-order comment names the lock `capture_scan()` is taken under, and the 14 runs say XeSS was on (95c0a7a5). ## Measurements **Verdict:** Fixed: with the render thread replaying jobs as they come, the frames with part of the picture dark went from 17, 16 and 27 of 3,600 to none (`framelog.py`: 57 events to 0); with the defaults there were none before or after, and the frame rate is the same (86.5 and 86.5 fps before, 86.4 and 86.6 after). Hardware: Windows 11 Pro 10.0.29704, Intel Arc B580 12 GB (driver 32.0.101.9154), Core i5-13600K (20 threads), 64 GB Scene: Kokiri Forest (child) at the debug teleport's arrival spot, standing; 1280x720 window, rendered at 960x540 and upscaled with Intel XeSS 2.0.2 (the Upscaler row's Intel XeSS, Ultra Quality: #459), no frame generation, `HOS_FPS_MAX=240` (corrected: this said "no upscaler" before; every run here had XeSS on, both sides alike) Method: each run a new start of the game, the teleport, 40 s to settle, 40 s measured: the frame rate from the log's 5 s `vi:` windows (the median of 8); dark frames from a region test in the renderer (a local patch, not in this PR: 16x9 regions of every presented frame, a frame counted when 2+ regions are more than 10 levels darker than in both neighbouring frames while those agree within 4), and in two runs from `scripts/framelog.py` on the frame log. The frame log on in every run, the same on both sides. Ten runs, interleaved: A1 B1 A0 B0 B1 A1 B0 A0 AK BK. Baseline: main's renderer (`MKWVK_API_VERSION` 24) does not load in this game folder, whose host is 690d8446's (22). Before is the renderer the folder runs: 690d8446 with the render-thread changes since merged to main (#761, #838, 64ed6a2f), and the region test: 5d3d6ed0, a local commit; After is the same with this PR's code (its first three commits), the same diff as here. How to reproduce: the steps above, once with each renderer; `RX_VK_FRAMELOG_DROP=1` drops the pictures when only the region test's count is wanted (the region test is not in this PR). | | Before: 690d8446 + #761, #838 @ 5d3d6ed0 | After: this PR @ fd5238f7 on the same | |---|---|---| | fps, defaults (two runs) | 86.5, 86.5 | 86.4, 86.6 | | dark frames of 3,000, defaults | 0, 0 | 0, 0 | | fps, `RX_VK_FRAME_GATE=0`, `RX_VK_PRESENT_AFTER_SIGNAL=1` (three runs) | 92.6, 93.1, 92.0 | 93.4, 92.5, 94.0 | | dark frames of 3,600, the same | 17, 16, 27 | 0, 0, 0 | | `scripts/framelog.py` events, the same (the third run) | 57 in 3,715 frames | 0 in 3,785 frames | | the game's thread waiting for `collect_mu`, four area loads a session (runs 11-14: After with a local counter) | n/a | defaults: 831 and 932 waits, 248 and 323 ms a session, the longest 1.0 and 11.9 ms (the second at an area load); `RX_VK_FRAME_GATE=0`: 35,165 and 33,267 waits, 2.65 and 2.51 s, the longest 20.8 and 20.7 ms (at area loads) | ``` Before, RX_VK_FRAME_GATE=0 (run 3), the region test: vk: region test: 3600 frames compared, 17 with 2+ regions darker than both neighbours by more than 10, 1 brighter; largest darkening 31.0 After, the same (run 4): vk: region test: 3600 frames compared, 0 with 2+ regions darker than both neighbours by more than 10, 0 brighter; largest darkening 0.9 Before (run 9), scripts/framelog.py: 3715 frames events: 57 on the game's frames, 0 on made frames frame 203 (game) darker 5 blocks, peak 28, at 2.20 s After (run 10), scripts/framelog.py: 3785 frames events: 0 on the game's frames, 0 on made frames After with the counter, the 600 collections over an area load (run 13, the defaults): vk: collect lock (600 collections): the capture waited 81 times, 61.274 ms in all, longest 11.943 ms; the collection waited 16 times, 0.652 ms in all, longest 0.169 ms; the window 212.286 ms in all, longest 20.726 ms (8201 pages at most) ``` The check, before the fix (696531d8) and with it (fd5238f7): ``` $ python -I -B tests/test_vk_collect_window.py --cxx clang++ At 696531d8, the check before the fix (the fixture exits 1): a capture just after the collection took the marks scanned at once; it took 0 pages, not page 9, page 9's mark set (for the next capture) FAIL:177 has9 At fd5238f7, the fix: a capture just after the collection took the marks waited for the collection; it took 1 pages, page 9 among them, page 9's mark cleared a capture as the collection marked the captured page again waited for the collection; it took 1 pages, page 9 among them, page 9's mark cleared PASS: the capture holds the page the collection was marking again, at both ends of the window ``` No screenshots: what changes is a one-frame dropout, which a still picture before and after doesn't show; the counts above stand for it. ### Flags and settings None. ## Testing **Tested:** Windows 11 Pro 10.0.29704, Arc B580 (driver 32.0.101.9154), the renderer built alone with llvm-mingw (clang 23.1.5): 14 runs in `docs/tested/` (`2026-11-16-collect-window-nyannoying-run01` to `run14`): Kokiri Forest standing before and after, with the defaults and with `RX_VK_FRAME_GATE=0`, then four area loads a session with a local counter of the waits; no device loss in any. `tests/test_vk_collect_window.py` (clang 23.1.5): fails at 696531d8 for the bug's reason, passes at fd5238f7 (11 runs of 11), and fails with each of three half fixes: the lock over taking the marks alone, over marking the pages again alone, and the capture's scan without it. `python -I -B tests/run_all.py`: see the checklist. **Not tested:** Linux, and any GPU but the B580; g++ (no g++ on this PC, the check compiled with clang 23.1.5 from llvm-mingw only). **Must be tested:** - Linux, any GPU (Test Farm: `@Beanstalk linux`): build the renderer and run `python -I -B tests/run_all.py test_vk_collect_window test_met_wait_capture`, expecting both to pass with g++; then Kokiri Forest and the Market, expecting no new hitch and no frame with part of the picture dark (by hand: the FPS counter's Dev level, a look). - Windows, AMD or NVIDIA (Test Farm: `@Beanstalk windows amd` / `nvidia`): Kokiri Forest, then the Market, with the defaults, expecting the frame rate within a run's noise of main's and no new hitch. ## Risks **What could break:** a new mutex between the render thread and the game's thread. A submit's capture made while the collection holds it waits for it (on the B580 with the defaults, about 900 times in a session with four area loads, 0.25 to 0.32 s in all, the longest 1 and 12 ms in two sessions (the 12 ms at an area load); with `RX_VK_FRAME_GATE=0` about 34,000 times, 2.5 to 2.7 s, the longest about 21 ms at each load); a collection that comes during a capture's scan waits for the scan. A deadlock would need a thread holding `cap_ranges_mu` or the host's map lock to take `collect_mu`; the only other taker, `capture_scan()`, takes it under its capture's order lock (`submit_order_mu`, `sync_order_mu` or `replay_mu`), none of which is taken under `collect_mu`, and takes none under it. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Issues None fits for certain; see Why for #588 and the reports that may be related. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). On Windows with clang 23.1.5 (`CXX=clang++`): 553 passed, 19 failed, 93 skipped (no Vulkan SDK headers or game folder here). All 19 fail on `main` (0691f81c) on this PC too: `test_auto_handheld`, `test_bug_report_defaults` (since #773, diagnosed there), `test_coverage_run` (its `tasklist` output, in this PC's code page, doesn't decode as cp1252), `test_crash_dialog`, `test_crt_options`, `test_failed_compile_keeps_objects`, `test_fastfp_fma`, `test_fpx_sticky`, `test_guest_imported` and `test_pending_guest_import` (`import_reach` undeclared), `test_linux_preflight`, `test_make_game_cache`, `test_missing_nup`, `test_pads_after_menu`, `test_run_all_changed`, `test_spawn_emit`, `test_tested_configurations` (a CRLF check of the command line's pipe), `test_vk_occlusion_queries` (its cut of `vk_replay.cpp` no longer finds the recording's end barrier), `test_walk_key`. `--strict --changed origin/main` at the move and at the fix: the same two import checks, and the four that need the Vulkan headers. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Windows (llvm-mingw): yes; Linux: not built here, the Test Farm's Linux run under Must be tested builds it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The scan of the captured ranges' write marks moves out of capture_make into a function of its own, unchanged: the next
commit's check runs it beside memory_collect_written on two threads. test_met_wait_capture.py finds the scan there: a
capture is still counted as begun before capture_scan collects the written pages.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
memory_collect_written and capture_scan on two threads, with stand-ins for the host's write marks: the capture comes
just after the collection took the marks, then as the collection marks the captured page again, and must hold the
captured page written before it began each time. The collection stays at that point until the capture has scanned or
waits for collect_mu (the fixture's mutex notes a capture that finds it held), so the check does not depend on timing.
It fails here: the capture scans at once and misses the page.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
memory_collect_written (the render thread) takes every page's write mark, then marks the captured ranges' pages again for
the next capture. A submit's capture on the game's thread whose scan fell between the two found those pages unmarked:
they were in neither capture, and the work the capture was taken for read an older capture's copy of them. With the
render thread replaying jobs as they come (RX_VK_FRAME_GATE=0) a third of the captures came in that window, and Kokiri
Forest's ground mist was gone for a frame in 16 to 89 of 3,600 frames; with whole frames (the default) the two met
while an area loaded (26 and 36 of ~650 captures after a teleport to Kokiri Forest, 0 or 1 of ~700 standing there). The
collection now holds collect_mu over that window and the capture's scan takes it. A capture that comes in the window
waits for the collection: with whole frames on an Arc B580, ~900 times in a session with four area loads, the longest
1 and 12 ms (at a load).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(tested): the capture's collect window fix, 14 runs on an Arc B580
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
7936a6dfd9
Kokiri Forest standing, before and after, with whole frames on and off (runs 1-10); four area loads in one session with a
local counter of collect_mu's waits (runs 11-14).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Contributor

Tested in game: Windows 11 Pro 10.0.29704, Intel Arc B580 12 GB, driver 32.0.101.9154: Kokiri Forest (child) standing, 10 interleaved runs before and after. With RX_VK_FRAME_GATE=0 the one-frame ground-mist dropouts went from 17, 16 and 27 frames of 3,600 to 0, 0 and 0 (scripts/framelog.py: 57 events to 0); with the defaults 0 before and after, 86.5 and 86.5 fps against 86.4 and 86.6. Then four sessions teleporting Kokiri Forest, the Market, Kokiri Forest, the Market, with a local counter of the lock's waits: no dark frame, no glitch, no device loss.

Tested in game: Windows 11 Pro 10.0.29704, Intel Arc B580 12 GB, driver 32.0.101.9154: Kokiri Forest (child) standing, 10 interleaved runs before and after. With RX_VK_FRAME_GATE=0 the one-frame ground-mist dropouts went from 17, 16 and 27 frames of 3,600 to 0, 0 and 0 (scripts/framelog.py: 57 events to 0); with the defaults 0 before and after, 86.5 and 86.5 fps against 86.4 and 86.6. Then four sessions teleporting Kokiri Forest, the Market, Kokiri Forest, the Market, with a local counter of the lock's waits: no dark frame, no glitch, no device loss.
Author
Contributor

@Beanstalk two runs for this PR, please:

  1. linux, any GPU: build the renderer (longshot_vk.so) from this branch and run python -I -B tests/run_all.py test_vk_collect_window test_met_wait_capture with g++. Expected: both pass. Then child Link at the debug teleport's arrival in Kokiri Forest, standing, 60 s after 40 s to settle, main against this PR, each with RX_VK_FRAME_GATE=0, RX_VK_PRESENT_AFTER_SIGNAL=1 and RX_VK_FRAMELOG=<file> (the log writes while <file>.go exists: create it for the 60 s, remove it after), then python -I scripts/framelog.py <file>. Expected: events (one-frame darker patches, the ground mist gone) on main, none with this PR; no device lost. Then Castle Town's market by day with the defaults, 60 s, main against this PR. Expected: the average fps and p99 frame time within a run's noise of main's, no new hitch.
  2. windows, amd or nvidia GPU: the same game runs as 1 (Kokiri Forest with the frame log, then the market with the defaults), the same expectations.

The frame log's pictures stay on the tester's PC; only framelog.py's counts are needed. What it changes: one mutex between the render thread's collection of the game's writes and a submit's capture on the game's thread (gpu/vk/vk_memory.cpp). On an Arc B580 it took the one-frame dropouts with RX_VK_FRAME_GATE=0 from 16-27 of 3,600 frames to 0, at no cost in fps with the defaults.

@Beanstalk two runs for this PR, please: 1. linux, any GPU: build the renderer (`longshot_vk.so`) from this branch and run `python -I -B tests/run_all.py test_vk_collect_window test_met_wait_capture` with g++. Expected: both pass. Then child Link at the debug teleport's arrival in Kokiri Forest, standing, 60 s after 40 s to settle, main against this PR, each with `RX_VK_FRAME_GATE=0`, `RX_VK_PRESENT_AFTER_SIGNAL=1` and `RX_VK_FRAMELOG=<file>` (the log writes while `<file>.go` exists: create it for the 60 s, remove it after), then `python -I scripts/framelog.py <file>`. Expected: events (one-frame darker patches, the ground mist gone) on main, none with this PR; no device lost. Then Castle Town's market by day with the defaults, 60 s, main against this PR. Expected: the average fps and p99 frame time within a run's noise of main's, no new hitch. 2. windows, amd or nvidia GPU: the same game runs as 1 (Kokiri Forest with the frame log, then the market with the defaults), the same expectations. The frame log's pictures stay on the tester's PC; only `framelog.py`'s counts are needed. What it changes: one mutex between the render thread's collection of the game's writes and a submit's capture on the game's thread (`gpu/vk/vk_memory.cpp`). On an Arc B580 it took the one-frame dropouts with `RX_VK_FRAME_GATE=0` from 16-27 of 3,600 frames to 0, at no cost in fps with the defaults.
Member

@nyannoying only members of BigBeans can start the farm.

One of them can approve this request by reacting 👍 to it. This request was for 7936a6d, and the PR has changed since (95c0a7a), so a 👍 no longer approves it. Comment @Beanstalk again to ask for the new commit.

@nyannoying only members of `BigBeans` can start the farm. ~~One of them can approve this request by reacting 👍 to it.~~ This request was for `7936a6d`, and the PR has changed since (`95c0a7a`), so a 👍 no longer approves it. Comment `@Beanstalk` again to ask for the new commit.
Author
Contributor

Correction to the description and the docs/tested/ rows: every run here had the Upscaler row on Intel XeSS (Ultra Quality: the scene rendered at 960x540 and upscaled to the 1280x720 window, #459), on both sides alike, not "no upscaler" as they said. Each run's log has Intel XeSS: 960x540 -> 1280x720 (Ultra Quality). No number changes. The description's Scene line is fixed now; the 14 rows get the same fix with this branch's next push, not now, so the Test Farm request above stays on this head.

Correction to the description and the `docs/tested/` rows: every run here had the Upscaler row on Intel XeSS (Ultra Quality: the scene rendered at 960x540 and upscaled to the 1280x720 window, #459), on both sides alike, not "no upscaler" as they said. Each run's log has `Intel XeSS: 960x540 -> 1280x720 (Ultra Quality)`. No number changes. The description's Scene line is fixed now; the 14 rows get the same fix with this branch's next push, not now, so the Test Farm request above stays on this head.
HumanBean left a comment

Code review only: I read the diff, the host side (host/rx_backend.c collect_in, host/nvdrv.c) and every caller, and ran nothing. It's WIP, so this is a comment, not an approval. I found nothing that has to change before merge.

What it does for a player: no more one-frame dropouts where part of the picture comes from an older capture (Kokiri Forest's ground mist). These are rare with the defaults, mostly while an area loads, and common with RX_VK_FRAME_GATE=0.

Correctness: the lock covers the window exactly. collect_mu is held from host->collect_written (taking every mark) to the last mark_written of a captured page, and the capture's scan takes it. shadow_written_collected, cap_view_bump and pages_collected stay outside the lock, which is right. I went through the possible orderings: a capture before the collection, during it (it waits) and after it. Each one ends with page P either in this capture or marked again for the next one.

Lock order: I found no cycle. Under collect_mu the collection takes only cap_ranges_mu (capture_ranges_merged → refresh_pool_ranges_locked → host->maps_generation/gpu_pools) and nvdrv's lock (hos_gpu_guest_ranges, hos_gpu_ptr). Both are leaves: neither calls back into the renderer. The scan is reached from three places:

  • memory_capture_submit on the game's thread, under submit_order_mu (or sync_order_mu with RX_VK_ASYNC=0).
  • memory_capture_now on the render thread, from resume_queue under replay_mu.
  • the synchronous path, where the capture and the collection run one after the other on the same thread.

capture_make lets go of cap_ranges_mu before capture_scan. Nothing under collect_mu takes submit_order_mu, sync_order_mu or replay_mu, so none of these can deadlock. The comment's "capture_scan takes collect_mu holding no other lock" isn't accurate, though (inline). The deadlock argument in Risks has the same wording.

Cost (a suggestion, your call). The game's thread now waits for the whole host->collect_written scan, over every GPU range. capture_make's own comment puts that at about 20 ms, and your longest waits (12 ms with the defaults at a load, about 21 ms with the gate off) match it. The wait also happens under submit_order_mu, so any other thread that submits waits behind it. design.md §4 rule 7 (the CPU saturates first) makes this worth watching on a slower CPU than the 13600K, so please make sure the Test Farm's AMD and NVIDIA runs include an area load. A follow-up could remove the window instead of serialising it: the collection would not take the marks inside the captured ranges at all. It would collect over the GPU ranges minus capture_ranges_merged(), which needs no re-mark and no lock. That takes a host entry (and an MKWVK_API_VERSION bump), and those pages would then reach shadow_written_collected only through the capture's own report. It's a design change, not something for this PR. #936 (faster collect_in) also shortens the window and so the waits.

The check. tests/test_vk_collect_window.py is good. It cuts with srccut.block, takes --cxx the way the other 147 checks do, and lets the capture in at each end of the window with a handshake instead of a timer, so it doesn't depend on timing. It asserts the right things: page 9 is in the capture with its mark cleared, and page 20 is stamped and not captured. test_met_wait_capture.py is updated, not weakened. I did not run either one.

Testing. The in-game run on the Arc B580 is posted (10 interleaved runs plus 4 sessions with the counter). Still missing: Linux (g++ and the game) and Windows on AMD or NVIDIA. Your Test Farm request is waiting for a BigBean's 👍. Noted that the 14 docs/tested/ rows still say "no upscaler" and that you'll fix them on your next push. I didn't touch them.

Mergeable. The forge says it conflicts, but main (a5b112b6) hasn't touched any of the PR's files since its base (0691f81c). A trial merge with main is textually clean, so the flag looks stale and your next push should recompute it. Nothing needed resolving, and I pushed nothing, which also keeps the head your Test Farm request is on.

Siblings and merge order:

  • #1034 (the write-mark oracle) adds its mark_audit_* calls on the lines around the scan call that this PR replaces. A trial merge with this PR is clean and keeps both: its audit calls end up around capture_scan(mirrored, &pages). Either order works. Landing this first is the smaller step, and its race fix gives 1034's oracle a cleaner baseline.
  • #936 changes only host/rx_backend.c. It merges cleanly with this PR and the two are independent, in either order.
  • #759 touches none of this PR's hunks. Its conflicts come from main itself, in the late-fix code.

Suggested order: this PR, then 1034. 936 and 759 are independent of it.

Code review only: I read the diff, the host side (`host/rx_backend.c` `collect_in`, `host/nvdrv.c`) and every caller, and ran nothing. It's WIP, so this is a comment, not an approval. I found nothing that has to change before merge. **What it does for a player:** no more one-frame dropouts where part of the picture comes from an older capture (Kokiri Forest's ground mist). These are rare with the defaults, mostly while an area loads, and common with `RX_VK_FRAME_GATE=0`. **Correctness: the lock covers the window exactly.** `collect_mu` is held from `host->collect_written` (taking every mark) to the last `mark_written` of a captured page, and the capture's scan takes it. `shadow_written_collected`, `cap_view_bump` and `pages_collected` stay outside the lock, which is right. I went through the possible orderings: a capture before the collection, during it (it waits) and after it. Each one ends with page P either in this capture or marked again for the next one. **Lock order: I found no cycle.** Under `collect_mu` the collection takes only `cap_ranges_mu` (`capture_ranges_merged` → `refresh_pool_ranges_locked` → `host->maps_generation`/`gpu_pools`) and nvdrv's `lock` (`hos_gpu_guest_ranges`, `hos_gpu_ptr`). Both are leaves: neither calls back into the renderer. The scan is reached from three places: - `memory_capture_submit` on the game's thread, under `submit_order_mu` (or `sync_order_mu` with `RX_VK_ASYNC=0`). - `memory_capture_now` on the render thread, from `resume_queue` under `replay_mu`. - the synchronous path, where the capture and the collection run one after the other on the same thread. `capture_make` lets go of `cap_ranges_mu` before `capture_scan`. Nothing under `collect_mu` takes `submit_order_mu`, `sync_order_mu` or `replay_mu`, so none of these can deadlock. The comment's "capture_scan takes collect_mu holding no other lock" isn't accurate, though (inline). The deadlock argument in Risks has the same wording. **Cost (a suggestion, your call).** The game's thread now waits for the whole `host->collect_written` scan, over every GPU range. `capture_make`'s own comment puts that at about 20 ms, and your longest waits (12 ms with the defaults at a load, about 21 ms with the gate off) match it. The wait also happens under `submit_order_mu`, so any other thread that submits waits behind it. design.md §4 rule 7 (the CPU saturates first) makes this worth watching on a slower CPU than the 13600K, so please make sure the Test Farm's AMD and NVIDIA runs include an area load. A follow-up could remove the window instead of serialising it: the collection would not take the marks inside the captured ranges at all. It would collect over the GPU ranges minus `capture_ranges_merged()`, which needs no re-mark and no lock. That takes a host entry (and an `MKWVK_API_VERSION` bump), and those pages would then reach `shadow_written_collected` only through the capture's own report. It's a design change, not something for this PR. [#936](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/936) (faster `collect_in`) also shortens the window and so the waits. **The check.** `tests/test_vk_collect_window.py` is good. It cuts with `srccut.block`, takes `--cxx` the way the other 147 checks do, and lets the capture in at each end of the window with a handshake instead of a timer, so it doesn't depend on timing. It asserts the right things: page 9 is in the capture with its mark cleared, and page 20 is stamped and not captured. `test_met_wait_capture.py` is updated, not weakened. I did not run either one. **Testing.** The in-game run on the Arc B580 is posted (10 interleaved runs plus 4 sessions with the counter). Still missing: Linux (g++ and the game) and Windows on AMD or NVIDIA. Your Test Farm request is waiting for a BigBean's 👍. Noted that the 14 `docs/tested/` rows still say "no upscaler" and that you'll fix them on your next push. I didn't touch them. **Mergeable.** The forge says it conflicts, but `main` (a5b112b6) hasn't touched any of the PR's files since its base (0691f81c). A trial merge with `main` is textually clean, so the flag looks stale and your next push should recompute it. Nothing needed resolving, and I pushed nothing, which also keeps the head your Test Farm request is on. **Siblings and merge order:** - [#1034](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1034) (the write-mark oracle) adds its `mark_audit_*` calls on the lines around the scan call that this PR replaces. A trial merge with this PR is clean and keeps both: its audit calls end up around `capture_scan(mirrored, &pages)`. Either order works. Landing this first is the smaller step, and its race fix gives 1034's oracle a cleaner baseline. - [#936](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/936) changes only `host/rx_backend.c`. It merges cleanly with this PR and the two are independent, in either order. - [#759](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/759) touches none of this PR's hunks. Its conflicts come from `main` itself, in the late-fix code. Suggested order: this PR, then 1034. 936 and 759 are independent of it.
@ -710,0 +717,4 @@
// waits for the collection. On an Arc B580, in sessions with four area loads (~110,000 collections each): with whole
// frames the captures waited ~900 times, 0.25 to 0.32 s in all, the longest 1 and 12 ms (at a load); with
// RX_VK_FRAME_GATE=0 ~34,000 times, ~2.6 s, the longest ~21 ms at each load. Lock order: collect_mu, then cap_ranges_mu
// (capture_ranges_merged) and the host's map lock (collect_written's ranges, gpu_ptr); capture_scan takes collect_mu
Owner

Nit: "capture_scan takes collect_mu holding no other lock" isn't quite right. It's taken under submit_order_mu (game thread, memory_capture_submit), sync_order_mu (RX_VK_ASYNC=0) and replay_mu (resume_queue → memory_capture_now, render thread). It's still safe, because nothing under collect_mu takes any of them. Maybe: "capture_scan takes it holding neither cap_ranges_mu nor the map lock (its callers' submit_order_mu, sync_order_mu or replay_mu, none of which is taken under collect_mu)". The deadlock line under Risks has the same wording.

Nit: "capture_scan takes collect_mu holding no other lock" isn't quite right. It's taken under `submit_order_mu` (game thread, `memory_capture_submit`), `sync_order_mu` (`RX_VK_ASYNC=0`) and `replay_mu` (`resume_queue` → `memory_capture_now`, render thread). It's still safe, because nothing under `collect_mu` takes any of them. Maybe: "capture_scan takes it holding neither cap_ranges_mu nor the map lock (its callers' submit_order_mu, sync_order_mu or replay_mu, none of which is taken under collect_mu)". The deadlock line under Risks has the same wording.
@ -760,1 +762,3 @@
continue;
int64_t n = 0;
{ // (from taking the marks to marking the captured pages again: no capture scans in between, collect_mu)
std::lock_guard lk(collect_mu);
Owner

This holds collect_mu over the whole host->collect_written scan of every GPU range (~20 ms per capture_make's own comment), and a capture that arrives then waits for the rest of it on the game's thread, under submit_order_mu. That matches your 12 ms and 21 ms longest waits. It's acceptable for the fix, but please make sure an area load is part of the Test Farm runs. A follow-up that collects only outside capture_ranges_merged() would have no window to lock (see the summary).

This holds `collect_mu` over the whole `host->collect_written` scan of every GPU range (~20 ms per capture_make's own comment), and a capture that arrives then waits for the rest of it on the game's thread, under `submit_order_mu`. That matches your 12 ms and 21 ms longest waits. It's acceptable for the fix, but please make sure an area load is part of the Test Farm runs. A follow-up that collects only outside `capture_ranges_merged()` would have no window to lock (see the summary).
docs(vk): capture_scan's lock order names the order lock it is taken under; the 14 runs had XeSS on
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 14s
CI / Workflow lint (actionlint) (pull_request) Successful in 15s
Third-party parts / As their notes say (pull_request) Successful in 18s
PR description / Description follows the template (pull_request_target) Failing after 8s
beanstalk/approval changed since the request: ask again
95c0a7a581
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean changed title from WIP: fix(vk): a submit's capture no longer misses the pages the replay's collection is marking again to fix(vk): a submit's capture no longer misses the pages the replay's collection is marking again 2026-11-16 09:28:47 +00:00
HumanBean approved these changes 2026-11-16 09:29:03 +00:00
HumanBean left a comment

Approved at 95c0a7a5. The code review (above) found nothing blocking. On top of the author's commits I pushed one fix, 95c0a7a5:

  • the lock-order comment now names the order lock capture_scan is taken under (submit_order_mu, sync_order_mu or replay_mu);
  • the 14 docs/tested/ rows now say XeSS was on, as nyannoying's correction asked;
  • the description's two lock-order sentences say the same.

Merged at a maintainer's request, without the Linux build or the AMD/NVIDIA and Linux game runs that Must be tested names. Those runs are still wanted against main.

Approved at 95c0a7a5. The code review (above) found nothing blocking. On top of the author's commits I pushed one fix, 95c0a7a5: - the lock-order comment now names the order lock `capture_scan` is taken under (`submit_order_mu`, `sync_order_mu` or `replay_mu`); - the 14 `docs/tested/` rows now say XeSS was on, as nyannoying's correction asked; - the description's two lock-order sentences say the same. Merged at a maintainer's request, without the Linux build or the AMD/NVIDIA and Linux game runs that Must be tested names. Those runs are still wanted against main.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1083
No description provided.