MagicBeanDev/Hookshot
Archived
Watch
2
0
Fork
You've already forked Hookshot
 0
Code Issues 125 Pull requests 205 Activity

perf(vk): a readback waits for the part that writes its range back, and no game thread waits behind a present for nothing #2602

Merged
hckrman101 merged 3 commits from perf/readback-part-wait into main 2026-11-12 02:42:39 +00:00
Conversation 31 Commits 3 Files changed 12 +679 -42
hckrman101 commented 2026-11-12 01:12:09 +00:00 (Migrated from github.com)

Hyrule Field runs about 15 fps slower on main than on v2.0.11-588 (ba86aee), and uses almost twice the CPU. One of the game's job threads sits blocked for 5.5 ms of every frame in a CPU read of GPU results, and the other job threads poll while they wait for it. This PR removes that wait without undoing what #2125 and #2419 fixed.

What regressed

RTX 3090, Ryzen 9 7900X, Linux, 2560x1440, uncapped, standing in Hyrule Field from the same save:

v2.0.11-588 (ba86aee) main (fce53453)
fps 79 64-71
CPU a frame, all threads 39-40 ms 64-73 ms
readbacks that waited, of 3000 ~1300, 0.13 ms each ~172, 5.5 ms each
time blocked in readbacks, per 3000 ~170 ms ~940 ms

GPU time and the replay's own time are the same on both.

Cause

An all-thread sample of both builds (HOS_SAMPLE_ALL) shows the extra CPU is one item: sched_yield in the host's short-sleep loop, 0.28 cores on the old build and 1.92 on main. It is the game's six job threads (ModuleSystemWorker1-5 and sead::MainThread), which poll with 20 us sleeps while one of them is blocked. On main a job thread is inside a readback at 38.7 % of all instants, with 3.9 threads polling meanwhile.

The blocked read is of the results that the first part of the scene's submission writes (two buffers, alternating by frame). That submission then stops at an event wait and is finished later, on the presenting thread. Three things held the read up:

  1. It waited for the whole submission. Since #2419 a submission's value maps to its last part's recording, which is right for a fence. A readback used the same map, so it waited for all of the scene's GPU work (3-4 ms) for pages the first part had already written back.
  2. It took sync_order_mu whenever any queue was stopped. #2125 added a look at the stopped queues before the read. The presenting thread holds that mutex for the whole replay of its submission and for its present, so the read waited for both (24 % of all instants, 3.9 ms a frame) for a look that let nothing on.
  3. It took dev.present_mu to flush held submits. Held submits exist only with batched submits (RADV). A present holds that mutex about 4 ms a frame. A readback whose recording was already done waited there, and so did the host's copy bookkeeping (rx_nvn_replayed, called under its own lock), which queued the other job threads behind it. The old build has this one too.

The fix

All of it is on by default; the two switches only turn parts off, for A/B.

  1. A readback waits for the part that writes its pages back. Each part of a split submission copies back, at its own end, what the GPU wrote in it. The page table now keeps that part with the page's value, and the read waits for the newest part of its pages. Fence waits are unchanged. RX_VK_READBACK_PART=0 restores the last part. The read still waits for its submission's last part, as on main, in two cases:
    • a page of its range was written in place (no mirror, as on the iGPUs of issue 997);
    • a part after its own reads guest memory in place: a pool bound with no mirror, a mirror's dirty pages that no capture holds, a mirror's first copy, a large texture unswizzled on the GPU, a texel buffer's view. The log's readbacks line counts these.
  2. The look at the stopped queues happens only when it can let something on. That is after a CPU signal (nvnEventSignal) nobody has looked at since, or while a queue is stopped at an event that neither the stream nor nvnEventSignal ever set, whose value is read from guest memory. A look counts as taken only once the burst it ran in has finished its recording, so its write-backs are in the page table. In the field 0-9 of 3000 readbacks look; on main every readback that found a queue stopped did. RX_VK_READBACK_SETTLE_ALWAYS=1 restores the look whenever anything is stopped.
  3. A flush with nothing held does not take dev.present_mu. An atomic says whether any submit is held. With batched submits a recording is now put where waiters find it only after its submit is in the held list, under the mutex, which also closes a window main has. A wait returns at once when the recordings' timeline has already reached it. No switch.

The trade-off against main

Two reads of guest memory at the GPU's time are deliberately not counted for the part wait. Both behave as they did in v2.0.14 and before, that is before #2419:

  • Shaders following pointers into a pool that was never bound, and so has no mirror. No call of the renderer's is made for those reads, so they cannot be noted. On a discrete GPU this is the remaining way to a frame of issue 997's kind that current main's whole-submission wait would have prevented: the game, let go after the first part, writes memory that a later part still reads through such a pointer. GPUs with no mirrors at all (997's iGPUs) are not affected: every page there is written in place and keeps the whole wait.
  • The refresh of readback pages into their mirrors (memory_rb_refresh, in every part's uploads). These copies are how a CPU write to a page it reads back reaches the mirror at all, and a later part that writes such a page again is waited for through its own part. Counting them would put an in-place read in every part and cancel the change. What they carry along is any other data that shares a 4 KB page with a readback range. mooki000 counted it on an RTX 4090 (see their comment in this PR). In Kokiri Forest 0 of 20,371 readbacks shared a page this way. In Hyrule Field 610 of 45,916 did, all with read-only bindings, and in every case the reading part's recording was already complete when the readback's wait returned.

Review

Three commits:

  • 798d625a: the three parts above.
  • 9daa0718: first review. The part wait gives way to in-place reads in later parts; the look also covers events whose value is in guest memory.
  • 4ba31740: mooki000's two findings on this PR, both real, and the second review.
    • A look was counted at its own end, before its burst's recording was finished; a readback in between took no look and read the page table without that recording's pages (#2125's hole again). Fixed: counted at the burst's end.
    • A held submit's recording could be found before the submit was in the held list, so a waiter skipped the flush and waited until something else sent the batch. Fixed as in part 3.
    • The second review added the unswizzle and texel-buffer reads, and bindings made outside a recording's uploads, to the in-place reads.

Measured

Hyrule Field, 30 s each, alternating builds in one session. Rounds 1-3 are 798d625a's renderer (built before its rebase onto 6f7ab2a7) on main's game binary (fce53453); the last two rows are the later commits, built as committed.

round fix v2.0.11-588 main
1 78.6 fps, 46.7 ms CPU 81.3 fps, 38.3 ms 63.1 fps, 71.0 ms
2 78.7 fps, 45.1 ms 73.9 fps, 40.0 ms 65.9 fps, 69.5 ms
3 69.9 fps, 50.5 ms (disturbed) 74.8 fps, 42.5 ms 67.2 fps, 68.8 ms
9daa0718 76.8 fps, 47.6 ms 66.4 fps, 42.4 ms (hitched) 60.9 fps, 73.4 ms
4ba31740 77.2 fps, 47.0 ms not run 61.1 fps, 75.5 ms

Round 3's fix run had 367,000 pages swapped in on the PC during its 30 s (the others 20,000-135,000), so I count it as disturbed. The PC was short of memory for the whole session, which is also why the old build swings between 66 and 81.

Time blocked in readbacks, per 3000: about 220 ms with the fix (1500-1540 waits of 0.14-0.16 ms), about 230 ms on the old build, about 1100 ms on main. The counts themselves are not comparable between builds: main counts a readback as waited whenever the replay was behind.

Kokiri Forest, 798d625a, two rounds: fix 65.3 / 58.2 fps at 68-69 ms of CPU a frame, main 40.5 (a hitched run) / 45.9 at 113-117 ms, the old build 50.8 / 56.0 at 84-88 ms. The number of draws there varies from run to run (955 to 1255) on every build, so these are rougher than the field's.

The CPU is still 6-8 ms a frame above the old build in the field. The read now makes one real wait a frame, for the first part on the GPU (1-2 ms), and about three job threads poll during it; on the old build those threads sat blocked on mutexes instead. That is the yield loop for sleeps under a millisecond. HOS_SHORT_SLEEP_TIMER=1 on main brings the CPU to 37.5 ms a frame at the same fps, so it is a separate follow-up in the host, not part of this PR.

SadTransGirl measured the cost of #2419 independently with an RX_VK_WAIT_FIRST_PART switch (branch) on an RTX 4070 SUPER, DLAA 1440p: about 4-7 fps (Town to Hyrule Field 55.9-56.2 before #2419, 47.4-47.5 on main, 52.7-54.2 with the switch).

Tested

  • python -I -B tests/run_all.py on Linux at 4ba31740: 447 passed, 3 failed, 25 skipped. The three are not this change: test_bake_portability and test_builder_log_read_only (no zstandard or tkinter on this PC) and test_env_on (HOS_FPS_LOG in host/oot_high_fps.c, on main).
  • New: tests/test_readback_part.py runs the page table, the part bookkeeping, the readback's choice of wait, the look at the stopped queues and the burst's end, cut from the source, and counts every copy out of a guest pool. tests/test_flush_nothing_held.py checks the order in which a held submit is listed, flagged and made findable. Both fail on the commit before the fix they cover.
  • Updated for the new code: tests/test_readback_settle.py, tests/test_recorded_wait_flush.py, and five fixtures that compile the changed functions.
  • Flashes, RX_VK_FRAMELOG with scripts/framelog.py, 30 s each, 798d625a against the same build with both switches off:
    • Hyrule Field: 0 events in 2374 frames with the fix, 0 in 1869 without.
    • Kokiri Forest: 0 events without; 4 with the fix, all one 20x20 block on four consecutive frames. I looked at them: a leaf falling across the sky.
    • Mean brightness is the same with and without (field 37.6 / 37.6, Kokiri 89.7 / 90.0), so nothing is hidden by a blown-out picture.
  • The same pass on 4ba31740, fix only: Hyrule Field 0 events in 2558 frames (brightness 37.6); Kokiri Forest 0 events in 1692 frames. That Kokiri run was the 964-draw variant of the scene, darker at 79.1, and I have no switches-off run of that variant to compare its brightness with.
  • The game ran on NVIDIA 620.70.06, Vulkan 1.4, RTX 3090 (24 GB), VRAM mirrors on, batched submits off. In these runs no readback fell back for in-place reads and no pages were read live from guest memory.

Not yet verified

  • 4ba31740 in the game beyond one field pair and one framelog pass (above): no Kokiri timing, no run of the old build beside it.
  • The batched-submit path (RADV) on hardware. Part 3 changed in 4ba31740, so a check of 798d625a does not cover it; it is covered by a source test only.
  • A GPU with no VRAM mirrors (issue 997's iGPUs), and a discrete GPU short of VRAM, where the in-place fallback would actually trigger.
  • Windows.
  • If a part's pages are noted and that part is then never submitted (a lost device), a read of them waits 250 ms before it falls back.
Hyrule Field runs about 15 fps slower on `main` than on v2.0.11-588 (`ba86aee`), and uses almost twice the CPU. One of the game's job threads sits blocked for 5.5 ms of every frame in a CPU read of GPU results, and the other job threads poll while they wait for it. This PR removes that wait without undoing what #2125 and #2419 fixed. ## What regressed RTX 3090, Ryzen 9 7900X, Linux, 2560x1440, uncapped, standing in Hyrule Field from the same save: | | v2.0.11-588 (`ba86aee`) | main (`fce53453`) | |---|---|---| | fps | 79 | 64-71 | | CPU a frame, all threads | 39-40 ms | 64-73 ms | | readbacks that waited, of 3000 | ~1300, 0.13 ms each | ~172, 5.5 ms each | | time blocked in readbacks, per 3000 | ~170 ms | ~940 ms | GPU time and the replay's own time are the same on both. ## Cause An all-thread sample of both builds (`HOS_SAMPLE_ALL`) shows the extra CPU is one item: `sched_yield` in the host's short-sleep loop, 0.28 cores on the old build and 1.92 on main. It is the game's six job threads (`ModuleSystemWorker1-5` and `sead::MainThread`), which poll with 20 us sleeps while one of them is blocked. On main a job thread is inside a readback at 38.7 % of all instants, with 3.9 threads polling meanwhile. The blocked read is of the results that the first part of the scene's submission writes (two buffers, alternating by frame). That submission then stops at an event wait and is finished later, on the presenting thread. Three things held the read up: 1. **It waited for the whole submission.** Since #2419 a submission's value maps to its last part's recording, which is right for a fence. A readback used the same map, so it waited for all of the scene's GPU work (3-4 ms) for pages the first part had already written back. 2. **It took `sync_order_mu` whenever any queue was stopped.** #2125 added a look at the stopped queues before the read. The presenting thread holds that mutex for the whole replay of its submission and for its present, so the read waited for both (24 % of all instants, 3.9 ms a frame) for a look that let nothing on. 3. **It took `dev.present_mu` to flush held submits.** Held submits exist only with batched submits (RADV). A present holds that mutex about 4 ms a frame. A readback whose recording was already done waited there, and so did the host's copy bookkeeping (`rx_nvn_replayed`, called under its own lock), which queued the other job threads behind it. The old build has this one too. ## The fix All of it is on by default; the two switches only turn parts off, for A/B. 1. **A readback waits for the part that writes its pages back.** Each part of a split submission copies back, at its own end, what the GPU wrote in it. The page table now keeps that part with the page's value, and the read waits for the newest part of its pages. Fence waits are unchanged. `RX_VK_READBACK_PART=0` restores the last part. The read still waits for its submission's last part, as on main, in two cases: - a page of its range was written in place (no mirror, as on the iGPUs of issue 997); - a part after its own reads guest memory in place: a pool bound with no mirror, a mirror's dirty pages that no capture holds, a mirror's first copy, a large texture unswizzled on the GPU, a texel buffer's view. The log's readbacks line counts these. 2. **The look at the stopped queues happens only when it can let something on.** That is after a CPU signal (`nvnEventSignal`) nobody has looked at since, or while a queue is stopped at an event that neither the stream nor `nvnEventSignal` ever set, whose value is read from guest memory. A look counts as taken only once the burst it ran in has finished its recording, so its write-backs are in the page table. In the field 0-9 of 3000 readbacks look; on main every readback that found a queue stopped did. `RX_VK_READBACK_SETTLE_ALWAYS=1` restores the look whenever anything is stopped. 3. **A flush with nothing held does not take `dev.present_mu`.** An atomic says whether any submit is held. With batched submits a recording is now put where waiters find it only after its submit is in the held list, under the mutex, which also closes a window main has. A wait returns at once when the recordings' timeline has already reached it. No switch. ## The trade-off against main Two reads of guest memory at the GPU's time are deliberately not counted for the part wait. Both behave as they did in v2.0.14 and before, that is before #2419: - **Shaders following pointers into a pool that was never bound, and so has no mirror.** No call of the renderer's is made for those reads, so they cannot be noted. On a discrete GPU this is the remaining way to a frame of issue 997's kind that current main's whole-submission wait would have prevented: the game, let go after the first part, writes memory that a later part still reads through such a pointer. GPUs with no mirrors at all (997's iGPUs) are not affected: every page there is written in place and keeps the whole wait. - **The refresh of readback pages into their mirrors** (`memory_rb_refresh`, in every part's uploads). These copies are how a CPU write to a page it reads back reaches the mirror at all, and a later part that writes such a page again is waited for through its own part. Counting them would put an in-place read in every part and cancel the change. What they carry along is any other data that shares a 4 KB page with a readback range. mooki000 counted it on an RTX 4090 (see their comment in this PR). In Kokiri Forest 0 of 20,371 readbacks shared a page this way. In Hyrule Field 610 of 45,916 did, all with read-only bindings, and in every case the reading part's recording was already complete when the readback's wait returned. ## Review Three commits: - `798d625a`: the three parts above. - `9daa0718`: first review. The part wait gives way to in-place reads in later parts; the look also covers events whose value is in guest memory. - `4ba31740`: mooki000's two findings on this PR, both real, and the second review. - A look was counted at its own end, before its burst's recording was finished; a readback in between took no look and read the page table without that recording's pages (#2125's hole again). Fixed: counted at the burst's end. - A held submit's recording could be found before the submit was in the held list, so a waiter skipped the flush and waited until something else sent the batch. Fixed as in part 3. - The second review added the unswizzle and texel-buffer reads, and bindings made outside a recording's uploads, to the in-place reads. ## Measured Hyrule Field, 30 s each, alternating builds in one session. Rounds 1-3 are `798d625a`'s renderer (built before its rebase onto `6f7ab2a7`) on main's game binary (`fce53453`); the last two rows are the later commits, built as committed. | round | fix | v2.0.11-588 | main | |---|---|---|---| | 1 | 78.6 fps, 46.7 ms CPU | 81.3 fps, 38.3 ms | 63.1 fps, 71.0 ms | | 2 | 78.7 fps, 45.1 ms | 73.9 fps, 40.0 ms | 65.9 fps, 69.5 ms | | 3 | 69.9 fps, 50.5 ms (disturbed) | 74.8 fps, 42.5 ms | 67.2 fps, 68.8 ms | | `9daa0718` | 76.8 fps, 47.6 ms | 66.4 fps, 42.4 ms (hitched) | 60.9 fps, 73.4 ms | | `4ba31740` | 77.2 fps, 47.0 ms | not run | 61.1 fps, 75.5 ms | Round 3's fix run had 367,000 pages swapped in on the PC during its 30 s (the others 20,000-135,000), so I count it as disturbed. The PC was short of memory for the whole session, which is also why the old build swings between 66 and 81. Time blocked in readbacks, per 3000: about 220 ms with the fix (1500-1540 waits of 0.14-0.16 ms), about 230 ms on the old build, about 1100 ms on main. The counts themselves are not comparable between builds: main counts a readback as waited whenever the replay was behind. Kokiri Forest, `798d625a`, two rounds: fix 65.3 / 58.2 fps at 68-69 ms of CPU a frame, main 40.5 (a hitched run) / 45.9 at 113-117 ms, the old build 50.8 / 56.0 at 84-88 ms. The number of draws there varies from run to run (955 to 1255) on every build, so these are rougher than the field's. The CPU is still 6-8 ms a frame above the old build in the field. The read now makes one real wait a frame, for the first part on the GPU (1-2 ms), and about three job threads poll during it; on the old build those threads sat blocked on mutexes instead. That is the yield loop for sleeps under a millisecond. `HOS_SHORT_SLEEP_TIMER=1` on main brings the CPU to 37.5 ms a frame at the same fps, so it is a separate follow-up in the host, not part of this PR. SadTransGirl measured the cost of #2419 independently with an `RX_VK_WAIT_FIRST_PART` switch ([branch](https://github.com/SadTransGirl/Hookshot-Gummi/tree/feat/wait-first-part-ab)) on an RTX 4070 SUPER, DLAA 1440p: about 4-7 fps (Town to Hyrule Field 55.9-56.2 before #2419, 47.4-47.5 on main, 52.7-54.2 with the switch). ## Tested - `python -I -B tests/run_all.py` on Linux at `4ba31740`: 447 passed, 3 failed, 25 skipped. The three are not this change: `test_bake_portability` and `test_builder_log_read_only` (no `zstandard` or `tkinter` on this PC) and `test_env_on` (`HOS_FPS_LOG` in `host/oot_high_fps.c`, on main). - New: `tests/test_readback_part.py` runs the page table, the part bookkeeping, the readback's choice of wait, the look at the stopped queues and the burst's end, cut from the source, and counts every copy out of a guest pool. `tests/test_flush_nothing_held.py` checks the order in which a held submit is listed, flagged and made findable. Both fail on the commit before the fix they cover. - Updated for the new code: `tests/test_readback_settle.py`, `tests/test_recorded_wait_flush.py`, and five fixtures that compile the changed functions. - Flashes, `RX_VK_FRAMELOG` with `scripts/framelog.py`, 30 s each, `798d625a` against the same build with both switches off: - Hyrule Field: 0 events in 2374 frames with the fix, 0 in 1869 without. - Kokiri Forest: 0 events without; 4 with the fix, all one 20x20 block on four consecutive frames. I looked at them: a leaf falling across the sky. - Mean brightness is the same with and without (field 37.6 / 37.6, Kokiri 89.7 / 90.0), so nothing is hidden by a blown-out picture. - The same pass on `4ba31740`, fix only: Hyrule Field 0 events in 2558 frames (brightness 37.6); Kokiri Forest 0 events in 1692 frames. That Kokiri run was the 964-draw variant of the scene, darker at 79.1, and I have no switches-off run of that variant to compare its brightness with. - The game ran on NVIDIA 620.70.06, Vulkan 1.4, RTX 3090 (24 GB), VRAM mirrors on, batched submits off. In these runs no readback fell back for in-place reads and no pages were read live from guest memory. ## Not yet verified - `4ba31740` in the game beyond one field pair and one framelog pass (above): no Kokiri timing, no run of the old build beside it. - The batched-submit path (RADV) on hardware. Part 3 changed in `4ba31740`, so a check of `798d625a` does not cover it; it is covered by a source test only. - A GPU with no VRAM mirrors (issue 997's iGPUs), and a discrete GPU short of VRAM, where the in-place fallback would actually trigger. - Windows. - If a part's pages are noted and that part is then never submitted (a lost device), a read of them waits 250 ms before it falls back.
mooki000 commented 2026-11-12 01:16:25 +00:00 (Migrated from github.com)

Testing this on Linux with an RTX 4090 (NVIDIA 615.78.08, i9-13900K, Arch); I'll post the numbers for this exact commit as each run is read. Two things from your "Not yet verified" list are what I can give: Kokiri Forest timing, and the flash check with RX_VK_FRAMELOG and scripts/framelog.py against the same build with both switches off. I am building 798d625a itself, so that is the third.

What I have already, measured tonight before this pull request existed, and it agrees with your "one item": in a forest scene the whole loss of main is the readback wait, and nothing else in the range costs anything there.

Kokiri Forest, a save with camera turns, 2048x1152, the game's own upscaler, uncapped (HOS_FPS_MAX=240), headless, a 30 s window after the load, builds alternated A B A B inside one exclusive hold each, the same counting patch in every build (it counts every buffer readback and how long it waited):

build fps mean frame 1 % low frames over 20 ms waited in readbacks, a frame load before → after
49f75dc8 78.6 12.73 ms 45.0 21 0.02 ms 8.0 → 7.8
49f75dc8 78.5 12.74 ms 38.9 18 0.02 ms 7.2 → 7.6
main 9df055f2 62.9 15.89 ms 34.0 425 4.96 ms 9.3 → 21.2
main 9df055f2 62.6 15.97 ms 37.1 445 5.02 ms 9.8 → 8.0
main 9df055f2 62.5 15.99 ms 36.2 432 4.89 ms 8.1 → 7.6
main, both taken back 79.2 12.63 ms 45.9 23 0.02 ms 7.8 → 7.2
main, both taken back 80.0 12.50 ms 47.1 19 0.02 ms 7.6 → 5.9
main, RX_VK_READBACK_NO_CATCHUP=1 69.0 14.49 ms 43.2 33 3.11 ms 7.6 → 7.2
main, RX_VK_READBACK_NO_CATCHUP=1 68.0 14.70 ms 42.3 35 3.23 ms 13.8 → 8.6
  • 49f75dc8 is v2.1.0-rc.3-53, the last build I had from before #2125 and #2419. "Both taken back" is main with those two pull requests' code hunks reverted, a local diagnostic only.
  • On main about one readback a frame waits, 4.8 to 5.0 ms; about 3.2 ms of that is in #2125's catch-up before the write-back lookup, which is what RX_VK_READBACK_NO_CATCHUP=1 removes. With both taken back the build is at the old build's rate (79.2 and 80.0 against 78.6 and 78.5 fps).
  • Left out: a fourth run of plain main at 54.4 fps (1 % low 11.5) that was disturbed. The main-against-old runs had a load of 8 to 21 from other work on this PC; the revert runs were quiet.
  • GPU time was the same in all nine (5.94 to 6.22 ms a frame).

Coming for 798d625a against its base 6f7ab2a7 and against 49f75dc8: the forest and an open-field scene uncapped, the forest at a 60 cap with DLAA, your two switches one at a time, the flash check, and your new tests with gcc 16.2 and clang 23.1. Not something I can test: Windows, AMD, an integrated GPU, RADV's batched submits.

Testing this on Linux with an RTX 4090 (NVIDIA 615.78.08, i9-13900K, Arch); I'll post the numbers for this exact commit as each run is read. Two things from your "Not yet verified" list are what I can give: Kokiri Forest timing, and the flash check with `RX_VK_FRAMELOG` and `scripts/framelog.py` against the same build with both switches off. I am building `798d625a` itself, so that is the third. What I have already, measured tonight before this pull request existed, and it agrees with your "one item": in a forest scene the whole loss of main is the readback wait, and nothing else in the range costs anything there. Kokiri Forest, a save with camera turns, 2048x1152, the game's own upscaler, uncapped (`HOS_FPS_MAX=240`), headless, a 30 s window after the load, builds alternated A B A B inside one exclusive hold each, the same counting patch in every build (it counts every buffer readback and how long it waited): | build | fps | mean frame | 1 % low | frames over 20 ms | waited in readbacks, a frame | load before → after | |---|---|---|---|---|---|---| | 49f75dc8 | 78.6 | 12.73 ms | 45.0 | 21 | 0.02 ms | 8.0 → 7.8 | | 49f75dc8 | 78.5 | 12.74 ms | 38.9 | 18 | 0.02 ms | 7.2 → 7.6 | | main 9df055f2 | 62.9 | 15.89 ms | 34.0 | 425 | 4.96 ms | 9.3 → 21.2 | | main 9df055f2 | 62.6 | 15.97 ms | 37.1 | 445 | 5.02 ms | 9.8 → 8.0 | | main 9df055f2 | 62.5 | 15.99 ms | 36.2 | 432 | 4.89 ms | 8.1 → 7.6 | | main, both taken back | 79.2 | 12.63 ms | 45.9 | 23 | 0.02 ms | 7.8 → 7.2 | | main, both taken back | 80.0 | 12.50 ms | 47.1 | 19 | 0.02 ms | 7.6 → 5.9 | | main, `RX_VK_READBACK_NO_CATCHUP=1` | 69.0 | 14.49 ms | 43.2 | 33 | 3.11 ms | 7.6 → 7.2 | | main, `RX_VK_READBACK_NO_CATCHUP=1` | 68.0 | 14.70 ms | 42.3 | 35 | 3.23 ms | 13.8 → 8.6 | - 49f75dc8 is v2.1.0-rc.3-53, the last build I had from before #2125 and #2419. "Both taken back" is main with those two pull requests' code hunks reverted, a local diagnostic only. - On main about one readback a frame waits, 4.8 to 5.0 ms; about 3.2 ms of that is in #2125's catch-up before the write-back lookup, which is what `RX_VK_READBACK_NO_CATCHUP=1` removes. With both taken back the build is at the old build's rate (79.2 and 80.0 against 78.6 and 78.5 fps). - Left out: a fourth run of plain main at 54.4 fps (1 % low 11.5) that was disturbed. The main-against-old runs had a load of 8 to 21 from other work on this PC; the revert runs were quiet. - GPU time was the same in all nine (5.94 to 6.22 ms a frame). Coming for `798d625a` against its base `6f7ab2a7` and against 49f75dc8: the forest and an open-field scene uncapped, the forest at a 60 cap with DLAA, your two switches one at a time, the flash check, and your new tests with gcc 16.2 and clang 23.1. Not something I can test: Windows, AMD, an integrated GPU, RADV's batched submits.
mooki000 commented 2026-11-12 01:22:39 +00:00 (Migrated from github.com)

One thing from reading the code while the timing runs build, in part 2 (the look only after a CPU signal). I think there is a window in which a readback can go through without waiting for work that schedule() has just let on. It comes from where ev_cpu_settled is stored. Lines are gpu/vk/vk_replay.cpp at 798d625a.

The order today

  1. A thread settles: sync_settle() (2140) takes sync_order_mu, burst_begin(0), schedule() (2144).
  2. schedule() lets the met queues on, which replays their work into the open recording, and at its end stores ev_cpu_settled = signals (2094).
  3. Only then burst_end() (2145) takes replay_mu (1920) and finish_recording() runs memory_end_submission (1931), which is where the pages that work writes back get their new value in the page table.

The window is between 2 and 3. A readback on another thread calls sync_settle_met() (2463): the two counters are equal (2159), so it returns without taking sync_order_mu. It then asks the page table (memory_writeback_value, 2476), gets the value from before the work that was just let on, finds it complete, and returns without a wait. That is the read #2125 was for: the range's writer is not recorded yet and its write-back not known. On main the readback took sync_order_mu whenever anything was stopped, so it could not get past a settle that was still open; with RX_VK_READBACK_SETTLE_ALWAYS=1 it still does. The window is short, but burst_end() waits for replay_mu inside it. The same holds for the other places that call schedule() between burst_begin and burst_end (2171, 4559, 4635), and drain() (2105) goes on recording forced queues after the store.

Shown without a game. A fixture that cuts schedule, sync_settle, sync_settle_met, the page table and the readback's choice of wait out of this branch and holds the settling thread at burst_end:

default:                  before burst publication: signals=1 settled=1 read_returned=1 waited_recording=0   FAIL
RX_VK_READBACK_SETTLE_ALWAYS:  before burst publication: signals=1 settled=1 read_returned=0                  ok (it then waits for recording 2)

The fixture (it imports the helpers of your tests/test_readback_part.py): python -I <fixture> --source <checkout of this branch> --output <a folder> [--always]. I ran it with g++ 16.2; I have not seen a wrong frame from this in the game, and the flash check is still to come.

A way to close it, if you agree it is one: store ev_cpu_settled after the burst's recording is finished and its pages are noted (after burst_end() in each caller, with schedule() returning the count it read), or let the readback's quick test also see that a settle is open.

One thing from reading the code while the timing runs build, in part 2 (the look only after a CPU signal). I think there is a window in which a readback can go through without waiting for work that `schedule()` has just let on. It comes from where `ev_cpu_settled` is stored. Lines are `gpu/vk/vk_replay.cpp` at `798d625a`. **The order today** 1. A thread settles: `sync_settle()` (2140) takes `sync_order_mu`, `burst_begin(0)`, `schedule()` (2144). 2. `schedule()` lets the met queues on, which replays their work into the open recording, and at its end stores `ev_cpu_settled = signals` (2094). 3. Only then `burst_end()` (2145) takes `replay_mu` (1920) and `finish_recording()` runs `memory_end_submission` (1931), which is where the pages that work writes back get their new value in the page table. **The window** is between 2 and 3. A readback on another thread calls `sync_settle_met()` (2463): the two counters are equal (2159), so it returns without taking `sync_order_mu`. It then asks the page table (`memory_writeback_value`, 2476), gets the value from before the work that was just let on, finds it complete, and returns without a wait. That is the read #2125 was for: the range's writer is not recorded yet and its write-back not known. On main the readback took `sync_order_mu` whenever anything was stopped, so it could not get past a settle that was still open; with `RX_VK_READBACK_SETTLE_ALWAYS=1` it still does. The window is short, but `burst_end()` waits for `replay_mu` inside it. The same holds for the other places that call `schedule()` between `burst_begin` and `burst_end` (2171, 4559, 4635), and `drain()` (2105) goes on recording forced queues after the store. **Shown without a game.** A fixture that cuts `schedule`, `sync_settle`, `sync_settle_met`, the page table and the readback's choice of wait out of this branch and holds the settling thread at `burst_end`: ``` default: before burst publication: signals=1 settled=1 read_returned=1 waited_recording=0 FAIL RX_VK_READBACK_SETTLE_ALWAYS: before burst publication: signals=1 settled=1 read_returned=0 ok (it then waits for recording 2) ``` [The fixture](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/397bffc3a030cea0f203bd8cf882ea57d31fd7c6/hookshot-2602-settle-publication-fixture.py) (it imports the helpers of your `tests/test_readback_part.py`): `python -I <fixture> --source <checkout of this branch> --output <a folder> [--always]`. I ran it with g++ 16.2; I have not seen a wrong frame from this in the game, and the flash check is still to come. **A way to close it**, if you agree it is one: store `ev_cpu_settled` after the burst's recording is finished and its pages are noted (after `burst_end()` in each caller, with `schedule()` returning the count it read), or let the readback's quick test also see that a settle is open.
mooki000 commented 2026-11-12 01:25:04 +00:00 (Migrated from github.com)

A second one, in part 3 (the flush with nothing held), and only on the batched path, which your description lists as checked by a source test only. Same file and commit.

What the flag is meant to guarantee (your comment at 397): a thread that finds a recording in the maps and waits for it never skips the flush its wait needs. The flag is raised before the maps are written (399) and again after the append (435). In between, a flush of older submits clears it (249).

The order that breaks it

  1. Producer: pend_held = true (399), its recording goes into the maps (406 to 424).
  2. Another thread flushes the older batch: queue_flush_locked() ends with pend_held = false (249).
  3. The producer waits for, or already holds, dev.present_mu (428) and has not appended yet (434) nor raised the flag again (435).
  4. A readback finds the new recording in the maps and waits for it: queue_flush_for(FR_WAIT) (2256) sees the flag down and returns without the mutex (263); the wait that follows has no time limit (2488, 2489) and is for work that is still to be held.

After that the readback waits until something else sends the batch: the producer's own batch_gpu_idle() check (443), the cap, a poll flush or a present. So it is a wait of up to a frame on the path this pull request frees from exactly that, not a wrong read. With the mutex always taken, as on main, the waiter in step 4 queued behind the producer when the producer already held the mutex, and flushed after it; when both were waiting for the mutex main could lose the same way, so main was not strict here either, but the quick test makes the skip certain for the whole of step 3.

Shown without a game. The real submit_recording, queue_flush_locked, queue_flush_for and wait_rec_seq bodies cut from this branch, Vulkan replaced by a model that says what was sent, the producer paused before its append with the mutex held:

default:                      new part mapped=2 after old flush pend_held=0 ... wait_ok=0 GPU=1 target=2 held_cbs=1   FAIL: waiter skipped its flush
quick test removed (control): new part mapped=2 after old flush pend_held=0 ... wait_ok=1 GPU=2 target=2 held_cbs=0   PASS

The fixture: python -I <fixture> --source <checkout of this branch> --output <a folder> [--locked-control]. It fails by default and passes with the control with g++ 16.2 and clang++ 23.1 here. (The first fixture now builds with clang++ too, same result: updated.) I have no RADV card, so I cannot say how often this order happens in play.

A way to close it: count the producers that have raised the flag and not appended yet (up at 399, down after 435 under the mutex), and let the quick test return only when the flag is down and that count is zero; or have the flush leave the flag up while that count is not zero.

A second one, in part 3 (the flush with nothing held), and only on the batched path, which your description lists as checked by a source test only. Same file and commit. **What the flag is meant to guarantee** (your comment at 397): a thread that finds a recording in the maps and waits for it never skips the flush its wait needs. The flag is raised before the maps are written (399) and again after the append (435). In between, a flush of older submits clears it (249). **The order that breaks it** 1. Producer: `pend_held = true` (399), its recording goes into the maps (406 to 424). 2. Another thread flushes the older batch: `queue_flush_locked()` ends with `pend_held = false` (249). 3. The producer waits for, or already holds, `dev.present_mu` (428) and has not appended yet (434) nor raised the flag again (435). 4. A readback finds the new recording in the maps and waits for it: `queue_flush_for(FR_WAIT)` (2256) sees the flag down and returns without the mutex (263); the wait that follows has no time limit (2488, 2489) and is for work that is still to be held. After that the readback waits until something else sends the batch: the producer's own `batch_gpu_idle()` check (443), the cap, a poll flush or a present. So it is a wait of up to a frame on the path this pull request frees from exactly that, not a wrong read. With the mutex always taken, as on main, the waiter in step 4 queued behind the producer when the producer already held the mutex, and flushed after it; when both were waiting for the mutex main could lose the same way, so main was not strict here either, but the quick test makes the skip certain for the whole of step 3. **Shown without a game.** The real `submit_recording`, `queue_flush_locked`, `queue_flush_for` and `wait_rec_seq` bodies cut from this branch, Vulkan replaced by a model that says what was sent, the producer paused before its append with the mutex held: ``` default: new part mapped=2 after old flush pend_held=0 ... wait_ok=0 GPU=1 target=2 held_cbs=1 FAIL: waiter skipped its flush quick test removed (control): new part mapped=2 after old flush pend_held=0 ... wait_ok=1 GPU=2 target=2 held_cbs=0 PASS ``` [The fixture](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/e84fdeb601d7bb1ce3a86d08957323f77372da4a/hookshot-2602-batch-publication-fixture.py): `python -I <fixture> --source <checkout of this branch> --output <a folder> [--locked-control]`. It fails by default and passes with the control with g++ 16.2 and clang++ 23.1 here. (The first fixture now builds with clang++ too, same result: [updated](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/e84fdeb601d7bb1ce3a86d08957323f77372da4a/hookshot-2602-settle-publication-fixture.py).) I have no RADV card, so I cannot say how often this order happens in play. **A way to close it:** count the producers that have raised the flag and not appended yet (up at 399, down after 435 under the mutex), and let the quick test return only when the flag is down and that count is zero; or have the flush leave the flag up while that count is not zero.
hckrman101 commented 2026-11-12 01:29:43 +00:00 (Migrated from github.com)

@mooki000 thanks for testing. Heads up: I pushed 9daa0718 on top of 798d625a. It fixes two review findings: the readback's part wait now gives way to the whole-submission wait while any part reads guest memory in place (direct-bound pools, live mirror syncs), and the look at stopped queues also covers events in guest memory. On a GPU with enough VRAM for mirrors it shouldn't change timing (no in-place reads in our Field/Kokiri logs). Your 798d625a numbers still count for perf; the flash check would be most useful on 9daa0718.

@mooki000 thanks for testing. Heads up: I pushed 9daa0718 on top of 798d625a. It fixes two review findings: the readback's part wait now gives way to the whole-submission wait while any part reads guest memory in place (direct-bound pools, live mirror syncs), and the look at stopped queues also covers events in guest memory. On a GPU with enough VRAM for mirrors it shouldn't change timing (no in-place reads in our Field/Kokiri logs). Your 798d625a numbers still count for perf; the flash check would be most useful on 9daa0718.
mooki000 commented 2026-11-12 01:31:13 +00:00 (Migrated from github.com)

@hckrman101 thanks, the flash check and the timing will be on 9daa0718 then (I build that one; the forest table above stays as the picture of main).

One note so the two comments above do not get lost under the new head: both still apply to 9daa0718 as far as I can see.

  • The settle counter: schedule() still stores ev_cpu_settled at its end (now line 2107) and sync_settle() still calls burst_end() after it, so the quick test in sync_settle_met() can still pass before the pages of the work just let on are noted. The new ev_wait_in_memory test sits beside it and does not change that order. My fixture cuts the old quick test out of the source and no longer builds against the new one; I'll adjust it and update the link.
  • The held-submit flag: lines 249, 263, 400 and 436 are as before, and the batch fixture gives the same result on 9daa0718 (fails by default, passes with the quick test removed; g++ 16.2).
@hckrman101 thanks, the flash check and the timing will be on `9daa0718` then (I build that one; the forest table above stays as the picture of main). One note so the two comments above do not get lost under the new head: both still apply to `9daa0718` as far as I can see. - The settle counter: `schedule()` still stores `ev_cpu_settled` at its end (now line 2107) and `sync_settle()` still calls `burst_end()` after it, so the quick test in `sync_settle_met()` can still pass before the pages of the work just let on are noted. The new `ev_wait_in_memory` test sits beside it and does not change that order. My fixture cuts the old quick test out of the source and no longer builds against the new one; I'll adjust it and update the link. - The held-submit flag: lines 249, 263, 400 and 436 are as before, and the batch fixture gives the same result on `9daa0718` (fails by default, passes with the quick test removed; g++ 16.2).
hckrman101 commented 2026-11-12 01:32:12 +00:00 (Migrated from github.com)

@mooki000 thanks, both are real as far as I can see from the code, and both are good catches.

  • Settle window (part 2): agreed. ev_cpu_settled is published before burst_end() notes the pages, so a readback can pass the quick test and read the old value: exactly the read #2125 is for. The fix will publish the count only after the burst's recording is finished and its pages are noted, for every schedule() caller and drain().
  • Held-submit flag (part 3): agreed. A flush of the older batch can drop the flag between the producer's mapping and its append, and the waiter then skips the flush it needs. Not a wrong read, but up to a frame of waiting on exactly the path this PR is meant to free. The fix will keep the quick path closed while any producer has raised the flag and not appended yet.

Both get tests of their own that fail on 9daa0718 and pass with the fix. I'll push them on top and ping here. Until then the PR stays in draft. Your flash check and timing on 9daa0718 are still useful, since neither change should move the frame rate on NVIDIA.

@mooki000 thanks, both are real as far as I can see from the code, and both are good catches. - **Settle window (part 2):** agreed. `ev_cpu_settled` is published before `burst_end()` notes the pages, so a readback can pass the quick test and read the old value: exactly the read #2125 is for. The fix will publish the count only after the burst's recording is finished and its pages are noted, for every `schedule()` caller and `drain()`. - **Held-submit flag (part 3):** agreed. A flush of the older batch can drop the flag between the producer's mapping and its append, and the waiter then skips the flush it needs. Not a wrong read, but up to a frame of waiting on exactly the path this PR is meant to free. The fix will keep the quick path closed while any producer has raised the flag and not appended yet. Both get tests of their own that fail on `9daa0718` and pass with the fix. I'll push them on top and ping here. Until then the PR stays in draft. Your flash check and timing on `9daa0718` are still useful, since neither change should move the frame rate on NVIDIA.
mooki000 commented 2026-11-12 01:33:56 +00:00 (Migrated from github.com)

Thanks. In case it helps for the two tests: the settle fixture adjusted to 9daa0718 is here. On that head, with g++ 16.2 and clang++ 23.1:

default:   before burst publication: signals=1 settled=1 read_returned=1 waited_recording=0   FAIL
--always:  before burst publication: signals=1 settled=1 read_returned=0                      ok (then waits for recording 2)

It models an event the CPU signalled (ev_wait_in_memory false) and asserts that no later in-place part hides the window, so 9daa0718's new guards are not what it trips over. The batch fixture linked above runs on 9daa0718 unchanged.

Timing and the flash check on 9daa0718 are building; numbers follow here.

Thanks. In case it helps for the two tests: the settle fixture adjusted to `9daa0718` is [here](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/198b82be9ce003b28e1a2a352b28f867352a4306/hookshot-2602-settle-publication-fixture.py). On that head, with g++ 16.2 and clang++ 23.1: ``` default: before burst publication: signals=1 settled=1 read_returned=1 waited_recording=0 FAIL --always: before burst publication: signals=1 settled=1 read_returned=0 ok (then waits for recording 2) ``` It models an event the CPU signalled (`ev_wait_in_memory` false) and asserts that no later in-place part hides the window, so `9daa0718`'s new guards are not what it trips over. The batch fixture linked above runs on `9daa0718` unchanged. Timing and the flash check on `9daa0718` are building; numbers follow here.
mooki000 commented 2026-11-12 01:36:53 +00:00 (Migrated from github.com)

A third one, on 9daa0718's new guard (the part wait gives way while a later part reads guest memory in place). The guard goes by note_in_place_read(), and one way of reading guest memory in place does not call it.

  • note_in_place_read() (gpu/vk/vk_memory.cpp 1652) is called at five places: 2443, 2750, 2782, 2810, 3118. Its comment names them: a pool bound with no mirror, a mirror's dirty pages no capture holds, a mirror's first copy.
  • gpu_buffer() (2578) hands out the imported guest buffer itself (p->buffer) and does not call it. Its two users give that buffer to the GPU to read at the GPU's time: the GPU unswizzle's source (gpu/vk/vk_texture.cpp 1142, bound as a storage buffer at 1177) and the texel buffer views (3580, 3601).
  • So a submission whose first part writes a page back and whose later part reads guest memory through one of those two leaves memory_in_place_part() where it was; in_place_after is false (vk_replay.cpp 2509) and the readback waits for the first part only (2511). That is the early return issue 997 was about, for these two readers.

Shown without a game. The real page table, readback choice, recording bookkeeping, marker and gpu_buffer cut from 9daa0718; two parts, the first writes the page back, the second takes it through gpu_buffer:

default:                         FAIL  a later raw guest-buffer read keeps the whole-submission wait   (waits for part 1)
note added at that one call (control):  ok                                                              (waits for the last part)

The fixture: python -I <fixture> --source <checkout> --output <a folder> [--noted-control]; g++ 16.2 and clang++ 23.1, same result. I am not claiming a wrong picture from it: whether the game overwrites a texture's source or a texel buffer's memory that soon I have not shown.

By reading only, not run: memory_rb_refresh copies pp->buffer into the mirror at the GPU's time (2567) without the note as well.

A way to close it: call note_in_place_read() in gpu_buffer() itself, so every present and future user of the raw buffer is covered, and at the refresh copy.

A third one, on `9daa0718`'s new guard (the part wait gives way while a later part reads guest memory in place). The guard goes by `note_in_place_read()`, and one way of reading guest memory in place does not call it. - `note_in_place_read()` (`gpu/vk/vk_memory.cpp` 1652) is called at five places: 2443, 2750, 2782, 2810, 3118. Its comment names them: a pool bound with no mirror, a mirror's dirty pages no capture holds, a mirror's first copy. - `gpu_buffer()` (2578) hands out the imported guest buffer itself (`p->buffer`) and does not call it. Its two users give that buffer to the GPU to read at the GPU's time: the GPU unswizzle's source (`gpu/vk/vk_texture.cpp` 1142, bound as a storage buffer at 1177) and the texel buffer views (3580, 3601). - So a submission whose first part writes a page back and whose later part reads guest memory through one of those two leaves `memory_in_place_part()` where it was; `in_place_after` is false (`vk_replay.cpp` 2509) and the readback waits for the first part only (2511). That is the early return issue 997 was about, for these two readers. **Shown without a game.** The real page table, readback choice, recording bookkeeping, marker and `gpu_buffer` cut from `9daa0718`; two parts, the first writes the page back, the second takes it through `gpu_buffer`: ``` default: FAIL a later raw guest-buffer read keeps the whole-submission wait (waits for part 1) note added at that one call (control): ok (waits for the last part) ``` [The fixture](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/071400e45f4e2d32bbc5d9bd406438f4f2636f40/hookshot-2602-direct-reader-fixture.py): `python -I <fixture> --source <checkout> --output <a folder> [--noted-control]`; g++ 16.2 and clang++ 23.1, same result. I am not claiming a wrong picture from it: whether the game overwrites a texture's source or a texel buffer's memory that soon I have not shown. By reading only, not run: `memory_rb_refresh` copies `pp->buffer` into the mirror at the GPU's time (2567) without the note as well. **A way to close it:** call `note_in_place_read()` in `gpu_buffer()` itself, so every present and future user of the raw buffer is covered, and at the refresh copy.
mooki000 commented 2026-11-12 01:39:42 +00:00 (Migrated from github.com)

The refresh copy from my last comment is now run as well, no longer only read: the same fixture with --refresh cuts the real memory_rb_refresh body, which records one copy from the imported guest buffer into the mirror, and the marker stays where it was:

--refresh:                  FAIL  a later guest-to-mirror refresh keeps the whole-submission wait   (waits for part 1)
--refresh --noted-control:  ok                                                                       (waits for the last part)

g++ 16.2 and clang++ 23.1 on 9daa0718, same result; the gpu_buffer case is unchanged.

The refresh copy from my last comment is now run as well, no longer only read: the [same fixture](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/6d8380600e98c44d2421a48cf1c457db982d1fc6/hookshot-2602-direct-reader-fixture.py) with `--refresh` cuts the real `memory_rb_refresh` body, which records one copy from the imported guest buffer into the mirror, and the marker stays where it was: ``` --refresh: FAIL a later guest-to-mirror refresh keeps the whole-submission wait (waits for part 1) --refresh --noted-control: ok (waits for the last part) ``` g++ 16.2 and clang++ 23.1 on `9daa0718`, same result; the `gpu_buffer` case is unchanged.
hckrman101 commented 2026-11-12 01:45:18 +00:00 (Migrated from github.com)

@mooki000 thanks again. Both of these were on 9daa0718; 4ba31740 (now the head) handles them:

  • gpu_buffer(): it now calls note_in_place_read() itself, as you suggested, so the GPU unswizzle's source and the texel buffer views are covered, and so is any future user of the raw buffer.
  • memory_rb_refresh: deliberately not noted, with the reason in the comment at note_in_place_read and in the PR description's "trade-off against main" section. Those copies run in every part's uploads and are how a CPU write to a page it reads back reaches the mirror at all. A later part that writes such a page again is waited for through its own part. Noting the refresh would put an in-place read in every part and cancel the change. So what it leaves open is other data sharing a 4 KB page with a readback range, the same as before #2419. If you think that residual is worse than it sounds (e.g. a readback range that regularly shares its page with data a later part reads), I'd like to hear it.

4ba31740 also has your two races fixed (the count is published at burst_end, and a held submit is mapped only after it is appended under present_mu), plus a second review. First field pair on it: fix 77.2 fps / 47.0 ms CPU, main 61.1 / 75.5. The flash check is running.

@mooki000 thanks again. Both of these were on `9daa0718`; `4ba31740` (now the head) handles them: - **`gpu_buffer()`:** it now calls `note_in_place_read()` itself, as you suggested, so the GPU unswizzle's source and the texel buffer views are covered, and so is any future user of the raw buffer. - **`memory_rb_refresh`:** deliberately not noted, with the reason in the comment at `note_in_place_read` and in the PR description's "trade-off against main" section. Those copies run in every part's uploads and are how a CPU write to a page it reads back reaches the mirror at all. A later part that writes such a page again is waited for through its own part. Noting the refresh would put an in-place read in every part and cancel the change. So what it leaves open is other data sharing a 4 KB page with a readback range, the same as before #2419. If you think that residual is worse than it sounds (e.g. a readback range that regularly shares its page with data a later part reads), I'd like to hear it. `4ba31740` also has your two races fixed (the count is published at `burst_end`, and a held submit is mapped only after it is appended under `present_mu`), plus a second review. First field pair on it: fix 77.2 fps / 47.0 ms CPU, main 61.1 / 75.5. The flash check is running.
hckrman101 commented 2026-11-12 01:45:36 +00:00 (Migrated from github.com)

Status: ready for review at 4ba31740.

Measured on 4ba31740 (Hyrule Field, RTX 3090 / 7900X, 2560x1440, uncapped, same session): fix 77.2 fps / 47.0 ms CPU a frame, main 61.1 fps / 75.5 ms. Earlier builds of the change: 76.8–78.7 fps in the field; Kokiri 65.3 / 58.2 fps against main's 40.5 / 45.9 (798d625a).

Review:

  • An independent review of 798d625a found the in-place reads in later parts and the events in guest memory; both fixed in 9daa0718.
  • @mooki000 found two races on 9daa0718: the look being counted before its burst's pages were noted, and a held submit being found before it was held. Both fixed in 4ba31740, each with a test that fails on 9daa0718.
  • The direct reader through gpu_buffer() is noted in 4ba31740. The readback-page refresh is left out on purpose; see "The trade-off against main" in the description.
  • A final review pass of 4ba31740 found it correct: no new lock-order problem (the one new edge is present_mu → rec_map_mu, and rec_map_mu is a leaf).

Still running: the framelog flash check on 4ba31740 in the field and Kokiri, and the batched-submit path (RADV, RX 580). Results will be posted here.

Not testable here: Windows, a no-mirror iGPU.

If something turns up after merging: RX_VK_READBACK_PART=0 and RX_VK_READBACK_SETTLE_ALWAYS=1 restore main's readback waits.

**Status: ready for review at `4ba31740`.** **Measured on `4ba31740`** (Hyrule Field, RTX 3090 / 7900X, 2560x1440, uncapped, same session): fix 77.2 fps / 47.0 ms CPU a frame, main 61.1 fps / 75.5 ms. Earlier builds of the change: 76.8–78.7 fps in the field; Kokiri 65.3 / 58.2 fps against main's 40.5 / 45.9 (`798d625a`). **Review:** - An independent review of `798d625a` found the in-place reads in later parts and the events in guest memory; both fixed in `9daa0718`. - @mooki000 found two races on `9daa0718`: the look being counted before its burst's pages were noted, and a held submit being found before it was held. Both fixed in `4ba31740`, each with a test that fails on `9daa0718`. - The direct reader through `gpu_buffer()` is noted in `4ba31740`. The readback-page refresh is left out on purpose; see "The trade-off against main" in the description. - A final review pass of `4ba31740` found it correct: no new lock-order problem (the one new edge is `present_mu` → `rec_map_mu`, and `rec_map_mu` is a leaf). **Still running:** the framelog flash check on `4ba31740` in the field and Kokiri, and the batched-submit path (RADV, RX 580). Results will be posted here. **Not testable here:** Windows, a no-mirror iGPU. **If something turns up after merging:** `RX_VK_READBACK_PART=0` and `RX_VK_READBACK_SETTLE_ALWAYS=1` restore main's readback waits.
hckrman101 commented 2026-11-12 01:48:44 +00:00 (Migrated from github.com)

Flash check on 4ba31740 (RX_VK_FRAMELOG + scripts/framelog.py): Hyrule Field 0 events in 2558 frames, Kokiri Forest 0 events in 1692 frames. Readbacks in the field pair: 1534-1551 of 3000 waited, at 0.10-0.12 ms each (main: 172-176 at 6.3-6.8 ms). 0-1 of 3000 took the look at the stopped queues, and 0 fell back to the whole-submission wait for in-place reads. The RADV batched-submit check is still to come.

Flash check on `4ba31740` (`RX_VK_FRAMELOG` + `scripts/framelog.py`): Hyrule Field 0 events in 2558 frames, Kokiri Forest 0 events in 1692 frames. Readbacks in the field pair: 1534-1551 of 3000 waited, at 0.10-0.12 ms each (main: 172-176 at 6.3-6.8 ms). 0-1 of 3000 took the look at the stopped queues, and 0 fell back to the whole-submission wait for in-place reads. The RADV batched-submit check is still to come.
mooki000 commented 2026-11-12 01:53:00 +00:00 (Migrated from github.com)

Checked 4ba31740 against the three fixtures, each adjusted to the moved code with its assertion unchanged (settle, held submit, in-place readers). g++ 16.2 and clang++ 23.1, same result with both:

fixture 9daa0718 4ba31740
the settle count and the burst's pages fails passes: the readback waits for recording 2
a held submit found before it is held fails passes: nothing is in the maps before the append, then the waiter flushes
gpu_buffer() as an in-place reader fails passes: waits for the last part
the refresh copy (--refresh) fails fails, as you decided: it is not noted

So the two races and the gpu_buffer() reader are closed as far as these can show. What the fixtures cannot show is a path that reaches the same state another way; I am reading the callers of schedule() and drain() and the two mutexes' order for that and will say what I find.

On the refresh copy: your argument makes sense to me from the code, and I would rather answer your question with counts than with a guess. I am taking them from this PC's runs (how often a readback range's page also holds data a later part reads through a refresh, forest and open field) and will post them with the timing and the flash check on 4ba31740.

Checked `4ba31740` against the three fixtures, each adjusted to the moved code with its assertion unchanged ([settle](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/cc76109ed471e872982e17987fe0f82cf5dc3453/hookshot-2602-settle-publication-fixture.py), [held submit](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/cc76109ed471e872982e17987fe0f82cf5dc3453/hookshot-2602-batch-publication-fixture.py), [in-place readers](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/cc76109ed471e872982e17987fe0f82cf5dc3453/hookshot-2602-direct-reader-fixture.py)). g++ 16.2 and clang++ 23.1, same result with both: | fixture | `9daa0718` | `4ba31740` | |---|---|---| | the settle count and the burst's pages | fails | passes: the readback waits for recording 2 | | a held submit found before it is held | fails | passes: nothing is in the maps before the append, then the waiter flushes | | `gpu_buffer()` as an in-place reader | fails | passes: waits for the last part | | the refresh copy (`--refresh`) | fails | fails, as you decided: it is not noted | So the two races and the `gpu_buffer()` reader are closed as far as these can show. What the fixtures cannot show is a path that reaches the same state another way; I am reading the callers of `schedule()` and `drain()` and the two mutexes' order for that and will say what I find. On the refresh copy: your argument makes sense to me from the code, and I would rather answer your question with counts than with a guess. I am taking them from this PC's runs (how often a readback range's page also holds data a later part reads through a refresh, forest and open field) and will post them with the timing and the flash check on `4ba31740`.
mooki000 commented 2026-11-12 02:06:53 +00:00 (Migrated from github.com)

First timing here, on an RTX 4090 (NVIDIA 615.78.08, i9-13900K, Arch Linux). This hold ran on 9daa0718, just before 4ba31740 came; the same arms on 4ba31740 are building and follow.

Kokiri Forest, uncapped (a save with camera turns, 2048x1152, the game's own upscaler, headless, 30 s after the load, base and fix in turn inside one exclusive hold, the same counting patch in both):

build fps mean frame 1 % low frames over 20 ms CPU a frame, all the game's threads waited in readbacks, a frame other work on the PC (cores, highest 5 s)
6f7ab2a7 (base) 63.2 15.81 ms 33.1 399 of 1895 77.0 ms 4.85 ms 1.8 (2.0)
9daa0718 80.5 12.42 ms 46.3 20 of 2415 54.9 ms 0.60 ms 1.6 (1.7)
6f7ab2a7 (base) 64.6 15.48 ms 37.0 375 of 1936 74.9 ms 4.71 ms 1.7 (2.1)
9daa0718 80.1 12.48 ms 46.5 17 of 2402 54.9 ms 0.59 ms 1.6 (1.8)

The fix: 3.20 ms a frame less (15.65 to 12.45), 16.4 fps more, 21.1 ms less CPU a frame, and the long frames gone (399 and 375 over 20 ms against 20 and 17). GPU time is the same in all four (5.91 to 5.99 ms a frame). For scale, in other holds tonight the build from before #2125 and #2419 (49f75dc8) ran this scene at 78.4 to 78.6 fps; the comparison with it in one hold comes on 4ba31740.

What main costs under a 60 cap, since most people play capped: the same scene with DLAA and the 60 cap, 49f75dc8 against main 9df055f2 (one exclusive hold, run before this pull request's builds were ready):

build fps mean frame 1 % low frames over 16.7 / 20 / 25 ms waited in readbacks, a frame load before → after
49f75dc8 58.4 17.12 ms 33.0 1024 / 85 / 11 of 1746 0.02 ms 7.9 → 19.4
main 9df055f2 54.5 18.34 ms 32.1 1005 / 456 / 43 of 1635 5.88 ms 19.4 → 9.9
49f75dc8 58.9 16.99 ms 41.1 1000 / 85 / 3 of 1764 0.02 ms 9.9 → 6.5
main 9df055f2 54.2 18.44 ms 31.6 1040 / 477 / 44 of 1626 5.97 ms 6.5 → 7.1

So even capped, main is 4.2 fps lower here, with about 5 times the frames over 20 ms. The same arm with the fix follows.

Reading 4ba31740 for the two races moved elsewhere: nothing found. All six callers of schedule() leave the count to burst_end() when a burst is open, and burst_end() publishes it after finish_recording(); drain()'s early return comes before any look and its forced resumes stay inside the burst. rec_map_mu is taken at four places, only ever inside present_mu and never the other way round.

Still to come from here: forest and open field on 4ba31740 against its base and against 49f75dc8, the 60-cap arm, your two switches, the flash check, and counts for the refresh question.

First timing here, on an RTX 4090 (NVIDIA 615.78.08, i9-13900K, Arch Linux). This hold ran on `9daa0718`, just before `4ba31740` came; the same arms on `4ba31740` are building and follow. **Kokiri Forest, uncapped** (a save with camera turns, 2048x1152, the game's own upscaler, headless, 30 s after the load, base and fix in turn inside one exclusive hold, the same counting patch in both): | build | fps | mean frame | 1 % low | frames over 20 ms | CPU a frame, all the game's threads | waited in readbacks, a frame | other work on the PC (cores, highest 5 s) | |---|---|---|---|---|---|---|---| | `6f7ab2a7` (base) | 63.2 | 15.81 ms | 33.1 | 399 of 1895 | 77.0 ms | 4.85 ms | 1.8 (2.0) | | `9daa0718` | 80.5 | 12.42 ms | 46.3 | 20 of 2415 | 54.9 ms | 0.60 ms | 1.6 (1.7) | | `6f7ab2a7` (base) | 64.6 | 15.48 ms | 37.0 | 375 of 1936 | 74.9 ms | 4.71 ms | 1.7 (2.1) | | `9daa0718` | 80.1 | 12.48 ms | 46.5 | 17 of 2402 | 54.9 ms | 0.59 ms | 1.6 (1.8) | The fix: 3.20 ms a frame less (15.65 to 12.45), 16.4 fps more, 21.1 ms less CPU a frame, and the long frames gone (399 and 375 over 20 ms against 20 and 17). GPU time is the same in all four (5.91 to 5.99 ms a frame). For scale, in other holds tonight the build from before #2125 and #2419 (49f75dc8) ran this scene at 78.4 to 78.6 fps; the comparison with it in one hold comes on `4ba31740`. **What main costs under a 60 cap**, since most people play capped: the same scene with DLAA and the 60 cap, 49f75dc8 against main `9df055f2` (one exclusive hold, run before this pull request's builds were ready): | build | fps | mean frame | 1 % low | frames over 16.7 / 20 / 25 ms | waited in readbacks, a frame | load before → after | |---|---|---|---|---|---|---| | 49f75dc8 | 58.4 | 17.12 ms | 33.0 | 1024 / 85 / 11 of 1746 | 0.02 ms | 7.9 → 19.4 | | main `9df055f2` | 54.5 | 18.34 ms | 32.1 | 1005 / 456 / 43 of 1635 | 5.88 ms | 19.4 → 9.9 | | 49f75dc8 | 58.9 | 16.99 ms | 41.1 | 1000 / 85 / 3 of 1764 | 0.02 ms | 9.9 → 6.5 | | main `9df055f2` | 54.2 | 18.44 ms | 31.6 | 1040 / 477 / 44 of 1626 | 5.97 ms | 6.5 → 7.1 | So even capped, main is 4.2 fps lower here, with about 5 times the frames over 20 ms. The same arm with the fix follows. **Reading `4ba31740`** for the two races moved elsewhere: nothing found. All six callers of `schedule()` leave the count to `burst_end()` when a burst is open, and `burst_end()` publishes it after `finish_recording()`; `drain()`'s early return comes before any look and its forced resumes stay inside the burst. `rec_map_mu` is taken at four places, only ever inside `present_mu` and never the other way round. Still to come from here: forest and open field on `4ba31740` against its base and against 49f75dc8, the 60-cap arm, your two switches, the flash check, and counts for the refresh question.
mooki000 commented 2026-11-12 02:31:33 +00:00 (Migrated from github.com)

Counts for the refresh question (whether leaving memory_rb_refresh out of the in-place readers can hand later work bytes newer than its submission expected).

How it was counted: 4ba31740 plus an observation-only counter, no wait changed, RX_VK_PARALLEL=0 so that a part belongs to one recording. It logs every readback after its wait, the refresh copies actually made (guest to mirror), the mirror bindings on pages that have had a readback, and each part's recording with whether it had completed. A readback is "sharing" when a later part of the same submission value refreshes its page and binds bytes of that page outside the readback's own range. RTX 4090, headless, one 30 s window a scene.

scene frames readbacks sharing of those, the binding part's recording not complete when the readback's wait returned
Kokiri Forest 930 20,371 0 0
Hyrule Field 2,707 45,916 610 (1.33 %; 0.23 a frame, on 305 frames) 0

All 610 are one shape: a 192-byte readback with a read-only 16-byte binding that starts 464 bytes before it and a read-only 64-byte binding that starts 192 bytes after it, both outside every readback range of that submission. In each case the recording of the part that binds them had completed when the readback's wait returned, so the game could not have changed those bytes under it.

So in these two scenes the choice costs nothing: the sharing is rare, read-only, and never still in flight. This is an answer in numbers for one card and two scenes, not a proof:

  • a bound range is an upper bound on what a shader reads, and a load through a pointer without a binding is not seen at all;
  • bindings made before a page first had a readback are not logged;
  • serial replay only, 30 s a scene.

The counter is a small local patch on top of 4ba31740 (off unless its switch is set). If you want it for another scene or for RADV, say so and I put it in a gist.

**Counts for the refresh question** (whether leaving `memory_rb_refresh` out of the in-place readers can hand later work bytes newer than its submission expected). How it was counted: `4ba31740` plus an observation-only counter, no wait changed, `RX_VK_PARALLEL=0` so that a part belongs to one recording. It logs every readback after its wait, the refresh copies actually made (guest to mirror), the mirror bindings on pages that have had a readback, and each part's recording with whether it had completed. A readback is "sharing" when a later part of the same submission value refreshes its page and binds bytes of that page outside the readback's own range. RTX 4090, headless, one 30 s window a scene. | scene | frames | readbacks | sharing | of those, the binding part's recording not complete when the readback's wait returned | |---|---|---|---|---| | Kokiri Forest | 930 | 20,371 | 0 | 0 | | Hyrule Field | 2,707 | 45,916 | 610 (1.33 %; 0.23 a frame, on 305 frames) | 0 | All 610 are one shape: a 192-byte readback with a read-only 16-byte binding that starts 464 bytes before it and a read-only 64-byte binding that starts 192 bytes after it, both outside every readback range of that submission. In each case the recording of the part that binds them had completed when the readback's wait returned, so the game could not have changed those bytes under it. So in these two scenes the choice costs nothing: the sharing is rare, read-only, and never still in flight. This is an answer in numbers for one card and two scenes, not a proof: - a bound range is an upper bound on what a shader reads, and a load through a pointer without a binding is not seen at all; - bindings made before a page first had a readback are not logged; - serial replay only, 30 s a scene. The counter is a small local patch on top of `4ba31740` (off unless its switch is set). If you want it for another scene or for RADV, say so and I put it in a gist.
hckrman101 commented 2026-11-12 02:40:22 +00:00 (Migrated from github.com)

Batched-submit path on hardware: RX 580 (RADV, GCN), with batching forced on (RX_VK_BATCH=1; it is off by default on GCN), Kokiri Forest, 150 s of play per run, headless.

run build / env fps (min) readbacks waited per 3000
A1 (cold) 4ba31740 defaults 21.3 (18.5) 401 × 12.5 ms
A2 (warm) 4ba31740 defaults 22.0 (19.7) 366 × 13.7 ms
B1 (warm) 4ba31740, RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1 23.3 (23.1) 164 × 30.5 ms
C1 (cold) main 6f7ab2a7 23.3 (21.6) 164 × 31.5 ms
  • No hang, no device loss, no Vulkan errors, and nothing misrendered in screenshots of every run. The held-submit order in 4ba31740 works on the batched path.
  • On this GPU-bound card the part wait does not help. Readbacks wait about 2.3× as often and each wait is about 2.3× shorter, so the total wait is unchanged, and fps is about 1.3 (5-6 %) lower than with the switches off or on main. That is one or two runs each, so it is a small gap, but it showed in both the cold and the warm pair. I'm looking into why.
  • Not covered: RDNA / Steam Deck, where batching is on by default.
Batched-submit path on hardware: RX 580 (RADV, GCN), with batching forced on (`RX_VK_BATCH=1`; it is off by default on GCN), Kokiri Forest, 150 s of play per run, headless. | run | build / env | fps (min) | readbacks waited per 3000 | |---|---|---|---| | A1 (cold) | `4ba31740` defaults | 21.3 (18.5) | 401 × 12.5 ms | | A2 (warm) | `4ba31740` defaults | 22.0 (19.7) | 366 × 13.7 ms | | B1 (warm) | `4ba31740`, `RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1` | 23.3 (23.1) | 164 × 30.5 ms | | C1 (cold) | main `6f7ab2a7` | 23.3 (21.6) | 164 × 31.5 ms | - No hang, no device loss, no Vulkan errors, and nothing misrendered in screenshots of every run. The held-submit order in `4ba31740` works on the batched path. - On this GPU-bound card the part wait does not help. Readbacks wait about 2.3× as often and each wait is about 2.3× shorter, so the total wait is unchanged, and fps is about 1.3 (5-6 %) lower than with the switches off or on main. That is one or two runs each, so it is a small gap, but it showed in both the cold and the warm pair. I'm looking into why. - Not covered: RDNA / Steam Deck, where batching is on by default.
luiz-august0 commented 2026-11-12 02:41:28 +00:00 (Migrated from github.com)

The same readback wait on Windows, with an 8 GB NVIDIA card and a 6-core CPU

My logs show the wait this PR removes, on a build that contains fce53453. These are the "main" numbers for another platform; I have not run this PR's build yet.

  • Build: v2.1.0-rc.3-191-gef81aee7 (nightly)
  • PC: RTX 4060 8 GB (NVIDIA 610.88, PCIe 3.0 x8 on an A320 board), Ryzen 5 5600X (6 cores, 12 threads), 32 GB, Windows 11 Pro 26200
  • Settings: fullscreen, output 1920x1080 on a 3840x2160 60 Hz display, DLSS Quality (1280x720 in), Reflex on + boost, V-Sync Auto (mailbox), frame generation off,
    RX_VK_WB_MAX_KB=1024 in hookshot_env.txt. VRAM budget Auto in session A, 4.8 GB in session B.
session A (hookshot_log.1.txt) session B (hookshot_log.txt)
frames counted 100,920 4,560
fps in gameplay, median (p10-p90) 41.7 (29.1-53.0) 38.5 (31.9-42.1)
GPU time a frame, median 9.9 ms 12.9 ms
readbacks that waited, of 3000 (median) 110 183
ms each (weighted mean) 9.1 12.5
time blocked in readbacks, per 3000 ~1000 ms ~2290 ms
waits a frame 0.96 0.94
time blocked a frame (a job thread, sometimes sead::MainThread) 8.6 ms 11.5 ms
  • The Presentation Thread's GPU-bound wait is 0.0 ms in every report, so this blocked read is where the GPU's time enters the frame here.
  • Each wait is about one frame's GPU time. That fits cause 1, the wait for the whole submission.
  • Compared with your main numbers (172-176 of 3000 at 6.3-6.8 ms on the 3090), each wait here is about twice as long, which fits the slower GPU.
  • Batched submits are RADV only, so I take it cause 3 does not apply here.
  • Session B also had the VRAM budget at 4.8 GB. That caps the buffer mirrors at 1 GB, and one 144 MB pool stayed in system memory, read over PCIe. So B's GPU time is higher, and so are
    its waits. That is a separate topic.

How I counted: from the log's readbacks of GPU results lines, against the N presents counter. Gameplay is the report periods with more than 300 draws a frame.

**The same readback wait on Windows, with an 8 GB NVIDIA card and a 6-core CPU** My logs show the wait this PR removes, on a build that contains `fce53453`. These are the "main" numbers for another platform; I have not run this PR's build yet. - **Build:** v2.1.0-rc.3-191-gef81aee7 (nightly) - **PC:** RTX 4060 8 GB (NVIDIA 610.88, PCIe 3.0 x8 on an A320 board), Ryzen 5 5600X (6 cores, 12 threads), 32 GB, Windows 11 Pro 26200 - **Settings:** fullscreen, output 1920x1080 on a 3840x2160 60 Hz display, DLSS Quality (1280x720 in), Reflex on + boost, V-Sync Auto (mailbox), frame generation off, `RX_VK_WB_MAX_KB=1024` in `hookshot_env.txt`. VRAM budget Auto in session A, 4.8 GB in session B. | | session A (`hookshot_log.1.txt`) | session B (`hookshot_log.txt`) | |---|---|---| | frames counted | 100,920 | 4,560 | | fps in gameplay, median (p10-p90) | 41.7 (29.1-53.0) | 38.5 (31.9-42.1) | | GPU time a frame, median | 9.9 ms | 12.9 ms | | readbacks that waited, of 3000 (median) | 110 | 183 | | ms each (weighted mean) | 9.1 | 12.5 | | time blocked in readbacks, per 3000 | ~1000 ms | ~2290 ms | | waits a frame | 0.96 | 0.94 | | time blocked a frame (a job thread, sometimes `sead::MainThread`) | 8.6 ms | 11.5 ms | - The Presentation Thread's `GPU-bound wait` is 0.0 ms in every report, so this blocked read is where the GPU's time enters the frame here. - Each wait is about one frame's GPU time. That fits cause 1, the wait for the whole submission. - Compared with your main numbers (172-176 of 3000 at 6.3-6.8 ms on the 3090), each wait here is about twice as long, which fits the slower GPU. - Batched submits are RADV only, so I take it cause 3 does not apply here. - Session B also had the VRAM budget at 4.8 GB. That caps the buffer mirrors at 1 GB, and one 144 MB pool stayed in system memory, read over PCIe. So B's GPU time is higher, and so are its waits. That is a separate topic. How I counted: from the log's `readbacks of GPU results` lines, against the `N presents` counter. Gameplay is the report periods with more than 300 draws a frame.
hckrman101 commented 2026-11-12 02:41:47 +00:00 (Migrated from github.com)

@mooki000 thank you, that's a lot of useful data.

  • Kokiri: +16 fps on 9daa0718 (63-65 → 80), 21 ms less CPU a frame, and the frames over 20 ms down from ~390 to ~20. That puts it just above your pre-#2125/#2419 build (78.4-78.6), which matches what we see in the field.
  • The 60-cap numbers are the most important part for players: main costs ~4 fps and about 5× the frames over 20 ms even when capped. I'm looking forward to the fix arm of that.
  • The refresh counts settle the trade-off well for these scenes: 0 sharing in Kokiri, 610 of 45,916 in the field, all read-only, all with the reading part's recording already complete. I'll link your comment from the description's trade-off section. No need for a gist for now; if the RX 580 slowdown leads us to touch that path, I'll ask.
  • And thanks for re-reading 4ba31740 for the races.

@luiz-august0 thanks, this is the Windows data we were missing. About one wait a frame at 9-12 ms, roughly one frame's GPU time, is the same cause-1 picture on a 4060 / 5600X. Right: batched submits are RADV only, so cause 3 doesn't apply to you. If you can run this PR's build (the Windows CI artifact of 4ba31740) with the same settings, the readbacks line and gameplay fps against that nightly would be very valuable. Windows is the one platform nobody has run it on yet. The 4.8 GB budget / 144 MB pool in system memory is worth its own issue.

@mooki000 thank you, that's a lot of useful data. - **Kokiri:** +16 fps on `9daa0718` (63-65 → 80), 21 ms less CPU a frame, and the frames over 20 ms down from ~390 to ~20. That puts it just above your pre-#2125/#2419 build (78.4-78.6), which matches what we see in the field. - **The 60-cap numbers** are the most important part for players: main costs ~4 fps and about 5× the frames over 20 ms even when capped. I'm looking forward to the fix arm of that. - **The refresh counts** settle the trade-off well for these scenes: 0 sharing in Kokiri, 610 of 45,916 in the field, all read-only, all with the reading part's recording already complete. I'll link your comment from the description's trade-off section. No need for a gist for now; if the RX 580 slowdown leads us to touch that path, I'll ask. - And thanks for re-reading `4ba31740` for the races. @luiz-august0 thanks, this is the Windows data we were missing. About one wait a frame at 9-12 ms, roughly one frame's GPU time, is the same cause-1 picture on a 4060 / 5600X. Right: batched submits are RADV only, so cause 3 doesn't apply to you. If you can run this PR's build (the Windows CI artifact of `4ba31740`) with the same settings, the readbacks line and gameplay fps against that nightly would be very valuable. Windows is the one platform nobody has run it on yet. The 4.8 GB budget / 144 MB pool in system memory is worth its own issue.
mooki000 commented 2026-11-12 03:00:31 +00:00 (Migrated from github.com)

The capped arm with the fix, as asked: Kokiri Forest at a player's settings (DLAA, the 60 cap, 2048x1152), headless, RTX 4090 (NVIDIA 615.78.08), base and fix in turn in one exclusive hold, all four runs counting. This hold ran on the pull request's head 4ba31740 just before the merge; the same on merged main is running now.

build fps mean frame 1 % low frames over 16.7 / 20 / 25 ms CPU a frame, all threads waited in readbacks, a frame
6f7ab2a7 (before the fix) 54.2 18.44 ms 28.6 1013 / 468 / 39 of 1625 83.3 ms 5.95 ms
4ba31740 (the fix) 59.3 16.87 ms 38.9 952 / 64 / 10 of 1777 57.3 ms 0.70 ms
6f7ab2a7 (before the fix) 53.9 18.55 ms 23.6 1009 / 484 / 47 of 1616 83.1 ms 5.95 ms
4ba31740 (the fix) 59.2 16.89 ms 39.6 966 / 71 / 7 of 1775 58.6 ms 0.70 ms

With the fix the scene holds 59.3 and 59.2 of 60 (before: 54.2 and 53.9), frames over 20 ms go from 468 and 484 to 64 and 71, over 25 ms from 39 and 47 to 10 and 7, and the game uses about 25 ms less CPU a frame. For scale, the build from before #2125 and #2419 in another hold tonight (so not a paired comparison): 58.4 and 58.9 fps, 85 and 85 frames over 20 ms.

Uncapped, the fix against the build from before the two changes, in one hold each (own upscaler), and the field against the base:

scene build fps (two runs) mean frame CPU a frame waited in readbacks, a frame
Kokiri Forest 4ba31740 (the fix) 79.2 / 81.2 12.63 / 12.31 ms 55.5 / 54.4 ms 0.57 / 0.56 ms
Kokiri Forest 49f75dc8 (before #2125 and #2419) 79.3 / 79.5 12.61 / 12.58 ms 54.6 / 53.6 ms 0.02 / 0.02 ms
Hyrule Field 4ba31740 (the fix) 92.6 / 92.1 10.80 / 10.86 ms 49.8 / 50.0 ms 0.33 / 0.33 ms
Hyrule Field 49f75dc8 (before #2125 and #2419) 91.5 / 91.9 10.93 / 10.88 ms 49.4 / 49.4 ms 0.02 / 0.02 ms
Hyrule Field 6f7ab2a7 (before the fix) 69.2 / 68.6 14.45 / 14.59 ms 71.1 / 71.8 ms 4.83 / 4.71 ms
Hyrule Field 4ba31740 (the fix) 92.8 / 92.7 10.78 / 10.79 ms 50.2 / 50.2 ms 0.41 / 0.31 ms

Against the old build the ranges overlap in both scenes, so nothing is left over on this PC: the fix gives back all of it. What remains of the readback wait is 0.6 to 0.7 ms a frame in the forest and about 0.3 ms in the field (0.02 ms on the old build).

Still coming from here, on merged main: this capped hold again, and the flash check with a window and the pixel frame log (a run with the readback wait switched off first, to show the check sees a flash at all).

**The capped arm with the fix**, as asked: Kokiri Forest at a player's settings (DLAA, the 60 cap, 2048x1152), headless, RTX 4090 (NVIDIA 615.78.08), base and fix in turn in one exclusive hold, all four runs counting. This hold ran on the pull request's head `4ba31740` just before the merge; the same on merged main is running now. | build | fps | mean frame | 1 % low | frames over 16.7 / 20 / 25 ms | CPU a frame, all threads | waited in readbacks, a frame | |---|---|---|---|---|---|---| | `6f7ab2a7` (before the fix) | 54.2 | 18.44 ms | 28.6 | 1013 / 468 / 39 of 1625 | 83.3 ms | 5.95 ms | | `4ba31740` (the fix) | 59.3 | 16.87 ms | 38.9 | 952 / 64 / 10 of 1777 | 57.3 ms | 0.70 ms | | `6f7ab2a7` (before the fix) | 53.9 | 18.55 ms | 23.6 | 1009 / 484 / 47 of 1616 | 83.1 ms | 5.95 ms | | `4ba31740` (the fix) | 59.2 | 16.89 ms | 39.6 | 966 / 71 / 7 of 1775 | 58.6 ms | 0.70 ms | With the fix the scene holds 59.3 and 59.2 of 60 (before: 54.2 and 53.9), frames over 20 ms go from 468 and 484 to 64 and 71, over 25 ms from 39 and 47 to 10 and 7, and the game uses about 25 ms less CPU a frame. For scale, the build from before #2125 and #2419 in another hold tonight (so not a paired comparison): 58.4 and 58.9 fps, 85 and 85 frames over 20 ms. **Uncapped, the fix against the build from before the two changes, in one hold each** (own upscaler), and the field against the base: | scene | build | fps (two runs) | mean frame | CPU a frame | waited in readbacks, a frame | |---|---|---|---|---|---| | Kokiri Forest | `4ba31740` (the fix) | 79.2 / 81.2 | 12.63 / 12.31 ms | 55.5 / 54.4 ms | 0.57 / 0.56 ms | | Kokiri Forest | 49f75dc8 (before #2125 and #2419) | 79.3 / 79.5 | 12.61 / 12.58 ms | 54.6 / 53.6 ms | 0.02 / 0.02 ms | | Hyrule Field | `4ba31740` (the fix) | 92.6 / 92.1 | 10.80 / 10.86 ms | 49.8 / 50.0 ms | 0.33 / 0.33 ms | | Hyrule Field | 49f75dc8 (before #2125 and #2419) | 91.5 / 91.9 | 10.93 / 10.88 ms | 49.4 / 49.4 ms | 0.02 / 0.02 ms | | Hyrule Field | `6f7ab2a7` (before the fix) | 69.2 / 68.6 | 14.45 / 14.59 ms | 71.1 / 71.8 ms | 4.83 / 4.71 ms | | Hyrule Field | `4ba31740` (the fix) | 92.8 / 92.7 | 10.78 / 10.79 ms | 50.2 / 50.2 ms | 0.41 / 0.31 ms | Against the old build the ranges overlap in both scenes, so nothing is left over on this PC: the fix gives back all of it. What remains of the readback wait is 0.6 to 0.7 ms a frame in the forest and about 0.3 ms in the field (0.02 ms on the old build). Still coming from here, on merged main: this capped hold again, and the flash check with a window and the pixel frame log (a run with the readback wait switched off first, to show the check sees a flash at all).
blushOFF commented 2026-11-12 03:18:57 +00:00 (Migrated from github.com)

Windows numbers, since Windows was on the "not yet verified" list. These ran on the first head (798d625a), not on the merged 4ba31740, so read them as the size of the effect on this platform, not as a check of the final code.

Windows 11, RTX 3070 8 GB (driver 610.88), i7-14700KF, the game's 2048x1152, RX_VK_IMAGE_BLOCK_MB=32 and RX_VK_WB_MAX_KB=1024. Teleport, then about 2 minutes standing, with arms alternated in one session (P O O P P O O P). P = the branch's defaults; O = the same build with RX_VK_READBACK_PART=0 and RX_VK_READBACK_SETTLE_ALWAYS=1 (so part 3 stays on in O).

Castle Town Market (uncapped in practice: GPU ~10.5 ms a frame, the replay is the limit):

arm fps replay a frame readbacks waited per 3000
P (4 runs) 54.3 / 54.2 / 55.9 / 54.5 13.8-14.4 ms 1114, 0.23 ms each
O (4 runs) 42.1 / 43.0 / 43.1 / 42.9 10.8-11.0 ms 171, 12.05 ms each (141 looked at the stopped queues)

So +27 % here (54.7 against 42.8 fps on average).

Hyrule Field at the teleport spot sits at the 60 cap with either arm (GPU ~9 ms), so it shows only the dips: the lowest 5 s window per run was 56.2-58.9 fps in P and 53.6-55.0 in O.

One O run in the field (the third) ended in VK_ERROR_DEVICE_LOST after 72 s; the 8 P runs and the other 7 O runs had none. O is the old wait path on a new build, so I would not pin it on this change, but it is the only device lost in the 43 runs on this card tonight.

Windows numbers, since Windows was on the "not yet verified" list. These ran on the first head (`798d625a`), not on the merged `4ba31740`, so read them as the size of the effect on this platform, not as a check of the final code. Windows 11, RTX 3070 8 GB (driver 610.88), i7-14700KF, the game's 2048x1152, `RX_VK_IMAGE_BLOCK_MB=32` and `RX_VK_WB_MAX_KB=1024`. Teleport, then about 2 minutes standing, with arms alternated in one session (P O O P P O O P). P = the branch's defaults; O = the same build with `RX_VK_READBACK_PART=0` and `RX_VK_READBACK_SETTLE_ALWAYS=1` (so part 3 stays on in O). **Castle Town Market** (uncapped in practice: GPU ~10.5 ms a frame, the replay is the limit): | arm | fps | replay a frame | readbacks waited per 3000 | |---|---|---|---| | P (4 runs) | 54.3 / 54.2 / 55.9 / 54.5 | 13.8-14.4 ms | 1114, 0.23 ms each | | O (4 runs) | 42.1 / 43.0 / 43.1 / 42.9 | 10.8-11.0 ms | 171, 12.05 ms each (141 looked at the stopped queues) | So +27 % here (54.7 against 42.8 fps on average). **Hyrule Field** at the teleport spot sits at the 60 cap with either arm (GPU ~9 ms), so it shows only the dips: the lowest 5 s window per run was 56.2-58.9 fps in P and 53.6-55.0 in O. One O run in the field (the third) ended in `VK_ERROR_DEVICE_LOST` after 72 s; the 8 P runs and the other 7 O runs had none. O is the old wait path on a new build, so I would not pin it on this change, but it is the only device lost in the 43 runs on this card tonight.
hckrman101 commented 2026-11-12 03:38:15 +00:00 (Migrated from github.com)

Follow-up on the RX 580 (RADV, GCN) runs: the earlier fix-on runs drew a different scene (about 2435 draws a frame against about 2050; the camera was farther back, from input timing on the way into play), so that 1.3 fps gap was partly a scene difference. I re-ran 4 alternating pairs of 4ba31740, warm, 120 s of play, Kokiri Forest:

batching fix draws a frame fps mean (lowest stretch) readbacks waited per 3000
on on 2424 / 2443 / 2431 22.2 / 22.0 / 21.9 (19.0-20.0) ~367 × 13.8 ms
on off 2422 / 2445 / 2424 22.6 / 22.5 / 22.4 (22.3-22.5) ~164 × 32 ms
off (GCN default) on 2446 21.1 (17.3) 444 × 10.8 ms
off (GCN default) off 2435 22.8 (22.7) 164 × 31.5 ms
  • Correctness: the same scene and the same camera framing in all 8 runs, so the fix does not change what the game reads back. No flashes, hangs or Vulkan errors.
  • Cost on this GPU-bound card: with batching on, about 0.5 fps (2 %), lower in every pair. With batching off, which is the GCN default, one pair shows about 7 % and lower lows. That is being repeated.
  • Why: when the GPU is a frame or more behind, a readback that returns after the first part meets the GPU still behind at the next one. Readbacks wait about 2.3× as often, each about 2.3× shorter, for about the same total. If the repeats confirm the cost, the follow-up is to fall back to the whole-submission wait when the GPU is more than one submission behind (or to keep the part wait off where the card is GPU-bound).
Follow-up on the RX 580 (RADV, GCN) runs: the earlier fix-on runs drew a different scene (about 2435 draws a frame against about 2050; the camera was farther back, from input timing on the way into play), so that 1.3 fps gap was partly a scene difference. I re-ran 4 alternating pairs of `4ba31740`, warm, 120 s of play, Kokiri Forest: | batching | fix | draws a frame | fps mean (lowest stretch) | readbacks waited per 3000 | |---|---|---|---|---| | on | on | 2424 / 2443 / 2431 | 22.2 / 22.0 / 21.9 (19.0-20.0) | ~367 × 13.8 ms | | on | off | 2422 / 2445 / 2424 | 22.6 / 22.5 / 22.4 (22.3-22.5) | ~164 × 32 ms | | off (GCN default) | on | 2446 | 21.1 (17.3) | 444 × 10.8 ms | | off (GCN default) | off | 2435 | 22.8 (22.7) | 164 × 31.5 ms | - **Correctness:** the same scene and the same camera framing in all 8 runs, so the fix does not change what the game reads back. No flashes, hangs or Vulkan errors. - **Cost on this GPU-bound card:** with batching on, about 0.5 fps (2 %), lower in every pair. With batching off, which is the GCN default, one pair shows about 7 % and lower lows. That is being repeated. - **Why:** when the GPU is a frame or more behind, a readback that returns after the first part meets the GPU still behind at the next one. Readbacks wait about 2.3× as often, each about 2.3× shorter, for about the same total. If the repeats confirm the cost, the follow-up is to fall back to the whole-submission wait when the GPU is more than one submission behind (or to keep the part wait off where the card is GPU-bound).
mooki000 commented 2026-11-12 03:51:45 +00:00 (Migrated from github.com)

The fix arm at the 60 cap, as asked, on the merged commit: once without a window and once in a real one; then the flash check. Linux 7.2.15-1-omarchy, RTX 4090 (NVIDIA 615.78.08), i9-13900K. Builds: merged main 09aac439 (this pull request as 76bbe4aa), the commit before it 6f7ab2a7, its last head 4ba31740, and 49f75dc8 (the last build I have from before #2125 and #2419); each is the exact commit plus one local counting patch that logs every buffer readback and how long it waited.

1. The 60 cap with DLAA, Kokiri Forest, 2048x1152 without a window

09aac439 59.4 and 59.3 fps against 6f7ab2a7 54.4 and 55.7 (-1.31 ms a frame, -7.2 %); frames over 20 ms 37 and 65 against 474 and 383, over 25 ms 4 and 7 against 44 and 24; 1 % low 41.7 and 39.4 against 32.2 and 32.5; CPU a frame 59.1 and 57.4 ms against 82.6 and 79.5; waited in readbacks a frame 0.78 and 0.69 ms against 5.97 and 5.48. The build from before #2125 and #2419 ran 58.4 and 58.9 fps here with 85 and 85 frames over 20 ms (the pair posted earlier, another hold), so the fix is back at that level.

hold/run build fps mean frame 1 % low frames over 16.7 / 20 / 25 ms CPU a frame, all threads waited in readbacks, a frame rest of the PC, cores (highest 5 s) counts
M1/1 6f7ab2a7 54.4 18.38 ms 32.2 1011 / 474 / 44 of 1631 82.6 ms 5.97 ms 1.8 (2.0) yes
M1/2 09aac439 59.4 16.84 ms 41.7 957 / 37 / 4 of 1780 59.1 ms 0.78 ms 1.7 (1.8) yes
M1/3 6f7ab2a7 55.7 17.96 ms 32.5 981 / 383 / 24 of 1669 79.5 ms 5.48 ms 1.9 (2.4) yes
M1/4 09aac439 59.3 16.88 ms 39.4 934 / 65 / 7 of 1776 57.4 ms 0.69 ms 1.8 (2.0) yes

2. The same pair in a real window (1920x1080 on a 60 Hz monitor, Hyprland on Wayland; the log says mailbox, V-Sync Auto)

09aac439 59.9 and 59.9 fps against 6f7ab2a7 55.9 and 56.2 (-1.15 ms a frame, -6.4 %); frames over 20 ms 82 and 67 against 665 and 616, over 25 ms 8 and 6 against 41 and 40, over 33 ms 0 and 0 against 1 and 1; 1 % low 41.0 and 42.5 against 35.4 and 35.0; CPU a frame 56.1 and 56.6 ms against 80.1 and 78.6.

hold/run build frames fps mean / median / p99 1 % low frames over 16.7 / 20 / 25 / 33 ms longest CPU a frame, all threads waited in readbacks, a frame counts
MW1/1 6f7ab2a7 3074 55.9 17.89 / 17.14 / 25.79 ms 35.4 1859 / 665 / 41 / 1 33.1 ms 80.1 ms 5.49 ms yes
MW1/2 09aac439 3294 59.9 16.70 / 16.65 / 21.79 ms 41.0 1573 / 82 / 8 / 0 30.5 ms 56.1 ms 0.59 ms yes
MW1/3 6f7ab2a7 3090 56.2 17.80 / 17.09 / 25.72 ms 35.0 1865 / 616 / 40 / 1 33.7 ms 78.6 ms 5.27 ms yes
MW1/4 09aac439 3295 59.9 16.69 / 16.64 / 21.56 ms 42.5 1592 / 67 / 6 / 0 25.7 ms 56.6 ms 0.57 ms yes

Runs of 105 s here, the window 15 to 70 s after the save's load, judged in two halves by the quiet rule below; frame times are the swapchain's present intervals. Memory available 56.7 GB before the hold and 75.5 GB after, 5058 pages swapped out during it.

3. Flash check (RX_VK_FRAMELOG and scripts/framelog.py of the merged commit; a real window as above, the 60 cap, the game's own upscaler, the frame log on for 76 s of play with the camera turns; not timing runs)

run build, scene frames logged missing triplets checked events of 1 / 2 / 3 / 4 blocks largest event a pulse or step of the whole picture's brightness (my two rules) the harness's window line
WF1 09aac439, RX_VK_NO_READBACK_WAIT=1, forest 4137 0 4135 164 143 / 15 / 5 / 1 4 of 144 blocks 0 window=ok active=same others=same screen=on
WF2 6f7ab2a7, forest 3668 0 3666 284 248 / 34 / 2 / 0 3 of 144 blocks 0 window=ok active=same others=same screen=on
WF3 09aac439, forest 4497 0 4495 63 63 / 0 / 0 / 0 1 of 144 blocks 0 window=ok active=same others=same screen=on
WF4 09aac439, field 4358 0 4356 93 68 / 18 / 5 / 2 4 of 144 blocks 0 window=ok active=same others=same screen=on

No event is larger than 4 of the picture's 144 blocks. I looked at the eight largest of each run (before, frame, after): Navi, falling leaves and, in the field, a tree against the sky crossing a block while the camera turns; no change of the whole picture in any of them. The number of these small events differs from run to run; my reading, not a measurement, is that it goes with how evenly the frames came in these shared runs (a small bright thing crosses more of a block when a frame is late), not with the build.

The control shows nothing, so this check has no proof on a real fault here. WF1 was meant to be that proof: the merged build run once on purpose with the wait of #296 switched off (RX_VK_NO_READBACK_WAIT=1; none of its 231,148 readbacks waited, by the counting patch), where the exposure is documented to step. On this card at the 60 cap it does not: its events are of the same kind and size as the normal builds' (164, between the 63 and 284 of the two normal runs of the same scene), and there is no step of the picture's brightness. So the rows above say that nothing of a flash's size happened in these runs; they do not show that the check would have caught the fault the wait is there for.

The script itself does report a flash that is in the log (the instrument's proof only): in 400 records of the merged run with a white rectangle of 12 blocks painted into one frame and one whole frame made 30 % brighter, it reports those two frames with 12 and 127 blocks; before the painting it reported none there.

What this cannot see: what the monitor showed (with mailbox the compositor takes the newest picture per refresh), a fault that needs another area or more than 76 s, and anything under the script's thresholds.

4. Also measured, on the last head 4ba31740 (without a window; the tables are folded below)

  • The 60 cap with DLAA, forest: 4ba31740 59.3 and 59.2 fps against 6f7ab2a7 54.2 and 53.9 (-1.62 ms a frame, -8.7 %); frames over 20 ms 64 and 71 against 468 and 484, over 25 ms 10 and 7 against 39 and 47; 1 % low 38.9 and 39.6 against 28.6 and 23.6; CPU a frame 57.2 and 58.6 ms against 83.3 and 83.0.
  • Uncapped (HOS_FPS_MAX=240, the game's own upscaler), forest: 4ba31740 80.1 fps against 6f7ab2a7 63.0 and 63.4 (-3.33 ms a frame, -21.1 %); CPU a frame 55.2 ms against 76.7 and 76.4; 4ba31740 79.2 and 81.2 fps against 49f75dc8 79.3 and 79.5 (-0.12 ms a frame, -1.0 %).
  • Uncapped, Hyrule Field: 4ba31740 92.8 and 92.7 fps against 6f7ab2a7 69.2 and 68.6 (-3.74 ms a frame, -25.7 %); CPU a frame 50.1 and 50.1 ms against 71.0 and 71.8; 4ba31740 92.6 and 92.1 fps against 49f75dc8 91.5 and 91.9 (-0.07 ms a frame, -0.7 %).
  • So against the build from before #2125 and #2419 nothing is left that these runs can see (-1.0 % in the forest, -0.7 % in the field, inside the runs' own spread). The time waited in readbacks is not back at that build's 0.02 ms a frame (0.57 and 0.56 ms in the forest), and that does not show in the frame time.
Every run of these holds

The 60 cap with DLAA, forest:

hold/run build fps mean frame 1 % low frames over 16.7 / 20 / 25 ms CPU a frame, all threads waited in readbacks, a frame rest of the PC, cores (highest 5 s) counts
Q5/1 6f7ab2a7 54.2 18.44 ms 28.6 1013 / 468 / 39 of 1625 83.3 ms 5.95 ms 1.8 (2.4) yes
Q5/2 4ba31740 59.3 16.87 ms 38.9 952 / 64 / 10 of 1777 57.2 ms 0.70 ms 1.7 (2.0) yes
Q5/3 6f7ab2a7 53.9 18.55 ms 23.6 1009 / 484 / 47 of 1616 83.0 ms 5.95 ms 2.1 (2.8) yes
Q5/4 4ba31740 59.2 16.89 ms 39.6 966 / 71 / 7 of 1775 58.6 ms 0.70 ms 2.4 (2.7) yes

Uncapped, forest:

hold/run build fps mean frame 1 % low frames over 20 ms CPU a frame, all threads waited in readbacks, a frame rest of the PC, cores (highest 5 s) counts
Q1/1 6f7ab2a7 63.0 15.87 ms 32.2 396 of 1889 76.7 ms 4.99 ms 1.8 (1.9) yes
Q1/2 4ba31740 80.1 12.49 ms 45.4 19 of 2401 55.2 ms 0.61 ms 1.7 (1.7) yes
Q1/3 6f7ab2a7 63.4 15.76 ms 35.0 381 of 1902 76.4 ms 4.93 ms 1.6 (1.8) yes
Q1/4 4ba31740 71.2 14.05 ms 10.1 78 of 2135 60.4 ms 0.63 ms 5.2 (19.8) no: others 5.2 cores; a 5 s sample of 19.8 cores
Q2/1 4ba31740 79.2 12.63 ms 24.7 34 of 2367 55.5 ms 0.57 ms 1.8 (2.0) yes
Q2/2 49f75dc8 79.3 12.61 ms 46.1 18 of 2379 54.5 ms 0.02 ms 1.7 (1.7) yes
Q2/3 4ba31740 81.2 12.31 ms 47.3 17 of 2435 54.3 ms 0.56 ms 1.7 (1.8) yes
Q2/4 49f75dc8 79.5 12.58 ms 43.3 24 of 2384 53.5 ms 0.02 ms 1.7 (1.8) yes

Uncapped, field:

hold/run build fps mean frame 1 % low frames over 20 ms CPU a frame, all threads waited in readbacks, a frame rest of the PC, cores (highest 5 s) counts
Q3/1 6f7ab2a7 69.2 14.45 ms 46.3 19 of 2075 71.0 ms 4.83 ms 1.6 (1.7) yes
Q3/2 4ba31740 92.8 10.78 ms 57.0 2 of 2782 50.1 ms 0.41 ms 1.7 (1.8) yes
Q3/3 6f7ab2a7 68.6 14.59 ms 43.3 32 of 2056 71.8 ms 4.71 ms 1.7 (1.8) yes
Q3/4 4ba31740 92.7 10.79 ms 57.6 2 of 2779 50.1 ms 0.31 ms 1.9 (2.7) yes
Q4/1 4ba31740 92.6 10.80 ms 57.3 0 of 2778 49.7 ms 0.33 ms 1.7 (1.8) yes
Q4/2 49f75dc8 91.5 10.93 ms 58.1 0 of 2745 49.3 ms 0.02 ms 1.8 (2.0) yes
Q4/3 4ba31740 92.1 10.86 ms 56.6 0 of 2762 49.9 ms 0.33 ms 1.9 (2.0) yes
Q4/4 49f75dc8 91.9 10.88 ms 57.6 1 of 2756 49.4 ms 0.02 ms 1.9 (2.0) yes

How the runs without a window were made. Each scene from a save, with the same scripted camera turns; runs of 82 s, the numbers from a 30 s window that starts 15 s after the save has loaded; two builds in turn, A B A B, inside one exclusive hold of the card (nothing else on the GPU, no builds); every copy with prepared shaders and one unmeasured run of each scene first. A run counts when over its window everything else on the PC used under 3 cores on average, no 5 s sample of that was over 6, and the two runs of an arm agree within 5 %. "CPU a frame" is the process's processor time over the window (all threads, from /proc) times the mean frame time; "waited in readbacks" is the time from a readback's entry into its wait to its end, summed over a frame. 4ba31740 and 76bbe4aa differ only by #2598's files; 09aac439 adds CI files.
Left out of every figure above, and shown in the tables: Q1/4 (4ba31740: others 5.2 cores; a 5 s sample of 19.8 cores). The cause: a desktop job of this PC that starts about 37 rg searches for 3 to 5 s every 15 minutes.

Not tested here: Windows, AMD, an integrated GPU or any card without VRAM mirrors, RADV's batched submits, frame generation, more than these two scenes. The game-free checks I ran with g++ 16.2.1 and clang 23.1.3 were on 9daa0718 (above); not again on the merged commit.

The fix arm at the 60 cap, as asked, on the merged commit: once without a window and once in a real one; then the flash check. Linux 7.2.15-1-omarchy, RTX 4090 (NVIDIA 615.78.08), i9-13900K. Builds: merged main `09aac439` (this pull request as `76bbe4aa`), the commit before it `6f7ab2a7`, its last head `4ba31740`, and `49f75dc8` (the last build I have from before #2125 and #2419); each is the exact commit plus one local counting patch that logs every buffer readback and how long it waited. **1. The 60 cap with DLAA, Kokiri Forest, 2048x1152 without a window** `09aac439` 59.4 and 59.3 fps against `6f7ab2a7` 54.4 and 55.7 (-1.31 ms a frame, -7.2 %); frames over 20 ms 37 and 65 against 474 and 383, over 25 ms 4 and 7 against 44 and 24; 1 % low 41.7 and 39.4 against 32.2 and 32.5; CPU a frame 59.1 and 57.4 ms against 82.6 and 79.5; waited in readbacks a frame 0.78 and 0.69 ms against 5.97 and 5.48. The build from before #2125 and #2419 ran 58.4 and 58.9 fps here with 85 and 85 frames over 20 ms (the pair posted earlier, another hold), so the fix is back at that level. | hold/run | build | fps | mean frame | 1 % low | frames over 16.7 / 20 / 25 ms | CPU a frame, all threads | waited in readbacks, a frame | rest of the PC, cores (highest 5 s) | counts | |---|---|---|---|---|---|---|---|---|---| | M1/1 | `6f7ab2a7` | 54.4 | 18.38 ms | 32.2 | 1011 / 474 / 44 of 1631 | 82.6 ms | 5.97 ms | 1.8 (2.0) | yes | | M1/2 | `09aac439` | 59.4 | 16.84 ms | 41.7 | 957 / 37 / 4 of 1780 | 59.1 ms | 0.78 ms | 1.7 (1.8) | yes | | M1/3 | `6f7ab2a7` | 55.7 | 17.96 ms | 32.5 | 981 / 383 / 24 of 1669 | 79.5 ms | 5.48 ms | 1.9 (2.4) | yes | | M1/4 | `09aac439` | 59.3 | 16.88 ms | 39.4 | 934 / 65 / 7 of 1776 | 57.4 ms | 0.69 ms | 1.8 (2.0) | yes | **2. The same pair in a real window** (1920x1080 on a 60 Hz monitor, Hyprland on Wayland; the log says mailbox, V-Sync Auto) `09aac439` 59.9 and 59.9 fps against `6f7ab2a7` 55.9 and 56.2 (-1.15 ms a frame, -6.4 %); frames over 20 ms 82 and 67 against 665 and 616, over 25 ms 8 and 6 against 41 and 40, over 33 ms 0 and 0 against 1 and 1; 1 % low 41.0 and 42.5 against 35.4 and 35.0; CPU a frame 56.1 and 56.6 ms against 80.1 and 78.6. | hold/run | build | frames | fps | mean / median / p99 | 1 % low | frames over 16.7 / 20 / 25 / 33 ms | longest | CPU a frame, all threads | waited in readbacks, a frame | counts | |---|---|---|---|---|---|---|---|---|---|---| | MW1/1 | `6f7ab2a7` | 3074 | 55.9 | 17.89 / 17.14 / 25.79 ms | 35.4 | 1859 / 665 / 41 / 1 | 33.1 ms | 80.1 ms | 5.49 ms | yes | | MW1/2 | `09aac439` | 3294 | 59.9 | 16.70 / 16.65 / 21.79 ms | 41.0 | 1573 / 82 / 8 / 0 | 30.5 ms | 56.1 ms | 0.59 ms | yes | | MW1/3 | `6f7ab2a7` | 3090 | 56.2 | 17.80 / 17.09 / 25.72 ms | 35.0 | 1865 / 616 / 40 / 1 | 33.7 ms | 78.6 ms | 5.27 ms | yes | | MW1/4 | `09aac439` | 3295 | 59.9 | 16.69 / 16.64 / 21.56 ms | 42.5 | 1592 / 67 / 6 / 0 | 25.7 ms | 56.6 ms | 0.57 ms | yes | Runs of 105 s here, the window 15 to 70 s after the save's load, judged in two halves by the quiet rule below; frame times are the swapchain's present intervals. Memory available 56.7 GB before the hold and 75.5 GB after, 5058 pages swapped out during it. **3. Flash check** (`RX_VK_FRAMELOG` and `scripts/framelog.py` of the merged commit; a real window as above, the 60 cap, the game's own upscaler, the frame log on for 76 s of play with the camera turns; not timing runs) | run | build, scene | frames logged | missing | triplets checked | events | of 1 / 2 / 3 / 4 blocks | largest event | a pulse or step of the whole picture's brightness (my two rules) | the harness's window line | |---|---|---|---|---|---|---|---|---|---| | WF1 | `09aac439`, `RX_VK_NO_READBACK_WAIT=1`, forest | 4137 | 0 | 4135 | 164 | 143 / 15 / 5 / 1 | 4 of 144 blocks | 0 | window=ok active=same others=same screen=on | | WF2 | `6f7ab2a7`, forest | 3668 | 0 | 3666 | 284 | 248 / 34 / 2 / 0 | 3 of 144 blocks | 0 | window=ok active=same others=same screen=on | | WF3 | `09aac439`, forest | 4497 | 0 | 4495 | 63 | 63 / 0 / 0 / 0 | 1 of 144 blocks | 0 | window=ok active=same others=same screen=on | | WF4 | `09aac439`, field | 4358 | 0 | 4356 | 93 | 68 / 18 / 5 / 2 | 4 of 144 blocks | 0 | window=ok active=same others=same screen=on | No event is larger than 4 of the picture's 144 blocks. I looked at the eight largest of each run (before, frame, after): Navi, falling leaves and, in the field, a tree against the sky crossing a block while the camera turns; no change of the whole picture in any of them. The number of these small events differs from run to run; my reading, not a measurement, is that it goes with how evenly the frames came in these shared runs (a small bright thing crosses more of a block when a frame is late), not with the build. **The control shows nothing, so this check has no proof on a real fault here.** WF1 was meant to be that proof: the merged build run once on purpose with the wait of #296 switched off (`RX_VK_NO_READBACK_WAIT=1`; none of its 231,148 readbacks waited, by the counting patch), where the exposure is documented to step. On this card at the 60 cap it does not: its events are of the same kind and size as the normal builds' (164, between the 63 and 284 of the two normal runs of the same scene), and there is no step of the picture's brightness. So the rows above say that nothing of a flash's size happened in these runs; they do not show that the check would have caught the fault the wait is there for. The script itself does report a flash that is in the log (the instrument's proof only): in 400 records of the merged run with a white rectangle of 12 blocks painted into one frame and one whole frame made 30 % brighter, it reports those two frames with 12 and 127 blocks; before the painting it reported none there. What this cannot see: what the monitor showed (with mailbox the compositor takes the newest picture per refresh), a fault that needs another area or more than 76 s, and anything under the script's thresholds. **4. Also measured, on the last head `4ba31740`** (without a window; the tables are folded below) - The 60 cap with DLAA, forest: `4ba31740` 59.3 and 59.2 fps against `6f7ab2a7` 54.2 and 53.9 (-1.62 ms a frame, -8.7 %); frames over 20 ms 64 and 71 against 468 and 484, over 25 ms 10 and 7 against 39 and 47; 1 % low 38.9 and 39.6 against 28.6 and 23.6; CPU a frame 57.2 and 58.6 ms against 83.3 and 83.0. - Uncapped (`HOS_FPS_MAX=240`, the game's own upscaler), forest: `4ba31740` 80.1 fps against `6f7ab2a7` 63.0 and 63.4 (-3.33 ms a frame, -21.1 %); CPU a frame 55.2 ms against 76.7 and 76.4; `4ba31740` 79.2 and 81.2 fps against `49f75dc8` 79.3 and 79.5 (-0.12 ms a frame, -1.0 %). - Uncapped, Hyrule Field: `4ba31740` 92.8 and 92.7 fps against `6f7ab2a7` 69.2 and 68.6 (-3.74 ms a frame, -25.7 %); CPU a frame 50.1 and 50.1 ms against 71.0 and 71.8; `4ba31740` 92.6 and 92.1 fps against `49f75dc8` 91.5 and 91.9 (-0.07 ms a frame, -0.7 %). - So against the build from before #2125 and #2419 nothing is left that these runs can see (-1.0 % in the forest, -0.7 % in the field, inside the runs' own spread). The time waited in readbacks is not back at that build's 0.02 ms a frame (0.57 and 0.56 ms in the forest), and that does not show in the frame time. <details><summary>Every run of these holds</summary> The 60 cap with DLAA, forest: | hold/run | build | fps | mean frame | 1 % low | frames over 16.7 / 20 / 25 ms | CPU a frame, all threads | waited in readbacks, a frame | rest of the PC, cores (highest 5 s) | counts | |---|---|---|---|---|---|---|---|---|---| | Q5/1 | `6f7ab2a7` | 54.2 | 18.44 ms | 28.6 | 1013 / 468 / 39 of 1625 | 83.3 ms | 5.95 ms | 1.8 (2.4) | yes | | Q5/2 | `4ba31740` | 59.3 | 16.87 ms | 38.9 | 952 / 64 / 10 of 1777 | 57.2 ms | 0.70 ms | 1.7 (2.0) | yes | | Q5/3 | `6f7ab2a7` | 53.9 | 18.55 ms | 23.6 | 1009 / 484 / 47 of 1616 | 83.0 ms | 5.95 ms | 2.1 (2.8) | yes | | Q5/4 | `4ba31740` | 59.2 | 16.89 ms | 39.6 | 966 / 71 / 7 of 1775 | 58.6 ms | 0.70 ms | 2.4 (2.7) | yes | Uncapped, forest: | hold/run | build | fps | mean frame | 1 % low | frames over 20 ms | CPU a frame, all threads | waited in readbacks, a frame | rest of the PC, cores (highest 5 s) | counts | |---|---|---|---|---|---|---|---|---|---| | Q1/1 | `6f7ab2a7` | 63.0 | 15.87 ms | 32.2 | 396 of 1889 | 76.7 ms | 4.99 ms | 1.8 (1.9) | yes | | Q1/2 | `4ba31740` | 80.1 | 12.49 ms | 45.4 | 19 of 2401 | 55.2 ms | 0.61 ms | 1.7 (1.7) | yes | | Q1/3 | `6f7ab2a7` | 63.4 | 15.76 ms | 35.0 | 381 of 1902 | 76.4 ms | 4.93 ms | 1.6 (1.8) | yes | | Q1/4 | `4ba31740` | 71.2 | 14.05 ms | 10.1 | 78 of 2135 | 60.4 ms | 0.63 ms | 5.2 (19.8) | no: others 5.2 cores; a 5 s sample of 19.8 cores | | Q2/1 | `4ba31740` | 79.2 | 12.63 ms | 24.7 | 34 of 2367 | 55.5 ms | 0.57 ms | 1.8 (2.0) | yes | | Q2/2 | `49f75dc8` | 79.3 | 12.61 ms | 46.1 | 18 of 2379 | 54.5 ms | 0.02 ms | 1.7 (1.7) | yes | | Q2/3 | `4ba31740` | 81.2 | 12.31 ms | 47.3 | 17 of 2435 | 54.3 ms | 0.56 ms | 1.7 (1.8) | yes | | Q2/4 | `49f75dc8` | 79.5 | 12.58 ms | 43.3 | 24 of 2384 | 53.5 ms | 0.02 ms | 1.7 (1.8) | yes | Uncapped, field: | hold/run | build | fps | mean frame | 1 % low | frames over 20 ms | CPU a frame, all threads | waited in readbacks, a frame | rest of the PC, cores (highest 5 s) | counts | |---|---|---|---|---|---|---|---|---|---| | Q3/1 | `6f7ab2a7` | 69.2 | 14.45 ms | 46.3 | 19 of 2075 | 71.0 ms | 4.83 ms | 1.6 (1.7) | yes | | Q3/2 | `4ba31740` | 92.8 | 10.78 ms | 57.0 | 2 of 2782 | 50.1 ms | 0.41 ms | 1.7 (1.8) | yes | | Q3/3 | `6f7ab2a7` | 68.6 | 14.59 ms | 43.3 | 32 of 2056 | 71.8 ms | 4.71 ms | 1.7 (1.8) | yes | | Q3/4 | `4ba31740` | 92.7 | 10.79 ms | 57.6 | 2 of 2779 | 50.1 ms | 0.31 ms | 1.9 (2.7) | yes | | Q4/1 | `4ba31740` | 92.6 | 10.80 ms | 57.3 | 0 of 2778 | 49.7 ms | 0.33 ms | 1.7 (1.8) | yes | | Q4/2 | `49f75dc8` | 91.5 | 10.93 ms | 58.1 | 0 of 2745 | 49.3 ms | 0.02 ms | 1.8 (2.0) | yes | | Q4/3 | `4ba31740` | 92.1 | 10.86 ms | 56.6 | 0 of 2762 | 49.9 ms | 0.33 ms | 1.9 (2.0) | yes | | Q4/4 | `49f75dc8` | 91.9 | 10.88 ms | 57.6 | 1 of 2756 | 49.4 ms | 0.02 ms | 1.9 (2.0) | yes | </details> **How the runs without a window were made.** Each scene from a save, with the same scripted camera turns; runs of 82 s, the numbers from a 30 s window that starts 15 s after the save has loaded; two builds in turn, A B A B, inside one exclusive hold of the card (nothing else on the GPU, no builds); every copy with prepared shaders and one unmeasured run of each scene first. A run counts when over its window everything else on the PC used under 3 cores on average, no 5 s sample of that was over 6, and the two runs of an arm agree within 5 %. "CPU a frame" is the process's processor time over the window (all threads, from /proc) times the mean frame time; "waited in readbacks" is the time from a readback's entry into its wait to its end, summed over a frame. `4ba31740` and `76bbe4aa` differ only by #2598's files; `09aac439` adds CI files. Left out of every figure above, and shown in the tables: Q1/4 (`4ba31740`: others 5.2 cores; a 5 s sample of 19.8 cores). The cause: a desktop job of this PC that starts about 37 `rg` searches for 3 to 5 s every 15 minutes. **Not tested here:** Windows, AMD, an integrated GPU or any card without VRAM mirrors, RADV's batched submits, frame generation, more than these two scenes. The game-free checks I ran with g++ 16.2.1 and clang 23.1.3 were on `9daa0718` (above); not again on the merged commit.
felipefl142 commented 2026-11-12 03:55:50 +00:00 (Migrated from github.com)

Results from an 8 GB laptop card on Linux. This PR fixes the main thing that held my game back: one job thread blocked in readbacks about a third of the time. At the same scene size, fps went up 30-50%.

Setup: RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (Linux). DLSS (built with the SDK), output 1920x1080, RX_VK_VRAM_MB=3072, RX_VK_COMPACT=1, built-in frame interpolation (DLSS-G needs an RTX 40). Both builds also have #2482 merged.

  • Before: main ef81aee + #2482 (v2.1.0-rc.3-195-geb0c4a2b), a 102-minute session.
  • After: main 09aac43 (this PR as 76bbe4aa) + #2482 (v2.1.0-rc.3-200-gd82f41c7), a 52-minute session: the Shadow Temple, then Lake Hylia and the Desert Colossus.

The commits in between are #2598, #2347 and #2556. On this card DLSS-G is not available, so none of those three should change performance. The two sessions are normal play, not an A/B run in one spot, and they took different routes. So I compare them by draw count rather than by place.

Readbacks

Before After
Readbacks that waited, of 3000 (median) 145 503
Each wait (median) 9.9 ms 0.1 ms
Time blocked, per 3000 readbacks (median) ~1100 ms ~60 ms
Replay submit a frame (median) 11.7 ms 8.9 ms
GPU time a frame (median) 11.6 ms 11.8 ms

More readbacks wait now, but each wait is about 100 times shorter. Before this PR the line came out about every 5.4 s. So a game thread was blocked for about 1.1 s of every 5.4, the same picture as in your Hyrule Field trace. GPU time did not change, so the gain is all on the CPU side.

Fps at the same scene size

Draws a frame Before After
400-700 34.8 fps 51.6
700-1000 30.7 46.3
1000-1300 28.3 37.7
1300-1700 28.1 33.0
1700-2200 24.4 30.8
2200+ 21.7 26.5

(Median of the presents lines' fps, grouped by their draw count.)

On an 8 GB card the gain is larger than the 3090 numbers in the description. Before this PR my waits were 10-20 ms each, not 5.5 ms. The gain shrinks as scenes get bigger, where the replay itself (#995) takes over.

I saw no visual problems: no flashes, no missing shadows or geometry. There was one 930 ms hitch, when VRAM ran out entering a large area (VRAM full x4, a 64 MB staging arena took 159 ms to make). It is memory pressure (#2412), not this PR.

Results from an 8 GB laptop card on Linux. This PR fixes the main thing that held my game back: one job thread blocked in readbacks about a third of the time. At the same scene size, fps went up 30-50%. **Setup:** RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (Linux). DLSS (built with the SDK), output 1920x1080, `RX_VK_VRAM_MB=3072`, `RX_VK_COMPACT=1`, built-in frame interpolation (DLSS-G needs an RTX 40). Both builds also have #2482 merged. - **Before:** main `ef81aee` + #2482 (`v2.1.0-rc.3-195-geb0c4a2b`), a 102-minute session. - **After:** main `09aac43` (this PR as `76bbe4aa`) + #2482 (`v2.1.0-rc.3-200-gd82f41c7`), a 52-minute session: the Shadow Temple, then Lake Hylia and the Desert Colossus. The commits in between are #2598, #2347 and #2556. On this card DLSS-G is not available, so none of those three should change performance. The two sessions are normal play, not an A/B run in one spot, and they took different routes. So I compare them by draw count rather than by place. ## Readbacks | | Before | After | |---|---|---| | Readbacks that waited, of 3000 (median) | 145 | 503 | | Each wait (median) | **9.9 ms** | **0.1 ms** | | Time blocked, per 3000 readbacks (median) | ~1100 ms | **~60 ms** | | Replay `submit` a frame (median) | 11.7 ms | 8.9 ms | | GPU time a frame (median) | 11.6 ms | 11.8 ms | More readbacks wait now, but each wait is about 100 times shorter. Before this PR the line came out about every 5.4 s. So a game thread was blocked for about 1.1 s of every 5.4, the same picture as in your Hyrule Field trace. GPU time did not change, so the gain is all on the CPU side. ## Fps at the same scene size | Draws a frame | Before | After | |---|---|---| | 400-700 | 34.8 fps | **51.6** | | 700-1000 | 30.7 | **46.3** | | 1000-1300 | 28.3 | **37.7** | | 1300-1700 | 28.1 | **33.0** | | 1700-2200 | 24.4 | **30.8** | | 2200+ | 21.7 | **26.5** | (Median of the `presents` lines' fps, grouped by their draw count.) On an 8 GB card the gain is larger than the 3090 numbers in the description. Before this PR my waits were 10-20 ms each, not 5.5 ms. The gain shrinks as scenes get bigger, where the replay itself (#995) takes over. I saw no visual problems: no flashes, no missing shadows or geometry. There was one 930 ms hitch, when VRAM ran out entering a large area (`VRAM full` x4, a 64 MB staging arena took 159 ms to make). It is memory pressure (#2412), not this PR.
DFN-master commented 2026-11-12 04:01:23 +00:00 (Migrated from github.com)

Windows test: Hyrule Field, stock and with our backend switches

Setup

  • Windows 11, 64-bit; Intel Core i7-11700 (8C/16T); 32 GB RAM; NVIDIA GeForce RTX 3080 10 GB; 1920x1080 @ 60 Hz.
  • Build: 55b976c0 (one commit past the PR's base 6f7ab2a7, only the DLSS frame-generation commit, no vk_replay/vk_memory change) + the three commits of this PR (798d625a, 9daa0718, 4ba31740), applied cleanly. Vanilla lift, no PGO. hookshot.exe 469,956,608 B, hookshot_vk.dll 11,190,272 B.
  • Same exe for every arm; the "main waits" arm is the same build with RX_VK_READBACK_PART=0 and RX_VK_READBACK_SETTLE_ALWAYS=1. Headless, fixed save, scripted menu input, teleport to HyruleField_Adult @ HyruleField_FrontOfLostWoodBridge, 330 s per run, ~1430 draws; fps counted after the teleport and 600 frames, readback lines from steady play.
  • All arms share the same 6.4 GB program cache and pipeline cache. No shader/pipeline failures.

Result (Hyrule Field, 1080p, same save and route)

arm fps avg (min-max) readbacks that waited, of 3000 ms each last-part fallback
stock waits (PART=0, SETTLE_ALWAYS=1) 48.12 (42.7-49.2) 172 5.53-5.68 n/a
stock fix (default) 59.29 (56.3-60.5) 1536-1552 0.02-0.03 0
fix + RX_VK_VRAM_MB=4096, RX_VK_COMPACT_PCT=90 59.83 (57.6-60.4) 1539-1550 0.02-0.03 0
fix + RX_VK_READBACK_NO_CATCHUP=1 59.96 (58.1-60.3) 1531-1550 0.02 0
fix + our full config without SYNC_LIVE (4 GB, pct 90, NO_CATCHUP, WB_MAX 1024) 59.77 (57.2-60.5) 1340 0.01 0

This is the same direction as your RTX 3090 result: +11.2 fps stock (+23%), 5.5 ms -> 0.02 ms, and the long frames go away (min 42.7 -> 56.3). Windows was on the "not yet verified" list; this covers it for one NVIDIA machine at 1080p.

Kakariko Village (same build and final config, ~2510 draws): 47.4 fps avg (40.4-50.0, last 48.2), 1227-1245 readbacks of 3000 at 0.15-0.16 ms, 0 fallback. The town is CPU-bound; the readback wait is no longer what limits it.

The one interaction we found: RX_VK_SYNC_LIVE=1

Our config sets RX_VK_SYNC_LIVE=1 (dirty pages not in the submission's capture are read from guest memory). With only that switch added to the stock build:

arm fps avg (min-max) readbacks that waited ms each last-part fallback
SYNC_LIVE=1 fix 51.39 (49.4-52.7) 482-1068 0.55-5.43 309-1022
our full config fix (SYNC_LIVE + 4 GB + pct 90 + NO_CATCHUP + WB_MAX) 53.25 (51.1-54.9) 838 0.83 820
our full config, main waits 53.86 (52.6-55.2) 753 0.91 n/a

The log's own counter shows why: with SYNC_LIVE=1 the mirror syncs read 80,954 pages "from guest memory as it was at the replay" per 600 submits (0 with it off), so a later part reads guest memory in place and the part wait gives way to the whole-submission wait for ~90% of the readbacks. That is exactly the guard you added in 9daa0718, not a wrong read; the switch and the part wait just cannot both apply. We will drop SYNC_LIVE in our config and keep the part wait.

RX_VK_VRAM_MB=4096 + RX_VK_COMPACT_PCT=90 alone and RX_VK_READBACK_NO_CATCHUP=1 alone do not trigger the fallback (both at 59.8-60.0 fps, 0 fallback), so the 4 GB texture budget is not the problem.

Checks

  • tests/test_readback_part.py, tests/test_flush_nothing_held.py, tests/test_readback_settle.py, tests/test_recorded_wait_flush.py: pass on Windows at this head.
  • 0 looked at the stopped queues first in every fix line matches your field count; SETTLE_ALWAYS=1 restores the look (172 of 3000 in the stock-waits arm), as documented.
  • No framelog/false-flash pass on Windows yet; that is the one check we have not run here. No screenshots were uploaded.

Limits

  • One NVIDIA machine, 1080p, one save and route; no AMD/Intel, no iGPU without mirrors.
  • The stock-waits arm is the same build with the two switches, not a separate main build; the third part (flush with nothing held) stays on in both arms and has no switch.
  • The byte sizes above are this build's; the PR's own base and lift may differ.
## Windows test: Hyrule Field, stock and with our backend switches ### Setup - Windows 11, 64-bit; Intel Core i7-11700 (8C/16T); 32 GB RAM; NVIDIA GeForce RTX 3080 10 GB; 1920x1080 @ 60 Hz. - Build: `55b976c0` (one commit past the PR's base `6f7ab2a7`, only the DLSS frame-generation commit, no `vk_replay`/`vk_memory` change) + the three commits of this PR (`798d625a`, `9daa0718`, `4ba31740`), applied cleanly. Vanilla lift, no PGO. `hookshot.exe` 469,956,608 B, `hookshot_vk.dll` 11,190,272 B. - Same exe for every arm; the "main waits" arm is the same build with `RX_VK_READBACK_PART=0` and `RX_VK_READBACK_SETTLE_ALWAYS=1`. Headless, fixed save, scripted menu input, teleport to `HyruleField_Adult @ HyruleField_FrontOfLostWoodBridge`, 330 s per run, ~1430 draws; fps counted after the teleport and 600 frames, readback lines from steady play. - All arms share the same 6.4 GB program cache and pipeline cache. No shader/pipeline failures. ### Result (Hyrule Field, 1080p, same save and route) | arm | fps avg (min-max) | readbacks that waited, of 3000 | ms each | last-part fallback | |---|---:|---:|---:|---:| | stock waits (`PART=0`, `SETTLE_ALWAYS=1`) | **48.12** (42.7-49.2) | 172 | 5.53-5.68 | n/a | | stock fix (default) | **59.29** (56.3-60.5) | 1536-1552 | 0.02-0.03 | 0 | | fix + `RX_VK_VRAM_MB=4096`, `RX_VK_COMPACT_PCT=90` | **59.83** (57.6-60.4) | 1539-1550 | 0.02-0.03 | 0 | | fix + `RX_VK_READBACK_NO_CATCHUP=1` | **59.96** (58.1-60.3) | 1531-1550 | 0.02 | 0 | | fix + our full config without `SYNC_LIVE` (4 GB, pct 90, NO_CATCHUP, WB_MAX 1024) | **59.77** (57.2-60.5) | 1340 | 0.01 | 0 | This is the same direction as your RTX 3090 result: +11.2 fps stock (+23%), 5.5 ms -> 0.02 ms, and the long frames go away (min 42.7 -> 56.3). Windows was on the "not yet verified" list; this covers it for one NVIDIA machine at 1080p. Kakariko Village (same build and final config, ~2510 draws): **47.4 fps avg** (40.4-50.0, last 48.2), 1227-1245 readbacks of 3000 at 0.15-0.16 ms, 0 fallback. The town is CPU-bound; the readback wait is no longer what limits it. ### The one interaction we found: `RX_VK_SYNC_LIVE=1` Our config sets `RX_VK_SYNC_LIVE=1` (dirty pages not in the submission's capture are read from guest memory). With only that switch added to the stock build: | arm | fps avg (min-max) | readbacks that waited | ms each | last-part fallback | |---|---:|---:|---:|---:| | `SYNC_LIVE=1` fix | **51.39** (49.4-52.7) | 482-1068 | 0.55-5.43 | 309-1022 | | our full config fix (`SYNC_LIVE` + 4 GB + pct 90 + NO_CATCHUP + WB_MAX) | **53.25** (51.1-54.9) | 838 | 0.83 | 820 | | our full config, main waits | **53.86** (52.6-55.2) | 753 | 0.91 | n/a | The log's own counter shows why: with `SYNC_LIVE=1` the mirror syncs read 80,954 pages "from guest memory as it was at the replay" per 600 submits (0 with it off), so a later part reads guest memory in place and the part wait gives way to the whole-submission wait for ~90% of the readbacks. That is exactly the guard you added in `9daa0718`, not a wrong read; the switch and the part wait just cannot both apply. We will drop `SYNC_LIVE` in our config and keep the part wait. `RX_VK_VRAM_MB=4096` + `RX_VK_COMPACT_PCT=90` alone and `RX_VK_READBACK_NO_CATCHUP=1` alone do not trigger the fallback (both at 59.8-60.0 fps, 0 fallback), so the 4 GB texture budget is not the problem. ### Checks - `tests/test_readback_part.py`, `tests/test_flush_nothing_held.py`, `tests/test_readback_settle.py`, `tests/test_recorded_wait_flush.py`: pass on Windows at this head. - `0 looked at the stopped queues first` in every fix line matches your field count; `SETTLE_ALWAYS=1` restores the look (172 of 3000 in the stock-waits arm), as documented. - No framelog/false-flash pass on Windows yet; that is the one check we have not run here. No screenshots were uploaded. ### Limits - One NVIDIA machine, 1080p, one save and route; no AMD/Intel, no iGPU without mirrors. - The stock-waits arm is the same build with the two switches, not a separate main build; the third part (flush with nothing held) stays on in both arms and has no switch. - The byte sizes above are this build's; the PR's own base and lift may differ.
github-sync commented 2026-11-12 06:23:30 +00:00
Owner

@DFN-master commented on GitHub · 2026-11-12 04:15 UTC · original

One more data point for the absolute level, same save and route, 1080p: my current playable build (4a53c897 + zstd + BC7, readbacks 0.02 ms) runs it at 59.67 fps avg (56.4-60.5, last 60.3), 1325/3000 readbacks at 0.02 ms, 7142 MB VRAM, 3490 MB textures. The PR at 55b976c0 with the 4 GB config runs it at 59.77 fps avg (57.2-60.5, last 60.3), 1340/3000 at 0.01 ms, 7280 MB VRAM, 3495 MB textures. So the PR brings newest main to the old build's level in this scene, against 48.12 fps on main without it.

> [@DFN-master](https://github.com/DFN-master) commented on GitHub · 2026-11-12 04:15 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6814333400) One more data point for the absolute level, same save and route, 1080p: my current playable build (`4a53c897` + zstd + BC7, readbacks 0.02 ms) runs it at **59.67 fps avg** (56.4-60.5, last 60.3), 1325/3000 readbacks at 0.02 ms, 7142 MB VRAM, 3490 MB textures. The PR at `55b976c0` with the 4 GB config runs it at **59.77 fps avg** (57.2-60.5, last 60.3), 1340/3000 at 0.01 ms, 7280 MB VRAM, 3495 MB textures. So the PR brings newest main to the old build's level in this scene, against 48.12 fps on main without it. <!-- github-sync:c:6814333400 -->
github-sync commented 2026-11-12 06:23:30 +00:00
Owner

@mooki000 commented on GitHub · 2026-11-12 04:24 UTC · original

An integrated GPU without VRAM mirrors, which the description listed as untested: Intel UHD Graphics 770 (RPL-S), Mesa 26.2.5 (ANV), Linux 7.2.15-1-omarchy, i9-13900K. Merged main 09aac439 plus a local counting patch, with what the game chose for this device at its first start (the log: the Minimum (4GB) preset, which sets Resolution to 50 %; VRAM mirror of buffer pools off; the compute fallback for mesh shaders), 2048x1152 output without a window, Kokiri Forest from a save, standing still. Started with RX_VK_PRECOMPILE=0 RX_VK_NO_WARM=1 RX_VK_NO_PREDICT=1: without them the driver's shader compiles take more memory than a run may have here.

The part wait does not act on this card, so there is nothing for it to cost.

  • In the code: a readback's part is noted only where a mirror's pages are copied back (wb_pages_note(..., part) in the write-back of gpu/vk/vk_memory.cpp); results the GPU writes in place are noted with part 0, and without mirrors that is every one of them. invalidate then has no part to wait for and waits for the whole submission, as before this pull request.
  • In the runs: the log's line counts 0 readbacks that "waited for their submission's last part" in every run (5 runs, 15 of its lines), with the defaults and with the switches alike.
  • The pair, in turn (defaults, switches, defaults, switches): fps 8.5 and 8.6 against 8.4 and 9.4; readbacks that waited, of 3000: 604 and 593 against 236 and 224, each 30.7 and 31.2 ms against 83.7 and 87.2 ms; of them looked at the stopped queues first 51 and 57 against 236 and 224.
run arm frames fps the log's 120-presents lines: fps mean (lowest .. highest) draws a frame GPU time a frame readbacks waited of 3000, each looked at the stopped queues first waited for their last part readbacks a frame, of them 0.1 ms or longer (mean) waited in readbacks a frame frame time rest of the PC, cores (highest 5 s)
I1/1 defaults 2463 10.5 11.4 (8.5 .. 19.8) 4322 84.6 ms 610, 26.8 ms 18 0 45.8, 2.20 (33.9 ms) 74.7 ms 95.4 ms 11.1 (30.8)
I2/1 defaults 328 8.5 8.6 (8.6 .. 8.6) 4335 97.2 ms 604, 30.7 ms 51 0 45.6, 2.06 (41.4 ms) 85.4 ms 117.0 ms 30.0 (30.5)
I2/2 RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1 335 8.4 8.4 (8.4 .. 8.4) 4325 97.7 ms 236, 83.7 ms 236 0 45.4, 1.04 (84.6 ms) 88.5 ms 119.0 ms 29.9 (30.3)
I2/3 defaults 344 8.6 8.6 (8.5 .. 8.6) 4342 97.3 ms 593, 31.2 ms 57 0 45.6, 1.92 (43.6 ms) 83.6 ms 116.0 ms 28.0 (30.3)
I2/4 RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1 374 9.4 9.4 (9.3 .. 9.4) 4328 94.8 ms 224, 87.2 ms 224 0 45.4, 1.02 (86.0 ms) 87.7 ms 106.4 ms 13.3 (17.9)

The card is GPU-bound in this scene (85 ms of GPU time in a 95 ms frame in the first, long run), and the game spends 75 ms of each frame waiting in readbacks: here that is the wait for the GPU's frame itself, the whole-submission wait your follow-up would fall back to.
How to read the table: the Intel device has no exclusive hold on this PC, so other work ran beside these runs (the last column; a run counts for me only under 3 cores on average with no 5 s over 6), and the frame rates are rough. I1 is the copy's first start on this device (one run of 440 s, the window 15 to 250 s after the load); I2 is the pair (105 s runs, the window 30 to 70 s after the load).

What does differ between the two arms is how the same wait is cut up, and by the code that can only be the settle switch, since no readback here has a part: with the defaults 604 and 593 of 3000 readbacks wait 30.7 and 31.2 ms each (51 and 57 of them after a look at the stopped queues); with both switches 236 and 224 wait 83.7 and 87.2 ms each, every one after that look. About 2.6 times as many waits, each about 2.8 times shorter, for about the same time a frame (85.4 and 83.6 ms against 88.5 and 87.7 ms): the shape you describe on the RX 580, on a card where the part wait cannot be its cause. Whether it costs frames here I cannot say: other work used 13 to 30 cores beside all four runs of the pair, so none of them counts as a timing.

Not a correct picture on this device, with or without this change: on today's main the UHD 770 draws this area in false colours (dispatches are left out while their compute pipelines compile; reported separately). "The same scene" above means the same draws a frame. No device lost, no i915 line in the kernel log, in any run.

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 04:24 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6814488950) An integrated GPU without VRAM mirrors, which the description listed as untested: Intel UHD Graphics 770 (RPL-S), Mesa 26.2.5 (ANV), Linux 7.2.15-1-omarchy, i9-13900K. Merged main `09aac439` plus a local counting patch, with what the game chose for this device at its first start (the log: `the Minimum (4GB) preset`, which sets Resolution to 50 %; `VRAM mirror of buffer pools off`; the compute fallback for mesh shaders), 2048x1152 output without a window, Kokiri Forest from a save, standing still. Started with `RX_VK_PRECOMPILE=0 RX_VK_NO_WARM=1 RX_VK_NO_PREDICT=1`: without them the driver's shader compiles take more memory than a run may have here. **The part wait does not act on this card, so there is nothing for it to cost.** - In the code: a readback's part is noted only where a mirror's pages are copied back (`wb_pages_note(..., part)` in the write-back of `gpu/vk/vk_memory.cpp`); results the GPU writes in place are noted with part 0, and without mirrors that is every one of them. `invalidate` then has no part to wait for and waits for the whole submission, as before this pull request. - In the runs: the log's line counts 0 readbacks that "waited for their submission's last part" in every run (5 runs, 15 of its lines), with the defaults and with the switches alike. - The pair, in turn (defaults, switches, defaults, switches): fps 8.5 and 8.6 against 8.4 and 9.4; readbacks that waited, of 3000: 604 and 593 against 236 and 224, each 30.7 and 31.2 ms against 83.7 and 87.2 ms; of them looked at the stopped queues first 51 and 57 against 236 and 224. | run | arm | frames | fps | the log's 120-presents lines: fps mean (lowest .. highest) | draws a frame | GPU time a frame | readbacks waited of 3000, each | looked at the stopped queues first | waited for their last part | readbacks a frame, of them 0.1 ms or longer (mean) | waited in readbacks a frame | frame time | rest of the PC, cores (highest 5 s) | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | I1/1 | defaults | 2463 | 10.5 | 11.4 (8.5 .. 19.8) | 4322 | 84.6 ms | 610, 26.8 ms | 18 | 0 | 45.8, 2.20 (33.9 ms) | 74.7 ms | 95.4 ms | 11.1 (30.8) | | I2/1 | defaults | 328 | 8.5 | 8.6 (8.6 .. 8.6) | 4335 | 97.2 ms | 604, 30.7 ms | 51 | 0 | 45.6, 2.06 (41.4 ms) | 85.4 ms | 117.0 ms | 30.0 (30.5) | | I2/2 | `RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1` | 335 | 8.4 | 8.4 (8.4 .. 8.4) | 4325 | 97.7 ms | 236, 83.7 ms | 236 | 0 | 45.4, 1.04 (84.6 ms) | 88.5 ms | 119.0 ms | 29.9 (30.3) | | I2/3 | defaults | 344 | 8.6 | 8.6 (8.5 .. 8.6) | 4342 | 97.3 ms | 593, 31.2 ms | 57 | 0 | 45.6, 1.92 (43.6 ms) | 83.6 ms | 116.0 ms | 28.0 (30.3) | | I2/4 | `RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1` | 374 | 9.4 | 9.4 (9.3 .. 9.4) | 4328 | 94.8 ms | 224, 87.2 ms | 224 | 0 | 45.4, 1.02 (86.0 ms) | 87.7 ms | 106.4 ms | 13.3 (17.9) | The card is GPU-bound in this scene (85 ms of GPU time in a 95 ms frame in the first, long run), and the game spends 75 ms of each frame waiting in readbacks: here that is the wait for the GPU's frame itself, the whole-submission wait your follow-up would fall back to. How to read the table: the Intel device has no exclusive hold on this PC, so other work ran beside these runs (the last column; a run counts for me only under 3 cores on average with no 5 s over 6), and the frame rates are rough. I1 is the copy's first start on this device (one run of 440 s, the window 15 to 250 s after the load); I2 is the pair (105 s runs, the window 30 to 70 s after the load). What does differ between the two arms is how the same wait is cut up, and by the code that can only be the settle switch, since no readback here has a part: with the defaults 604 and 593 of 3000 readbacks wait 30.7 and 31.2 ms each (51 and 57 of them after a look at the stopped queues); with both switches 236 and 224 wait 83.7 and 87.2 ms each, every one after that look. About 2.6 times as many waits, each about 2.8 times shorter, for about the same time a frame (85.4 and 83.6 ms against 88.5 and 87.7 ms): the shape you describe on the RX 580, on a card where the part wait cannot be its cause. Whether it costs frames here I cannot say: other work used 13 to 30 cores beside all four runs of the pair, so none of them counts as a timing. Not a correct picture on this device, with or without this change: on today's main the UHD 770 draws this area in false colours (dispatches are left out while their compute pipelines compile; reported separately). "The same scene" above means the same draws a frame. No device lost, no i915 line in the kernel log, in any run. <!-- github-sync:c:6814488950 -->
github-sync commented 2026-11-12 07:28:05 +00:00
Owner

@dankman42 commented on GitHub · 2026-11-12 07:04 UTC · original

Windows run for @hckrman101's request: main against the last nightly before this PR, on an RTX 3080.

This complements DFN-master's RTX 3080 run above: there, one build with the switches that restore the old waits, at a 60 Hz cap; here, two separate builds (main and the nightly itself), uncapped, at the same teleport spot. The nightly matches DFN-master's "stock waits" arm (172 of 3000, ~5.6 ms each), and without the cap main's gain shows in full.

Setup. Windows 11, RTX 3080 10 GB (driver 621.14), i9-12900K, 64 GB. Both built from source with llvm-mingw, no DLSS: main 87991bc8 and 55b976c0 (this PR's parent, the last commit whose Windows and Linux nightly runs both succeeded before it). Main also carries about 50 other merges after #2602, so this is main against that nightly, not #2602 alone. Window 1280x720 (mailbox, V-Sync Auto), default options, HOS_FPS_MAX=1000 (uncapped). Scene: the same save, teleported to Hyrule Field (adult, by the Lost Woods bridge, the teleport's own view), standing still; 45 s to settle, then 120 s measured. A warm-up run in each build first, so no pipeline was compiled during the measurement (1481-1495 before and after). Two pairs, interleaved main, nightly, main, nightly.

nightly 55b976c0 main 87991bc8
readbacks that waited, of 3000 171-182 (median 172) 1542-1573 (median 1550)
wait each 4.92-6.34 ms (medians 5.80, 5.56) 0.03-0.10 ms (medians 0.07, 0.04)
fps, 120 s 59.7 / 60.8 78.4 / 79.3
median frame time 16.39 / 16.32 ms 12.18 / 12.23 ms
p99 frame time 21.46 / 18.98 ms 17.18 / 18.06 ms
frames over 20 ms 264 / 22 30 / 37
draws a frame 1355-1445 1335-1453

So on this card about +31 % fps (median frame 16.35 -> 12.2 ms), in line with the 3090's 63-66 -> 77-79. On main, more readbacks wait (about half, against 6 %), but each one for 0.04-0.10 ms instead of ~5.7 ms. "Looked at the stopped queues first" and "waited for their submission's last part" are 0 in every line of both main runs.

Log lines, the first five of each measured window (55 and 55 lines on main, 42 and 43 on the nightly):

main, pair 1
vk: readbacks of GPU results: 1550 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1548 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1548 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1551 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
nightly 55b976c0, pair 1
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.87 ms each
vk: readbacks of GPU results: 174 of 3000 waited for their write-back, 5.74 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.86 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.88 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.89 ms each
main, pair 2
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1552 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1552 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
nightly 55b976c0, pair 2
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.74 ms each
vk: readbacks of GPU results: 173 of 3000 waited for their write-back, 5.70 ms each
vk: readbacks of GPU results: 173 of 3000 waited for their write-back, 5.72 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.64 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.77 ms each

Not tested: DLSS, frame generation, Reflex, a frame rate cap, other areas, moving through the field.

> [@dankman42](https://github.com/dankman42) commented on GitHub · 2026-11-12 07:04 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6817135289) Windows run for @hckrman101's request: main against the last nightly before this PR, on an RTX 3080. This complements DFN-master's RTX 3080 run above: there, one build with the switches that restore the old waits, at a 60 Hz cap; here, two separate builds (main and the nightly itself), uncapped, at the same teleport spot. The nightly matches DFN-master's "stock waits" arm (172 of 3000, ~5.6 ms each), and without the cap main's gain shows in full. **Setup.** Windows 11, RTX 3080 10 GB (driver 621.14), i9-12900K, 64 GB. Both built from source with llvm-mingw, no DLSS: main `87991bc8` and `55b976c0` (this PR's parent, the last commit whose Windows and Linux nightly runs both succeeded before it). Main also carries about 50 other merges after #2602, so this is main against that nightly, not #2602 alone. Window 1280x720 (mailbox, V-Sync Auto), default options, `HOS_FPS_MAX=1000` (uncapped). Scene: the same save, teleported to Hyrule Field (adult, by the Lost Woods bridge, the teleport's own view), standing still; 45 s to settle, then 120 s measured. A warm-up run in each build first, so no pipeline was compiled during the measurement (1481-1495 before and after). Two pairs, interleaved main, nightly, main, nightly. | | nightly `55b976c0` | main `87991bc8` | |---|---|---| | readbacks that waited, of 3000 | 171-182 (median 172) | 1542-1573 (median 1550) | | wait each | 4.92-6.34 ms (medians 5.80, 5.56) | 0.03-0.10 ms (medians 0.07, 0.04) | | fps, 120 s | 59.7 / 60.8 | 78.4 / 79.3 | | median frame time | 16.39 / 16.32 ms | 12.18 / 12.23 ms | | p99 frame time | 21.46 / 18.98 ms | 17.18 / 18.06 ms | | frames over 20 ms | 264 / 22 | 30 / 37 | | draws a frame | 1355-1445 | 1335-1453 | So on this card about +31 % fps (median frame 16.35 -> 12.2 ms), in line with the 3090's 63-66 -> 77-79. On main, more readbacks wait (about half, against 6 %), but each one for 0.04-0.10 ms instead of ~5.7 ms. "Looked at the stopped queues first" and "waited for their submission's last part" are 0 in every line of both main runs. Log lines, the first five of each measured window (55 and 55 lines on main, 42 and 43 on the nightly): ``` main, pair 1 vk: readbacks of GPU results: 1550 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1548 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1548 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1551 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) nightly 55b976c0, pair 1 vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.87 ms each vk: readbacks of GPU results: 174 of 3000 waited for their write-back, 5.74 ms each vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.86 ms each vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.88 ms each vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.89 ms each main, pair 2 vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1552 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1552 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs) nightly 55b976c0, pair 2 vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.74 ms each vk: readbacks of GPU results: 173 of 3000 waited for their write-back, 5.70 ms each vk: readbacks of GPU results: 173 of 3000 waited for their write-back, 5.72 ms each vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.64 ms each vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.77 ms each ``` Not tested: DLSS, frame generation, Reflex, a frame rate cap, other areas, moving through the field. <!-- github-sync:c:6817135289 -->
github-sync commented 2026-11-12 07:28:05 +00:00
Owner

@Brett-D commented on GitHub · 2026-11-12 07:24 UTC · original

Windows numbers for this one, after the merge: it recovered the loss from #2125/#2419 on an RTX 5090. (#2700 has RTX 3080 data on the same question.)

Setup. i7-13700K (24 threads), 98 GB, Windows 11, RTX 5090 (driver 621.14), 165 Hz. Every build was made from source with the Hookshot Builder's toolchain and v2.1.0's profile and hot list. Kokiri Forest, the same autosave loaded by a scripted input each run, 2048x1152. Frame generation off and HOS_FPS_MAX=240 ("frame rate cap 240" in the log). Frame times come from HOS_PRESENT_TRACE, 40-70 s after the first present, about 2,330 draws. Builds alternated within each round, with a warm-up run each; 3 runs per build (6 for main).

build fps p50 / p95 / p99 ms readback wait, ms a frame CPU, cores
main 9776c36d (with this PR) 61.9 (65.5 in a second set) 16.0 / 18.9 / 20.9 2.4-2.9 5.8-6.0
55b976c0 (main just before this PR) 47.6 20.9 / 23.9 / 27.1 8.2-9.4 (about 8 ms each) 6.2-6.3
49f75dc8 (before #2125/#2419) 61.9 15.9 / 18.9 / 22.2 0.09-0.10 5.1-5.3
v2.1.0 61.2 16.1 / 19.4 / 22.8 0.09-0.14 5.2-5.3

The readback wait is the sum of "waited x ms each" from the readbacks of GPU results lines, divided by the game's frames in play (frames while the scene draws 1,000 or more draws). Edited: the first version divided by every frame of the run, title screen included, which halved these numbers. CPU is the process's CPU time over the measured window.

With the player's own settings (frame generation Auto) the game's limiter holds every build at 58-60 fps, so this is the uncapped view. No run had a stall, a "timed out" line or a device loss, and screenshots of main and 49f75dc8 in the same view match.

What's left after this PR

  • Readbacks still wait 2.4-2.9 ms a frame, against about 0.1 ms before #2125. That is about the one real wait a frame for the first part that the description expects.
  • Main uses about 0.65 more cores at the same frame rate. A sampled profile (scripts/cpu_report.py) puts the extra in os/sleep-and-timers (44% of a core against 21% on 49f75dc8: the job workers yielding in their 20 us sleeps while one waits in a readback) and render/replay-and-submit (28% against 12%).
  • HOS_SHORT_SLEEP_TIMER=1 on main, 2 runs each: 0.9 fewer cores, but about 2 fps lower with worse p95/p99 on this 24-thread CPU. So it's not a default here; CPUs with fewer cores may differ.
> [@Brett-D](https://github.com/Brett-D) commented on GitHub · 2026-11-12 07:24 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6817467208) Windows numbers for this one, after the merge: it recovered the loss from #2125/#2419 on an RTX 5090. (#2700 has RTX 3080 data on the same question.) **Setup.** i7-13700K (24 threads), 98 GB, Windows 11, RTX 5090 (driver 621.14), 165 Hz. Every build was made from source with the Hookshot Builder's toolchain and v2.1.0's profile and hot list. Kokiri Forest, the same autosave loaded by a scripted input each run, 2048x1152. Frame generation off and `HOS_FPS_MAX=240` ("frame rate cap 240" in the log). Frame times come from `HOS_PRESENT_TRACE`, 40-70 s after the first present, about 2,330 draws. Builds alternated within each round, with a warm-up run each; 3 runs per build (6 for main). | build | fps | p50 / p95 / p99 ms | readback wait, ms a frame | CPU, cores | |---|---|---|---|---| | main 9776c36d (with this PR) | 61.9 (65.5 in a second set) | 16.0 / 18.9 / 20.9 | 2.4-2.9 | 5.8-6.0 | | 55b976c0 (main just before this PR) | **47.6** | 20.9 / 23.9 / 27.1 | 8.2-9.4 (about 8 ms each) | 6.2-6.3 | | 49f75dc8 (before #2125/#2419) | 61.9 | 15.9 / 18.9 / 22.2 | 0.09-0.10 | 5.1-5.3 | | v2.1.0 | 61.2 | 16.1 / 19.4 / 22.8 | 0.09-0.14 | 5.2-5.3 | The readback wait is the sum of "waited x ms each" from the `readbacks of GPU results` lines, divided by the game's frames in play (frames while the scene draws 1,000 or more draws). *Edited: the first version divided by every frame of the run, title screen included, which halved these numbers.* CPU is the process's CPU time over the measured window. With the player's own settings (frame generation Auto) the game's limiter holds every build at 58-60 fps, so this is the uncapped view. No run had a stall, a "timed out" line or a device loss, and screenshots of main and 49f75dc8 in the same view match. **What's left after this PR** - Readbacks still wait 2.4-2.9 ms a frame, against about 0.1 ms before #2125. That is about the one real wait a frame for the first part that the description expects. - Main uses about 0.65 more cores at the same frame rate. A sampled profile (`scripts/cpu_report.py`) puts the extra in os/sleep-and-timers (44% of a core against 21% on 49f75dc8: the job workers yielding in their 20 us sleeps while one waits in a readback) and render/replay-and-submit (28% against 12%). - `HOS_SHORT_SLEEP_TIMER=1` on main, 2 runs each: 0.9 fewer cores, but about 2 fps lower with worse p95/p99 on this 24-thread CPU. So it's not a default here; CPUs with fewer cores may differ. <!-- github-sync:c:6817467208 -->
github-sync commented 2026-11-12 10:34:43 +00:00
Owner

@DFN-master commented on GitHub · 2026-11-12 10:14 UTC · original

On top of this PR I built the stack I actually play (zstd program cache + BC7 big textures + the 4 GB texture budget) and ran the same save and route:

scene fps avg (min-max) readbacks/3000 ms each last-part fallback VRAM
Hyrule Field 60.0 (57.2-60.5) 1343-1360 0.01 0 7402/9085 MB
Lake Hylia (adult) 45.94 (41.6-51.1, last 46.4) 1356-1363 0.03 0 6704/8944 MB
Kakariko Village 50-51 (last 51.1) 1227-1261 0.42-0.46 1221-1261 7502/9017 MB

One cross-PR note from the Kakariko run: the BC7 path's GPU unswizzle reads the source through gpu_buffer(), which now notes an in-place read (4ba31740), so the part wait gives way to the last part for essentially every readback there (1221-1261 of 3000, 0.42-0.46 ms). Field and Lake show 0 fallback. It is the guard working as designed; it just means BC7 users lose the part wait in texture-heavy scenes. Town fps is CPU-bound and unchanged.

No device loss on this stack; the earlier crash in my runs was stale pipeline play-cache files from older builds (mkw_vk_pipelines_play_*.bin), not this PR; the zstd program cache was fine.

> [@DFN-master](https://github.com/DFN-master) commented on GitHub · 2026-11-12 10:14 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6820820722) On top of this PR I built the stack I actually play (zstd program cache + BC7 big textures + the 4 GB texture budget) and ran the same save and route: | scene | fps avg (min-max) | readbacks/3000 | ms each | last-part fallback | VRAM | |---|---:|---:|---:|---:|---:| | Hyrule Field | **60.0** (57.2-60.5) | 1343-1360 | 0.01 | 0 | 7402/9085 MB | | Lake Hylia (adult) | **45.94** (41.6-51.1, last 46.4) | 1356-1363 | 0.03 | 0 | 6704/8944 MB | | Kakariko Village | **50-51** (last 51.1) | 1227-1261 | 0.42-0.46 | 1221-1261 | 7502/9017 MB | One cross-PR note from the Kakariko run: the BC7 path's GPU unswizzle reads the source through `gpu_buffer()`, which now notes an in-place read (`4ba31740`), so the part wait gives way to the last part for essentially every readback there (1221-1261 of 3000, 0.42-0.46 ms). Field and Lake show 0 fallback. It is the guard working as designed; it just means BC7 users lose the part wait in texture-heavy scenes. Town fps is CPU-bound and unchanged. No device loss on this stack; the earlier crash in my runs was stale pipeline play-cache files from older builds (`mkw_vk_pipelines_play_*.bin`), not this PR; the zstd program cache was fine. <!-- github-sync:c:6820820722 -->
github-sync commented 2026-11-12 13:40:05 +00:00
Owner

@mooki000 commented on GitHub · 2026-11-12 12:45 UTC · original

Thanks @dankman42 for the uncapped two-build RTX 3080 comparison, @Brett-D for including v2.1.0 and separating the remaining readback/CPU cost on the RTX 5090, and @DFN-master for the RTX 3080 switch comparisons and follow-up with your play stack. The Linux three-column comparison is on #2595, and I'm measuring Brett-D's remaining cost on Linux now. I'm also checking the BC7 interaction you reported, without drawing a conclusion yet.

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 12:45 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6823980184) Thanks @dankman42 for the uncapped two-build RTX 3080 comparison, @Brett-D for including v2.1.0 and separating the remaining readback/CPU cost on the RTX 5090, and @DFN-master for the RTX 3080 switch comparisons and follow-up with your play stack. The Linux three-column comparison is [on #2595](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2595), and I'm measuring Brett-D's remaining cost on Linux now. I'm also checking the BC7 interaction you reported, without drawing a conclusion yet. <!-- github-sync:c:6823980184 -->
github-sync commented 2026-11-12 17:58:15 +00:00
Owner

@mooki000 commented on GitHub · 2026-11-12 17:26 UTC · original

The Linux follow-up I promised above, on the cost that is left: the extra CPU cores Brett-D measured on Windows do not show up here. Main uses about 0.21 fewer cores in the forest and 0.45 fewer in Kakariko than v2.1.0, despite the remaining readback wait. This differs from Brett-D’s Windows result; main already includes #2637, which changes Linux’s guest short sleeps to timers while leaving the Windows default unchanged on his CPU.

Setup: RTX 4090, NVIDIA 615.78.08, i9-13900K, Arch Linux 7.2.15-1-omarchy; main 46780503 versus v2.1.0, matched release profile/toolchain and equivalent readback logging. Headless, 2048×1152, game’s upscaler, cap 240, frame generation off, warmed caches. Each scene ran A/B/A/B in one exclusive hold; 30s measured after loading. All eight rows below passed the quiet/repeat checks. Other work used 0.90–1.96 cores on average, with no five-second peak above 2.33.

Scene / run Build FPS Mean frame CPU cores Readback wait/frame
Forest, fixed view, 1 main 46780503 62.14 16.092 ms 3.479 1.3574 ms
Forest, fixed view, 2 v2.1.0 8ef84611 59.70 16.750 ms 3.719 0.0185 ms
Forest, fixed view, 3 main 46780503 62.00 16.128 ms 3.470 1.2864 ms
Forest, fixed view, 4 v2.1.0 8ef84611 61.35 16.300 ms 3.641 0.0178 ms
Kakariko entrance, 1 main 46780503 101.88 9.815 ms 3.874 0.3289 ms
Kakariko entrance, 2 v2.1.0 8ef84611 101.45 9.857 ms 4.340 0.0142 ms
Kakariko entrance, 3 main 46780503 99.79 10.021 ms 3.883 0.3252 ms
Kakariko entrance, 4 v2.1.0 8ef84611 102.41 9.765 ms 4.325 0.0153 ms

The forest is a fixed, roughly 4,000-draw view; it is not the earlier moving-camera, roughly 2,000-draw measurement. The remaining wait there is 1.29–1.36ms/frame, not the earlier ~0.5ms. Two earlier forest attempts were not used for a conclusion: one had repeat spread above 5%, and one had a run remain at the title screen. Those failed rows were retained. The small frame-rate differences above do not establish a new speed gain beyond the paired repeat spread.

Where the CPU goes: in Kakariko the five module workers together use 0.933–0.934 cores on main versus 1.296–1.311 on the release; the main thread uses 0.307–0.309 versus 0.393. Sampled stacks support the sleep-policy difference: workers yielding inside their sleep calls account for 5.05% of their release samples and zero on main, where 7.99% instead show the timer wait. Those are stack shares, not CPU-core estimates; the cores in the table come from per-thread/process CPU time. The Presentation thread does not show a Windows-sized CPU increase.

The remaining wait is real. The sampled large-result read sometimes reaches a writer recording that has not completed, with later recordings still beyond it. The game then imports frame-tagged result data. Returning without that completion would lose the guarantee that the CPU receives the GPU’s finished results. The stopped-queue publication guarantee from #2125 and whole-submission fences from #2419 still matter. I found no demonstrated safe no-wait change to offer; Linux’s short-sleep follow-up is already merged. “Readback wait” above is only the instrumented invalidate path, not all CPU waiting on the GPU or an additive frame-time penalty.

BC7 interaction: I could enable BC7, but did not reproduce the near-universal fallback in this Kakariko entrance sweep:

Settings Last-part fallbacks / 3000 calls Share of calls counted as waited
Defaults, BC3 0–1 0.01%
RX_VK_ASTC_BC7=1 0–1 0.01%
BC7 + 4 GB texture budget, compaction 90%, write-back limit 1024 KB 0–2 0.03%
Same budget/settings, BC3 0–4 0.04%

The budget arm used RX_VK_VRAM_MB=4096 RX_VK_COMPACT_PCT=90 RX_VK_WB_MAX_KB=1024; neither SYNC_LIVE nor READBACK_NO_CATCHUP was set. Both BC7 runs confirmed BC7 in the log. This is not DFN-master’s card, exact save/view or custom stack: my sweep averages about 1,150–1,350 draws, not his heavy town scene. It does not refute his result.

The fallback is the intended safety cost when a later part reads guest memory directly. One attribution detail: on this pinned main the BC7 path stages its encoded source, while the separate raw GPU-unswizzle path excludes ASTC/BC textures. The counter alone cannot identify the BC7 encoder as the reader that triggered it; other raw readers or mirror pressure in that stack may be responsible.

Limits: no Windows, AMD/no-mirror GPU, exact tester stack, long play or continuous flash test. Inspected captures show the intended scenes and ordinary HUDs, not proof against one-frame faults. No synchronization was disabled and no product change was tested. All game runs ended without a device loss or kernel GPU fault.

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 17:26 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2602#issuecomment-6830885748) The Linux follow-up I promised above, on the cost that is left: **the extra CPU cores Brett-D measured on Windows do not show up here**. Main uses about **0.21 fewer cores in the forest and 0.45 fewer in Kakariko** than v2.1.0, despite the remaining readback wait. This differs from Brett-D’s Windows result; main already includes #2637, which changes Linux’s guest short sleeps to timers while leaving the Windows default unchanged on his CPU. **Setup:** RTX 4090, NVIDIA 615.78.08, i9-13900K, Arch Linux 7.2.15-1-omarchy; main 46780503 versus v2.1.0, matched release profile/toolchain and equivalent readback logging. Headless, 2048×1152, game’s upscaler, cap 240, frame generation off, warmed caches. Each scene ran A/B/A/B in one exclusive hold; 30s measured after loading. All eight rows below passed the quiet/repeat checks. Other work used 0.90–1.96 cores on average, with no five-second peak above 2.33. | Scene / run | Build | FPS | Mean frame | CPU cores | Readback wait/frame | |---|---|---:|---:|---:|---:| | Forest, fixed view, 1 | main `46780503` | 62.14 | 16.092 ms | 3.479 | 1.3574 ms | | Forest, fixed view, 2 | v2.1.0 `8ef84611` | 59.70 | 16.750 ms | 3.719 | 0.0185 ms | | Forest, fixed view, 3 | main `46780503` | 62.00 | 16.128 ms | 3.470 | 1.2864 ms | | Forest, fixed view, 4 | v2.1.0 `8ef84611` | 61.35 | 16.300 ms | 3.641 | 0.0178 ms | | Kakariko entrance, 1 | main `46780503` | 101.88 | 9.815 ms | 3.874 | 0.3289 ms | | Kakariko entrance, 2 | v2.1.0 `8ef84611` | 101.45 | 9.857 ms | 4.340 | 0.0142 ms | | Kakariko entrance, 3 | main `46780503` | 99.79 | 10.021 ms | 3.883 | 0.3252 ms | | Kakariko entrance, 4 | v2.1.0 `8ef84611` | 102.41 | 9.765 ms | 4.325 | 0.0153 ms | The forest is a fixed, roughly 4,000-draw view; it is not the earlier moving-camera, roughly 2,000-draw measurement. The remaining wait there is 1.29–1.36ms/frame, not the earlier ~0.5ms. Two earlier forest attempts were not used for a conclusion: one had repeat spread above 5%, and one had a run remain at the title screen. Those failed rows were retained. The small frame-rate differences above do not establish a new speed gain beyond the paired repeat spread. **Where the CPU goes:** in Kakariko the five module workers together use 0.933–0.934 cores on main versus 1.296–1.311 on the release; the main thread uses 0.307–0.309 versus 0.393. Sampled stacks support the sleep-policy difference: workers yielding *inside their sleep calls* account for 5.05% of their release samples and zero on main, where 7.99% instead show the timer wait. Those are stack shares, not CPU-core estimates; the cores in the table come from per-thread/process CPU time. The Presentation thread does not show a Windows-sized CPU increase. **The remaining wait is real.** The sampled large-result read sometimes reaches a writer recording that has not completed, with later recordings still beyond it. The game then imports frame-tagged result data. Returning without that completion would lose the guarantee that the CPU receives the GPU’s finished results. The stopped-queue publication guarantee from #2125 and whole-submission fences from #2419 still matter. I found no demonstrated safe no-wait change to offer; Linux’s short-sleep follow-up is already merged. “Readback wait” above is only the instrumented invalidate path, not all CPU waiting on the GPU or an additive frame-time penalty. **BC7 interaction:** I could enable BC7, but did not reproduce the near-universal fallback in this Kakariko entrance sweep: | Settings | Last-part fallbacks / 3000 calls | Share of calls counted as waited | |---|---:|---:| | Defaults, BC3 |0–1|0.01%| | `RX_VK_ASTC_BC7=1` |0–1|0.01%| | BC7 + 4 GB texture budget, compaction 90%, write-back limit 1024 KB |0–2|0.03%| | Same budget/settings, BC3 |0–4|0.04%| The budget arm used `RX_VK_VRAM_MB=4096 RX_VK_COMPACT_PCT=90 RX_VK_WB_MAX_KB=1024`; neither `SYNC_LIVE` nor `READBACK_NO_CATCHUP` was set. Both BC7 runs confirmed BC7 in the log. This is **not DFN-master’s card, exact save/view or custom stack**: my sweep averages about 1,150–1,350 draws, not his heavy town scene. It does not refute his result. The fallback is the intended safety cost when a later part reads guest memory directly. One attribution detail: on this pinned main the BC7 path stages its encoded source, while the separate raw GPU-unswizzle path excludes ASTC/BC textures. The counter alone cannot identify the BC7 encoder as the reader that triggered it; other raw readers or mirror pressure in that stack may be responsible. **Limits:** no Windows, AMD/no-mirror GPU, exact tester stack, long play or continuous flash test. Inspected captures show the intended scenes and ordinary HUDs, not proof against one-frame faults. No synchronization was disabled and no product change was tested. All game runs ended without a device loss or kernel GPU fault. <!-- github-sync:c:6830885748 -->
Commenting is not possible because the repository is archived.
Reviewers
No reviewers
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
autorelease: pending

   
autorelease: tagged

   
bug
Something isn't working

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
testing wanted

   
wontfix
This will not be worked on

No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Hookshot!2602
No description provided.