fix(vk): the late pages' fix no longer puts older words over the game's newer ones #526
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!526
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:fix/nightly-flicker"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Linux (a game-free test, and RTX 4090 runs of the nightly 140 build for the late-page counts), Windows 11 on an Intel Arc laptop (frame rate and dark frames, first fault's fix only) · Needs: Windows with an NVIDIA card (T-LUBF)
Regression: black flashes on Link and other characters since nightly 123, caused by #248. Nightly 80 was fine. Workaround until it is fixed in a nightly:
RX_VK_LATE_FIX=0inlongshot_env.txt.Work in progress (14 November): two faults in the late pages' fix, both fixed here. The first, a read of a page whose fix an earlier part of the same submission recorded, was found first; two testers' logs on Windows with NVIDIA show that path was not taken in their runs, and one still saw a few flashes with it. The second, in the write-back shadow (below), fits every switch result so far: the flashes need both the late pages' fix and
RX_VK_SHADOW_FOLLOWS_SYNC, and stop with either off. Neither is shown in a game yet. The log linelate pages' fix (600 recordings with one)now ends with a count for every path a late page takes, so one run on an affected PC shows which path a flash goes through. This stays work in progress until such a run.Why
Players on Windows with NVIDIA cards see short black flashes on Link and other characters, in Zora's Fountain and Hyrule Field among other places, with frame generation off too; setting
RX_VK_LATE_FIX=0stops them for every player who tried it (#366). The cause is in the late pages' fix from #248 (nightly 123): when the game reads a page whose GPU fix is still pending, the capture could end with older words than the game had written. The second fault is in the write-back shadow those fixes read from: a page could be counted as equal to the game's memory while the game had written newer words to it. This changes that read and that count.What is measured: a game-free test of the sequence fails on main and passes here (below). Not measured: the flash itself. It was not seen in a game run on Linux, and nothing was run on Windows or on a GPU that is slow enough to make late pages frequent.
How to reproduce
Seen on: players on Windows with NVIDIA cards (an RTX 4080 on nightly 143; an RTX 4050 Laptop on nightly 127; an RTX 3060, a GTX 1660 Ti and a further player on nightly 143). It was not seen in a game run on Linux or on the Intel Arc laptop used here.
Settings and switches: default settings and no
RX_VK_LATE_FIXline; frame generation off shows it too. SettingRX_VK_LATE_FIX=0stops the flashes for every player who tried it, and three players who set the three switches of #324 back still saw them.Steps: play in Zora's Fountain or Hyrule Field, among other places, and watch Link and other characters for brief black flashes.
Expected: no flashes (nightly 80 was fine).
Actual: short black flashes on Link and other characters since nightly 123.
What changed
This fixes two faults in the late pages' fix. First,
late_fix_cpu_readingpu/vk/vk_memory.cppnow waits for an earlier part of the same submission (the newvk_part_waitingpu/vk/vk_replay.cpp), so its fix has run and the capture is read as the fix left it, instead of copying the page again from the game's memory and leaving the fix to run over it. Second, withRX_VK_SHADOW_FOLLOWS_SYNCon (the default), a mirror sync no longer leaves as known the pages the replay or the collection wrote since (shadow_synced_known,shadows_forget_written), including when the forget's page lists overflowed, so the write-back shadow is filled again from the game's memory. Thelate pages' fix (600 recordings with one)log line also ends with a count for every path, andtests/test_late_fix_build.pycovers both faults.late_fix_cpu_readingpu/vk/vk_memory.cpp: the replay reads a late page of a capture on the CPU (for a draw's descriptors and constants) and the page has a fix. Before, a fix recorded in an earlier part of the same submission was neither waited for nor dropped; the function waited for the write-back, copied the page again from the game's memory and left the fix to run. Now that part is waited for (it is already submitted, so the wait is for work that is on its way), so its fix has run, and the capture is read as the fix left it, with nothing written to it (what a fix of an earlier recording already did). The wait has a hang guard of 5 s, logged; a part that ran past it is not waited for again. Three cases keep the old copy, with the fix's table counts set to 0 first so the fix writes nothing over it: the current part (not submitted, so the GPU has not read its table), a part past the guard, and a part that cannot be waited for. The last is only the case ofRX_VK_PER_QUEUE_FENCES=0or a recordings' timeline that could not be created (gpu/vk/vk_replay.cpp,per_queue_fences_off,replay_start): no GPU, driver or OS decides it, so NVIDIA's driver on Windows takes the wait unless that switch is set. That fallback keeps the race of zeroing a table a running part may be reading; it is rare by construction and counted (next bullet).vk_part_waitingpu/vk/vk_replay.cpp, declared ingpu/vk/vk_internal.h: waits for one part of a submission on the recordings' timeline, the same signal the readback wait already uses (wait_recorded_part); returns 1 done, 0 ran out, -1 cannot be waited for.late pages' fix (600 recordings with one)log line gets two counts at its end: of the reads on the CPU, how many waited for an earlier part's fix and how many copied the page again instead, so a player's log shows which way it went.tests/test_late_fix_build.py: the function is cut out of the source and run beside the existing cases, with a stand-in forshaders/late_fix.comprun over the fix table. Cases: the fix of an earlier part, the current part, a part that cannot be waited for (and its second page), an earlier recording done and not done.shadow_synced_knownandshadows_forget_writteningpu/vk/vk_memory.cpp: withRX_VK_SHADOW_FOLLOWS_SYNC(on by default) a mirror sync copies a capture's pages into a pool's write-back shadow, and after the write-back's forget those pages were marked known, meaning the shadow equals the game's memory. The forget has just cleared that mark for the pages the replay's own writes or the collection reported after the capture, where the game's memory holds newer words than the shadow, and the sync set it again. A known page is not read from the game's memory into the shadow before the next write-back, so two readers took the capture's older words: the write-back's diff pass (shaders/wb_diff.comp), which writes a whole 512-byte span of the mirror's words when any word in it differs, and the late pages' fix, whose words come from that shadow. Now the forget keeps those two page lists and the sync leaves those pages unknown, so the next write-back that writes one back reads it from the game's memory first. When the lists overflowed (more pages written between two write-backs than they keep, so the forget clears every page), the sync leaves every page it copied unknown too; a tester found that case. Pages the capture alone holds are known after the sync as before.late pages' fix (600 recordings with one)line now also ends with a count for every path: reads on the CPU with the fix done, of an earlier recording, not recorded yet and of this part; pages fixed from stamps and from snapshots; and for the shadows, the pages synced from captures, made known by it, and left unknown because the replay or the collection wrote them since. Plain counters; no behaviour change.tests/test_late_fix_build.py: the realshadows_forget_writtenis added to the fixture with a sequence of a sync of four pages, two of them written again by the replay and by the collection, the forget and the known marking; the two must stay unknown. It fails on the commit before the fix (failed line 1077: !is_known(a) && !is_known(c)) and passes with it. A second sequence does the same after a forget whose lists overflowed; it fails on the commit before its fix (failed line 1098: !(pool.shadow_known[i >> 6] >> (i & 63) & 1)) and passes with it. The counts are asserted too.RX_VK_LATE_FIXand ofRX_VK_SHADOW_FOLLOWS_SYNC.The chain of causes
capture_make,gpu/vk/vk_memory.cpp). Since #248 the capture copies it at once and a small compute pass (shaders/late_fix.comp), recorded in the uploads of the submission's first part, writes the words the write-backs wrote into the capture before the mirror is made from it.maybe_split,submit_recordingingpu/vk/vk_replay.cpp). The part that recorded the fix can be running, or run after what the replay thread does next.memory_read_as_submitted), which for a late page goes throughlate_fix_cpu_read. For a fix recorded in an earlier part it took the lastelsebranch: wait for the write-back, copy the page again from guest memory, leave the fix.shaders/late_fix.compreads the pool's write-back shadow) over them.The other branches of the function do not do this: a fix already run, or recorded by an earlier recording, only reads the fixed buffer; a fix not recorded yet, or recorded in the current part, is dropped before the copy.
Reviewer notes:
shadow_synced_knownand the new sequence at the end oftests/test_late_fix_build.py.late_fix_cpu_read(about 2384 on main) and the new case j intests/test_late_fix_build.py.RX_VK_PER_QUEUE_FENCES=0.Measurements
Verdict: the game-free test of the sequence fails on main (
failed line 868: slot[kX] == kA && slot[kY] == kCpu) and passes with this PR, the other new cases pass, and the shadow test fails on the commit before the fix (failed line 1077: !is_known(a) && !is_known(c)) and, for a forget whose page lists overflowed, before the overflow fix (failed line 1098: !(pool.shadow_known[i >> 6] >> (i & 63) & 1)), and passes with it. The flash itself is not measured: it was not seen in a game run here, with or without the fix. On the Intel Arc laptop (Windows, the first fault's fix only) the frame rate is not slower, inside the spread between runs, and no walking picture had a dark Link or character (0 of 685 against 0 of 661); the shadow fix is not measured yet.Branch state: on main
546dc21208e0331648cc208c10bd3039d4f15fdc; head69fb33a1460236116eae943eeaf9ff4bd635e83c; whole game-free suite on this head:554 passed, 6 failed, 16 skipped(main546dc21208e0331648cc208c10bd3039d4f15fdc:554 passed, 6 failed, 16 skipped). The 6 failing tests fail on main the same way:tests/test_appimage_player_build.py,tests/test_cpm_discovery.py,tests/test_lift_cleanup_error.py,tests/test_linux_preflight.py,tests/test_make_game_cache.py,tests/test_walk_key.py. The commits are unchanged by rebases apart from the results table.main:
543846ebb0(2026-11-14)this PR: fix/nightly-flicker @
4610ffddaHardware: game-free test, Arch Linux 7.2.15, Core i9-13900K, g++ with
-Wall -Wextra -Werror; no GPUScene: none (a unit test of the function over a made-up page)
Method:
python -I -B tests/test_late_fix_build.py, run on the test commit alone (test only, main's source) and on this headHow to reproduce: from the branch,
git checkout a1929ba56 -- tests/test_late_fix_build.pyis the test alone on main's source (it fails);python -I -B tests/test_late_fix_build.pyon the head passes543846ebb4610ffddatests/test_late_fix_build.py, case j: a CPU read of a late page whose fix an earlier part recorded; the write-back landed and the game wrote a newer wordfailed line 868: slot[kX] == kA && slot[kY] == kCpu(the page was copied again, then the fix ran over it)What the players' logs show, as evidence that the path is hit (not a measurement of this change):
late pages' fix (600 recordings with one)lines of the flickering run read 674 captures with late pages, 659 fixed, 82.8 pages a capture, 235 read on the CPU first, and 764 captures, 759 fixed, 113.8 pages, 587 read on the CPU first. Its run withRX_VK_LATE_FIX=0does not flicker.RX_VK_LATE_FIX=0stops the flicker, on, off, on.RX_VK_LATE_FIX=0; three players who set the three switches of #324 back still saw the flicker.RX_VK_SHADOW_FOLLOWS_SYNC=0and the late fix on. That fits the second fault (fixed here): with it on, the pages a mirror sync took are "known" to the write-back shadow and are not filled again (shadow_sync_copy,shadow_synced_known), so the fix and the write-back's diff pass read shadow words older than guest memory; with it off those pages are unknown at the next write-back and are filled again from guest memory on the GPU, so the shadow follows the newer words. Not proven in a game.The second fault: the shadow test
The related focused tests pass with it:
tests/test_wb_shadow_follows_sync.py,tests/test_wb_shadow_remake.py,tests/test_wb_diff_cpu_writes.py,tests/test_late_fix_shaders.py,tests/test_wb_diff_split.py.Arc laptop: frame rate and dark frames, main against this pull request (Windows)
Baseline: main @
543846ebb, this pull request's base when it was measured. Measured at4610ffddabefore the carry onto the current main; the carry replays the same two commits unchanged, and main's changes since (the replay's reserved core on Linux, the per-CPU game build) do not touch the late pages' fix, so the numbers were not taken again.main:
543846ebb0(2026-11-14)this PR: fix/nightly-flicker @
4610ffddaHardware: Intel Arc Graphics, integrated (Core Ultra 9 185H laptop, shared memory), driver 32.0.101.9175; 64 GB; Windows 11 Pro 10.0.26300
Scene: Kokiri Forest (child), the same autosave resumed on the forest floor, a 50 s walk of fixed stick inputs between the houses, camera centering off; frame generation off; 1280x720 window; Windows builds of both commits (same compiler and settings), shader caches warmed and restored to the same saved state before each measured run
Method: the two builds alternated, main / this PR / main / this PR. Picture-free runs: no pictures during the walk; fps from the game's own
vi: ... frames presented (... fps over the last 5.0 s)lines inside the walk (ten per run). Dense-picture runs: a picture of every frame the game's control bridge could take (about half of the presented frames, 306 to 365 per run); a program flags frames where part of the picture is much darker than in both neighbours, and every flagged frame was looked at.How to reproduce: Kokiri Forest, an autosave on the forest floor, the same 50 s walk on both builds, alternated; the late-page counters are in the game's log (
a capture taken (ms): ... GPU wait for late pages (N of 600 takes)), the fix's own summary line (late pages' fix (600 recordings with one)) did not appear in any run.543846ebb4610ffddaRX_VK_LATE_FIX=0as control: 13.35)RX_VK_LATE_FIX=0: 0 of 347)late pages' fix (600 recordings with one)lines, and the new counts "waited for an earlier part's fix" / "copied the page again instead"How to read the frame-rate rows: four picture-free runs, two per build, is a small sample. The two runs of one build differ by up to 1.1 fps; this PR's medians are 0.6 fps higher on average, inside that spread, so the claim is "not slower", not "faster". The walks end in different places (real-time physics) and first-time shader translations differ (0 to 35 per walk), so differences of about 1 fps are not meaningful. The late-page path is hardly used in these walks (0.3% of captures wait for a late page, about 12 to 14% in the menus before the walk), so these rows show no regression and the frame-rate comparison, not the fault or its fix. The flash was not seen on this GPU, with or without the fix.
Screenshots
Nothing on screen: the flash was not captured here.
Flags and settings
None.
RX_VK_LATE_FIX=0stays the way to get the old wait. No flag is added.Testing
Tested: the new cases both ways (table above) on Linux; the test commit alone fails; the rest of
tests/test_late_fix_build.pyis unchanged and passes. Game runs on Linux (RTX 4090, NVIDIA 615.78.08, windowless) of the nightly 140 build only, for the late-page counts; the build of this change was made but not run in the game, since this card shows too few late pages for a run to say anything. No flash was seen and none was expected: the card finishes a write-back before the next capture is taken, so late pages are rare in play (a walk through the field, the market and Kokiri Forest with area loads gave no late-pages summary line at all, which needs 600 recordings with a fix) and come in bursts at loads (618 captures with late pages, 371 read on the CPU first, at 5120x2880). Whole game-free suite on this head:554 passed, 1 failed, 16 skipped(Linux,--jobs 6 --strict, 153 s). The one failure,tests/test_walk_key.py(a link error,undefined reference to hos_oot_autoload_input), fails the same way on main @543846ebb; this change does not touch it, and #497 fixes it.Built for Windows and Linux at the current head, not run: At
69fb33a1460236116eae943eeaf9ff4bd635e83c, the Windows x86-64 Vulkan renderer DLL and all 50 selected host translation units were cross-built on Linux with llvm-mingw 20261104 (Clang 23.1.5), using the project's production flags and pinned dependencies (renderer: 35 newly compiled / 170 reused; host: 50 newly compiled / 0 reused; lifted C: 0 newly compiled / 80 reused; all Windows units: 245 newly compiled / 404 reused from matching object-cache entries), the renderer DLL linked and the full game executable linked; nothing was run on Windows or under Wine, and this is neither the project's hosted builder run nor a Windows UTF-8 run-time path test.The exact same head
69fb33a1460236116eae943eeaf9ff4bd635e83calso compiled and linked its full native Linux executable and renderer with Clang 23.1.3 through the documented resume path (490 newly compiled units, 1 matching Linux-cache units, source stamp584bae88), using the verified target-neutral C from lift414f56e226d2be90a288ff2eb2fa172f7d4e6c93rather than carrying the patch onto another commit; neither Linux binary was run.CPU target (this revision builds the game for the build PC's own CPU, #524; the two builds differ on purpose): the Windows cross-build keeps the builder's pinned
-march=x86-64-v3 -ffp-contract=off(488 of its 649 compile commands carry it; the other 161 carry no-marchflag; none carries-march=native,-mtuneor-DLONGSHOT_NATIVE_CPU), so the Windows binaries are not built for this PC's CPU. The native Linux build used the revision's default-march=native, which clang resolved on this PC (13th Gen Intel(R) Core(TM) i9-13900K) toraptorlake_raptorlake_5fe3e3650f(build log line:build: the game is built for this PC's CPU (raptorlake_raptorlake_5fe3e3650f)), so the Linux binaries are built for that CPU's instruction set and tuning; neither binary was run.Renderer profile (this revision has the committed renderer play profile, #569,
assets/profiles/longshot_vk.profdata.xz, 159448 bytes, git blob01238f45954c): both runners chose the profile with the revision's ownmake_game.vk_profile_for, as the project's build does, and passed it asRX_VK_PGO. Windows renderer (llvm-mingw Clang): built with it: 43 of its 649 compile commands (the renderer's own sources, not Eden's recompiler or sirit) carry-fprofile-instr-use=<the unpacked profile>and-DLONGSHOT_VK_PROFILE_SHA1=d965e746b947f357, all exit 0; build log line:the Vulkan back end: built with the profile longshot_vk.shipped.profdata. Linux renderer (native Clang): built with it (build log line:the Vulkan back end: built with the profile longshot_vk.shipped.profdata). The game's own lifted code is built with the olderlongshot_pieces.profprofile as before; neither binary was run.The earlier heads
444a376ad74fe7ac769f005f51c81d8a1379462e(before the overflow fix),fea96c39c04c802882c545107202068fb7b5b32f(the first fault's fix on main7ff5256e0) and4610ffddad50771e1b26b63bfe9e06837765c32f(on main543846ebb) were built the same way.Tested by a player on the shadow fix (
444a376ad7, before the overflow fix), 14 November: Windows 11, RTX 4070, NVIDIA 621.01, Zora's Fountain, the same checkpoint and a scripted camera and walk, about 90 s per run, renderer-only builds of the control (the counters without the shadow fix) and of the fix, no workaround switch. The observer saw at least one split-second black flash on Link in the control run and none in the fixed run; the rest (uneven frame pacing) was in both. The new counts show the second fault's path in play: in the control, 57,753 pages a mirror sync made known although the replay or the collection had written them after the capture; in the fixed run, 100,214 such pages left unknown; 0 reads waited for an earlier part's fix in either. Frame-time context from that one pair, not a measurement: 83.88 against 81.75 fps on average, p99 16.38 against 18.54 ms, no frame over 50 ms in either (#526 (comment)).Tested by players, 14 November, on the first fault's fix only (before the shadow fix): two posted logs (an RTX 4070 and an RTX 4050 Laptop on Windows) show 0 reads through the earlier-part path; one player still saw a few flashes, harder to notice; one saw no clear flash on either build. Separately, on a build of this pull request, a player who saw the flashes on the nightly reports that the flickering in Hyrule Field is gone and that Zora's Fountain shows none either (the team's Discord, 13:41 local).
Not tested: a game run on Windows or with NVIDIA's Windows driver; any card slow enough for late pages to be frequent in play; frame generation; a before and after picture of a flash (none was captured here); whether the part wait costs frame time where many late pages are read in later parts (the old way waited for the write-back instead; no frame times were taken); the fallback paths (a part that cannot be waited for or past the guard) in a real game, and the race that fallback keeps.
Must be tested:
RX_VK_LATE_FIXline, on a build of the current head: watch Link and other characters for brief black flashes, expecting none, and send thelate pages' fix (600 recordings with one)lines of the log (test request T-LUBF; by hand).Risks
What could break: The wait for an earlier part can stall the replay thread for as long as that part runs on the GPU (about 120 draws) the first time a later part reads one of its late pages; the old way waited for the write-back instead. A wait past 5 s (a hang guard that is logged) falls back to the old copy and that part is not waited for again.
Turning it off:
RX_VK_LATE_FIX=0turns the whole late pages' fix off; reverting the commit is enough.Compatibility:
longshot_options.iniand the shader and pipeline caches: unchanged. The text says the shaders, the stamps and the defaults ofRX_VK_LATE_FIXandRX_VK_SHADOW_FOLLOWS_SYNCare unchanged; the diff adds no shader file and touches no save or options code.gpu/vk/vk_memory.cpp,gpu/vk/vk_replay.cppandgpu/vk/vk_internal.h(the internal declaration ofvk_part_wait), notgpu/vk/vk_api.h.late pages' fix (600 recordings with one)line gets more counts at its end, so a player's log shows which path a late page took.Issues
Fixes #366
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.543846ebbagainst the first fault's fix @4610ffdda, two alternated runs each, medians. The shadow fix's frame-time cost is being measured now on that laptop and on Linux, main against the current head, interleaved; one tester's single pair, under Testing, is context only.)mainand passing here.tests/that reads changed source updated in this PR, not weakened.69fb33a1460236116eae943eeaf9ff4bd635e83c(under Testing); nothing was run on Windows at this head, so UTF-8 path behavior at run time is not claimed.New on this forge; not opened on GitHub.
Arc laptop check of this pull request: no regression seen
This is a check for regressions and frame rate on a slow integrated GPU. It does not show the fault or the fix: late pages turned out to be rare in these walks (see below), so I never saw the black flash with or without the change.
Hardware: Intel Arc Graphics (the integrated GPU of a Core Ultra 9 185H laptop, shared memory), driver 32.0.101.9175, Windows 11 Pro 10.0.26300, 64 GB RAM.
Built: this pull request's head
4610ffddad, and main543846ebb0(the pull request's base; the pull request is main plus its two commits). Both are Windows builds made from those exact commits with the same compiler and settings.What I did, all in Kokiri Forest, frame generation off, window output 1280x720, no co-op, no audio, camera centering off:
RX_VK_LATE_FIX=0as a control. A small program compared each picture with the ones before and after it and flagged any frame where part of the picture was much darker than in both neighbours (the kind of dip a black Link would make). I looked at every flagged frame in a strip of before, flagged and after: 83 single frames and 26 pairs of frames.Late pages in these walks:
late pages' fix (600 recordings with one)line, so fewer than 600 recordings with a fix happen in a whole run of 2 to 5 minutes. For the same reason the two new counts this pull request adds to that line never appeared.GPU wait for late pages (N of 600 takes)line is the other sign of them: in the walks 51 of 16,200 captures waited on main (0.3%), 43 of 16,800 on this pull request (0.3%), 11 of 4,200 withRX_VK_LATE_FIX=0(0.3%). In the title screen, menus and loading before each walk it is about 12 to 14% (medians of 74 to 87 per 600 captures) on all three.Results (D1 to D5 dense-picture runs, T1 to T4 picture-free runs, each one 50 second walk):
RX_VK_LATE_FIX=0Not tested: NVIDIA cards or any Windows run on NVIDIA; Zora's Fountain; Hyrule Field with pictures (only the late-page survey on main); frame generation (off in every run); runs longer than a few minutes; the flash itself, which I did not see on this GPU in any run, with or without
RX_VK_LATE_FIX=0.4610ffddadfea96c39c0Carried onto main
7ff5256e0(main gained the replay's reserved-core fix and the per-CPU game build since); new headfea96c39c. The two commits are unchanged. The whole suite on this exact head: 553 passed, 2 failed, 16 skipped. The build box is unticked until this pushed head has been built for Windows and Linux; it is ticked again then.WIP: fix(vk): a late page the game reads is no longer overwritten with older words by its pending fixto fix(vk): a late page the game reads is no longer overwritten with older words by its pending fixBuilt at the current head
fea96c39c04c802882c545107202068fb7b5b32f, nothing run: the Windows executable and renderer cross-built with llvm-mingw (-march=x86-64-v3, as the Windows builds are pinned), and the Linux executable and renderer built natively (-march=native, which since #524 is this build PC's own CPU). Both linked without errors; the details are under Testing, and the build box is ticked for this head. With the laptop measurement above, every box is done, so this is ready for review. A Windows run on an NVIDIA card withoutRX_VK_LATE_FIX=0is under way through test request T-LUBF.Result from a player who saw the flashes (Tommy, in the team's Discord), on a build of this pull request: the flickering he had in Hyrule Field on the nightly is gone, and after warping to Zora's Fountain there was no flickering there either. With the game-free test, the builds of this head and the Intel laptop's no-regression run, that is everything this side planned; it is ready to merge from here. Test request T-LUBF is still claimed by another tester, whose result will be added when it comes.
Test request T-LUBF: mixed / visual fix inconclusive.
Windows regression check passes; these game runs do not verify the reported black-flash fix. The new earlier-part wait path was not exercised in the logged intervals. The person watching also reported uncertain flashes/artifacting, so this is not a clean visual pass or a merge approval.
Hardware: RTX 4070 12 GB, NVIDIA 621.01, i7-13700F, 32 GB RAM, Windows 11 build 26300; llvm-mingw clang 23.1.5, bundled Python 3.14.7.
Exact-head game-free checks: PR
fea96c39c04c802882c545107202068fb7b5b32f: 532 passed, 5 failed, 34 skipped. Control5765a767(the test-only commit on main7ff5256e): 531 passed, 6 failed, 34 skipped. Its extra failure is the intended regression:failed line 868: slot[kX] == kA && slot[kY] == kCpu; the fixed test passes. The five shared Windows failures aretest_cpm_discovery.py,test_oot_state_tap.py,test_release.py,test_tested_configurations.py, andtest_walk_key.py. No PR-only failure.Runtime provenance: renderer-only A/B packs on the same existing full-build host, main
dc390e593c432a00c77414a24542b27a3f1cd4ec, using the documented dev-packs workflow and disposable save copies. This was a local backport of the PR's three renderer-file changes, not a full game built at PR head. Fixedvk_memory.cppandvk_internal.hare byte-identical to PR head;vk_part_waitis identical too. The remainingvk_replay.cppdifference is existing diagnostic wait logging. The public renderer API is unchanged between this host and the PR base. The Windows renderer pack compiled/linked with the local optional NGX SDK, 0 failures. No shared base was moved.Settings and route: Custom, 2560x1440 output, 100% render size, Game's own upscaler, SDR, uncapped, V-Sync off, frame generation off; 1920x1080 window. These are controlled comparison settings, not Auto defaults. No
RX_VK_LATE_FIX=0workaround, no per-queue-fence override. Each fresh process loads the same copied field checkpoint; Fountain runs teleport toZoraFountain_Childbefore settling. Camera rotation, then a tight movement loop: 120 s in Hyrule Field, 90 s in Fountain. The first field baseline was repeated because the observer missed most of it. All five runs exited 0, passed quiet-machine guards, and had no fatal/device-loss/5-second-guard diagnostic.Human observations (qualification preserved):
Coverage: the fixed field log summaries contain 3,757 CPU reads of late pages; fixed Fountain 5,411. Both have 0 earlier-part waits and 0 earlier-part fallback copies. These are sums of completed log-summary intervals, including startup/loading, not counters bounded to the timed segment or every read before shutdown. The specific changed branch therefore remains unverified on this GPU. A representative fixed Fountain line ends:
Frame-time context only:
All selected trace ranges are complete and consecutive. The two fixed runs each have one interval over 50 ms (none over 100 ms); the baseline windows have none over 50 ms. A few screenshot readbacks occurred in every timed run, and there is only one fixed run per scene, so these are diagnostic observations, not a performance gain/regression claim or proof of the reported pacing issue. Raw traces, builds, saves and captures remain local.
Still needed: an affected Windows/NVIDIA reproduction at the current full build, with the new log counter nonzero and a clear character-black-flash comparison. The unit regression is resolved on Windows; the visual target remains inconclusive here. No approving review based on this result.
Windows/NVIDIA follow-up from Zora's Fountain: visual result inconclusive.
Hardware: RTX 4050 Laptop GPU (6 GB), NVIDIA 621.14, Windows 11 Home build 26200, 32 GB RAM.
Build:
dev_packsbase543846ebb(nightly 145) and a renderer-only pack from this PR at4610ffdda, before its rebase. The three renderer source files are unchanged at the current headfea96c39c; this was not a full build of that rebased head. The same base executable and runtime were used; onlylongshot_vk.dllwas swapped.tests/test_late_fix_build.pypassed on Windows with the pinned PR source.Scene/settings: same copied
ZoraFountain_Childsave, 1920x1080 display, same options file and warmed cache,RX_VK_FRAMEGEN=0, noRX_VK_LATE_FIXline. This local build has no DLSS. The controller was detected through SDL3 in both runs.Observation: the player saw rare black flickers around Link on the base and still saw a few with the PR DLL, subjectively harder to notice. Camera movement and run lengths were not scripted or matched; there is no capture to establish whether these are the same fault. This is not a measured reduction, a visual pass, or proof that the proposed fix failed.
Completed
late pages' fixlog summaries: base 1 summary / 40 CPU-first reads; patched 3 summaries / 458 CPU-first reads, 0 earlier-part waits and 0 fallback copies. The summaries include startup and loading, and omit reads after the last summary. Both runs reachedZoraFountain_Childand closed without fatal or device-loss lines. The changed branch was not observed on this GPU in these runs, similar to the mixed T-LUBF result already posted. I kept both full logs locally and can repeat a specific route if one is suggested to make the new wait count nonzero.fix(vk): a late page the game reads is no longer overwritten with older words by its pending fixto WIP: fix(vk): a late page the game reads is no longer overwritten with older words by its pending fixThank you both, these logs are exactly what was needed. Both show thousands of pages read on the CPU first and 0 reads that waited for an earlier part's fix, so the path this pull request changes did not run in your runs, and the flashes (rarer, uncertain, but still there for one of you) come from somewhere else in the late pages' fix. The ordering fault fixed here is real (its test fails on main), but I am not claiming it fixes the flashes. I have set this back to work in progress. Next: the copy of the written-back words the fix reads from (the write-back shadow). One run with
RX_VK_SHADOW_FOLLOWS_SYNC=0also stopped the flashes, which points there. A build with a count for each way a late page can be read and fixed follows, so a run shows which path a flash goes through. Until thenRX_VK_LATE_FIX=0stays the workaround.Follow-up from the same isolated Windows RTX 4050 test copy of PR #526 (
4610ffddarenderer on the543846ebbfull-build base):RX_VK_LATE_FIX=0, the player watched Link and moved the camera for about 2 minutes and saw no black flash.The observation periods and camera paths were not identical, so this does not measure a reduction or prove the switch removes every flash. It does reinforce the late-fix feature as the lead while showing that the PR's newly fixed earlier-part-wait path did not run in this observation. No frame of the flash was captured. I retained both full logs locally and can run the planned per-path diagnostic build on this GPU when available.
fea96c39c0444a376ad7A second fault, in the write-back shadow, and its fix are now on this pull request, with a count for every path in the log.
Why the shadow: the flashes stop with
RX_VK_LATE_FIX=0for seven setups, and Rogasse's short run withRX_VK_SHADOW_FOLLOWS_SYNC=0also stopped them. So the fault needs both the late pages' fix and the write-back shadow that follows the mirror's sync.What I found by reading the code and checked with a test that needs no game: when a mirror sync copies a capture's page into a pool's write-back shadow, the shadow is marked "known" (equal to the game's memory) after the write-back's forget. The forget had just cleared that mark for pages the replay's own writes or the collection reported after the capture, because the game's memory holds newer words there. The sync then set the mark again. A page marked known is not read from the game's memory into the shadow before the write-back, so the shadow kept the capture's older words, and two readers use that shadow: the write-back pass, which writes the mirror's words of a whole 512-byte span when any word in it changed, and the late pages' fix, which takes its words from the shadow. With
RX_VK_SHADOW_FOLLOWS_SYNC=0that mark is never set, which fits Rogasse's short run.The new test fails on the branch as it was and passes with the change: the pages the capture alone holds stay known, the pages written again after the capture stay unknown and are read from the game's memory at the next write-back that writes them back.
I have not seen the flash on the machine I test on (an RTX 4090 on Linux), so this is not yet shown to end it on Windows. To tell, the line that starts
late pages' fix (600 recordings with one)now ends with counts for every path: reads on the CPU by case, pages fixed from stamps and from snapshots, and for the shadows the pages synced from captures, made known, and left unknown because they were written again. If you can run the next build on Windows with an NVIDIA card in Zora's Fountain or Hyrule Field, default settings and noRX_VK_LATE_FIXline, please send that line from a run with a flash and from one without.Not tested: Windows, NVIDIA's Windows driver, frame generation. Two properties of the write-back that I read but did not change: a mirror sync can copy an older capture of a page over a newer GPU result (it was there before this work), and the write-back pass writes whole spans from the mirror.
Now on main
414f56e22, head444a376ad; whole game-free suite on it: 553 passed, 4 failed, 16 skipped. The builds for both systems follow.WIP: fix(vk): a late page the game reads is no longer overwritten with older words by its pending fixto WIP: fix(vk): the late pages' fix no longer puts older words over the game's newer onesBuilt at the current head
444a376ad74fe7ac769f005f51c81d8a1379462efor Windows (llvm-mingw cross-build) and Linux (native); both linked without errors, nothing was run. The build box is ticked for this head and the details are under Testing. This stays work in progress until a run on an affected PC shows the flash is gone (or which path it still takes).T-LUBF testing follow-up on
444a376ad74fe7ac769f005f51c81d8a1379462e: the revised shadow regression passes on native Windows, but an additional authored case finds a residual invalidation case.shadow_written_to()setsshadow_written_all=trueand clears all three detailed lists when they exceed8u << 20entries. In that reachable state,shadows_forget_written()clears every pool's known bits, consumes/resetsshadow_written_all, and leavesshadow_forgotten_lastempty (or containing only writes recorded after the overflow).shadow_synced_known()then sets the synced capture page's known bit again because the detailed list no longer identifies it. That discards the conservative invalidation when guest memory may hold newer words than the capture/shadow.I added this case after the existing cases in the branch's source-extracted
test_late_fix_build.pyfixture, without modifying production source:Native Windows 11 build 26300, llvm-mingw clang 23.1.5: compiled successfully, then failed the second assertion (
failed line 1100). Preserving theallinvalidation through the following known-marking pass would cover the lost-list case; please include it in the regression fixture. This is an incomplete edge of the existing fix, not a demonstrated new runtime regression. I have not measured how often the overflow occurs in play, or linked it to the players' flashes.The normal new regression fails on the pre-shadow-fix control
ff8ce525at!is_known(a) && !is_known(c)and passes on this head. The visual comparison is prepared but has not run yet; the observer requested an explicit readiness confirmation before launch.Thank you, futondestroyer: that case is right. After a forget whose page lists overflowed, the sync's known marking set the synced pages known again because the list it checks was empty. The new head keeps whether the lists overflowed and leaves every page synced since unknown after such a forget; your sequence is case (r) of
tests/test_late_fix_build.py, as you wrote it plus a forget with whole lists afterwards. It fails on the previous head (failed line 1098, the same assertion you saw) and passes here; the other write-back shadow tests pass too. As you say, how often the lists overflow in play is not known; the summary line counts these pages with the ones left unknown.Now on main
546dc2120, head69fb33a14; whole game-free suite on it: 554 passed, 6 failed, 16 skipped. The builds for both systems follow.444a376ad769fb33a146Built at the current head
69fb33a1460236116eae943eeaf9ff4bd635e83cfor Windows (llvm-mingw cross-build) and Linux (native); both linked without errors, nothing was run. The build box is ticked for this head and the details are under Testing. This stays work in progress until a run on an affected PC shows whether the flash is gone or which path it still takes.T-LUBF, revised PR
444a376ad74fe7ac769f005f51c81d8a1379462e: positive targeted visual A/B result; mixed overall because the reported overflow regression remains unresolved.The observer identified at least one split-second black flash on Link in the control. No black flash was seen in the fixed run. The remaining minor artifacts were described as mostly stutter/uneven frame pacing, present in both; the observer reported nothing new in the fixed run. This is one approximately 90-second observation per arm, not proof that all flashes or pacing issues are eliminated.
RTX 4070 12 GB, NVIDIA 621.01, i7-13700F/32 GB, Windows 11 build 26300. Same copied checkpoint, scripted camera rotation then movement in
ZoraFountain_Child; Custom 2560x1440, 100%, Game's own upscaler, SDR, uncapped, V-Sync off, frame generation off, 1920x1080 window. No late-fix or shadow-follow workaround. No screenshot readbacks during the timed segments. Both runs exited 0, passed quiet-machine guards, and had no fatal, device-loss or five-second wait-guard diagnostic.This time the second fix's shadow path was exercised:
ff8ce525)444a376a)Those totals establish path coverage; they are not equal-duration rates or a performance comparison. The earlier-part wait path still did not run, while the second shadow fix did.
Frame-time context for the completed timed ranges (continuous, consecutive trace frames):
One pair does not establish the size or cause of a performance difference. The observer's pacing complaint is retained; no pacing improvement or zero-cost claim is made.
Provenance/limits: renderer-only A/B packs on the same existing
dc390e593c432a00c77414a24542b27a3f1cd4ecfull-build host, not full games built at the PR heads. Control/fixedvk_memory.cppmatch their respective public revisions byte for byte; the fixed wrapper and interface match as previously reported. Local pack commits are35cf26df(diagnostic control) and5fd2b860(revised fix). An initial control attempt stopped before the timed check because a blocking menu input timed out; it was excluded, the startup input was made nonblocking, and the completed control retry above was used. No game files, builds, saves or raw traces leave the PC.Outstanding before calling the whole change verified: the authored
shadow_written_alloverflow case still fails this head, as reported at #526 (comment) . The positive normal-path visual result does not settle that case. No approving review is implied by this report.Thank you, futondestroyer, this is the result that was missing: the control's counts show the shadow path in play (57,753 pages made known although written again after the capture), the fixed run left 100,214 such pages unknown, and the observer saw a flash on Link only in the control. The overflow case from your earlier comment is fixed in the current head
69fb33a146, with your sequence as case (r) of the test (comment #526 (comment)); it is built for Windows and Linux. The frame-time difference in your one pair (about 2.5% lower average, higher p99) is noted in the description as context; whether the extra page reads cost frame time is the open measurement before this is called done.WIP: fix(vk): the late pages' fix no longer puts older words over the game's newer onesto fix(vk): the late pages' fix no longer puts older words over the game's newer onesApproved at
69fb33a146. The overflow issue I reported is resolved: the forget preserves its all-pages invalidation through the sync's known marking, and the following non-overflow forget resets that state. Case (r) covers both transitions.Independent Windows 11 / RTX 4070 test follow-up with llvm-mingw clang 23.1.5: the current source-extracted
tests/test_late_fix_build.pycompiles and passes against this exact head. The same fixture against the previous444a376asource fails the overflow assertion, as expected. The current-head description check passes. I have not repeated the complete suite on this head; the earlier Windows suite comparison and author's current-head builds/suite remain the evidence for those checks.Our affected-hardware visual comparison is recorded at #526 (comment) : a definite brief black flash on Link in the control, none observed with the second shadow fix, and no new visual issue reported. The normal shadow-fix path was exercised. That runtime comparison used renderer packs of the earlier revision on the documented older host, not a full game built at this head; the new overflow change has been verified here with the source-extracted regression, not another game run.
The previous report's overflow blocker is therefore cleared. Repeated interleaved performance runs remain useful follow-up: the single pair's approximately 2.5% FPS difference does not establish a regression or a zero-cost result. This approval makes no pacing-improvement claim, and does not claim runtime coverage of the earlier-part wait path or prove that every black flash is eliminated.
Windows/NVIDIA follow-up on PR #526, renderer source
444a376ad(the shadow fix and per-path counters), on the same isolated full-build host at543846ebb(Nightly 145 base). This is a renderer-only update, not a full game built at the PR head. The renderer source filegpu/vk/vk_memory.cppmatched444a376byte for byte; the updated DLL loaded and the new log counters appeared.RX_VK_FRAMEGEN=0);RX_VK_LATE_FIXon (no override); no shadow-follow override. This local renderer build lacks DLSS.ZoraFountain_Child, source recording 2:05–2:13; (2) outside the Castle Town gate inHyruleField_Child, source 3:25–3:35. The player describes it as milder and rarer than before, but this was one free-play run, not a matched rate comparison. The short clips were cut from the same recording and recompressed for upload.late pages' fix (600 recordings with one)summaries counted 170 CPU-first reads in total, 0 earlier-part waits and 0 fallback copies. The third summary counted 127,124 shadow pages left unknown after later replay/collection writes. These are aggregate intervals including loading, with no exact timestamp link to either flash; they show path activity, not causation.The current PR head has since moved to
69fb33a14, including an overflow-case fix that this run did not test. Also, issue 366 now has a trace of a potentially separate CPU-event capture path for dark-Link frames. I cannot tell from this run whether either clip has that cause. The two clips and the exact three summary lines are below.Exact late-pages summary lines from this run
Video clips
Exploratory Windows/NVIDIA follow-up on PR #526 head
69fb33a1460236116eae943eeaf9ff4bd635e83c: I built its renderer over a separate copy of Nightly 170 (6c9edffe). This is a renderer-only test build, not a full game build from the PR head.RX_VK_LATE_FIXenabled by default, no shadow override, Frame Generation off. I moved the camera and played around Link in Zora's Fountain, including near Jabu-Jabu. I saw no confirmed black flash in this run.vk: late pages' fix on, but contains nolate pages' fix (600 recordings with one)summary. The latest capture summaries in both logs sayGPU wait for late pages 0.000 (0 of 600 takes). I have no captured flash from these runs.Result: inconclusive. Neither run showed a confirmed reproduction, the camera routes and Frame Generation settings were not matched, and the PR log does not show the targeted path running. This is not a visual pass for #526 or evidence of an FPS change. The earlier player-reviewed clips on renderer
444a376remain in my previous comment.