Vulkan: no one-frame fog/exposure flashes from OoT's queue ordering #415
Closed
hckrman101 wants to merge 2 commits from
flicker-queue-order into main
pull from: flicker-queue-order
merge into: MagicBeanDev:main
MagicBeanDev:main
MagicBeanDev:github/pr/2893
MagicBeanDev:github/pr/2890
MagicBeanDev:github/pr/2887
MagicBeanDev:github/pr/2878
MagicBeanDev:github/pr/2875
MagicBeanDev:github/pr/2874
MagicBeanDev:github/pr/2873
MagicBeanDev:github/pr/2871
MagicBeanDev:github/pr/2756
MagicBeanDev:github/pr/2897
MagicBeanDev:github/pr/2896
MagicBeanDev:github/pr/2718
MagicBeanDev:github/pr/2842
MagicBeanDev:github/pr/2691
MagicBeanDev:github/pr/2895
MagicBeanDev:github/pr/2743
MagicBeanDev:github/pr/2765
MagicBeanDev:github/pr/2708
MagicBeanDev:github/pr/2732
MagicBeanDev:github/pr/2755
MagicBeanDev:github/pr/2817
MagicBeanDev:github/pr/2876
MagicBeanDev:release-please--branches--main
MagicBeanDev:release-please--branches--main--release-notes
MagicBeanDev:github/pr/2778
MagicBeanDev:github/pr/2888
MagicBeanDev:github/pr/2866
MagicBeanDev:github/pr/2877
MagicBeanDev:github/pr/2800
MagicBeanDev:github/pr/2829
MagicBeanDev:github/pr/2850
MagicBeanDev:github/pr/2889
MagicBeanDev:github/pr/2882
MagicBeanDev:github/pr/2860
MagicBeanDev:github/pr/2849
MagicBeanDev:github/pr/2886
MagicBeanDev:github/pr/2794
MagicBeanDev:github/pr/2840
MagicBeanDev:github/pr/2830
MagicBeanDev:github/pr/2834
MagicBeanDev:github/pr/2719
MagicBeanDev:github/pr/2742
MagicBeanDev:github/pr/2741
MagicBeanDev:github/pr/2820
MagicBeanDev:github/pr/2771
MagicBeanDev:github/pr/2859
MagicBeanDev:github/pr/2858
MagicBeanDev:github/pr/2823
MagicBeanDev:github/pr/2757
MagicBeanDev:github/pr/2769
MagicBeanDev:github/pr/2844
MagicBeanDev:github/pr/2843
MagicBeanDev:github/pr/2673
MagicBeanDev:github/pr/2855
MagicBeanDev:github/pr/2739
MagicBeanDev:github/pr/2807
MagicBeanDev:github/pr/2851
MagicBeanDev:github/pr/2789
MagicBeanDev:github/pr/2814
MagicBeanDev:github/pr/2818
MagicBeanDev:github/pr/2774
MagicBeanDev:github/pr/2772
MagicBeanDev:github/pr/2750
MagicBeanDev:github/pr/2786
MagicBeanDev:github/pr/2785
MagicBeanDev:github/pr/2782
MagicBeanDev:github/pr/2779
MagicBeanDev:github/pr/2775
MagicBeanDev:github/pr/2706
MagicBeanDev:github/pr/2766
MagicBeanDev:github/pr/2747
MagicBeanDev:github/pr/2759
MagicBeanDev:github/pr/2745
MagicBeanDev:github/pr/2740
MagicBeanDev:github/pr/2733
MagicBeanDev:github/pr/2725
MagicBeanDev:github/pr/2727
MagicBeanDev:github/pr/2728
MagicBeanDev:github/pr/2729
MagicBeanDev:github/pr/2721
MagicBeanDev:github/pr/2722
MagicBeanDev:github/pr/2715
MagicBeanDev:github/pr/2714
MagicBeanDev:github/pr/2707
MagicBeanDev:github/pr/2703
MagicBeanDev:github/pr/2705
MagicBeanDev:github/pr/2697
MagicBeanDev:github/pr/2679
MagicBeanDev:github/pr/2676
MagicBeanDev:github/pr/2669
MagicBeanDev:docs/what-hookshot-connects-to
MagicBeanDev:fix/hdr-encode-test
MagicBeanDev:fix/main-tests
MagicBeanDev:fix/play-cache-files
MagicBeanDev:perf/game-barrier-order
MagicBeanDev:feat/heap-lazy
MagicBeanDev:feat/texture-detail
MagicBeanDev:perf/astc-opaque-bc1
MagicBeanDev:fix/fs-dir-lock
MagicBeanDev:fix/framegen-native-kept
MagicBeanDev:fix/update-no-downgrade
MagicBeanDev:port/k10-gcn-f16-mirrors
MagicBeanDev:fix/framegen-mult-cap
MagicBeanDev:chore/release-as-2.1.1
MagicBeanDev:perf/pcache-load-switch
MagicBeanDev:fix/prepare-progress-log
MagicBeanDev:fix/update-closes-stale-game
MagicBeanDev:fix/hdr-scaled-sampled
MagicBeanDev:perf/first-start-preset
MagicBeanDev:perf/zero-upload
MagicBeanDev:fix/vcore-escape-2s
MagicBeanDev:test/fake-gh-atomic
MagicBeanDev:perf/batch-idle-flush
MagicBeanDev:test/finish-stub-a
MagicBeanDev:chore/record-2.1.0
MagicBeanDev:fix/main-red-tests
MagicBeanDev:fix/test-display-startup-driver-cache
MagicBeanDev:feat/mesh-shader-mode
MagicBeanDev:fix/menu-probes-at-startup
MagicBeanDev:fix/fps-counter-classic
MagicBeanDev:fix/test-log-batch-flake
MagicBeanDev:fix/narrow-screen-hud
MagicBeanDev:ci/prereleases
MagicBeanDev:release/v2.0.15
MagicBeanDev:hotfix/v2.0.15
MagicBeanDev:fix/present-ahead-replay
MagicBeanDev:release/v2.0.13
MagicBeanDev:fix/bracket-paths-2013
MagicBeanDev:fix/hdr-overlays-2013
MagicBeanDev:fix/fps-counter-scale-2013
MagicBeanDev:fix/stick-seen-at-rest
MagicBeanDev:hotfix/menu-fits-screen
MagicBeanDev:fix/f1-always-opens-menu
MagicBeanDev:fix/ini-keep-lines
MagicBeanDev:claude/cranky-pike-7cf247
MagicBeanDev:perf/texture-answer-flat
MagicBeanDev:fix/stamp-test-modules
MagicBeanDev:fix/appimage-fonts
MagicBeanDev:fix/display-awake
MagicBeanDev:release/v2.0.12
MagicBeanDev:fix/discord-windows
MagicBeanDev:rc/v2.0.12-tests
MagicBeanDev:fix/dlss-keep-feature
MagicBeanDev:rc/v2.0.12
MagicBeanDev:button-icons
No reviewers
Labels
Clear labels
accessibility
Barrier affecting people with disabilities
autorelease: pending
autorelease: tagged
bug
Something isn't working
dependencies
Pull requests that update a dependency file
documentation
Improvements or additions to documentation
duplicate
This issue or pull request already exists
enhancement
New feature or request
github_actions
Pull requests that update GitHub Actions code
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
question
Further information is requested
testing wanted
wontfix
This will not be worked on
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!415
Loading…
Reference in a new issue
No description provided.
Delete branch "flicker-queue-order"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes most of the remaining one-frame flashes on Linux (#371): haze and sun shafts appearing or vanishing for one frame, with the environment brighter or darker while the HUD and characters stay the same.
OoT's presentation queues submit a frame's work before two things it waits for are ready: the scene work (through event waits) and the CPU's writes for that frame (the CPU signals an event when it has finished). Two paths replayed that work against the previous frame's data.
Forced waits. Every NVN queue shared one fence timeline, and it only advanced as far as the oldest submission stopped at a wait. Three things waited on that timeline and pushed the presentation queues' stopped work past its waits:
Fences are now per queue:
nvnQueueFenceSynctakes the queue's own last submission. They are waited on through a second timeline that numbers the recordings. The readbacks no longer force anything through.RX_VK_PER_QUEUE_FENCES=0restores the old behaviour.Stale captures. The memory capture was taken at submit time, so it held the previous frame's data. Now a submission stopped at a wait for a CPU-signalled event takes a fresh capture of the CPU's writes when it resumes, because the console's GPU reads memory after its wait.
RX_VK_RESUME_CAPTURE=0turns this off.Measured on Linux with an RTX 3090: Kokiri Forest save, standing still, 60 s frame log on a 75 Hz monitor, one-frame events counted with
scripts/framelog.py.ba594dac)The frame rate goes up because the main thread no longer blocks on the other queues' stopped work.
Diagnosis:
Known gaps:
The branch is rebased onto current main (after #416). The test now checks
take_capture(), which both capture paths share, forcap_view_bump(). The numbers above were measured onba594dac.Tested on Windows 11, AMD Radeon RX 9070 XT (Vulkan driver 0x8001a0, 32.0.32015.7029), Ryzen 7 5800X3D, 3440x1440 fullscreen. Built from
no-visual-studiowith this PR plus #423, #435, #436, #329 and #205.With this PR the fps goes up, but the screen gets very fast, subtle white flashes in Kokiri Forest (not there on the same build without it). Results:
RX_VK_RESUME_CAPTURE=0only: still flashing.RX_VK_PER_QUEUE_FENCES=0andRX_VK_RESUME_CAPTURE=0: no flashing, but the fps gain is gone.So the per-queue fences seem to cause the flashing on this AMD card. I'm happy to test a new version or send a log.
Going to rework this and ensure it's better
Hang on RADV with batched submits
This PR hangs at startup on RADV, where batched submits (#413) are on by default. The game's main thread blocks forever in
nvnSyncWait, before the first movie:Cause
submit_recordingputs therec_timelinesignal insignals[], and only the unbatchedvkQueueSubmit2uses that array.batch_on(), the command buffers go intopend, andqueue_flush_locked()signals onlytimelineandbatch_tl.rec_timelinenever advances, and the firstwait_recorded()that reaches it never returns.It doesn't show with batching off, which is why it wouldn't show on NVIDIA.
Reproduction
RX 7800 XT, RADV (Mesa, Vulkan 1.4), Fedora 44, Linux 7.2. This PR merged onto current main.
RX_VK_PER_QUEUE_FENCES=0RX_VK_RESUME_CAPTURE=0The guest memory was a memfd window, from #448. With anonymous memory (
HOS_GUEST_MEMFD=0) it didn't hang in one run, so timing seems to decide whether a wait reaches the new path. The missing signal is there either way.Fix
Carry the highest recording number in the batch, and signal it at the flush. With this, the game starts every time and plays (90 s and 5.5 min runs).
I'm measuring this PR's effect on the flashes on RADV now and will post the numbers here.
Found and patched with Claude (Anthropic); I ran the tests on my machine.
Thanks @azlekayn, that's right: held submissions never signalled
rec_timeline. I've pushed your fix as its own commit, credited to you. I could reproduce the hang on NVIDIA by forcingRX_VK_BATCH=1: it hung before the first present, and now it starts and plays.@Kirkidoo, thanks for testing on the RX 9070 XT. Your result matches what I measure: the per-queue fences are what bring the white flashes back. They change when the CPU reads OoT's exposure meter, and the meter then sometimes reads 0 or twice its value for a frame. I'm working on a fix for the meter itself, and on how to measure it reliably, because one-frame flash counts vary too much between runs to compare small changes. I'll post numbers here when I have them. Until then,
RX_VK_PER_QUEUE_FENCES=0keeps the other half of this PR, which roughly halved the flashes in my runs, without the fences.I'm closing this for now. I'll open a new PR once I have a fix I'm confident in, which should be soon.
What I've found since opening it, on an RTX 3090 (Linux):
435a0138b9013040) samples the HDR scene image after it has been drawn that frame, on bad frames too. On a bad frame it measures the same brighter version of the scene every time, so the problem is earlier, in what draws the scene that frame.Thanks to @azlekayn for the batched-submit fix (it's in this branch) and to @Kirkidoo for testing. I'll link the new PR here.
Tested on an RTX 4080 / Windows 11 / Ryzen 9 7900X:
main4cfa08fwith this PR merged, warm caches, Kokiri Forest standing still, 60 sRX_VK_FRAMELOGper run, windowed 1280x720, frame generation off. One run per build, plus two runs of plainmain.main(2 runs)main+ #415No regression in fps, no white frames or device lost. Plain
mainbarely flickers on this card, so I can't confirm the fix here. One note: #415 showed 8 small darker blips/min against main's 3-4. That's a single run each and may be noise, but it lines up with Kirkidoo's report of new flashes with the per-queue fences.The follow-up is #513: it includes both commits from here, rebased, plus the fix for the remaining flashes (the CPU running a frame ahead and overwriting data the GPU still had to read). A/B: 5 flashes to 0, brightness unchanged.
Update: this PR's commits are now in #523 (rebased, with the RADV batched-submit fix); #513 only has the fix for the remaining race.
A follow-up to the RADV hang fix above, in case it's useful for this PR: https://github.com/azlekayn/Hookshot/tree/fix/readback-wait-when-behind
The branch is this PR plus two commits:
e5dc399: the hang fix from my earlier comment (batched submits also signal the recordings' timeline).bdf58d3: on this PR's unforced readback path, a readback waits for its write-back only while the replay is behind. #296's wait applies only then, not on every readback.RX_VK_READBACK_WAIT_ALWAYS=1restores always waiting.Results on an RX 7800 XT (RADV), Kokiri gameplay, 75 s frame logs. A flash is a frame where at least 20 of 144 screen blocks jump the same way:
The last row shows the change needs this PR's per-queue fences. Without them, "behind" can't be detected reliably. Feel free to cherry-pick either commit.
Correction to my comment above (the
fix/readback-wait-when-behindbranch): please disregard it. I posted it without seeing that this PR had been replaced by #523.I re-tested on current main (
97e7c55, with #513 and #523, plus my open #448 for guest memory on this setup). RX 7800 XT, RADV, Kokiri, one run of about 45-55 s per setting. A flash is a frame where at least 20 of 144 screen blocks jump the same way:So the behind-only wait no longer helps, and I am not proposing it. The RADV hang fix from my earlier comment is already in main through #523.