perf(vk): with frame generation, the replay no longer waits behind each present, and a capture waits for less of the frame before #2633
Open
hckrman101 wants to merge 3 commits from
hckrman101/perf/framegen-present into main
pull from: hckrman101/perf/framegen-present
merge into: MagicBeanDev:main
MagicBeanDev:main
MagicBeanDev:github/pr/2893
MagicBeanDev:github/pr/2890
MagicBeanDev:github/pr/2887
MagicBeanDev:github/pr/2878
MagicBeanDev:github/pr/2875
MagicBeanDev:github/pr/2874
MagicBeanDev:github/pr/2873
MagicBeanDev:github/pr/2871
MagicBeanDev:github/pr/2756
MagicBeanDev:github/pr/2897
MagicBeanDev:github/pr/2896
MagicBeanDev:github/pr/2718
MagicBeanDev:github/pr/2842
MagicBeanDev:github/pr/2691
MagicBeanDev:github/pr/2895
MagicBeanDev:github/pr/2743
MagicBeanDev:github/pr/2765
MagicBeanDev:github/pr/2708
MagicBeanDev:github/pr/2732
MagicBeanDev:github/pr/2755
MagicBeanDev:github/pr/2817
MagicBeanDev:github/pr/2876
MagicBeanDev:release-please--branches--main
MagicBeanDev:release-please--branches--main--release-notes
MagicBeanDev:github/pr/2778
MagicBeanDev:github/pr/2888
MagicBeanDev:github/pr/2866
MagicBeanDev:github/pr/2877
MagicBeanDev:github/pr/2800
MagicBeanDev:github/pr/2829
MagicBeanDev:github/pr/2850
MagicBeanDev:github/pr/2889
MagicBeanDev:github/pr/2882
MagicBeanDev:github/pr/2860
MagicBeanDev:github/pr/2849
MagicBeanDev:github/pr/2886
MagicBeanDev:github/pr/2794
MagicBeanDev:github/pr/2840
MagicBeanDev:github/pr/2830
MagicBeanDev:github/pr/2834
MagicBeanDev:github/pr/2719
MagicBeanDev:github/pr/2742
MagicBeanDev:github/pr/2741
MagicBeanDev:github/pr/2820
MagicBeanDev:github/pr/2771
MagicBeanDev:github/pr/2859
MagicBeanDev:github/pr/2858
MagicBeanDev:github/pr/2823
MagicBeanDev:github/pr/2757
MagicBeanDev:github/pr/2769
MagicBeanDev:github/pr/2844
MagicBeanDev:github/pr/2843
MagicBeanDev:github/pr/2673
MagicBeanDev:github/pr/2855
MagicBeanDev:github/pr/2739
MagicBeanDev:github/pr/2807
MagicBeanDev:github/pr/2851
MagicBeanDev:github/pr/2789
MagicBeanDev:github/pr/2814
MagicBeanDev:github/pr/2818
MagicBeanDev:github/pr/2774
MagicBeanDev:github/pr/2772
MagicBeanDev:github/pr/2750
MagicBeanDev:github/pr/2786
MagicBeanDev:github/pr/2785
MagicBeanDev:github/pr/2782
MagicBeanDev:github/pr/2779
MagicBeanDev:github/pr/2775
MagicBeanDev:github/pr/2706
MagicBeanDev:github/pr/2766
MagicBeanDev:github/pr/2747
MagicBeanDev:github/pr/2759
MagicBeanDev:github/pr/2745
MagicBeanDev:github/pr/2740
MagicBeanDev:github/pr/2733
MagicBeanDev:github/pr/2725
MagicBeanDev:github/pr/2727
MagicBeanDev:github/pr/2728
MagicBeanDev:github/pr/2729
MagicBeanDev:github/pr/2721
MagicBeanDev:github/pr/2722
MagicBeanDev:github/pr/2715
MagicBeanDev:github/pr/2714
MagicBeanDev:github/pr/2707
MagicBeanDev:github/pr/2703
MagicBeanDev:github/pr/2705
MagicBeanDev:github/pr/2697
MagicBeanDev:github/pr/2679
MagicBeanDev:github/pr/2676
MagicBeanDev:github/pr/2669
MagicBeanDev:docs/what-hookshot-connects-to
MagicBeanDev:fix/hdr-encode-test
MagicBeanDev:fix/main-tests
MagicBeanDev:fix/play-cache-files
MagicBeanDev:perf/game-barrier-order
MagicBeanDev:feat/heap-lazy
MagicBeanDev:feat/texture-detail
MagicBeanDev:perf/astc-opaque-bc1
MagicBeanDev:fix/fs-dir-lock
MagicBeanDev:fix/framegen-native-kept
MagicBeanDev:fix/update-no-downgrade
MagicBeanDev:port/k10-gcn-f16-mirrors
MagicBeanDev:fix/framegen-mult-cap
MagicBeanDev:chore/release-as-2.1.1
MagicBeanDev:perf/pcache-load-switch
MagicBeanDev:fix/prepare-progress-log
MagicBeanDev:fix/update-closes-stale-game
MagicBeanDev:fix/hdr-scaled-sampled
MagicBeanDev:perf/first-start-preset
MagicBeanDev:perf/zero-upload
MagicBeanDev:fix/vcore-escape-2s
MagicBeanDev:test/fake-gh-atomic
MagicBeanDev:perf/batch-idle-flush
MagicBeanDev:test/finish-stub-a
MagicBeanDev:chore/record-2.1.0
MagicBeanDev:fix/main-red-tests
MagicBeanDev:fix/test-display-startup-driver-cache
MagicBeanDev:feat/mesh-shader-mode
MagicBeanDev:fix/menu-probes-at-startup
MagicBeanDev:fix/fps-counter-classic
MagicBeanDev:fix/test-log-batch-flake
MagicBeanDev:fix/narrow-screen-hud
MagicBeanDev:ci/prereleases
MagicBeanDev:release/v2.0.15
MagicBeanDev:hotfix/v2.0.15
MagicBeanDev:fix/present-ahead-replay
MagicBeanDev:release/v2.0.13
MagicBeanDev:fix/bracket-paths-2013
MagicBeanDev:fix/hdr-overlays-2013
MagicBeanDev:fix/fps-counter-scale-2013
MagicBeanDev:fix/stick-seen-at-rest
MagicBeanDev:hotfix/menu-fits-screen
MagicBeanDev:fix/f1-always-opens-menu
MagicBeanDev:fix/ini-keep-lines
MagicBeanDev:claude/cranky-pike-7cf247
MagicBeanDev:perf/texture-answer-flat
MagicBeanDev:fix/stamp-test-modules
MagicBeanDev:fix/appimage-fonts
MagicBeanDev:fix/display-awake
MagicBeanDev:release/v2.0.12
MagicBeanDev:fix/discord-windows
MagicBeanDev:rc/v2.0.12-tests
MagicBeanDev:fix/dlss-keep-feature
MagicBeanDev:rc/v2.0.12
MagicBeanDev:button-icons
No reviewers
Labels
Clear labels
accessibility
Barrier affecting people with disabilities
autorelease: pending
autorelease: tagged
bug
Something isn't working
dependencies
Pull requests that update a dependency file
documentation
Improvements or additions to documentation
duplicate
This issue or pull request already exists
enhancement
New feature or request
github_actions
Pull requests that update GitHub Actions code
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
question
Further information is requested
testing wanted
wontfix
This will not be worked on
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!2633
Loading…
Reference in a new issue
No description provided.
Delete branch "hckrman101/perf/framegen-present"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Draft, not yet run in the game. With frame generation on, the game's replay spends several milliseconds of every frame waiting on things that are not its own work: a lock the display thread holds while the driver waits inside each present, and a wait for the whole of the frame before's GPU work. This removes the first and narrows the second.
Measured on
mainwith DLSS, frame interpolation 4x and NVIDIA Reflex on, cap 120, Hyrule Field (RTX 3090, Ryzen 9 7900X, Linux, NVIDIA 620.70.06): the replay'ssubmitis 11.5 ms a frame against 6.8 ms with default options in the same place. 3.8 ms of the difference is the lock, 2.3 ms the capture's wait.1. A present on the replay's queue waits for its copy with the queue's lock let go
143b5e54vkQueuePresentKHR, on the CPU, until the frame's copy into the window is done on the GPU.dev.present_muthrough that wait. Every submit of the replay takes the same lock.submit_recording's wait for the lock at 17.2 % of all instants, while the display thread was insidepresent_out, in the driver's wait, at 34.5 %.present_outwaits for the copy's fence first, with the lock let go, and takes it again for the present itself. The present goes out when the driver would have let it: it waited for the same copy.RX_VK_PRESENT_COPY_WAIT=0/=1. On by default only with NVIDIA's driver on Linux, where the wait was seen. Off on Windows, where a note in the code says the same idea was tried and was worse, and off for other Linux drivers, which were not seen to wait for the copy in the present.2. A capture's late pages wait for the part that writes them back
d533cd18RX_VK_CAPTURE_WAIT=0has none.RX_VK_CAPTURE_PART=0restores the old wait.RX_VK_LOG_CAPTURE_LATE=1logs each late wait.What change 2 does and does not do (measured on the #2622 stack, which makes the game run ahead in the same way; default options, Hyrule Field and Kokiri Forest,
RX_VK_LOG_CAPTURE_LATE=1):Tests
tests/test_present_copy_wait.py: both callers givepresent_outtheir copy's fence and take it away; a failed submit gives none; the lock is let go for the wait and taken again before the present; the wait has a limit and sees a lost device; the default per platform and vendor; the presenter thread's hand-off is unchanged.tests/test_capture_part.py: the choice of a wait, cut from the source and run (a page whose part is done; the newest part; a page in place; a part not known; the switch), and the order incapture_makeandtake_capture.tests/test_present_semaphores.pyupdated for the changed function.tests/test_present_device_loss.pynow covers the new fence wait (ten sites).python -I -B tests/run_all.py): every failure it has is also a failure onmainitself at7297d93d, checked by running each one there:test_readback_part,test_dlss_checks,test_framegen_preset,test_hdr_encode,test_audio_biquad,test_audio_aux,test_mods_size_table,test_rumble_menu,test_mouse_native,test_linux_docker_packages,test_wayland_libdecor_hint,test_display_startup_failure,test_texel_buffer_ownership, and three known on this PC (test_bake_portability,test_builder_log_read_only,test_env_on).hookshot_vk.sobuilds onmainata016b7a6with both commits and passes the ABI check.Not yet tested in the game
Nothing here has been run with frame generation yet. For testers with frame generation on (any multiplier, DLSS-G or the renderer's own), NVIDIA on Linux first:
RX_VK_PRESENT_COPY_WAIT=0 RX_VK_CAPTURE_PART=0, same place, same settings.slow submitlines (how many, and how many saymemory sync).vk: presents on the replay's queue ...andvk: a capture taken (ms): ...(the late pages' numbers).RX_VK_PRESENT_COPY_WAIT=1there would say whether it helps.WIP: perf(vk): with frame generation, the replay no longer waits behind each present, and a capture waits for less of the frame beforeto perf(vk): with frame generation, the replay no longer waits behind each present, and a capture waits for less of the frame beforeTested in the game on NVIDIA Linux with frame generation, A/B against the same build with
RX_VK_PRESENT_COPY_WAIT=0 RX_VK_CAPTURE_PART=0.Setup: RTX 4070, NVIDIA 615.71.09, Ryzen 7 5800X3D, Ubuntu 26.04.1 (kernel 7.0), GNOME Wayland, 2560x1440 at 144 Hz. CI AppImage
v2.1.0-rc.3-237-g6be69c79. DLSS on, Reflex on, frame pacing on, V-Sync Auto.Log lines asked for: with the PR,
vk: presents on the replay's queue wait for their copy first, the queue's lock let go; with both switches at 0,...made with the queue's lock held. So change 1 engages on this machine.2x, DLSS Frame Gen making every frame, uncapped, 60 s standing still, same spot and facing. Run order: PR on, PR off, PR on.
The first run was about 10% slower in both places, and the PR-on repeat matched PR off, so I read that as warm-up. Standing still, the fps is the same with and without the PR. Change 2 took whole-submission waits to 0 and removed almost all late waits in Hyrule Field. In Kokiri the count and the time per wait stayed about the same, which matches your note about the exposure page.
Free play, 2x uncapped (about 19 min PR off, about 14 min PR on; Kokiri, Hyrule Field, load zones):
The routes weren't identical, but every measure leaned slightly toward the PR. The PR-on run's one
slow submit(387.5 ms, a texture upload, notmemory sync) and its two windows over 250 ms came during a pipeline warm-up at start-up that the PR-off run didn't have.Also seen: about one
a capture's wait ... write-back timed outper run, early in the run, with the PR on and off (on: 0 to 2 a run, off: 1), each with an 80 to 176 ms sync wait.No
slow submitlines in any standing run. Shown-frame spacing was 90 to 99% within 7 to 12 ms with and without the PR. No flashes or exposure steps I could tie to the PR in Kokiri or Hyrule Field, and input with Reflex felt the same throughout.4x, cap 144: the game held 36.0 fps with and without the PR, with no
slow submitlines. 4x doesn't measure the PR on this machine: DLSS-G's runtime here makes at most 1 frame, so the renderer's own generator makes them, but V-Sync Auto still picks FIFO, which holds the game at 144 / 4. Opened as #2695.Not tested: frame pacing off, V-Sync On, Windows, AMD/Intel.
Timing result with frame generation on: no gain in the first pair, so this stays a draft.
One pair, standing in Hyrule Field, RTX 3090 + Ryzen 9 7900X, Linux (NVIDIA, Wayland), 2560x1440 window, DLSS, frame generation 4x, Reflex on, cap 120. DLSS, 4x interpolation and Reflex were active in both logs:
Steam Deck data for part 2 (a capture's late pages wait for the part that writes them back). It probably doesn't change much on GPUs without mirrors.
Setup: Steam Deck OLED (Galileo, APU 0932, RADV, Game Mode, handheld profile, docked), main
6749ba21plus the Deck patches, Kokiri Forest standing scene, 30 s frame logs per start.note_direct_write(part 0, written in place). Yourlate_page()sends those to the whole-submission wait (whole = true), so this PR leaves the Deck's waits as they are: 177-195 of 600 takes wait, ~3.5-4.5 ms each.So on mirrorless iGPUs, the late-page wait isn't where the Deck's frame time goes. I won't open a PR for the deferral. Your part-based wait looks right for the mirrored case it targets.
Tested in the game on Linux + NVIDIA with frame interpolation on: no change in frame rate here.
Hardware: RTX 3060 12 GB, NVIDIA 615.71.09, Manjaro (kernel 6.12), Core Ultra 7 265KF (8 P-cores + 12 E-cores), Wayland. Built on main
51da0464: each PR merged onto it, onlyhookshot_vk.sobuilt (container build,RX_MARCH=x86-64-v3 RX_RELEASE=1) and swapped into the same game folder; main'shookshot_vk.sobuilt the same way as the baseline.Settings: 3440x1440 monitor, picture 2576x1080 (Aspect ratio: monitor's), frame interpolation 2x, Reflex on, V-Sync off, no frame rate cap; otherwise unchanged between runs.
Scene: Hyrule Field from the same save, standing, no input, 3.5 to 7 minutes a run; only the 5 s windows after loading (3,370-3,760 draws in every run) are counted. One run each, in the order main, #2633, #2644, #2559.
(replay columns: medians of the last 20
a frame's replay timelines;GPU-bound waitwas 0.0 ms in all runs)presents on the replay's queue wait for their copy first, the queue's lock let go.drawtime is within run-to-run noise: #2559, which changes only GPU barriers, shows the same drop. With 2x interpolation, the present lock does not seem to be what limits the frame on this machine; memory sync (~9.5 ms) is the largest part of the replay in every run.