feat(vk): RX_VK_PRESENT_PROBE and RX_VK_QUEUE_PRIORITY, switches to see where a present waits on the GPU #2182
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2182
Loading…
Reference in a new issue
No description provided.
Delete branch "Jiro113/Longshot:feat/fg-present-wait-diag"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
With frame generation (DLSS Frame Gen 2x at 60 fps, V-Sync on/FIFO, a 240 Hz VRR display) the frames reach the screen
unevenly: the game frame and the generated frame do not share the 16.7 ms between two game frames evenly. To see where
the game frame's present waits on the GPU, this adds three diagnosis switches, all off unless set:
RX_VK_PRESENT_PROBE=<file>: one CSV line a probed present (its header names the columns). Timestamps are submittedalone on the present's queue right before and right after
vkQueuePresentKHR, and one at the end of its image's copy,with the CPU clock (QueryPerformanceCounter ticks on Windows) at the submits, the call, the return and each probe's
fence, and whether the copy's fence was signalled at the call and at the return. A thread of the probe's own waits for
the fences and writes the file.
present_commitmakes are probed: a fixed multiplier's sets presented ahead(
present_ahead_wanted:pace && shown >= 2 && swap_spare >= shown), the game frame included. In the 2x run ofthis PR's head, 12,256 probe lines over 105 s against 12,259 presents in
RX_VK_FG_LATENCY_LOG's file over the sameclock range, half with the game frame's Reflex id. What
present_imagehands to the presenter thread (on every PCsince #1763: no frame generation, or sets not presented
ahead) is not probed.
call/returnbracket the CPU call,copy_done_at_*are booleans, andbefore_seen/after_seenare when the probe's thread saw its own fences. The present's waits are inframes.csv(
slot_wait_ms,gpu_behind_ms,shown_ms).RX_VK_QUEUE_PRIORITY=low|medium|high|realtime: the global priority asked for the replay's queue(
VK_KHR_global_priority, elseVK_EXT_global_priority), where the queue's family lists that level. WithVK_EXT_global_priorityalone (neitherVK_KHR_global_prioritynorVK_EXT_global_priority_query) the levels cannotbe listed, and the level is asked unlisted. A device refused with it is made again without it, and the log says what
was asked and what the device was made with.
RX_VK_PRESENT_QUEUE_PRIORITY: the same for the presents' own queue (RX_VK_PRESENT_QUEUE=1).Unset, the device is made as on
main(nopNext, no extension) and no present is probed. Set, the probe adds twosubmits and two fences a probed present, and
RX_VK_QUEUE_PRIORITYchanges how the GPU schedules the replay's queueagainst other work: that is what it is for.
These are what found the wait the follow-up pull request #2183 fixes (the replay's queue at global priority low
with a fixed frame generation multiplier on NVIDIA's Windows driver). That pull request carries these two commits too,
so the one merged first brings them.
Read first:
gpu/vk/vk_replay.cpp(PresentProbe,probe_present,probe_copy_stamp) andgpu/vk/vk_main.cpp(
create_device: the levels asked, the device made again without them).Testing
Tested: Windows 11, RTX 4090, NVIDIA 621.24, a 240 Hz VRR display; DLSS Frame Gen 2x, the game capped at 60 fps,
V-Sync on, full screen, Hyrule Field with the camera panning, PresentMon recording. Six runs with the switches unset and two with
RX_VK_QUEUE_PRIORITY=lowandRX_VK_PRESENT_PROBEset, against six ofmain(Measurements). WithRX_VK_PRESENT_PROBEset the file had one line a present; withRX_VK_QUEUE_PRIORITY=lowthe log saidthe device was made with the queues' global priority asked (VK_KHR_global_priority; the replay's low, the presents' own unasked). No device was lost. This PR's head, onmaina2dc53859, ran again 30 s in a windowwith both switches set, in Hyrule Field: the same line, frame interpolation ready at 2x, the probe file written (5467
lines), no device lost.
mise run test(tests/run_all.py --strict) on this PC, Windows with llvm-mingw's clang++: 37 checks fail onmaina2dc53859here (listed below), and the same 37 fail here, on the same lines, and no other (test_sampler_admission'sconcurrent cases fail a varying number of times from run to run on
main: 2 of 147 in its full run, 3 in this PR's). The four checks this PR changes,test_present_acquire_ahead,test_present_device_loss(with and without the probe),test_present_semaphoresandtest_present_thread, passwith clang++ and with g++ (11.4, Linux). The renderer builds on Linux (
RX_VK_ONLY=1).The 37 checks that fail on
maina2dc53859on this PCnx2/test_import_windows,test_auto_handheld,test_bug_report,test_close_waits_for_save,test_compile_zero_byte,test_crash_dialog,test_damaged_rom_files,test_dev_ab,test_dev_build,test_dev_migrate,test_dev_packs,test_draw_stream_visibility,test_fs_save_commit,test_fsr3fi_hud_layer,test_fsr3fi_hudless_colour,test_fsr_uniform_ownership,test_game_menu_mouse_buttons,test_guest_copies,test_hold_toggle,test_input_movie,test_linux_preflight,test_lower_atomg_cas,test_menu_i18n,test_menu_keys,test_menu_slider_ends,test_pads_after_menu,test_perf_report,test_portable_builder_home,test_portable_builder_startup,test_recording_admission,test_sampler_admission,test_sdiv_overflow,test_sim_turf_pools,test_switch_table,test_tls_objects,test_vibration_menu,test_work_lockStill to test:
RX_VK_QUEUE_PRIORITY=lowandRX_VK_PRESENT_PROBE=probe.csvwith framegeneration at a fixed 2x, expecting the log's line of the level asked and made, and a probe file of one line a present
where the sets are presented ahead (that needs the surface to give
minImageCount + 2images or more, and the pacingon: still to confirm on those surfaces) (Test Farm)
takes either):
RX_VK_QUEUE_PRIORITY=realtime, expecting the device made again without it and the log saying so(by hand)
Measurements
Verdict: unset, no change beyond the runs' spread. Over six runs each, 8.4 % of the intervals on screen were within
0.5 ms of 8.33 ms (6.0 to 10.6) against 10.0 % on
main(7.3 to 12.3). The runs of both fall in two groups, theintervals' standard deviation near 2.3 ms (9.3 to 12.3 % within 0.5 ms) or near 3.0 ms (6.0 to 7.3 %):
mainhad onerun of six in the second group, this PR three, which is where the means differ. Set,
RX_VK_QUEUE_PRIORITY=low(withRX_VK_PRESENT_PROBE) took it to 26.4 % (26.0 and 26.8, two runs): what #2183 builds on.Setup: Windows 11, RTX 4090, NVIDIA 621.24, 240 Hz VRR display; DLSS Frame Gen 2x, 60 fps cap, FIFO, full screen,
Hyrule Field with the camera panning for 60 s after a teleport, PresentMon over the pan (the frames' intervals on
screen: the share within 0.5 ms of 8.33 ms, and their standard deviation); every run with the same logging switches on
(
HOS_PRESENT_TRACE,RX_VK_FG_LATENCY_LOG,RX_VK_REFLEX_LOG),RX_VK_PRESENT_QUEUE=0. Runs interleaved with theother heads of this work, after one warm-up run each; the mean of each head's runs, (lowest-highest). One more run of
mainwas left out: PresentMon saw 5309 of its 13240 frames composed by the desktop (Composed: Flip), not flippedstraight to the display.
8aa749834The frame rate with the switches unset, by
dev.py ab(this PR's pack andmain's on the same base, three interleavedpairs, 60 s of the frame trace each, in a window, uncapped with
HOS_FPS_MAX=1000, frame generation and V-Sync off,Render resolution 67 %): no difference shown, 136.9 against 133.8 fps (the median of three runs;
main128.7 to 137.4,this PR 133.0 to 137.9), within the runs' spread. The runs' logs list none of the three switches. (
dev.py abrefusedits write-up because the control bridge's pipe name,
HOS_CONTROL, differs from run to run; the write-up was madeagain from its results with that one left out of the comparison.)
8aa74983Measured on
main8aa749834with this PR's two commits on it (eb135e974). This PR is now onmaina2dc53859, itscommits' changes the same (checked with
git range-diff) but intest_present_device_loss, which now takesmain'slist of the present routine's calls and adds the probe's to it;
main's commits since change neither the presents'timing nor the queues; in the renderer, one keeps the HUD's colours with FSR 3.1 frame generation in Native HDR (DLSS
here), one starts the replay's threads through a guard that reports their C++ exceptions, one logs fewer memory
imports, #2171 moves two includes, spells
rx_nvn.h's static assertionsso g++ builds them and changes menu texts, and #2174 keeps the simulated-turf lists at the game's own size in areas without turf, so that Hyrule
Field, the scene measured, no longer stalls 250 ms at a time on the way in: that changes frame times there, and both
sides above ran without it. #1722 adds a save.
Risks
Nothing changes unless a switch is set. The probe sees only the sets presented ahead (Summary): a run without frame
generation, or whose sets are not presented ahead, makes no file, and its log has no
vk: present probe:line (theprobe is made at the first present it sees). Set,
RX_VK_QUEUE_PRIORITY(orRX_VK_PRESENT_QUEUE_PRIORITY) at low makes thereplay give the GPU up to other programs, and a device refused with it costs one more
vkCreateDevice. The probe addstwo submits and two fences a present, and a thread writing the file. Nothing in
longshot_options.ini, the saves orthe caches changes.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.No screenshots: nothing on screen changes with the switches unset. No docs: diagnosis switches are documented by their
rows in
gpu/vk/switches.def.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Tested in game: Windows 11, RTX 4090, NVIDIA 621.24, a 240 Hz VRR display, full screen; DLSS Frame Gen 2x at a 60 fps cap, V-Sync on, Hyrule Field with the camera panning, PresentMon recording. Switches unset, six runs against six of main: the intervals on screen as on main within the runs' spread. With RX_VK_QUEUE_PRIORITY=low and RX_VK_PRESENT_PROBE set (two runs): the log said the device was made with the replay's queue at global priority low, the probe file had one line a present. No device was lost. The final head, on main
a2dc53859, ran again 30 s in a window with both switches set: the same log line, frame interpolation ready at 2x, the probe file written, no device lost.Possibly related, for context:
WIP: feat(vk): RX_VK_PRESENT_PROBE and RX_VK_QUEUE_PRIORITY, switches to see where a present waits on the GPUto feat(vk): RX_VK_PRESENT_PROBE and RX_VK_QUEUE_PRIORITY, switches to see where a present waits on the GPUThe probe does not see the game frame's present on current main — read from the source, not run here.
present_outreturns before the probe path whenever the copy is handed to the presenter thread:gpu/vk/vk_replay.cpp:6924const bool hand_off = present_copy_done && present_thread_on();:6931if (hand_off) { … return true; }— it returns before:6964probe_ready(...)and:6968/:6970probe_present.present_image_with_trialalways setspresent_copy_done = probe.copy_fence = f.done(:6830) before callingpresent_out, so every normal game-frame present takes that return.present_thread_on()(:7168) returns true on every PC:RX_VK_PRESENT_THREADisGONE(gpu/vk/switches.def:797, since 11-19).So
probe_presentis reached only frompresent_commit's direct presents, wherepresent_copy_doneis unset — the frame-generation paced presents, not the game frame's. The 5,467 lines in the Windows run are that set. The description says "the presenter thread's presents … are not probed", but does not say that on main the presenter thread owns the game frame's present. As written the probe cannot answer the question the PR asks (where the game frame's present waits). Could you wrap the presenter'svkQueuePresentKHR/copy-fence wait/present-wait, or carry the probe timestamps through the hand-off? And add a column that says which frame set a line belongs to:reflex_framealone does not say game or generated.Columns against
frames.csv(the #1988 reads): no duplicate names, butcall/returnare host-clock brackets of the CPU call,copy_done_at_call/returnare 0/1 booleans, andbefore_seen/after_seenare the probe's own fence observations — none is a wait duration. There is no acquire column and no present-shown column;frames.csv'sshown_ms/shown_at_nsis the present-wait confirmation, andslot_wait_ms/gpu_behind_msalready name those waits. A reader joining the two files could takeafter_seen - callfor the present's GPU wait; the header or description should say the probe is not a wait-duration source, or the copy-fence/present-wait durations should be added.Tests:
test_present_threadandtest_present_acquire_aheadfail on main because they require theprobe.copy_fencestring;test_present_device_lossandtest_present_semaphoresonly gain stubs/fields and pass on main.test_switch_tablepasses on main (no new names) and on the PR, so nothing fails on main forRX_VK_QUEUE_PRIORITY/RX_VK_PRESENT_QUEUE_PRIORITY; no test covers the level parse, the family gate or the device retry. A red-first check for one switch read and one gate would be cheap.Minor, not the blocker: the probe setup returns
falsewith no log whenvkAllocateCommandBuffers/vkCreateFencefails (:6241,:6248), so a driver that refuses them silently leaves the probe off. And with onlyVK_EXT_global_priority(no_query),can_listis false and the code assumes the level is listed (in_list = true,vk_main.cpp:596) and asks anyway, so "only where the family lists that level" is not true on that driver; the device refusal is logged and retried, so that is a wording/gate gap rather than silence.No fix claim in the title or description: it is diagnostic, and #2183 is named as the fix. The one wording to change is "With them set, nothing else the game does changes" —
RX_VK_QUEUE_PRIORITYchanges queue scheduling by design.@dazemc thanks, checked against the source and against the runs' files. You are right about the code path, and the description was wrong about it.
What the probe sees. Since #1763,
present_thread_on()is always true. A present frompresent_imagecarriespresent_copy_done. It goes to the presenter thread, andpresent_outreturns beforeprobe_ready. Onlypresent_commit's presents stay inpresent_outand are probed.With a fixed multiplier whose sets are presented ahead (
present_ahead_wanted:pace && shown >= 2 && swap_spare >= shown),present_commitpresents the whole set, the game frame included. So in the case this PR was measured on, every present is probed. Counts from this PR's head, DLSS Frame Gen 2x:RX_VK_FG_LATENCY_LOG's file over the same clock range;FSR 3.1 and the built-in frame generation, in mailbox, give the same: 8,420 to 14,041 lines, 3 to 10 short of the presents.
Without frame generation, or when the sets are not presented ahead, nothing is probed. The description now says so, in place of the line about
RX_VK_PRESENT_THREAD(a switch gone since #1763). The base the PR was measured on,main8aa749834, already has #1763, so the measured runs behaved the same. The Measurements' numbers come from PresentMon, not from the probe.Columns. Agreed: none is a wait duration.
call/returnbracket the CPU call.copy_done_at_*are booleans.before_seen/after_seenare when the probe's thread saw its own fences.The description now says the probe is not a source of wait durations, and points to
frames.csv'sslot_wait_ms,gpu_behind_msandshown_msfor those.Tests. Your reading matches mine:
test_present_threadandtest_present_acquire_aheadfail onmain.RX_VK_QUEUE_PRIORITY's parse, the family gate or the device made again.Minor points. Both are true.
VK_KHR_global_priorityandVK_EXT_global_priority_query, so the runs here never took thecan_listfalse path.VK_EXT_global_priorityalone the level is asked unlisted, and that a refusal makes the device again and is logged.RX_VK_QUEUE_PRIORITYchanges the GPU's scheduling, which is its point.Next, on this PR:
reflex_framehappens to be the game frame's id or 0 in these runs, but that is not a contract;mainfor the level's read and the family gate.Probing the presenter thread (
presenter_loop, or carrying the stamps through the hand-off) would cover the path without frame generation. That path is the one #1518 needs. This PR's question is the frame generation sets, and they are all probed, so I would rather make that a change of its own. Say if you think it belongs here.Description corrected after @dazemc's review (answered in the comment above):
RX_VK_PRESENT_PROBE: which presents are probed (onlypresent_commit's, a fixed multiplier's sets presented ahead, the game frame included, with the counts that show it), in place of the line aboutRX_VK_PRESENT_THREAD, a switch gone since #1763. A second point says the columns are not wait durations, and where those are.RX_VK_QUEUE_PRIORITY: withVK_EXT_global_priorityalone the level is asked unlisted.RX_VK_QUEUE_PRIORITYchanges the GPU's scheduling, which is its point.No code changed. The code changes listed in the answer come as their own commits, with a note here.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.