vk: Add an opt-in recorder thread for the serial replay #1473
No reviewers
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!1473
Loading…
Reference in a new issue
No description provided.
Delete branch "vk-recorder-main"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this fixes
CPU-bound Hyrule leaves GPU headroom while Vulkan calls on the presentation thread limit frame rate.
Why merge
RX_VK_RECORDER=thread; default and synchronous path unchanged.Change
gpu/vk/vk_recorder.h: queue, copies, drains.gpu/vk/vk_work.h: route calls.gpu/vk/vk_replay.cpp: lock release, pinning.gpu/vk/vk_gprof.cpp: wrap queries.gpu/vk/vk_menu.cpp: drain before ImGui.gpu/vk/vk_reflex.cpp: wrap calls.gpu/vk/vk_vma.cpp: wrap lifecycle calls.gpu/vk/vk_dlss.cpp: drain before NGX.gpu/vk/vk_fsr4.cpp: drain before SDK dispatch.tests/recorder_sync_test.cpp: ordering checks.tests/recorder_exit_test.cpp: exit regression.tests/test_recorder_sync.py: census, negative controls, compiler declaration.tests/test_fps_overlay_order.py: ImGui ordering.Tested
python -I -B tests/test_recorder_sync.py --cxx clang++ --headers _deps/vulkan-headers/include: routing, ordering, lifetime and opaque-call checks. Fails on the old code, passes now.HOS_FPS_MAX=240, clock≈1.0: Hyrule 54.2→62.6 fps (6 pairs); Kokiri 82.2→80.8 (4). CPU cost +14%/+8%; both arms still had hitches.Not covered
02251. No full SyncVal, pixel equality, save/reload or soak.Went through this with the threading in mind and didn't find a real bug. The invariant holds up: the recorder only runs while some thread holds
replay_mu, only for that thread, andDrainingMutex::unlockwaits for it, so any thread that takes the lock sees an idle recorder. Command pools are all made at start-up (replay_start/init_pools_make), so there's no allocate-vs-record race on a pool. Every recording that's guarded by another mutex (readback_begin/readback_submit_wait,present_image) ends and submits through a boundary before it lets go. Descriptor alloc/update and pipeline creation on the producer don't need to be synchronized with bind calls on the recorder. Destroys, resets, waits, submits andvkDeviceWaitIdleare all boundaries, and the fourunordered()exceptions look right to me.Off path: macros go through
Route::call, which is one relaxed load and a branch. The three files that now includevk_work.h(gprof, reflex, vma) only make state, timestamp and boundary calls, sonote_work()/full_barrierbehave the same. The option is env-only (RX_VK_RECORDER*), so there's no settings id to collide with. The test is registered intests/run_all.py, and CI is green, including both Windows Vulkan builds.A few small things, none blocking:
cmdthrough its own function pointers needs aB_DIRECTbefore it. The source check only catches NGX andui::record*, plus any opaque call followed by a hand-writtennote_work(). #1008 (FSR 4.1 SDK) and #1590 (DLSS 5) will need that when they land after this.Recorder::start()zeroesWthenR. Aforeign_wait()that read the oldWjust before can then spin until the new ring has run that many bytes. It's only reachable after avkDestroyDevicestop plus a restart, so it's theoretical. Keeping the counters monotonic across restarts would make it go away.stubs/agent-recorder2/RESULT.md, which isn't in the repo (inline).@ -0,0 +1,1133 @@// (hookshot) RX_VK_RECORDER: the command buffer calls as packets, step 1 of a recorder thread (the "sync" phase).Nit:
stubs/agent-recorder2/RESULT.mdisn't in the tree, and theRESULT.mdmentions on the two cases below point at it too. Maybe inline the one-line reasons (they're already there) and drop the reference.Nit: a
foreign_wait()that loaded the oldWjust before this, and then readsR == 0, spins until the new ring has run that many bytes. It's only reachable on a restart aftervkDestroyDevice, so it's very unlikely. Continuing from the oldW/R(offsets are& maskanyway) instead of zeroing would avoid it.Heads-up: this conflicts with main now (a lot landed today). Could you merge main into the branch (or rebase) and resolve the conflicts? Thanks!
Merged current main (
d224dc0) in, and addressed the points:9e89988):tests/run_all.pytakes main's version;test_recorder_sync.pynow declares its ownRUN_ALL_ARGS.vk_dlss.cpptakes main's version, with the twoB_DIRECTboundaries put back before NGX's create and evaluate.7515e76): FSR 4 is merged now, and the census caught itsffxDispatch:note_work()after an unknown recording. It gets aB_DIRECTbefore the dispatch. The test registers it (FFX_RECORD_CALL) and checks it, with a control that removes the boundary. #1590 will need the same once it lands; the census will stop it until then.Recorder::start()zeroingW/R(8edc411): the counters now continue fromW, withRset to it, so aforeign_wait()that read the oldWseesRreach it at once. No test exercises a restart aftervkDestroyDevice; the change is checked by reading every reader ofR/W/w/w_pub/r_seen.RESULT.mdreferences (8edc411): removed. The one-line reasons beside eachunordered()case stay.Locally:
tests/run_all.pywith Vulkan headers passes, includingtest_recorder_sync.py(262 s),test_ui_record.pyandtest_fps_overlay_order.py. Four tests fail here on main as well (test_nup_out_of_space,test_nup_paths,test_fpx_sticky,test_precompile_compute_guard): environment, not this branch. An independent review of the three commits found no defects.@ -0,0 +1,1133 @@// (hookshot) RX_VK_RECORDER: the command buffer calls as packets, step 1 of a recorder thread (the "sync" phase).Removed in
8edc411: the doc comment and the four case comments no longer point at RESULT.md; the one-line reason beside each case stays.🤖 Addressed by Claude Code
@ -0,0 +1,1133 @@// (hookshot) RX_VK_RECORDER: the command buffer calls as packets, step 1 of a recorder thread (the "sync" phase).Done in
8edc411: start() keeps the counters going: R is set to the current W and w/w_pub/r_seen follow, nothing is zeroed, so a foreign_wait() that read the old W sees R reach it at once. Anything a stopped recorder left unrun is dropped, as the fresh ring did.🤖 Addressed by Claude Code
CI on
8edc411failed in two places; both are fixed inbe325f7:test_recorder_sync.py: the status-read check was timing-dependent. On a loaded runner the recorder sometimes ran the few packets in front of a read before the read happened. The fake driver now holds the recorder for 50 ms around those reads. Four parallel runs gave every read with packets in flight, and the broken variants are still caught.nx2/test_draw_parameters.py: this fails on main as well. The block it compiles now readsctx.mesh_fallback(d1d6caa), which its stand-in context lacked. It's a one-line fixture fix in its own commit; it fails before the fix and passes after (64 combinations, with spirv-val). Glad to drop it if you'd rather fix it on main directly.Earlier API12 benchmark and native-check details (preserved from the long description; these are not new-head API13 hardware results):
python -I -B tests/test_recorder_sync.py --cxx clang++ --headers _deps/vulkan-headers/include: poisoned arguments, ordering, exit, census. Fails without the recorder, passes with clang and MinGW g++; 23 broken variants caught.RX_VK_RECORDER=thread, same binaries (main88b554ehost with #1696, this PR's renderer),HOS_FPS_MAX=240with the game clock checked byHOS_FPS_LOG=1. RX 9070 XT, driver 32.0.32015.12012, Ryzen 7 5700X3D, 32:9; alternating pairs, 20 s windows, medians:Thanks for the merge and the follow-ups. I re-checked at
32bea7e.Merge with main: nothing of main's is lost. Against main, the diff is only the recorder change.
vk_dlss.cppkeeps main's version plus the two boundaries, the draw-parameters fixture commit dropped out once main had its own fix, and GitHub reports the branch mergeable.Direct SDK recordings: I grepped the vk/ sources at head for anything that records into
cmdoutside vk_work.h. That turns up NGX create/evaluate (vk_dlss.cpp), FSR 4'sffxDispatch(vk_fsr4.cpp) and the threeui::record*calls invk_menu.cpp. Each of them now hasboundary(B_DIRECT)as the statement right before it, and the source check tests each one with a control. FSR 4's other loader calls (query/create) don't take a command buffer. FSR 3.1 upscaling and frame generation are our own dispatches through vk_work.h, so the census already covers them. Anti-Lag and Reflex don't record. The newnote_work()registry is a good catch-all for the next SDK (#1590).present_image (#1515/#1720): it still records only through vk_work.h or the menu's B_DIRECT. Its end, submit and present are boundary macros, so the earlier reasoning still holds. A producer drains before it submits, and the frame-generation thread (no
replay_mu) records directly.Tests:
test_recorder_sync.pydeclaresRUN_ALL_ARGSwith{cxx}/{vulkan_headers}, which the current run_all.py resolves.test_fps_overlay_order.pyexpects the newdstep. The Windows timing fix for the status-read check looks right.The two nits: both are fixed.
start()now continues fromW(R = w = w_pub = r_seen = W), so aforeign_wait()that read the oldWno longer spins. TheRESULT.mdreferences are gone.CI is all green on this head. Main's own latest completed source-check run is red, so this is ahead of it.
One small thing, not blocking: as the description says, the merged API13 code hasn't been run in the game yet. A short
RX_VK_RECORDER=threadrun (Hyrule plus opening the menu) before merge would be nice, and an FSR 4 one if you have the hardware. It's opt-in, so the default path doesn't depend on it.LGTM.
A portability finding from clean main
8a8ac27:test_recorder_sync.pyfails on Windows 11 build 26300 with llvm-mingw clang 23.1.5 (x86_64-w64-windows-gnu). The healthy recording/mode/stress checks pass, but the old-destructor negative control also exits 0 and prints the sentinel, so the runner reportsFAIL: the exit test passes the old destructor.This reproduced in the full suite and an isolated run (33.2 s). The isolated command was
python -I -X utf8 -B tests/test_recorder_sync.py --cxx clang++ --headers <Vulkan-Headers/include>, with the llvm-mingw bin directory on PATH. Both current and broken fixtures printrecorder parked in idle() holding sleep_mu: 1; ExitProcessandDLL detach: the global destructors ran to the end.I have not changed or skipped the assertion. This shows that the negative control is not discriminating on this toolchain; it does not establish a native gameplay shutdown failure. Is this exit control intended to require static CRT linkage? The fixture currently builds without
-static.Thanks @Arkex-ai, reproduced with llvm-mingw 20261104. It isn't about static linking:
-staticchanged nothing. The sentinel has to be destroyed after the recorder's global, butg_recis an inline variable, so its initialization order against the fixture's statics isn't fixed. With llvm-mingw it was made first, and the sentinel printed before the old destructor ended the process. Fixed in #2240 withinit_priorityon the sentinel. Checked with llvm-mingw, MSYS2 clang++ and g++, and MSVC-target clang: the current code passes and the old destructor is caught on each.