WIP: perf: Hookshot's new defaults and speed-ups merged after Longshot branched, restored #288
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!288
Loading…
Reference in a new issue
No description provided.
Delete branch "TrashPanda/Longshot:restore/defaults-after-rc4"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Hookshot merged new defaults and speed-ups after Longshot branched: texture memory (BC1 for opaque ASTC, a Texture
detail row, 32 MB image blocks, the 7/8 trim), barrier-ordered passes on NVIDIA, staging and view caches, host hot-path
costs, ReplayPin on Linux, pipeline lists and Auto run. They change defaults for players, so they wait for after rc.4,
or for Longshot numbers. Split out of #209.
What changed
Split out of #209 as its reviewer asked (review on #209, point 4, group e). Each commit is the merged Hookshot pull request under its original author, carried through Longshot's rename; a
Restored-from:trailer names it, and the bare Hookshot numbers in the messages and the restored text are now full URLs.Credit to the authors of the restored work: Dewey Harry D. bartolo, getzabelz, NotRexed, Mooki, David Sandner, FIocker, Maksim Klimovich, Erfboom, Juliann, jarynclouatre, yassine milal, klikso, RufflezAU.
large excess cut to 7/8 again; render targets covered by newer ones let go; no pool import for one unswizzle; pool
import without a heap walk; udmabuf fallback; staging tail suballocation
(https://github.com/NotRexed/Hookshot/pull/2559). This is a new default for every NVIDIA player with no Longshot
measurement yet: dispatches back to back get no barrier of their own, so a hazard the game's barriers miss would
show as flicker or wrong pixels. It needs a measurement on Longshot (and a look for artefacts) before it merges;
RX_VK_GAME_ORDER=0turns it offprewarm half of that PR is left out: the four 64 MB arenas it made in
replay_startwent throughtexture_frame'skeep and idle rules, so on a card with 6.5 GB or less three were freed at the first frame and all of them 30 s later,
for 256 MB of host-visible memory at start. Keeping them needs an exemption on
Arenaand a check of its own; thatcan follow separately
HOS_OPT_AUTO_RUN): upstream it was 55, which Longshot's Damage taken row (#138)now has. An option number, not a change to
MkwVkHostorMkwVkRendererSplit siblings: tools, checks and docs #284, test-request pinning #285, rc.4 player fixes #286, renderer fixes #287.
Commits, in order:
4ec9a872build(release): a shipped pipeline list from a player's progress on Linux AMD, 1364 pipelines (Hookshot 3029, by Dewey Harry D. bartolo)8dadb975build(release): a shipped pipeline list from an overnight tour of 21 areas, 3388 pipelines (Hookshot 3030, by Dewey Harry D. bartolo)bba74c19feat(input): Auto run, the sprint goes on after A is let go (Hookshot 2747, by getzabelz)ea98c37afix(vk): one pipeline play file a session, and the next start puts them together (Hookshot 2625, by NotRexed)794f067bfix(vk): report GPU resource waits beyond recording-slot reuse (Hookshot 2897, by Mooki)53d4ed32fix(linux): enable ReplayPin on the reserved core and reset child thread affinity (Hookshot 2903, by David Sandner)5404f36ffix(shaders): the noperspective workaround baked offline too; RX_VK_LOWER_SKIP in the program stamp (Hookshot 2782, by FIocker)21ae09b1perf(vk): opaque ASTC textures kept as BC1, half the video memory of BC3 (Hookshot 2554, by NotRexed)e4d17e98feat(menu): a Texture detail row (Auto, Full, Medium, Low) for cards short of video memory (Hookshot 2557, by NotRexed)01608e79perf(vk): back-to-back passes and dispatches ordered by the game's barriers (NVIDIA by default) (Hookshot 2559, by NotRexed)ccfb04b6fix(vk): report native shader keys in draw statistics (Hookshot 2896, by Maksim Klimovich)43b799c7perf(vk): suballocate untouched tails of in-flight staging arenas (Hookshot 2469, by Erfboom)cffd08e2perf(vk): cache mesh shader picks in thread-local program caches (Hookshot 2476, by Erfboom)ea6dc5f5fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation (Hookshot 2482, by Erfboom)dcd63673perf(host): three per-call costs on hot paths removed (own-thread lookups, a Signal with no waiter, switches read once) (Hookshot 2708, by Maksim Klimovich)6ca3c01cfix(vk): let render targets go as newer ones together cover them, and reuse their images at once (Hookshot 2766, by Juliann)4fda7ba9perf(vk): no pool import for one GPU unswizzle, and the CPU unswizzle threaded from 4 MB (Hookshot 2785, by FIocker)117992f8perf(vk): a pool import does not pay a walk over the game's heap (Hookshot 3043, by jarynclouatre)98edff67fix(vk): image pool blocks of 32 MB by default, not 128 (Hookshot 2967, by yassine milal)df66d827fix(vk): a large excess over the 90% rule or the driver's limit is cut to 7/8 again, only a small one is trimmed (Hookshot 3020, by klikso)b3e80733fix(vk): answer repeated texture views from a front cache (Hookshot 2960, by RufflezAU)2fb964eafix: write 8/16-bit global stores directly instead of a compare-exchange loop (Hookshot 2976, by yassine milal)f9f3eedatest: ViewFast stand-ins in the checks that cut draw_begin_recording (mine)b543f0b4docs: the restored Hookshot references as full URLs (mine)9fe72e19docs(development): the new renderer and host switches of the restored speed-ups (mine)5ce3b68atest(vk): the occlusion wiring check names StatsQuery's arguments as the draw statistics now pass them (mine)Flags and settings
RX_VK_ASTC_BC1=0RX_VK_GAME_ORDER=1/=0RX_VK_IMAGE_BLOCK_MB=<n>RX_VK_VIEW_FAST=0RX_VK_NATIVE_NARROW=0RX_VK_IMPORT_FENCE=0RX_VK_UNSWIZZLE_IMPORT=1RX_VK_LOWER_SKIP=<key prefix,...>HOS_CV_SIGNAL_ALWAYS=1Measurements
No Longshot numbers yet: each change was measured upstream on Hookshot's tree (its commit message and tested-configuration
rows). These are new defaults, so this PR needs Longshot numbers before it goes ready; until then the game-free checks and
builds below, main and this PR on the same PC.
main:
d9648863(2026-11-13)this PR: restore/defaults-after-rc4 @ 6c477f22 (the runs; now
5ce3b68a, after the docs-only rebase onto main's514591a3)Hardware: RTX 4070 Ti (driver 595.99.02), Ryzen 7 9800X3D, Ubuntu 26.04.1
Scene: none (no game run)
Method:
python3 -I -B tests/run_all.py --jobs 6, once on each; the builds belowHow to reproduce: the same command on each commit
d9648863run_all.py: passed / failed / skippedThe numbers still hold on
514591a3: main's #274 changed a Discord link in 11 docs files and nothing these checks or builds compile; the checks that read those docs were run again on this head.Testing
Tested:
python3 -I -B tests/run_all.py --jobs 6on Linux (Ubuntu 26.04.1, Python 3.14, Vulkan and SPIR-V headers found): this PR @6c477f22: 523 passed, 2 failed, 18 skipped (test_lift_cache.py, test_menu_i18n.py); main @d9648863on the same PC: 513 passed, 2 failed, 18 skipped (test_lift_cache.py, test_menu_i18n.py).test_lift_cachefails on main the same way (the lift's scripts do not import here), andtest_menu_i18nfails on that main too. Main then gained514591a3(#274, a Discord link in 11 docs files only); after the rebase onto it, the checks that read those docs passed again at5ce3b68a.Built at
6c477f22with no game, on Linux (system clang 21, lld) and for Windows (cross-built here with tools.lock's llvm-mingw 20261104 and its windows-x64 CPM cache, which has FFmpeg and dav1d): (before the docs-only rebase onto #274)nx2/build.py --standalonewith an empty lift compiled everything in the repository that the game links, the host, its libraries and the Vulkan back end (406 files on Linux, 405 for Windows), with no failure, and its link stopped only at the symbols the game's lifted code provides (18 on Linux, 19 on Windows: the same list as main @d9648863).RX_VK_ONLY=1 RX_RELEASE=1linkedlongshot_vk.soandlongshot_vk.dll, andbuilder/setup/setup.clinked asLongshotBuilder-Setup.exewith build-windows.yml's command. The five split branches merged together locally (the API version taken as 23) compiled and linked the same way, so they also build together.Not tested: a run in the game, and any measurement on Longshot: the game build needs the game's lifted code.
Needs other testers:
RX_VK_GAME_ORDER=1against=0, twointerleaved runs each: GPU time and fps, and anything drawn wrong (T-Y5MJ, claimable once a maintainer adds the
testing wantedlabel)Screenshots
None: no run in the game yet (the Texture detail and Auto run rows show on screen).
Risk and rollback
Every player's defaults change (NVIDIA's barriers, texture formats and memory). Each has a switch above to turn it off,
and each commit reverts on its own.
Reviewer notes
Please merge with rebase, not squash, so that each author keeps their commit. One commit of mine adds
ViewFaststand-ins in the three checks that cut
draw_begin_recording. Two conflicts with main were resolved keeping main'scode: in
build_descriptorsthe front cache's timers (https://github.com/NotRexed/Hookshot/pull/2960) sit inside#107's handle-table branch, and the draw statistics'
StatsQuery(cmd, p, d)(https://github.com/NotRexed/Hookshot/pull/2896) keeps #260's
OccGuardafter it. https://github.com/NotRexed/Hookshot/pull/2559 could alsomove to its own PR if you'd rather let the rest go first.
Issues
See #209
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.docs/tested-configurations.mdfor each measured run. (The restored rows, from the upstream runs.)tests/that reads changed source updated in this PR, not weakened.Two costs found by profiling a 4-core handheld (Steam Deck, RADV Van Gogh, 15 W), both generic — no Deck-specific defaults, both behind an env switch that keeps the old behaviour available. Making a staging arena while ~2.5 GB of pools and textures are held took **12–25 ms** — and an 8 MB one could take **116 ms**, so it is the allocator, not the size. The arena the last burst used goes stale after `RX_VK_STAGING_IDLE_S` (30 s) and is given back, so the first upload of the next burst makes another. A camera turn brings new textures in, so it paid for one every time: **28 slow creations in one session**, each a stall of the replay thread. Four are now made at replay start, on an empty device, where they are cheap (`RX_VK_STAGING_PREWARM`, default 4; 0 restores the old behaviour). Measured over a 240 s run, same save and settings: **28 slow creations before, 1 after.** This is the same failure the comment above the idle free already describes on other hardware ("a scene change made several right after the idle ones were freed"), so it should help anywhere memory pressure makes a 64 MB host-visible buffer expensive. `texture_view_of` was **0.99 ms a frame** for ~10,600 lookups, each a hash-map find, and the same slots come round again and again (14.9 descriptors a draw, 725 draws). A 1024-entry direct-mapped cache now answers the repeats. It holds the key, the slot's version, the destroy generation and the texture epoch, and does the epoch's touch itself — so a slot re-registered or a texture destroyed misses it at once and takes the map's path. **The map stays the authority**; the cache only answers a lookup whose every input is the one it was filled with. `RX_VK_VIEW_FAST=0` restores the old path. Measured back to back, same session, same scene, 75 % internal resolution: | | fps | p50 | p99 | 1 % low | |---|---|---|---|---| | without | 24.8 | 40.32 ms | 42.37 ms | 23.6 | | **with** | **24.9** | 40.16 ms | **41.32 ms** | **24.2** | The average is within noise; the tail is a real millisecond. - `RX_VK_PROF_FINE` now splits the texture path (storage-pointer test, handle, view, fingerprint), and the frame statistics say which binding moved `bind_gen` — counted: ubo 2159, tex 1794, ssbo 601, program 431, img 84 a frame. That last one is worth knowing: the game genuinely rebinds ~7 things per draw, which is why 0.9 of draws rebuild their descriptor set and why the reuse check cannot be relaxed. - `scripts/chain.py` no longer fails on a thread line without a name. Based on current `main` (e0e60920); both commits cherry-picked cleanly. I have not been able to build this branch locally (my build machine is on the 127 commits behind base), so CI is the first build — say the word if you would rather I build it first and reopen. Co-authored-by: RufflezAU <[email protected]> Ported to Longshot: the texture-view front cache and the statistics only. The staging prewarm (four 64 MB arenas made in replay_start, RX_VK_STAGING_PREWARM) is left out: the arenas it made went through texture_frame's keep and idle rules like any other, so on a card with 6.5 GB or less three of the four were freed at the first frame, and all of them 30 s after they were made. Keeping them needs an exemption of its own and a check; until then it cost 256 MB of host-visible memory at start for little (review of #209). Restored-from: https://github.com/NotRexed/Hookshot/pull/2960🤖 A look at the Texture detail row (
e4d17e98, Hookshot 2557) as restored here, at head5ce3b68a. We were about to port it on its own for players who miss it from late Hookshot builds, found it here instead, and are leaving it to this PR. Three things it doesn't carry over:gpu/vk/menu_i18n.hhas nothing for the row's new texts in any of the 13 tables (pt_BR, es_419, es, fr, de, it, nl, pl, ru, ja, ko, zh_Hans, zh_Hant): "Texture detail", "Full", the row's help and the four lines ofhn_texture_detail. "Low" and "Medium" are already translated, so in another language the row shows an English name with half its values translated.test_menu_i18ndoesn't catch it, since it checks only that each table's texts are the menu's, not that every menu text has a translation.low_vram_texture_mips()is also true atopts.preset == kAutoPreset && opts.budget == 1. So on a driver budget of 4.5 GB or less, the row at Auto acts as Medium under Auto (this PC) at Minimum's tier as well as under Minimum. The code does the right thing, buthn_texture_detail's Auto line ("…at the Minimum preset: as Medium.") and the comment abovetexture_detail_dimsinvk_texture.cpponly name Minimum. The row ispreset_exemptand no preset sets it, so a player's Full/Medium/Low stays as they chose it under every preset, Auto (this PC) included. Worth saying so in the description, because that's the interaction a reviewer will ask about.tx_texture_detailwent in between/// Anisotropic filtering: the game's own, or 2x to 16x …andtx_anisotropy, so that comment now sits above the wrong function.Checks run on this head (Windows, no game):
test_menu_sectionsandtest_preset_exempt(68 rows) pass.test_menu_i18nfails, but on a Debug tools text and the same way on main (#273 fixes it).test_low_vram_texture_mipsandtest_option_idsweren't run because there's no C++ compiler on this PC.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.