perf(runtime): reduce Market frame stalls and Vulkan capture overhead #1553
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
6 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1553
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:perf/market-render-rewrite"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: high · Tested on: CachyOS Linux, RTX 3090 · Needs: controlled main comparison, Windows, AMD, Intel and low-VRAM hardware
TL;DR
Market could miss the frame budget while game workers waited for a virtual core, and a recording diagnostic could stall other workers for hundreds of milliseconds. This series changes how Longshot schedules CPU work, captures submitted memory and reuses Vulkan resources. The player reported smoother gameplay on the final build. Two Market visits are measured below.
Breakdown
Why
In a Market playtest, a recorded frame took 717 ms. While a recording worker held its table lock and a virtual core inside a log write, audio waited 321 ms for that core. Other slow frames coincided with resource decompression and waits for game workers. An earlier renderer path could also reject an otherwise usable host-memory import range. Related work includes #155, #952, #977 and #1402.
How to reproduce
Load child Market and traverse the busy area. Record presents through the local profiling queue, including area entry separately. For a fixed-route comparison, use the same save, camera path, warmed caches, GPU driver and graphics settings on both revisions.
What changed
The review follow-up at
fdee2857broadcasts an event when multiple registered sleepers may need different guest cores. Auto-clear still permits one consumer. Pending and completed GPU buffer writes each retain at most 65,536 exact ranges. Excess writes retain conservative completed-timeline evidence for guest-backed textures. GPU-drawn textures require a real ordered byte overlap before their pixels can be replaced, and an upload acknowledges a write only after it succeeds.Measurements
Verdict: Interactive results support further testing. A controlled comparison against main is pending. Runs below used different player routes, and the queue detected external build activity in the final session.
Hardware: Ryzen 9 7900X, RTX 3090, NVIDIA 620.70.06, CachyOS Linux, 32 GB RAM. Settings included 2560×1440 configured output, fullscreen, V-Sync off and frame generation off. Timings use consecutive displayed-frame intervals from the built-in profiler.
5d6590e2, Market, 60 s28fb02e2, Market, 60 s1820ed9b, first Market visit, 20 s1820ed9b, return to Market, 20 sThe final windows start at the first periodic present-count log following each Market marker and end before the next area transition. They exclude entry and loading. The final build visited Hyrule Field, Death Mountain Crater and Ganon's Castle as well.
The rebased Linux build with the two review fixes (
65cf2a90) ran headed in Market and exited normally. A roughly 50-second interactive sample measured 56.9 FPS and a 27.1 ms p99. A separate native smoke run built from the final integration source exited normally after 8,136 presents. Its 900 settled Market frames measured 75.0 FPS and a 17.3 ms p99. The pack builder recorded the final smoke source as a working copy ond87dd600, tree fingerprint03d9b04d. The two runs followed different routes and do not establish a speedup.Flags and settings
The profile used HOS_PROFILE=1 and HOS_RESERVE_CORE=0 on each build, with saved player settings and warmed pipeline caches. No new player-facing switch was added.
Testing
Tested: Linux native game build and interactive playtest, normal exit. The focused capture, late-read and uniform-cache checks passed. Both changed translation units from the last commit passed a Windows syntax check. The Linux full suite during the last development pass produced 711 passes, 43 failures and 19 skips. All 43 failures also appeared in the preceding rewrite suite. The affected checks were rerun after final boundary fixes.
Review verification: The new disjoint-core event test and pending-allocation test both reproduced the findings at
d87dd600and passed after the fixes. The write-history fixture covers 196,608 disjoint writes, many outstanding timeline values, peak publication allocations, neighboring GPU-owned images, and writes on either side of a draw in one submission. Related event, multiwait, virtual-core, Tracy, Linux condition-variable, drawn-page and readback checks passed. An integration-suite run with the default tool search reported 638 passed, 35 failed and 104 skipped. The dependency-enabled complete suite atfdee2857reported 715 passed, 43 failed and 19 skipped. Running all 43 failed checks on upstream68d7757ereproduced 42. The remaining check,test_astc_reuse.py, found its cleanup call outside a hard-coded 200-character excerpt. Test-only commitcbedca6cuses the full function and the check passed on rerun. The runtime source matchesfdee2857.42 checks that failed on both revisions
test_bug_report_defaults,test_capture_readback_visibility,test_compute_warm_apart,test_cpu_report,test_crt_options,test_damage_debug_menu,test_dev_packs,test_display_startup_failure,test_dlss_feature_size,test_draw_stream_visibility,test_failed_compile_keeps_objects,test_fsr_image_retirement,test_fsr_uniform_ownership,test_guest_imported,test_hud_layout_row,test_invalidate_quick,test_invalidate_vcore,test_libc_hooks,test_linux_preflight,test_linux_pressure,test_linux_update,test_log_open_failure,test_mirror_budget,test_missing_nup,test_mods_size_table,test_nonuniform_image_arrays,test_parallel_init_order,test_pending_guest_import,test_pipeline_play_cache,test_present_source_crop,test_present_thread,test_profiling_mode,test_psychov35_fsr3_fg,test_readback_timeout_source,test_replay_write_order,test_run_all_changed,test_slow_frames,test_staging_burst,test_switch_table,test_upscaler_rows,test_vk_occlusion_queries,test_walk_keyNot tested: Final runtime on Windows, non-NVIDIA GPUs or low-VRAM cards. Controlled performance comparisons against current main remain open.
Must be tested:
Risks
What could break: Guest event ordering, texture snapshots, import fallback and resource retirement are sensitive to thread timing and driver memory behavior.
Review follow-up: With opt-in parallel replay, a later worker may record a buffer write before an earlier worker first creates a texture copy destination at the same address. The destination may miss that write. The ordered-worker fixture covers images already registered with the renderer. The default serial path creates these images in stream order. Excess ranges can also cause extra uploads of guest-backed textures. The CPU cost of the live image index under heavy aliasing still needs measurement.
Trace interpretation: Large native Zstandard decodes contribute to the virtual-core-away interval counted in blocked time. Repeated event signals can update the recorded timestamp when the event is already ready, so the last reported signal does not always identify the producer that woke the waiter.
Turning it off: The work spans host and renderer changes. Revert the series for a full rollback.
Compatibility: The renderer ABI and save format are unchanged. The updated compute shader has a regenerated checked-in header. This PR contains source and tests only.
Checklist
Everyone
79d4a143before the review fixes. Main then advanced. A local merge ofd1a307cband the published headcbedca6chas no source conflicts; Forgejo reports the WIP PR as non-mergeable.68d7757e. Testing lists the 42 checks and the corrected integration-specific check.When it applies
Tested in game: CachyOS Linux, Ryzen 9 7900X, RTX 3090, NVIDIA 620.70.06. The final build ran for 14 minutes, visited child Market twice, Hyrule Field, Death Mountain Crater and Ganon’s Castle, and exited normally. The player reported smoother gameplay. The two 20-second Market samples measured 65.2 and 67.1 FPS.
Possibly related, for context:
@Beanstalk
Beanstalk test farm · an earlier run: commit
1820ed9(the newer results are in a comment below)Commit
1820ed9· tier reviewed · approved by HumanBean · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)Two interleaved runs per revision compare Market entry, a fixed movement route, and a return visit after Hyrule Field, taking about 26 minutes per tester. The plan measures frame times, hitches, load times and VRAM, and compares images for rendering regressions. Crashes, significant performance or memory regressions, and substantial screenshot differences fail the plan.
Started at 05:28 UTC.
Estimate: done in about 60 min, from the queue and the testers online now.
@Beanstalk I want a windows tester, an amd tester, a steam deck tester
Beanstalk test farm · an earlier run: commit
1820ed9, run 2 (the newer results are in a comment below)Commit
1820ed9· tier reviewed · approved by HumanBean · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)The plan runs on Windows, AMD and Steam Deck targets, comparing two interleaved runs per revision through child Market, Hyrule Field and Death Mountain Crater before returning to Market. Each target takes about 28 minutes of game runs, measuring frame times, hitches, load times and memory use and comparing screenshots with main. Crashes, timeouts, device-loss or out-of-memory logs, substantial frame-time, load-time or memory regressions, and large screenshot differences fail the plan.
Started at 05:30 UTC. Run 2: re-planned by HumanBean.
Estimate: done in about 55 min, from the queue and the testers online now; some jobs can't start yet.
Testing this now on two machines. Each compares the PR head against its base, so only this series differs. That's now
cbedca6c(withfdee2857's fixes) against main79d4a143.Both report p95/p99, the worst frame and long-frame counts alongside FPS, since this targets stalls. Results will follow here.
Edited: retargeted from
1820ed9b/310e7c9d, then fromd87dd600, to the current head.A code-review note on the scheduling part at
1820ed9b, from reading the source; it hasn't been reproduced yet:An auto-clear event now wakes one sleeper, chosen by the host, but that sleeper must get a guest core back before it can claim the event.
host/nn_os_sync.c:680wakes a single sleeper withWakeConditionVariable.event_wait(:743) can only take the event afterhos_cv_sleepreturns.host/hos_thread.c:1989–1992reacquires the guest core first.Say sleeper A may only run on busy core 0 and sleeper B may run on free core 1. The host can wake A and leave B asleep, while the event stays
ready=1. A higher-priority holder of core 0 can then delay A until the starvation/escape handling (hos_thread.c:1817–1820,1878–1885). That's a progress/priority-inversion risk, not a lost wake-up or a proven deadlock. Main's broadcast let both sleepers compete.A possible fix: broadcast when more than one sleeper is registered (the CAS still lets only one consume it), or hand the event to an eligible sleeper explicitly.
tests/test_event_handoff.py:53–57waits on a plaincondition_variable_anywithouthos_vcore_back, so it can't catch this. A case with two sleepers on disjoint core masks would.For anyone profiling this:
hos_zstd.c:406–410now runs the CPU-bound decode between away and back, andhos_thread.c:1956,1967count that time as blocked. So a rise inmain_blocked_msdoesn't necessarily mean more waiting.HOS_CHAINSIGNAL records andsig_nsare still overwritten when the event is already ready. That's old behaviour, but now that path wakes nobody, so treat "last SIGNAL" as the waking producer with care.No problem found so far in capture-range reuse or the one-page snapshot read. The Deck and Windows A/B runs mentioned above are still going ahead.
A second note from reading
1820ed9b(a static finding, not seen in game): the new GPU write history has no bound on unfinished writes.gpu/vk/gpu_write_history.h:49–65puts every disjoint write interval intopending[value]with no node or byte limit.:68–75caps only finished ranges, and only after a whole completed batch has been published.texture_gpu_wroteinvk_texture.cppcapped the retained map at 65,536 on each insert.So one long recording or load with N disjoint SSBO or texel-buffer writes at the same
rec_valuekeeps N map nodes.complete(value)can build the second representation before its final trim. The periodic completion call (vk_replay.cpp:837, every 32 parts) doesn't bound one unfinished batch or a GPU backlog.Please don't fix this by dropping unfinished records: that would bring back the stale-texture invalidation this history exists to prevent. A conservative overflow policy for pending entries and the publication peak would do instead, for example a coarse dirty-epoch/range fallback; your choice. A demonstrated upstream bound on how many there can be would also do.
A test:
test_wrote_regionscurrently uses a few tiny ranges. A case with more than 65,536 disjoint writes under one pending value, plus overlapping finished and unfinished writes, would cover this.Otherwise: no stale-capture problem found in the capture-range reuse, the snapshot fast paths or the capture-spare reservations.
1820ed9b3dd87dd6000fBeanstalk test farm · an earlier run: commit
d87dd60(the newer results are in a comment below)Commit
d87dd60· tier reviewed · approved by HumanBean · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)Two alternating runs compare main and the PR in child Market, Hyrule Field and Death Mountain Crater, then return to Market to exercise resource reuse. The automated plan takes about 27 minutes per PC and measures frame times, hitches, load times, VRAM and process memory while comparing pictures. Crashes, timeouts, substantial performance or memory regressions, and significant picture differences fail the plan.
Started at 05:37 UTC.
Estimate: done in about 45 min, from the queue and the testers online now.
@Beanstalk do broad test
Both notes above still apply at the rebased head
d87dd600:host/nn_os_sync.c:716, insideevent_signalstarting at:704. The lines moved with the rebase onto current main.gpu/vk/gpu_write_history.his byte-identical to1820ed9b, so the line numbers in the second note are unchanged.Beanstalk test farm
Commit
cbedca6· tier reviewed · approved by HumanBean · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)The plan compares main and the PR across Market, Hyrule Field, Death Mountain Crater and Ganon’s Castle, including a return to Market to exercise resource reuse. It takes about 15 minutes per tester and measures frame times, memory and loading, with screenshots for visual regressions. Crashes, timeouts, substantial performance or memory regressions, and excessive screenshot differences fail the run.
Done (started 06:14 UTC, last report 06:43 UTC). Comment
@Beanstalkto run it again.NVIDIA GeForce RTX 2060, 621.24.0Measurements · any
Verdict: could not be judged on any (1/1 report).
Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 2060, 621.24.0AMD Ryzen 9 5900XT 16-Core Processor, 32 GB RAMScene: market-scene-reuse, from a new save (vsync=2, frame_generation=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); market-scene-reuse: 1 game start a side, market-first 30 s after 5 s warm-up, field 20 s after 5 s warm-up, crater 20 s after 5 s warm-up, castle 20 s after 5 s warm-up, market-return 30 s after 5 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)Nothing was measured: no metric came back.
Flags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Both review findings reproduced on the rebased head
d87dd600and are addressed infdee2857.The description now distinguishes the original playtest, the headed review build and the final integration smoke. It also records the scope of the profiler's blocked-time and SIGNAL timestamps. One follow-up remains in opt-in parallel replay when an earlier segment first creates a destination after a later worker has already recorded an aliased buffer write.
The branch was rebased on
79d4a143and combines with current main68d7757ewithout source conflicts. It remains WIP for measurement and wider hardware coverage.The complete dependency-enabled suite on
fdee2857reported 715 passed, 43 failed and 19 skipped. All 43 failed checks were rerun on upstream68d7757eand 42 failed there too. I corrected the ASTC reuse check to inspect the full retirement function, then reran it successfully. The PR description lists the common failures. The new headcbedca6cchanges only the test, so the Market smoke still covers the current runtime source.@Beanstalk broad test
Thanks. I read
fdee2857against both notes, and both look addressed. This is a code-reading check of those two fixes only, not a review of the whole PR; nothing was run.e->lk, and the signal wakes one sleeper only when exactly one is registered, otherwise all of them. The auto-clear CAS still admits one consumer. The revisedtest_event_handoff.pyforces the wake-one choice onto the busy core and would fail if that came back.complete()only advances a finished dirty floor, and trimming happens per publish. Nothing is dropped, and neighbouring rendered images aren't dirtied just because the history overflowed. The upload-failure retry is checked by a source assertion rather than a real failed upload in the renderer, as far as I can see.RX_VK_PARALLELchecked unset in both arms.WIP: perf(runtime): reduce Market frame stalls and Vulkan capture overheadto perf(runtime): reduce Market frame stalls and Vulkan capture overheadSteam Deck OLED, the child Market on a fixed route: no difference resolved between main before this PR and main with it, and no regression. One run a side, so read it as a bound, not a measurement of the change.
d1a307cb430eef4f(this PR in it)GPU timeline, both sides, about 2,100 to 2,250 draws), a long way from the 65 to 75 fps of the PR's own numbers. A wait of a millisecond or two for a virtual core is a small part of such a frame.Hardware: Steam Deck OLED (AMD Custom APU 0932, 4 cores / 8 threads, RADV VANGOGH, Mesa 26.2.0-devel), SteamOS 3.9.5, 16 GB, on the charger, the GPU at a manual 1600 MHz.
Builds: both built on the Deck by the Linux builder (
scripts/linux/appimage.sh, thenAppRun --cli --update,--jobs 4), withLONGSHOT_CHUNK_OPT=alland a local hot list of 918 functions, the same on both sides, so not the default build. The update from one to the other compiled 107 files, none of the lifted code, and prepared no shader again.Settings, the same on both sides: the handheld profile, a fixed 3D resolution of 1536x864 (
HOS_RES_LOG=1:vi: res: fixed 1 (scale 1.000) ... 3D scale 1.000in both logs), the game's own upscaler, headless,HOS_FPS_MAX=1000, and the player's switches:RX_VK_READBACK_LATE=newest,3980,RX_VK_LATE_DEFER=0,RX_VK_READBACK_EARLY_GAP=120,HOS_RESERVE_CORE=0,RX_VK_VRAM_MB=2800,RX_VK_TEX_MAX_DIM=1024,RX_VK_TEX_SKIP_MB=4,RX_VK_LOAD=create,RX_VK_LOW_SPEC=1,RX_VK_TEX_AHEAD=1,HOS_NATIVE_ZSTD=0.Route: from a save in Kokiri Forest, through the control bridge: the debug teleport to
Market_Childthree times, Hyrule Field between two visits; in each visit 30 s of a walk that comes back to its start, with camera swings (longshot_ctl.py stick: left stick forward for 90 frames, right stick for 60, back for 90, right stick the other way for 60, left for 60, right for 60, again until the time is up). Every present is taken from the bridge'spresentevents with its time of arrival; a visit's frames are the presents of 200 draws or more between the teleport's answer and the end of the walk.Limits:
HOS_NATIVE_ZSTD=0is in the player's settings, so the game's own decoder took the streams in both runs: what this PR changes around the host's decode during a load is not measured here.The route is three small scripts; I can run it again on other settings (the game's dynamic resolution, a default build,
HOS_NATIVE_ZSTDon) if one of them would help.Veix referenced this pull request2026-11-18 10:34:33 +00:00
Results, following up our testing notice. This PR was merged before our runs finished, so the planned PR-only comparison was replaced. These results compare main
79d4a143with430eef4f: a nine-commit bundle that includes this PR, #1564 and #1567. They are not this PR in isolation. The planned Deck Market block wasn't run in this series.Steam Deck OLED
Windows, RTX 5090: three interleaved pairs, warm caches
p0lux's Market traversal (comment 48525) used different builds and settings, with native zstd off. It complements these results; we haven't pooled the two.
Follow-up on the Field admission refusals: the recurring “1 jobs queued” pattern aligns with the renderer’s scheduled glibc heap check/trim (
mallinfo2/malloc_trim), which also exists on544229a2; window alignment explains why it was sampled in some runs and not others. It is not evidence of a newly translated Field shader. Our original admission decisions and exclusions are unchanged; separate program-census refusals remain, so there is still no admitted Field comparison pair.Testing note from the Steam Deck LCD campaign (VanGogh iGPU, no VRAM mirrors — the opposite regime from the RTX 3090 route in the description): our campaign overlay branch (perf/1477-bc1-packed-export, PR #1561) is based on pre-1553 main, and #1553 lands under every campaign measurement made in the last two days. The Deck campaign therefore cannot treat post-1553 main as an interchangeable control without one matched pair first (
0ffbb8b3vs7daafa7a, same Deck, same route). We will run that pair when the campaign reaches its next rebase; flagging it now so nobody compares across the squash by accident. No defect claimed — just regime coverage: the snapshot-read + late-page path and the ASTC upload repartition both run differently on UMA than on the 3090.Instrumentation caveat for the Steam Deck numbers: recurring long frames have nearly the same cadence as our /proc memory sampler. We are investigating an observer effect; no sampler-off control has established its size or whether it changes the arm contrasts. The Deck p99, long-frame counts, maxima and FPS comparisons above include that instrumentation and must not yet be treated as unsampled player experience. The raw results and exclusions are unchanged. This caveat does not concern the separate Windows runs.
One more data point from the Windows side, from a profile on main
430eef4f(RTX 5090, the child Market). It's descriptive and isn't an FPS result.Five complete 600-present blocks inside the marked Market window averaged 6.72 ms a frame of timestamp gap before the first draw-bearing recording, against 1.56 ms after it. The former includes the output copy; upload times were logged separately. The main-vcore wait was about 0.02 ms a frame.
This points further investigation at the interval before the first draw-bearing recording, not at vcore arbitration. It doesn't identify the signalling dependency or show reclaimable GPU idle time, and it isn't a Deck result. The capture was headless, sampled every 2 ms and warm-cache, with 15.2% other CPU on the PC. No wait chain was obtained.
Follow-up to the instrumentation caveat above: a sampler-cadence control on the same Deck moved the recurring long frames with the memory sampler's period (5.8 → 17.8 → 5.8 s), with every Kokiri interval over 100 ms intersecting its
smaps_rollupread. That strengthens the observer-effect caveat for our Deck tail numbers; it doesn't estimate unsampled FPS or show that the arm contrasts cancel. Details: #1477 (comment)