WIP: feat(profiling): find what a stutter is: the slow-frame recorder (HOS_SLOW_FRAMES), the main thread's time in the frame trace, and the cause of a long wait for a virtual core #1218
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1218
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:feat/slow-frame-recorder"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Linux, RTX 3090 (NVIDIA 620.70.06) · Needs: a Windows run of
HOS_SLOW_FRAMESWhy
A stutter is a rare, short frame, and the tools we have miss it:
HOS_SAMPLE_ALLsamples between a start and a stop that someone has to time by hand.HOS_CHAINrecords everything until its file is taken away.vcores:line gives a longest wait, but not whose it was or why.In a play session on an RTX 3090 the game's main thread showed 99–100% of a core and a 274 ms wait for a virtual core, and nothing said what either was. This PR adds three tools that catch a stutter as it happens and say where its time went.
What changed
HOS_SLOW_FRAMES=<path>(host/slow_frames.c, both all-thread samplers inhost/posix/prof.candhost/mkw_glue.c,rx_nvn_present):<path>-<n>.txt. The format isHOS_SAMPLE_ALL's, one time per sweep, soscripts/crit.pyandscripts/hotlist.pyread it as they are.HOS_SAMPLE_ALLis set (one sampler at a time).HOS_SLOW_FRAME_MS,HOS_SLOW_FRAMES_EVERY_MSandHOS_SLOW_FRAMES_MAX.RX_VK_FRAME_TRACE=1,logs/longshot_frames.csv), 10 new columns after the existing ones, which keep their order:game_ms…wait_main_ms), now measured while the trace is on whatever the counter shows (fps_traced).main_cpu_ms), blocked in waits and sleeps (main_blocked_ms), and waiting for a virtual core (main_vcore_ms). Whatever is left of the frame is time spent runnable but kept off the CPU.MkwVkHostcallback,main_thread_times.MKWVK_API_VERSION25 → 26.hos_vcore_awayandhos_vcore_back, and only whilehos_fps_timingis on, so it costs nothing otherwise.HOS_VCORE_LOG_MS(default 50 ms; 0: off;host/hos_thread.c):vcores:line counts them all.docs/development.mdrows forHOS_SLOW_FRAMES,HOS_VCORE_LOG_MSandRX_VK_FRAME_TRACE's columns.tests/test_slow_frames.py(new) compilesslow_frames.cwith a stub host and checks the dump, the cut at a whole frame, and the shared sweep time. It also checks that both samplers, the present and the vcore and trace counters feed it.tests/test_frame_trace_waits.pytakes the new columns and reads a format string that spans several lines.tests/test_audio_output_devices.pytakes version 26 and the new last callback.Measurements
Nothing to measure as a gain. These are diagnostics: the recorder is off by default, and the trace's new timing runs only while the trace is on. The vcore line is the only part on by default, and it does a few stores per core handoff. Its 5 s
vcores:line in the session below showed 70,000 to 137,000 waits every 5 s.What the three found in one play session on this PC (RTX 3090, DLAA 1440p, Auto threading; 36,657 frames, 56 of them 50 ms or longer):
main_blocked_ms54.6,main_cpu_ms8.8,main_vcore_ms0.1.ARes Loading 0/1,OptimizeChangeBanc) are busy: these stutters come with the game's streaming.PreActorModelCompose (priority 26) waited 838.8 ms for a core: given by sead::MainThread (priority 16), which held it 838.8 ms and was never asked.GeneralFormatAsync_ waited 879.8 ms … given by BaseProcCreateAndDeleteThread, which held it 879.6 ms.GeneralFormatAsync_,PreActorModelCompose,aglTextureDataMgr::Compaction,tera::FileResourceMgr), none the main thread.Testing
Tested: Linux, RTX 3090, NVIDIA 620.70.06, in the game: a play session of about 6 minutes with
HOS_SLOW_FRAMESandRX_VK_FRAME_TRACE=1.crit.pyread the dumps.Two things that session showed are fixed here, and the build was checked after them:
crit.py. A sweep's samples now share its time.Not tested:
python -I -B tests/run_all.py: not run on this PC (no test suites run on it); left to CI.Must be tested:
HOS_SLOW_FRAMES=<path>, expectingprof: a slow framelines and files thatscripts/crit.pyreads.Risks
What could break:
MkwVkHostgrows a callback (version 26), so a host and a renderer from different builds refuse each other, as at every bump. With the trace on, the FPS counter's Breakdown timing runs whatever the counter shows (as when the counter is at Breakdown).HOS_VCORE_LOG_MSwrites up to 200 log lines in a run with long waits.Turning it off: leave
HOS_SLOW_FRAMESandRX_VK_FRAME_TRACEunset (the defaults);HOS_VCORE_LOG_MS=0.Compatibility:
longshot_frames.csvgains columns at its end (scripts that read the old ones by name or position keep working). No cache, save or options file changes.Issues
Prior work: none found (searched "slow frame", "stutter", "vcore", "HOS_SAMPLE", "HOS_CHAIN", "frame breakdown", "hitch"). Builds on #928 (the FPS counter's Breakdown, whose parts the trace now writes) and on the samplers of
HOS_SAMPLE_ALL. Related: #1044 and #682 (the chain recorder), #594 (virtual-core starvation).Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Not run on this PC (no test suites run on it); left to CI.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.MKWVK_API_VERSIONbumped (gpu/vk/vk_api.h'sMkwVkHostgrewmain_thread_times).🤖 Generated with Claude Code
Tested in game: RTX 3090, NVIDIA 620.70.06, Linux: a 6-minute play session with HOS_SLOW_FRAMES and RX_VK_FRAME_TRACE=1: 17 slow frames written and logged, 36,657 trace lines with the new columns, 170 long virtual-core waits logged with their cause; scripts/crit.py read the dumps
@Beanstalk Windows, any GPU: play a few minutes through a load or two (Kokiri Forest into Hyrule Field, then the Market) with HOS_SLOW_FRAMES=/f set. Expect "prof: a slow frame" lines in the log and a f-.txt file for each, which scripts/crit.py reads without errors; and with RX_VK_FRAME_TRACE=1, logs/longshot_frames.csv ending in the main_cpu_ms, main_blocked_ms and main_vcore_ms columns.
Beanstalk test farm · an earlier run: commit
68e901b(the newer results are in a comment below)Commit
68e901b· tier reviewed · approved by HumanBean · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)On Windows, this plan runs a short Kokiri Forest, Hyrule Field and Market session on main and the pull request, enabling the slow-frame recorder only on the pull request. It compares gameplay images and frame times across two loads, and fails on crashes, substantial frame-time regressions or image differences. The farm cannot set the requested folder path or frame-trace switch, inspect recorder files or CSV columns, or run crit.py, so those diagnostic outputs remain unverified.
f(PR only).Started.
68e901ba7e44787ae129Beanstalk test farm
Commit
44787ae· tier untrusted · approved by nobody yet · dashboardA member of
BigBeanscan review this commit and comment@Beanstalkto test it.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.