feat(profiling): find what a stutter is: the slow-frame recorder (HOS_SLOW_FRAMES), the main thread's time in the frame trace, and the cause of a long wait for a virtual core #1218

Merged
HumanBean merged 1 commit from HumanBean/Longshot:feat/slow-frame-recorder into main 2026-11-16 23:22:39 +00:00
Owner

Type: feat · Risk: low · Tested on: Linux, RTX 3090 (NVIDIA 620.70.06) · Needs: a Windows run of HOS_SLOW_FRAMES

Why

A stutter is a rare, short frame, and the tools we have miss it:

  • HOS_SAMPLE_ALL samples between a start and a stop that someone has to time by hand.
  • HOS_CHAIN records everything until its file is taken away.
  • The 5 s vcores: line gives a longest wait, but not whose it was or why.

In a play session on an RTX 3090 the game's main thread showed 99–100% of a core and a 274 ms wait for a virtual core, and nothing said what either was. This PR adds three tools that catch a stutter as it happens and say where its time went.

What changed

Three diagnostics for stutters, and one switch that turns them on: HOS_RECORD_STUTTERS=1 records each frame of 50 ms or more as it happens into logs/, the frame trace says where the main thread's time went in each frame, and the log names the cause of each long wait for a virtual core. All off by default except the vcore line, which only writes on a wait over 50 ms.

  • One switch for all of it, HOS_RECORD_STUTTERS=1 (host/hos_main.c, record_stutters_env), the line players are asked to add:
    • Sets HOS_SLOW_FRAMES to the game folder's logs/slow_frame (files logs/slow_frame-<n>.txt) and RX_VK_FRAME_TRACE=1, each only when it isn't set already. The vcore line is on by default anyway.
    • Read after longshot_env.txt, in both mains, so the file can set it. longshot_env.txt can't take HOS_SLOW_FRAMES=<path> itself: it names a file to write, so it's now in env_line_allowed's list beside HOS_SAMPLE_ALL and HOS_CHAIN.
    • One env: HOS_RECORD_STUTTERS: log line says what it turned on. Off by default.
  • The slow-frame recorder, HOS_SLOW_FRAMES=<path> (host/slow_frames.c, both all-thread samplers in host/posix/prof.c and host/mkw_glue.c, rx_nvn_present):
    • Samples every game and renderer thread every 4 ms for the whole run, into a ring in memory. Nothing is written while frames are on time.
    • When a present comes 50 ms or more after the one before it, that frame's samples and the 250 ms before it go to <path>-<n>.txt. The format is HOS_SAMPLE_ALL's, one time per sweep, so scripts/crit.py and scripts/hotlist.py read it as they are.
    • At most 40 files a run, at least a second apart, each one named in the log.
    • It costs frame rate, so it's off by default and stands down when HOS_SAMPLE_ALL is set (one sampler at a time).
    • Tuned with HOS_SLOW_FRAME_MS, HOS_SLOW_FRAMES_EVERY_MS and HOS_SLOW_FRAMES_MAX.
  • The main thread's time in the frame trace (RX_VK_FRAME_TRACE=1, logs/longshot_frames.csv), 10 new columns after the existing ones, which keep their order:
    • The FPS counter's Breakdown parts (game_ms … wait_main_ms), now measured while the trace is on whatever the counter shows (fps_traced).
    • The game's main thread's own time in the frame: on the CPU (main_cpu_ms), blocked in waits and sleeps (main_blocked_ms), and waiting for a virtual core (main_vcore_ms). Whatever is left of the frame is time spent runnable but kept off the CPU.
    • The host gives the main thread's times through a new MkwVkHost callback, main_thread_times. MKWVK_API_VERSION 26 → 27 (main's 26 is perf_info; main_thread_times comes after it).
    • Blocks are timed between hos_vcore_away and hos_vcore_back, and only while hos_fps_timing is on, so it costs nothing otherwise.
  • The cause of a long wait for a virtual core, HOS_VCORE_LOG_MS (default 50 ms; 0: off; host/hos_thread.c):
    • Each wait that long gets a log line naming the thread whose release ended it: its priority, how long it had held the core, and how long after being asked it let go (or that it was never asked).
    • If the wait ended another way, the line says so (found free, or ran without a core), and it lists the holders of the cores the waiter may use.
    • The first 200 a run. The vcores: line counts them all.
    • It's on by default because it costs a few stores per core handoff and only writes on a wait over 50 ms.
  • docs:
    • Players: Report slow performance has an If it stutters section (the one line, what it writes, what to send, and that the files hold no personal data). The README's low-FPS checklist points to it.
    • Developers: docs/development.md has a new Recording a stutter section under Time and profiles: what the switch turns on, how to read the files (crit.py, cpu_report.py, RX_MAPS) and what the trace's main-thread columns say. Also rows for HOS_RECORD_STUTTERS, HOS_SLOW_FRAMES (environment only), HOS_VCORE_LOG_MS and RX_VK_FRAME_TRACE's columns, and API version 27 in the host-interface paragraph.
  • tests:
    • tests/test_slow_frames.py (new) compiles slow_frames.c with a stub host and checks the dump, the cut at a whole frame, and the shared sweep time. It also checks that both samplers, the present and the vcore and trace counters feed it, and that HOS_RECORD_STUTTERS is read after longshot_env.txt, keeps switches already set, and that the file refuses HOS_SLOW_FRAMES.
    • tests/test_frame_trace_waits.py takes the new columns and reads a format string that spans several lines.
    • tests/test_audio_output_devices.py takes version 27 and the new last callback after perf_info (with an offset check for it).

Measurements

Nothing to measure as a gain. These are diagnostics: the recorder is off by default, and the trace's new timing runs only while the trace is on. The vcore line is the only part on by default, and it does a few stores per core handoff. Its 5 s vcores: line in the session below showed 70,000 to 137,000 waits every 5 s.

What the three found in one play session on this PC (RTX 3090, DLAA 1440p, Auto threading; 36,657 frames, 56 of them 50 ms or longer):

  • The slow frames are the main thread waiting, not computing, and not for a virtual core. Median of the 56: main_blocked_ms 54.6, main_cpu_ms 8.8, main_vcore_ms 0.1.
  • In their samples, no game thread runs at all in 47% of the sweeps. The main thread waits in the same game function as in the steady profile, while the loading threads (ARes Loading 0/1, OptimizeChangeBanc) are busy: these stutters come with the game's streaming.
  • The long vcore waits are the loading threads queuing on one virtual core each. For example:
    • PreActorModelCompose (priority 26) waited 838.8 ms for a core: given by sead::MainThread (priority 16), which held it 838.8 ms and was never asked.
    • GeneralFormatAsync_ waited 879.8 ms … given by BaseProcCreateAndDeleteThread, which held it 879.6 ms.
    • Of the 170 lines, the longest are all loaders (GeneralFormatAsync_, PreActorModelCompose, aglTextureDataMgr::Compaction, tera::FileResourceMgr), none the main thread.

Flags and settings

Adds four diagnostic switches (off by default) and one on by default (HOS_VCORE_LOG_MS), and widens RX_VK_FRAME_TRACE's file.

Flag Before this PR After this PR Effect Where it helps Recommendation
HOS_RECORD_STUTTERS - off 1: sets HOS_SLOW_FRAMES=logs/slow_frame and RX_VK_FRAME_TRACE=1 for the run, where not set already a player's stutter report, from longshot_env.txt off (costs frame rate)
HOS_SLOW_FRAMES=<path>, HOS_SLOW_FRAME_MS (50), HOS_SLOW_FRAMES_EVERY_MS (4), HOS_SLOW_FRAMES_MAX (40) - off; the path from the environment only each frame of 50 ms or more, sampled on every thread, to <path>-<n>.txt finding what a stutter was off
RX_VK_FRAME_TRACE off; the older columns off; 10 more columns when on the Breakdown's parts and the main thread's CPU, blocked and vcore time a frame where a slow frame's main-thread time went off
HOS_VCORE_LOG_MS - 50 a log line for each vcore wait that long, naming who held the core virtual-core starvation 50 (a few stores per handoff; writes only on a long wait)

Testing

Tested: Linux, RTX 3090, NVIDIA 620.70.06, in the game: a play session of about 6 minutes with HOS_SLOW_FRAMES and RX_VK_FRAME_TRACE=1.

  • 17 slow frames were written and logged.
  • The trace had 36,657 lines with the new columns.
  • 170 long-wait lines were logged.
  • crit.py read the dumps.

Two things that session showed are fixed here, and the build was checked after them:

  • A sample longer than the ring's line was cut through a frame. It now ends at its last whole frame.
  • Each sample carried its own time, so a sweep looked like many instants to crit.py. A sweep's samples now share its time.

Not tested:

  • HOS_RECORD_STUTTERS (added after the session above, with the rebase on main): not built or run.
  • The rebase onto main (perf_info at API 26, the performance summary's perf in fps_presented): not built.
  • python -I -B tests/run_all.py: not run on this PC (no test suites run on it); left to CI.
  • Windows: the recorder's Windows sampler changes follow the Linux ones line for line, but haven't been built or run.
  • The fixed build in the game: the two fixes were checked only by building.

Must be tested:

  • By hand, any OS: a run with only HOS_RECORD_STUTTERS=1 in longshot_env.txt, expecting the env: HOS_RECORD_STUTTERS: line, logs/slow_frame-<n>.txt files and logs/longshot_frames.csv with the main-thread columns.
  • Test Farm (@Beanstalk), Windows, any GPU: a play session with HOS_SLOW_FRAMES=<path>, expecting prof: a slow frame lines and files that scripts/crit.py reads.

Risks

What could break: MkwVkHost grows a callback (version 26), so a host and a renderer from different builds refuse each other, as at every bump. With the trace on, the FPS counter's Breakdown timing runs whatever the counter shows (as when the counter is at Breakdown). HOS_VCORE_LOG_MS writes up to 200 log lines in a run with long waits.

Turning it off: leave HOS_RECORD_STUTTERS, HOS_SLOW_FRAMES and RX_VK_FRAME_TRACE unset (the defaults); HOS_VCORE_LOG_MS=0.

Compatibility: longshot_frames.csv gains columns at its end (scripts that read the old ones by name or position keep working). No cache, save or options file changes.

Issues

Prior work: none found (searched "slow frame", "stutter", "vcore", "HOS_SAMPLE", "HOS_CHAIN", "frame breakdown", "hitch"). Builds on #928 (the FPS counter's Breakdown, whose parts the trace now writes) and on the samplers of HOS_SAMPLE_ALL. Related: #1044 and #682 (the chain recorder), #594 (virtual-core starvation).

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Not run on this PC (no test suites run on it); left to CI.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • MKWVK_API_VERSION bumped to 27 (gpu/vk/vk_api.h's MkwVkHost grew main_thread_times).
  • Switches documented (docs/development.md) and shipped at their recommended value.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** low · **Tested on:** Linux, RTX 3090 (NVIDIA 620.70.06) · **Needs:** a Windows run of `HOS_SLOW_FRAMES` ## Why A stutter is a rare, short frame, and the tools we have miss it: - `HOS_SAMPLE_ALL` samples between a start and a stop that someone has to time by hand. - `HOS_CHAIN` records everything until its file is taken away. - The 5 s `vcores:` line gives a longest wait, but not whose it was or why. In a play session on an RTX 3090 the game's main thread showed 99–100% of a core and a 274 ms wait for a virtual core, and nothing said what either was. This PR adds three tools that catch a stutter as it happens and say where its time went. ## What changed Three diagnostics for stutters, and one switch that turns them on: `HOS_RECORD_STUTTERS=1` records each frame of 50 ms or more as it happens into `logs/`, the frame trace says where the main thread's time went in each frame, and the log names the cause of each long wait for a virtual core. All off by default except the vcore line, which only writes on a wait over 50 ms. - **One switch for all of it, `HOS_RECORD_STUTTERS=1`** (`host/hos_main.c`, `record_stutters_env`), the line players are asked to add: - Sets `HOS_SLOW_FRAMES` to the game folder's `logs/slow_frame` (files `logs/slow_frame-<n>.txt`) and `RX_VK_FRAME_TRACE=1`, each only when it isn't set already. The vcore line is on by default anyway. - Read after `longshot_env.txt`, in both mains, so the file can set it. `longshot_env.txt` can't take `HOS_SLOW_FRAMES=<path>` itself: it names a file to write, so it's now in `env_line_allowed`'s list beside `HOS_SAMPLE_ALL` and `HOS_CHAIN`. - One `env: HOS_RECORD_STUTTERS:` log line says what it turned on. Off by default. - **The slow-frame recorder, `HOS_SLOW_FRAMES=<path>`** (`host/slow_frames.c`, both all-thread samplers in `host/posix/prof.c` and `host/mkw_glue.c`, `rx_nvn_present`): - Samples every game and renderer thread every 4 ms for the whole run, into a ring in memory. Nothing is written while frames are on time. - When a present comes 50 ms or more after the one before it, that frame's samples and the 250 ms before it go to `<path>-<n>.txt`. The format is `HOS_SAMPLE_ALL`'s, one time per sweep, so `scripts/crit.py` and `scripts/hotlist.py` read it as they are. - At most 40 files a run, at least a second apart, each one named in the log. - It costs frame rate, so it's off by default and stands down when `HOS_SAMPLE_ALL` is set (one sampler at a time). - Tuned with `HOS_SLOW_FRAME_MS`, `HOS_SLOW_FRAMES_EVERY_MS` and `HOS_SLOW_FRAMES_MAX`. - **The main thread's time in the frame trace** (`RX_VK_FRAME_TRACE=1`, `logs/longshot_frames.csv`), 10 new columns after the existing ones, which keep their order: - The FPS counter's Breakdown parts (`game_ms` … `wait_main_ms`), now measured while the trace is on whatever the counter shows (`fps_traced`). - The game's main thread's own time in the frame: on the CPU (`main_cpu_ms`), blocked in waits and sleeps (`main_blocked_ms`), and waiting for a virtual core (`main_vcore_ms`). Whatever is left of the frame is time spent runnable but kept off the CPU. - The host gives the main thread's times through a new `MkwVkHost` callback, `main_thread_times`. **`MKWVK_API_VERSION` 26 → 27** (main's 26 is `perf_info`; `main_thread_times` comes after it). - Blocks are timed between `hos_vcore_away` and `hos_vcore_back`, and only while `hos_fps_timing` is on, so it costs nothing otherwise. - **The cause of a long wait for a virtual core, `HOS_VCORE_LOG_MS`** (default 50 ms; 0: off; `host/hos_thread.c`): - Each wait that long gets a log line naming the thread whose release ended it: its priority, how long it had held the core, and how long after being asked it let go (or that it was never asked). - If the wait ended another way, the line says so (found free, or ran without a core), and it lists the holders of the cores the waiter may use. - The first 200 a run. The `vcores:` line counts them all. - It's on by default because it costs a few stores per core handoff and only writes on a wait over 50 ms. - **docs:** - Players: [Report slow performance](docs/performance-report.md) has an *If it stutters* section (the one line, what it writes, what to send, and that the files hold no personal data). The README's low-FPS checklist points to it. - Developers: `docs/development.md` has a new *Recording a stutter* section under *Time and profiles*: what the switch turns on, how to read the files (`crit.py`, `cpu_report.py`, `RX_MAPS`) and what the trace's main-thread columns say. Also rows for `HOS_RECORD_STUTTERS`, `HOS_SLOW_FRAMES` (environment only), `HOS_VCORE_LOG_MS` and `RX_VK_FRAME_TRACE`'s columns, and API version 27 in the host-interface paragraph. - **tests:** - `tests/test_slow_frames.py` (new) compiles `slow_frames.c` with a stub host and checks the dump, the cut at a whole frame, and the shared sweep time. It also checks that both samplers, the present and the vcore and trace counters feed it, and that `HOS_RECORD_STUTTERS` is read after `longshot_env.txt`, keeps switches already set, and that the file refuses `HOS_SLOW_FRAMES`. - `tests/test_frame_trace_waits.py` takes the new columns and reads a format string that spans several lines. - `tests/test_audio_output_devices.py` takes version 27 and the new last callback after `perf_info` (with an offset check for it). ## Measurements Nothing to measure as a gain. These are diagnostics: the recorder is off by default, and the trace's new timing runs only while the trace is on. The vcore line is the only part on by default, and it does a few stores per core handoff. Its 5 s `vcores:` line in the session below showed 70,000 to 137,000 waits every 5 s. What the three found in one play session on this PC (RTX 3090, DLAA 1440p, Auto threading; 36,657 frames, 56 of them 50 ms or longer): - **The slow frames are the main thread waiting, not computing, and not for a virtual core.** Median of the 56: `main_blocked_ms` 54.6, `main_cpu_ms` 8.8, `main_vcore_ms` 0.1. - **In their samples, no game thread runs at all in 47% of the sweeps.** The main thread waits in the same game function as in the steady profile, while the loading threads (`ARes Loading 0/1`, `OptimizeChangeBanc`) are busy: these stutters come with the game's streaming. - **The long vcore waits are the loading threads queuing on one virtual core each.** For example: - `PreActorModelCompose (priority 26) waited 838.8 ms for a core: given by sead::MainThread (priority 16), which held it 838.8 ms and was never asked`. - `GeneralFormatAsync_ waited 879.8 ms … given by BaseProcCreateAndDeleteThread, which held it 879.6 ms`. - Of the 170 lines, the longest are all loaders (`GeneralFormatAsync_`, `PreActorModelCompose`, `aglTextureDataMgr::Compaction`, `tera::FileResourceMgr`), none the main thread. ### Flags and settings Adds four diagnostic switches (off by default) and one on by default (`HOS_VCORE_LOG_MS`), and widens `RX_VK_FRAME_TRACE`'s file. | Flag | Before this PR | After this PR | Effect | Where it helps | Recommendation | |---|---|---|---|---|---| | `HOS_RECORD_STUTTERS` | - | off | `1`: sets `HOS_SLOW_FRAMES=logs/slow_frame` and `RX_VK_FRAME_TRACE=1` for the run, where not set already | a player's stutter report, from `longshot_env.txt` | off (costs frame rate) | | `HOS_SLOW_FRAMES=<path>`, `HOS_SLOW_FRAME_MS` (50), `HOS_SLOW_FRAMES_EVERY_MS` (4), `HOS_SLOW_FRAMES_MAX` (40) | - | off; the path from the environment only | each frame of 50 ms or more, sampled on every thread, to `<path>-<n>.txt` | finding what a stutter was | off | | `RX_VK_FRAME_TRACE` | off; the older columns | off; 10 more columns when on | the Breakdown's parts and the main thread's CPU, blocked and vcore time a frame | where a slow frame's main-thread time went | off | | `HOS_VCORE_LOG_MS` | - | 50 | a log line for each vcore wait that long, naming who held the core | virtual-core starvation | 50 (a few stores per handoff; writes only on a long wait) | ## Testing **Tested:** Linux, RTX 3090, NVIDIA 620.70.06, in the game: a play session of about 6 minutes with `HOS_SLOW_FRAMES` and `RX_VK_FRAME_TRACE=1`. - 17 slow frames were written and logged. - The trace had 36,657 lines with the new columns. - 170 long-wait lines were logged. - `crit.py` read the dumps. Two things that session showed are fixed here, and the build was checked after them: - A sample longer than the ring's line was cut through a frame. It now ends at its last whole frame. - Each sample carried its own time, so a sweep looked like many instants to `crit.py`. A sweep's samples now share its time. **Not tested:** - `HOS_RECORD_STUTTERS` (added after the session above, with the rebase on main): not built or run. - The rebase onto main (`perf_info` at API 26, the performance summary's `perf` in `fps_presented`): not built. - `python -I -B tests/run_all.py`: not run on this PC (no test suites run on it); left to CI. - Windows: the recorder's Windows sampler changes follow the Linux ones line for line, but haven't been built or run. - The fixed build in the game: the two fixes were checked only by building. **Must be tested:** - By hand, any OS: a run with only `HOS_RECORD_STUTTERS=1` in `longshot_env.txt`, expecting the `env: HOS_RECORD_STUTTERS:` line, `logs/slow_frame-<n>.txt` files and `logs/longshot_frames.csv` with the main-thread columns. - Test Farm (`@Beanstalk`), Windows, any GPU: a play session with `HOS_SLOW_FRAMES=<path>`, expecting `prof: a slow frame` lines and files that `scripts/crit.py` reads. ## Risks **What could break:** `MkwVkHost` grows a callback (version 26), so a host and a renderer from different builds refuse each other, as at every bump. With the trace on, the FPS counter's Breakdown timing runs whatever the counter shows (as when the counter is at Breakdown). `HOS_VCORE_LOG_MS` writes up to 200 log lines in a run with long waits. **Turning it off:** leave `HOS_RECORD_STUTTERS`, `HOS_SLOW_FRAMES` and `RX_VK_FRAME_TRACE` unset (the defaults); `HOS_VCORE_LOG_MS=0`. **Compatibility:** `longshot_frames.csv` gains columns at its end (scripts that read the old ones by name or position keep working). No cache, save or options file changes. ## Issues Prior work: none found (searched "slow frame", "stutter", "vcore", "HOS_SAMPLE", "HOS_CHAIN", "frame breakdown", "hitch"). Builds on #928 (the FPS counter's Breakdown, whose parts the trace now writes) and on the samplers of `HOS_SAMPLE_ALL`. Related: #1044 and #682 (the chain recorder), #594 (virtual-core starvation). ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). Not run on this PC (no test suites run on it); left to CI. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] `MKWVK_API_VERSION` bumped to 27 (`gpu/vk/vk_api.h`'s `MkwVkHost` grew `main_thread_times`). - [x] Switches documented (docs/development.md) and shipped at their recommended value. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(profiling): find what a stutter is: the slow-frame recorder (HOS_SLOW_FRAMES), the main thread's time in the frame trace, and the cause of a long wait for a virtual core
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 13s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 15s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Workflow lint (actionlint) (pull_request) Successful in 17s
Third-party parts / As their notes say (pull_request) Successful in 23s
beanstalk/approval approved by HumanBean (reviewed)
beanstalk/windows 0/1 pass, 1 running, 0 queued
68e901ba7e
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8dzpak9jUZJcRrqmxhM
Author
Owner

Tested in game: RTX 3090, NVIDIA 620.70.06, Linux: a 6-minute play session with HOS_SLOW_FRAMES and RX_VK_FRAME_TRACE=1: 17 slow frames written and logged, 36,657 trace lines with the new columns, 170 long virtual-core waits logged with their cause; scripts/crit.py read the dumps

Tested in game: RTX 3090, NVIDIA 620.70.06, Linux: a 6-minute play session with HOS_SLOW_FRAMES and RX_VK_FRAME_TRACE=1: 17 slow frames written and logged, 36,657 trace lines with the new columns, 170 long virtual-core waits logged with their cause; scripts/crit.py read the dumps
Author
Owner

@Beanstalk Windows, any GPU: play a few minutes through a load or two (Kokiri Forest into Hyrule Field, then the Market) with HOS_SLOW_FRAMES=/f set. Expect "prof: a slow frame" lines in the log and a f-.txt file for each, which scripts/crit.py reads without errors; and with RX_VK_FRAME_TRACE=1, logs/longshot_frames.csv ending in the main_cpu_ms, main_blocked_ms and main_vcore_ms columns.

@Beanstalk Windows, any GPU: play a few minutes through a load or two (Kokiri Forest into Hyrule Field, then the Market) with HOS_SLOW_FRAMES=<a folder>/f set. Expect "prof: a slow frame" lines in the log and a f-<n>.txt file for each, which scripts/crit.py reads without errors; and with RX_VK_FRAME_TRACE=1, logs/longshot_frames.csv ending in the main_cpu_ms, main_blocked_ms and main_vcore_ms columns.
Member
Beanstalk test farm · an earlier run: commit 68e901b (the newer results are in a comment below)

Commit 68e901b · tier reviewed · approved by HumanBean · dashboard

What will be tested · AI plan (gpt-6.1-terra)

On Windows, this plan runs a short Kokiri Forest, Hyrule Field and Market session on main and the pull request, enabling the slow-frame recorder only on the pull request. It compares gameplay images and frame times across two loads, and fails on crashes, substantial frame-time regressions or image differences. The farm cannot set the requested folder path or frame-trace switch, inspect recorder files or CSV columns, or run crit.py, so those diagnostic outputs remain unverified.

  • Where: windows (1 report).
  • Matrix: 1 cell, each on main and the PR: HOS_SLOW_FRAMES f (PR only).
  • forest-field-market: from a new save, then measure forest for 30 s (after 5 s warm-up), take the picture forest to compare with main, teleport to Hyrule Field, move the stick, measure field for 30 s (after 5 s warm-up), take the picture field to compare with main, teleport to Market, measure market for 30 s (after 5 s warm-up), take the picture market to compare with main; 1 run on each side.
  • Measured: crashes, frame times, stutters and hitches, load times.
  • The PR fails if: it crashes; or a picture differs from main by more than 10%; or frame time (p99) is more than 50% above main.
  • Time: about 16 min of game runs per tester (2 game starts), plus the two builds.

Started.

Target Reports Crash Frame p99 (ms) vs main VRAM peak (MB) vs main Result
windows 0/1 — — — running
<!-- beanstalk:previous --> <details><summary>Beanstalk test farm · an earlier run: commit <code>68e901b</code> (the newer results are in a comment below)</summary> Commit `68e901b` · tier **reviewed** · approved by HumanBean · [dashboard](https://beanstalk.magicbean.dev/prs/1218) **What will be tested** · AI plan (`gpt-6.1-terra`) On Windows, this plan runs a short Kokiri Forest, Hyrule Field and Market session on main and the pull request, enabling the slow-frame recorder only on the pull request. It compares gameplay images and frame times across two loads, and fails on crashes, substantial frame-time regressions or image differences. The farm cannot set the requested folder path or frame-trace switch, inspect recorder files or CSV columns, or run crit.py, so those diagnostic outputs remain unverified. - **Where:** windows (1 report). - **Matrix:** 1 cell, each on main and the PR: HOS_SLOW_FRAMES `f` (PR only). - **forest-field-market:** from a new save, then measure forest for 30 s (after 5 s warm-up), take the picture forest to compare with main, teleport to Hyrule Field, move the stick, measure field for 30 s (after 5 s warm-up), take the picture field to compare with main, teleport to Market, measure market for 30 s (after 5 s warm-up), take the picture market to compare with main; 1 run on each side. - **Measured:** crashes, frame times, stutters and hitches, load times. - **The PR fails if:** it crashes; or a picture differs from main by more than 10%; or frame time (p99) is more than 50% above main. - **Time:** about 16 min of game runs per tester (2 game starts), plus the two builds. Started. | Target | Reports | Crash | Frame p99 (ms) vs main | VRAM peak (MB) vs main | Result | | --- | --- | --- | --- | --- | --- | | windows | 0/1 | — | — | — | running | </details>
HumanBean force-pushed feat/slow-frame-recorder from 68e901ba7e
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 13s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 15s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Workflow lint (actionlint) (pull_request) Successful in 17s
Third-party parts / As their notes say (pull_request) Successful in 23s
beanstalk/approval approved by HumanBean (reviewed)
beanstalk/windows 0/1 pass, 1 running, 0 queued
to 44787ae129
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
beanstalk/approval not approved yet
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 17s
2026-11-16 20:03:24 +00:00
Compare
Member

Beanstalk test farm

Commit d99af5f · tier untrusted · approved by nobody yet · dashboard

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `d99af5f` · tier **untrusted** · approved by nobody yet · [dashboard](https://beanstalk.magicbean.dev/prs/1218) A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
HumanBean force-pushed feat/slow-frame-recorder from 44787ae129
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
beanstalk/approval not approved yet
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 17s
to d99af5f51d
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 17s
PR description / Description follows the template (pull_request_target) Failing after 4s
beanstalk/approval not approved yet
2026-11-16 23:15:45 +00:00
Compare
Author
Owner

Rebased on main (e335bd5c). Main took API version 26 for perf_info, so this moves to 27 with main_thread_times after it. fps_presented keeps both the performance summary's perf and the trace's Breakdown level.

Added HOS_RECORD_STUTTERS=1, one switch that turns on the slow-frame recorder (into logs/slow_frame-<n>.txt) and the frame trace, and that longshot_env.txt accepts. HOS_SLOW_FRAMES=<path> is now refused from the file, as other file-writing switches are. Docs: an If it stutters section in docs/performance-report.md for players, and a Recording a stutter section in docs/development.md for developers.

Not built or run: the rebase and the new switch. Tests weren't run on this PC.

Rebased on main (e335bd5c). Main took API version 26 for `perf_info`, so this moves to 27 with `main_thread_times` after it. `fps_presented` keeps both the performance summary's `perf` and the trace's Breakdown level. Added `HOS_RECORD_STUTTERS=1`, one switch that turns on the slow-frame recorder (into `logs/slow_frame-<n>.txt`) and the frame trace, and that `longshot_env.txt` accepts. `HOS_SLOW_FRAMES=<path>` is now refused from the file, as other file-writing switches are. Docs: an *If it stutters* section in docs/performance-report.md for players, and a *Recording a stutter* section in docs/development.md for developers. Not built or run: the rebase and the new switch. Tests weren't run on this PC.
HumanBean changed title from WIP: feat(profiling): find what a stutter is: the slow-frame recorder (HOS_SLOW_FRAMES), the main thread's time in the frame trace, and the cause of a long wait for a virtual core to feat(profiling): find what a stutter is: the slow-frame recorder (HOS_SLOW_FRAMES), the main thread's time in the frame trace, and the cause of a long wait for a virtual core 2026-11-16 23:22:34 +00:00
HumanBean deleted branch feat/slow-frame-recorder 2026-11-16 23:22:40 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1218
No description provided.