feat(profiling): HOS_PROFILE=1, one switch for every profiling tool, into a folder of its own, and scripts/profile_report.py to read it #1279

Merged
HumanBean merged 1 commit from HumanBean/Longshot:feat/profile-mode into main 2026-11-17 00:14:43 +00:00
Owner

Type: feat · Risk: low · Tested on: Linux, RTX 3090 (NVIDIA 620.70.06) · Needs: a Windows run of HOS_PROFILE=1

Why

Finding what to optimize next, or what a stutter was, took several switches set by hand, files in several places, and samplers started and stopped at the right moment. Most of those files had no common clock, so a stutter could not be matched to what the threads were doing. This PR makes it one switch, HOS_PROFILE=1, and adds one command that reads the result into a report and a profiler's timeline. It replaces HOS_RECORD_STUTTERS (#1218).

What changed

One switch records a whole run into a folder of its own, and one script reads it: a markdown report of where the time goes and what each stutter was, a Perfetto timeline of every thread, and speedscope flame graphs.

  • HOS_PROFILE=1 (host/profiling.c; allowed in longshot_env.txt). It writes logs/profile-<date>-<time>/, a new folder each run:
    • all.txt: every game and renderer thread's stack every 2 ms, from the start. Packed (HOS_SAMPLE_PACKED): each distinct stack is written once, about a tenth of the plain format's size.
    • frames.csv: the frame trace, with a new last column at_ns, the present on the host's clock (host->now_ns).
    • presents.csv, cpu.csv (process CPU and memory every second) and events.txt (each long virtual-core wait, with its time).
    • process.json (switches; paths outside the game folder are left out) and longshot_options.ini.
    • In the log: the busiest threads and thread CPU every 5 s, HOS_FPS_LOG, RX_VK_PASS_TIMING=1 and RX_VK_SLOW_LOG=50.
    • A switch that is already set keeps its value.
  • The samplers (host/posix/prof.c, host/mkw_glue.c) take their thread list again every 5 s in file mode too, as the slow-frame ring already did. They write through hos_sample_write.
  • HOS_PROFILE=<seconds> (the busiest threads' log alone) is now HOS_PROFILE_EVERY. An old HOS_PROFILE=5 turns the whole mode on and says so in the log. HOS_RECORD_STUTTERS only logs a note naming HOS_PROFILE=1.
  • scripts/profile_report.py <folder> writes report.md:
    • at a glance: fps and frame times, what limits the frame, the hottest code on the serial threads, what stutters are spent on and what runs more in them;
    • over time, the frame trace's parts, threads, CPU by function, the stutters one by one, and GPU pass timing.
  • scripts/profile_trace.py (also run by the report) writes trace.json.gz for Perfetto (a flame chart per thread over time, frames, stutters, virtual-core waits, counters) and speedscope.json.
  • scripts/samples.py reads both sample formats. crit.py, hotlist.py, vkprof.py, chain.py and cpu_report.py use it.
  • docs: docs/development.md has a new "The profiling mode" section and rows for the switches; docs/performance-report.md gives players the one line; README, CONTRIBUTING.md and cpu-profile.md are updated.
  • tests:
    • tests/test_profiling_mode.py (new) compiles the packed writer, reads it back, and checks the report and the trace on a made-up folder. It also has source checks.
    • test_frame_trace_waits.py covers at_ns; test_slow_frames.py and test_vcore_escape.py are updated.

Measurements

Nothing to measure as a gain: a diagnostic, off by default.

Flags and settings

Adds the profiling mode and renames the busiest-threads log; both are off by default.

Flag Before this PR After this PR Effect Where it helps Recommendation
HOS_PROFILE =<seconds>: the busiest threads' log =1: the profiling mode, off by default every profiling tool, into logs/profile-<date>-<time>/ finding what to optimize; a player's stutter report off (costs frame rate)
HOS_PROFILE_EVERY - off; 5 with HOS_PROFILE=1 the busiest threads every n s (what HOS_PROFILE=<n> did) - off
HOS_SAMPLE_PACKED - off; on with HOS_PROFILE=1 each distinct stack written once in HOS_SAMPLE_ALL's file long sampled runs as the mode sets it
HOS_RECORD_STUTTERS the slow-frame recorder and the frame trace removed (logs a note) - - HOS_PROFILE=1
HOS_PROFILE_DIR - set by the mode (not from longshot_env.txt) the renderer's frames.csv goes into the folder - -

Testing

Tested: Linux, RTX 3090, NVIDIA 620.70.06, CachyOS kernel 7.3.3. A pack over the base ac778751, run with scripts/memtour.py --repo-save kokiri-start --dests 13 --dwell 10 --env HOS_PROFILE=1 (headless).

  • The folder held all of its files, with every thread sampled: the loaders, mkw_vk_* and the rest.
  • profile_report.py read 96 s of play (6,246 frames, 65 fps; 28 frames of 50 ms or more, around the Zora's Domain load) in 29 s.
  • The stutters were the main thread blocked while the renderer's workers created pipelines, and the loaders. 32 of the 79 long vcore waits fell in a stutter.
  • trace.json.gz (14 MB) opened in Perfetto with the thread, frame, wait and counter tracks.

Not tested:

  • python -I -B tests/run_all.py: not run on this PC (no test suites run on it); left to CI.
  • Windows: not built or run. The Windows sampler's change mirrors the Linux one, and the Windows-only parts of profiling.c (GetProcessTimes, K32GetProcessMemoryInfo, _putenv) haven't been compiled.

Must be tested:

  • By hand, Windows, any GPU: a run with HOS_PROFILE=1 in longshot_env.txt, expecting a logs/profile-... folder whose all.txt names the game's and renderer's threads, and that scripts/profile_report.py reads (with --maps to the build's maps).

Risks

What could break:

  • Scripts and docs that set HOS_PROFILE=<seconds> now get the whole mode, which costs frame rate. The log says so.
  • The frame trace gains a last column, at_ns.
  • The samplers take their thread list again every 5 s when writing to a file.

Turning it off: leave HOS_PROFILE unset (the default).

Compatibility: longshot_frames.csv gains a column at its end. HOS_SAMPLE_ALL's plain format is unchanged unless HOS_SAMPLE_PACKED is set. No cache, save or options file changes.

Issues

Prior work: #1218 (the slow-frame recorder, the frame trace's main-thread columns, the vcore wait log), which this builds on; #928 (the Breakdown); HOS_SAMPLE_ALL and scripts/cpu_report.py.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Not run on this PC (no test suites run on it); left to CI.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Switches documented (docs/development.md) and shipped at their recommended value.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** low · **Tested on:** Linux, RTX 3090 (NVIDIA 620.70.06) · **Needs:** a Windows run of `HOS_PROFILE=1` ## Why Finding what to optimize next, or what a stutter was, took several switches set by hand, files in several places, and samplers started and stopped at the right moment. Most of those files had no common clock, so a stutter could not be matched to what the threads were doing. This PR makes it one switch, `HOS_PROFILE=1`, and adds one command that reads the result into a report and a profiler's timeline. It replaces `HOS_RECORD_STUTTERS` (#1218). ## What changed One switch records a whole run into a folder of its own, and one script reads it: a markdown report of where the time goes and what each stutter was, a Perfetto timeline of every thread, and speedscope flame graphs. - **`HOS_PROFILE=1`** (`host/profiling.c`; allowed in `longshot_env.txt`). It writes `logs/profile-<date>-<time>/`, a new folder each run: - `all.txt`: every game and renderer thread's stack every 2 ms, from the start. Packed (`HOS_SAMPLE_PACKED`): each distinct stack is written once, about a tenth of the plain format's size. - `frames.csv`: the frame trace, with a new last column `at_ns`, the present on the host's clock (`host->now_ns`). - `presents.csv`, `cpu.csv` (process CPU and memory every second) and `events.txt` (each long virtual-core wait, with its time). - `process.json` (switches; paths outside the game folder are left out) and `longshot_options.ini`. - In the log: the busiest threads and thread CPU every 5 s, `HOS_FPS_LOG`, `RX_VK_PASS_TIMING=1` and `RX_VK_SLOW_LOG=50`. - A switch that is already set keeps its value. - **The samplers** (`host/posix/prof.c`, `host/mkw_glue.c`) take their thread list again every 5 s in file mode too, as the slow-frame ring already did. They write through `hos_sample_write`. - **`HOS_PROFILE=<seconds>`** (the busiest threads' log alone) is now `HOS_PROFILE_EVERY`. An old `HOS_PROFILE=5` turns the whole mode on and says so in the log. `HOS_RECORD_STUTTERS` only logs a note naming `HOS_PROFILE=1`. - **`scripts/profile_report.py <folder>`** writes `report.md`: - at a glance: fps and frame times, what limits the frame, the hottest code on the serial threads, what stutters are spent on and what runs more in them; - over time, the frame trace's parts, threads, CPU by function, the stutters one by one, and GPU pass timing. - **`scripts/profile_trace.py`** (also run by the report) writes `trace.json.gz` for Perfetto (a flame chart per thread over time, frames, stutters, virtual-core waits, counters) and `speedscope.json`. - **`scripts/samples.py`** reads both sample formats. `crit.py`, `hotlist.py`, `vkprof.py`, `chain.py` and `cpu_report.py` use it. - **docs:** `docs/development.md` has a new "The profiling mode" section and rows for the switches; `docs/performance-report.md` gives players the one line; README, CONTRIBUTING.md and cpu-profile.md are updated. - **tests:** - `tests/test_profiling_mode.py` (new) compiles the packed writer, reads it back, and checks the report and the trace on a made-up folder. It also has source checks. - `test_frame_trace_waits.py` covers `at_ns`; `test_slow_frames.py` and `test_vcore_escape.py` are updated. ## Measurements Nothing to measure as a gain: a diagnostic, off by default. ### Flags and settings Adds the profiling mode and renames the busiest-threads log; both are off by default. | Flag | Before this PR | After this PR | Effect | Where it helps | Recommendation | |---|---|---|---|---|---| | `HOS_PROFILE` | `=<seconds>`: the busiest threads' log | `=1`: the profiling mode, off by default | every profiling tool, into `logs/profile-<date>-<time>/` | finding what to optimize; a player's stutter report | off (costs frame rate) | | `HOS_PROFILE_EVERY` | - | off; 5 with `HOS_PROFILE=1` | the busiest threads every n s (what `HOS_PROFILE=<n>` did) | - | off | | `HOS_SAMPLE_PACKED` | - | off; on with `HOS_PROFILE=1` | each distinct stack written once in `HOS_SAMPLE_ALL`'s file | long sampled runs | as the mode sets it | | `HOS_RECORD_STUTTERS` | the slow-frame recorder and the frame trace | removed (logs a note) | - | - | `HOS_PROFILE=1` | | `HOS_PROFILE_DIR` | - | set by the mode (not from `longshot_env.txt`) | the renderer's `frames.csv` goes into the folder | - | - | ## Testing **Tested:** Linux, RTX 3090, NVIDIA 620.70.06, CachyOS kernel 7.3.3. A pack over the base ac778751, run with `scripts/memtour.py --repo-save kokiri-start --dests 13 --dwell 10 --env HOS_PROFILE=1` (headless). - The folder held all of its files, with every thread sampled: the loaders, `mkw_vk_*` and the rest. - `profile_report.py` read 96 s of play (6,246 frames, 65 fps; 28 frames of 50 ms or more, around the Zora's Domain load) in 29 s. - The stutters were the main thread blocked while the renderer's workers created pipelines, and the loaders. 32 of the 79 long vcore waits fell in a stutter. - `trace.json.gz` (14 MB) opened in Perfetto with the thread, frame, wait and counter tracks. **Not tested:** - `python -I -B tests/run_all.py`: not run on this PC (no test suites run on it); left to CI. - Windows: not built or run. The Windows sampler's change mirrors the Linux one, and the Windows-only parts of `profiling.c` (`GetProcessTimes`, `K32GetProcessMemoryInfo`, `_putenv`) haven't been compiled. **Must be tested:** - By hand, Windows, any GPU: a run with `HOS_PROFILE=1` in `longshot_env.txt`, expecting a `logs/profile-...` folder whose `all.txt` names the game's and renderer's threads, and that `scripts/profile_report.py` reads (with `--maps` to the build's maps). ## Risks **What could break:** - Scripts and docs that set `HOS_PROFILE=<seconds>` now get the whole mode, which costs frame rate. The log says so. - The frame trace gains a last column, `at_ns`. - The samplers take their thread list again every 5 s when writing to a file. **Turning it off:** leave `HOS_PROFILE` unset (the default). **Compatibility:** `longshot_frames.csv` gains a column at its end. `HOS_SAMPLE_ALL`'s plain format is unchanged unless `HOS_SAMPLE_PACKED` is set. No cache, save or options file changes. ## Issues Prior work: #1218 (the slow-frame recorder, the frame trace's main-thread columns, the vcore wait log), which this builds on; #928 (the Breakdown); `HOS_SAMPLE_ALL` and `scripts/cpu_report.py`. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). Not run on this PC (no test suites run on it); left to CI. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Switches documented (docs/development.md) and shipped at their recommended value. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(profiling): HOS_PROFILE=1, one switch for every profiling tool, into a folder of its own, and scripts/profile_report.py to read it
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Failing after 11s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 17s
Website / Website on Cloudflare (pull_request_target) Has been skipped
f8124bb97e
HOS_PROFILE=1 records a whole run into logs/profile-<date>-<time>/: every
thread's stack every 2 ms (packed: each distinct stack once), the frame trace
with the host's clock (at_ns), the presents, CPU and memory every second, and
the long waits for a virtual core with their time; and in the log the busiest
threads, thread CPU, the frame limiter, GPU pass timing and slow frames.
scripts/profile_report.py reads it into report.md (frame times, what limits
the frame, hot code by thread, each stutter's causes), trace.json.gz for
Perfetto and speedscope.json. The old HOS_PROFILE=<seconds> is
HOS_PROFILE_EVERY; HOS_RECORD_STUTTERS is gone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Owner

Tested in game: Linux, RTX 3090, NVIDIA 620.70.06, CachyOS: HOS_PROFILE=1 with scripts/memtour.py (kokiri-start, teleport to Zora's Domain, 10 s dwell, headless). The profile folder held every file, with every game and renderer thread sampled; profile_report.py read 96 s of play (6,246 frames, 65 fps, 28 frames over 50 ms around the load) and its trace.json.gz opened in Perfetto.

Tested in game: Linux, RTX 3090, NVIDIA 620.70.06, CachyOS: HOS_PROFILE=1 with scripts/memtour.py (kokiri-start, teleport to Zora's Domain, 10 s dwell, headless). The profile folder held every file, with every game and renderer thread sampled; profile_report.py read 96 s of play (6,246 frames, 65 fps, 28 frames over 50 ms around the load) and its trace.json.gz opened in Perfetto.
Collaborator

Possibly related, for context:

  • PR #1044 WIP: fix(profiling): the Windows all-thread sampler stamps each sample with its own time under HOS_CHAIN — Overlapping Windows all-thread sampler changes; confirm compatibility
Possibly related, for context: - PR #1044 WIP: fix(profiling): the Windows all-thread sampler stamps each sample with its own time under HOS_CHAIN — Overlapping Windows all-thread sampler changes; confirm compatibility
HumanBean deleted branch feat/profile-mode 2026-11-17 00:14:44 +00:00
Member

Tested on Windows, the "Must be tested" item: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600, main @ 8e9cfc3a (this PR's merge commit, the same commit as nightly-397). The build was scripts/dev_build.py base, so the Windows-only parts of host/profiling.c compile with llvm-mingw. The run was longshot_ctl.py launch --headless --profile test --repo-save kokiri-start --env HOS_PROFILE=1, about 11 minutes. The control client never got the game past the menu here, so it is the menu at ~226 fps plus the start: no teleports. That is a harness issue on my side, not this PR.

Works on Windows:

  • logs/profile-<date>-<time>/ holds every file: all.txt (packed), frames.csv with at_ns, presents.csv, cpu.csv, events.txt (24 vcore waits), process.json (paths outside the game folder left out) and longshot_options.ini. The log names the folder.
  • all.txt names the game's and the renderer's threads (sead::MainThread, mkw_vk_replay, mkw_vk_recorder, Presentation_Thread, ARes_Loading_* and the rest; 60+ of them). It starts with 1 thread and picks up the rest as they appear (the 5 s retake).
  • profile_report.py reads it with the build's longshot.map / longshot_vk.map found by default, and writes report.md, trace.json.gz and speedscope.json. The GPU pass timing block comes through too.

Found on the way (not blockers; I'll take the first two as follow-ups unless someone has them):

  1. main_cpu_ms is per-tick on Windows (from #1218, read by this report). GetThreadTimes advances in scheduler ticks, so in 154,200 frames the column took only 0, 15.6 and 31.2 ms (139,570 / 14,613 / 16 frames). Per frame, main CPU + blocked comes out larger than the frame: steady frames average 1.48 + 3.36 ms in 4.39 ms frames, and the stutters show 358.9 ms of CPU in 164 ms frames. The report's "main thread on the CPU / blocked / off the CPU" split can't be trusted on Windows until that is a real per-frame time (e.g. QueryThreadCycleTime, scaled by a measured TSC rate).
  2. Two report.md table issues:
    • Threads split in two: the Threads table matches the log's prof: threads in names (mkw_vk replay) against the samples' names (mkw_vk_replay, spaces made _ by both samplers), so each such thread gets two rows: one with the CPU %, one with the samples. That likely happens on Linux too.
    • Unreadable names: on Windows the llvm-mingw maps hold Itanium names (_ZN5mkwvk12_GLOBAL__N_14waitEyy), and prof.demangle only knows MSVC's, so the C++ frames stay mangled.
  3. Size: this run's all.txt is 494 MB for ~11 min (about 45 MB a minute with every thread at 2 ms, packed), and the report took 278 s and wrote a 76 MB trace. It might be worth a line in docs/performance-report.md asking players for a few minutes around the problem rather than a whole session.
  4. No log in harness runs: with longshot_ctl.py the game's log goes to the run folder, not logs/longshot_log.txt, so the report finds no log. The Threads CPU % and GPU sections come out empty unless the log is copied in as game.log. Players' runs aren't affected.
Tested on Windows, the "Must be tested" item: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600, main @ 8e9cfc3a (this PR's merge commit, the same commit as nightly-397). The build was `scripts/dev_build.py base`, so the Windows-only parts of `host/profiling.c` compile with llvm-mingw. The run was `longshot_ctl.py launch --headless --profile test --repo-save kokiri-start --env HOS_PROFILE=1`, about 11 minutes. The control client never got the game past the menu here, so it is the menu at ~226 fps plus the start: no teleports. That is a harness issue on my side, not this PR. **Works on Windows:** - `logs/profile-<date>-<time>/` holds every file: `all.txt` (packed), `frames.csv` with `at_ns`, `presents.csv`, `cpu.csv`, `events.txt` (24 vcore waits), `process.json` (paths outside the game folder left out) and `longshot_options.ini`. The log names the folder. - `all.txt` names the game's and the renderer's threads (`sead::MainThread`, `mkw_vk_replay`, `mkw_vk_recorder`, `Presentation_Thread`, `ARes_Loading_*` and the rest; 60+ of them). It starts with 1 thread and picks up the rest as they appear (the 5 s retake). - `profile_report.py` reads it with the build's `longshot.map` / `longshot_vk.map` found by default, and writes `report.md`, `trace.json.gz` and `speedscope.json`. The GPU pass timing block comes through too. **Found on the way (not blockers; I'll take the first two as follow-ups unless someone has them):** 1. **`main_cpu_ms` is per-tick on Windows** (from #1218, read by this report). `GetThreadTimes` advances in scheduler ticks, so in 154,200 frames the column took only 0, 15.6 and 31.2 ms (139,570 / 14,613 / 16 frames). Per frame, main CPU + blocked comes out larger than the frame: steady frames average 1.48 + 3.36 ms in 4.39 ms frames, and the stutters show 358.9 ms of CPU in 164 ms frames. The report's "main thread on the CPU / blocked / off the CPU" split can't be trusted on Windows until that is a real per-frame time (e.g. `QueryThreadCycleTime`, scaled by a measured TSC rate). 2. **Two `report.md` table issues:** - **Threads split in two:** the Threads table matches the log's `prof: threads in` names (`mkw_vk replay`) against the samples' names (`mkw_vk_replay`, spaces made `_` by both samplers), so each such thread gets two rows: one with the CPU %, one with the samples. That likely happens on Linux too. - **Unreadable names:** on Windows the llvm-mingw maps hold Itanium names (`_ZN5mkwvk12_GLOBAL__N_14waitEyy`), and `prof.demangle` only knows MSVC's, so the C++ frames stay mangled. 3. **Size:** this run's `all.txt` is 494 MB for ~11 min (about 45 MB a minute with every thread at 2 ms, packed), and the report took 278 s and wrote a 76 MB trace. It might be worth a line in `docs/performance-report.md` asking players for a few minutes around the problem rather than a whole session. 4. **No log in harness runs:** with `longshot_ctl.py` the game's log goes to the run folder, not `logs/longshot_log.txt`, so the report finds no log. The Threads CPU % and GPU sections come out empty unless the log is copied in as `game.log`. Players' runs aren't affected.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1279
No description provided.