diag(host): HOS_AWAY_LOG_MS logs a serial thread's released-core stretches #1666

Open
Molier wants to merge 1 commit from Molier/Longshot:diag/away-log-ms into main
Contributor

Summary

A diagnostic: HOS_AWAY_LOG_MS=<ms> logs every released-core stretch (hos_vcore_away .. hos_vcore_back) of a
serial thread that lasts at least that many ms, with the stretch's CPU time and the host call site that started it
(away: lines, symbolised module+offset) — the same quantity the frame trace's main_blocked_ms column
already totals, but per occurrence instead of per frame. Off (unset or negative): one cached load and a branch, no
behaviour change. Read first: host/hos_thread.c (away_log_ms, away_cpu_ns, away_log, and the two call sites
in hos_vcore_away/hos_vcore_back).

Testing

Tested: Linux (Ubuntu), host compile clean (-fsyntax-only, host/hos_thread.c; one pre-existing unrelated
warning, unused function 'pick_cache_cores', not from this change). tests/run_all.py away_log:
tests/test_away_log.py passes on the rebased head.

Still to test:

  • Linux or Windows, in the game: HOS_AWAY_LOG_MS=1 during a known stall (an area change, a long load) and confirm
    the away: line's call site matches a thread actually blocked there in a Tracy capture (by hand).

Measurements

Nothing to measure: a diagnostic switch, off by default, with no effect on the game's own behaviour when unset.

Risks

None when unset. When set, extra CPU time (one clock_gettime/GetThreadTimes call and a log line) only on the
stretches it reports; HOS_AWAY_LOG_MS is meant for short diagnostic sessions, not left on.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC (game-free checks; no in-game effect when unset to screenshot).
  • tests/run_all.py away_log passes.
  • Nothing to measure (a diagnostic).
  • No game files, build output or hand-edited generated files; third_party/ untouched.

When it applies

  • Docs updated (HOS_AWAY_LOG_MS row added to docs/dev/reference/switches.md, #1802's location).
  • Host code builds on Linux with UTF-8 paths; Windows llvm-mingw not run here (the #if defined(_WIN32) /
    __linux__ CPU-time split mirrors existing patterns elsewhere in the file).
  • AI assistance recorded with Assisted-by: trailers.

Ready to merge

  • The in-game call-site cross-check (Still to test) has run.
  • Approved in review, every comment answered.
  • CI passes on the head.
## Summary A diagnostic: `HOS_AWAY_LOG_MS=<ms>` logs every released-core stretch (`hos_vcore_away` .. `hos_vcore_back`) of a serial thread that lasts at least that many ms, with the stretch's CPU time and the host call site that started it (`away:` lines, symbolised module+offset) — the same quantity the frame trace's `main_blocked_ms` column already totals, but per occurrence instead of per frame. Off (unset or negative): one cached load and a branch, no behaviour change. Read first: `host/hos_thread.c` (`away_log_ms`, `away_cpu_ns`, `away_log`, and the two call sites in `hos_vcore_away`/`hos_vcore_back`). ## Testing **Tested:** Linux (Ubuntu), host compile clean (`-fsyntax-only`, `host/hos_thread.c`; one pre-existing unrelated warning, `unused function 'pick_cache_cores'`, not from this change). `tests/run_all.py away_log`: `tests/test_away_log.py` passes on the rebased head. **Still to test:** - Linux or Windows, in the game: `HOS_AWAY_LOG_MS=1` during a known stall (an area change, a long load) and confirm the `away:` line's call site matches a thread actually blocked there in a Tracy capture (by hand). ## Measurements Nothing to measure: a diagnostic switch, off by default, with no effect on the game's own behaviour when unset. ## Risks None when unset. When set, extra CPU time (one `clock_gettime`/`GetThreadTimes` call and a log line) only on the stretches it reports; `HOS_AWAY_LOG_MS` is meant for short diagnostic sessions, not left on. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC (game-free checks; no in-game effect when unset to screenshot). - [x] `tests/run_all.py away_log` passes. - [x] Nothing to measure (a diagnostic). - [x] No game files, build output or hand-edited generated files; `third_party/` untouched. ### When it applies - [x] Docs updated (`HOS_AWAY_LOG_MS` row added to `docs/dev/reference/switches.md`, #1802's location). - [x] Host code builds on Linux with UTF-8 paths; Windows llvm-mingw not run here (the `#if defined(_WIN32)` / `__linux__` CPU-time split mirrors existing patterns elsewhere in the file). - [x] AI assistance recorded with `Assisted-by:` trailers. ### Ready to merge - [ ] The in-game call-site cross-check (Still to test) has run. - [ ] Approved in review, every comment answered. - [ ] CI passes on the head.
Molier force-pushed diag/away-log-ms from 5f478884b1
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 4ec23e9629
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 34s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-18 23:46:33 +00:00
Compare
Contributor

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. This PR's head 4ec23e96 and #1667's a871f3ff, both cherry-picked onto their common base 2f1c0c9c, in one test build (longshot abc-probes (source 166c4da4)), with HOS_AWAY_LOG_MS=50 and HOS_PROFILE=1 (which sets RX_VK_FRAME_TRACE=1). No errors, hangs or device losses in any run.

The probe (this PR's Needs item): it logs as described on a real GPU: 72 and 81 away: lines, each with its thread, wall and CPU time and call site, and the call sites resolve against the build's symbols with nm. Three sites cover sead::MainThread:

sead::MainThread's stretches of 50 ms or more, run 1 / run 2 lines longest on a CPU where
n_nvnSyncWait 18 / 19 752 / 409 ms 1.5 / 0.8 ms the area changes before the measured part
n_nvnQueueSubmitCommands 9 / 9 122 / 773 ms 102 / 758 ms 3 of the 4 frames of 100 ms or more in the measured parts
hos_cv_sleep 3 / 2 275 / 129 ms 0.4 / 0.2 ms
  • At n_nvnQueueSubmitCommands the stretch is the submit-time capture: its wall time equals the slow-capture line logged just before it (#1667's probe), for example 773.0 ms against vk: slow capture 773.0 ms.
  • At n_nvnSyncWait the game waits on a GPU sync with almost no CPU, and each long one is most of a slow frame: 751 ms in a 764 ms frame, 408 in 454, 303 in 314, 284 in 323. The Presentation Thread waits through the same stretch in n_nvnQueuePresentTexture (754 ms against the main thread's 752). It isn't GPU execution: the longest submission in the log's GPU time lines is 20.6 and 18.9 ms. Each long wait comes right after a frame of 100 to 300 MB of texture uploads (95 to 379 uploads, many of them 57 MB ASTC textures re-encoded as BC3). In the 752 ms one the replay thread was 73% in programs_load_wait (program_cache_load under program_pick); in most of the others (407, 409, 175 ms) it was idle. So the work the fence waits for is held somewhere before the replay, and where is still open. HOS_NVN_SYNC_LOG should show which fence each wait wants against how far the replay is; that's our next run.
  • Against the frame trace, the stretches don't map one to one onto main_blocked_ms, because a stretch can span a present: the 148 ms frame shows main_blocked_ms 69 against a 122 ms stretch ending in it. Splitting a stretch at the present would make the two comparable.
  • The Presentation Thread's hos_cv_sleep lines (20 and 29, up to 9.9 s at start) are that thread waiting for work, with almost no CPU. A cutoff per thread, or leaving out a known idle wait, would cut that noise.
[hos sead::MainThread] away: sead::MainThread 751.5 ms (on a CPU 1.5 ms) from longshot+0x1f73a4b8, ended at 23510505785078 ns
[hos Presentation Thread] away: Presentation Thread 754.4 ms (on a CPU 1.0 ms) from longshot+0x1f741e17, ended at 23510509034673 ns
[hos sead::MainThread] away: sead::MainThread 69.4 ms (on a CPU 56.8 ms) from longshot+0x1f74147d, ended at 23668421140256 ns

(In this build +0x1f73a4b8 is in n_nvnSyncWait, +0x1f741e17 in n_nvnQueuePresentTexture, +0x1f74147d in n_nvnQueueSubmitCommands.)

Not compared against main: the probe changes nothing while it's off, so these are its readings, not a measurement of this PR.

Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). RX_VK_CAPTURE_SLOW_MS=50 (#1667) in every run.
Method: a scripted route through the control bridge from the repository save kokiri-start: the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps along Link's facing between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 2 runs; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. This PR's head `4ec23e96` and #1667's `a871f3ff`, both cherry-picked onto their common base `2f1c0c9c`, in one test build (`longshot abc-probes (source 166c4da4)`), with `HOS_AWAY_LOG_MS=50` and `HOS_PROFILE=1` (which sets `RX_VK_FRAME_TRACE=1`). No errors, hangs or device losses in any run. **The probe (this PR's Needs item):** it logs as described on a real GPU: 72 and 81 `away:` lines, each with its thread, wall and CPU time and call site, and the call sites resolve against the build's symbols with `nm`. Three sites cover sead::MainThread: | sead::MainThread's stretches of 50 ms or more, run 1 / run 2 | lines | longest | on a CPU | where | |---|---|---|---|---| | `n_nvnSyncWait` | 18 / 19 | 752 / 409 ms | 1.5 / 0.8 ms | the area changes before the measured part | | `n_nvnQueueSubmitCommands` | 9 / 9 | 122 / 773 ms | 102 / 758 ms | 3 of the 4 frames of 100 ms or more in the measured parts | | `hos_cv_sleep` | 3 / 2 | 275 / 129 ms | 0.4 / 0.2 ms | | - **At `n_nvnQueueSubmitCommands` the stretch is the submit-time capture:** its wall time equals the slow-capture line logged just before it (#1667's probe), for example 773.0 ms against `vk: slow capture 773.0 ms`. - **At `n_nvnSyncWait` the game waits on a GPU sync with almost no CPU,** and each long one is most of a slow frame: 751 ms in a 764 ms frame, 408 in 454, 303 in 314, 284 in 323. The Presentation Thread waits through the same stretch in `n_nvnQueuePresentTexture` (754 ms against the main thread's 752). It isn't GPU execution: the longest submission in the log's GPU time lines is 20.6 and 18.9 ms. Each long wait comes right after a frame of 100 to 300 MB of texture uploads (95 to 379 uploads, many of them 57 MB ASTC textures re-encoded as BC3). In the 752 ms one the replay thread was 73% in `programs_load_wait` (`program_cache_load` under `program_pick`); in most of the others (407, 409, 175 ms) it was idle. So the work the fence waits for is held somewhere before the replay, and where is still open. `HOS_NVN_SYNC_LOG` should show which fence each wait wants against how far the replay is; that's our next run. - **Against the frame trace,** the stretches don't map one to one onto `main_blocked_ms`, because a stretch can span a present: the 148 ms frame shows `main_blocked_ms` 69 against a 122 ms stretch ending in it. Splitting a stretch at the present would make the two comparable. - **The Presentation Thread's `hos_cv_sleep` lines** (20 and 29, up to 9.9 s at start) are that thread waiting for work, with almost no CPU. A cutoff per thread, or leaving out a known idle wait, would cut that noise. ``` [hos sead::MainThread] away: sead::MainThread 751.5 ms (on a CPU 1.5 ms) from longshot+0x1f73a4b8, ended at 23510505785078 ns [hos Presentation Thread] away: Presentation Thread 754.4 ms (on a CPU 1.0 ms) from longshot+0x1f741e17, ended at 23510509034673 ns [hos sead::MainThread] away: sead::MainThread 69.4 ms (on a CPU 56.8 ms) from longshot+0x1f74147d, ended at 23668421140256 ns ``` (In this build `+0x1f73a4b8` is in `n_nvnSyncWait`, `+0x1f741e17` in `n_nvnQueuePresentTexture`, `+0x1f74147d` in `n_nvnQueueSubmitCommands`.) Not compared against main: the probe changes nothing while it's off, so these are its readings, not a measurement of this PR. Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). `RX_VK_CAPTURE_SLOW_MS=50` (#1667) in every run. Method: a scripted route through the control bridge from the repository save `kokiri-start`: the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps along Link's facing between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 2 runs; Link's hearts protected (`HOS_OOT_PROTECT_HEARTS=1`).
Molier force-pushed diag/away-log-ms from 4ec23e9629
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 34s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 338d0ab083
Some checks failed
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Failing after 16s
2026-11-20 14:38:06 +00:00
Compare
Molier changed title from WIP: diag(host): HOS_AWAY_LOG_MS logs a serial thread's released-core stretches to diag(host): HOS_AWAY_LOG_MS logs a serial thread's released-core stretches 2026-11-20 14:38:23 +00:00
Author
Contributor

Rebased onto main @ 89c3d21e. One real conflict in host/hos_thread.c (main reworded the neighbouring comment): kept main's wording, this PR's functions unchanged after it. Docs row moved to docs/dev/reference/switches.md (#1802). tests/test_away_log.py passes; host compile clean. Off by default, a diagnostic only, so out of WIP; body rewritten in the new template.

Rebased onto `main` @ `89c3d21e`. One real conflict in `host/hos_thread.c` (main reworded the neighbouring comment): kept main's wording, this PR's functions unchanged after it. Docs row moved to `docs/dev/reference/switches.md` (#1802). `tests/test_away_log.py` passes; host compile clean. Off by default, a diagnostic only, so out of WIP; body rewritten in the new template.
Some checks failed
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Failing after 16s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'diag/away-log-ms:Molier-diag/away-log-ms'
git switch 'Molier-diag/away-log-ms'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1666
No description provided.