fix(profiling): a Windows profile names its C++ code, counts its waits as waits, and gives a thread one row #1321

Open
madebyjordan wants to merge 2 commits from madebyjordan/Longshot:fix/profile-report-names into main
Member

Type: fix · Risk: low · Tested on: Windows 11, RX 6700 XT · Needs: none

Why

On Windows, the profiling mode's report (#1279, HOS_PROFILE=1) mixes two kinds of names, and that puts wrong numbers in three places (found testing #1279 on Windows, #1279 (comment)):

  • Unreadable C++ names: the Windows build's linker maps (llvm-mingw) hold Itanium names, but prof.demangle only knew MSVC's. The report's hot code showed _ZN5mkwvk12_GLOBAL__N_14waitEyy where the Linux report shows mkwvk::(anonymous namespace)::wait(...).
  • Waits counted as work: a sample counts as "running" unless its innermost Longshot frame ends with a name in WAIT_FRAMES. Mangled names never matched, and on either system a C++ name with its parameters (...condition_variable::wait(std::__1::unique_lock<...>&), as nm -C gives it) didn't either. The host's sleeps now block in sleep_away, which wasn't in the list. So threads asleep all run long came out running 99-100 % of the time (the renderer's replay and recorder threads, the vibration thread), and the main thread 63 %.
  • One thread, two rows: the Threads table matched the log's prof: threads in names (mkw_vk replay) against the samplers' (mkw_vk_replay). Each thread with a space in its name got one row with its CPU % and another with its samples.

How to reproduce

Seen on: Windows 11, RX 6700 XT, main @ 8e9cfc3a (#1279's merge commit).
Steps: run the game with HOS_PROFILE=1 (a minute is enough), then python -I scripts/profile_report.py <game folder>/logs/profile-<date>-<time> --no-trace.
Expected: the C++ frames named as on Linux (mkwvk::...), a thread asleep in a wait not counted as running, one Threads row a thread.
Actual: _ZN... names; the renderer's replay and recorder threads running 99-100 % of their samples while the log gives them ~30-35 % of a core; mkw_vk replay and mkw_vk_replay on two rows.

What changed

The report now names a Windows build's C++ code the way the Linux report does, so its waits are recognised and a sleeping thread no longer counts as busy. Each thread also gets a single row in the Threads table.

  • scripts/prof.py:
    • load_map demangles a map's Itanium names (_Z...) as nm -C does on Linux (demangle_all). The demangler is llvm-cxxfilt or c++filt (demangler()): from the PATH, LONGSHOT_MINGW, or the llvm-mingw of a build tools folder on the PC (scripts/dev_build.py's, the Longshot Launcher's). A name that isn't Itanium (C, MSVC's, a lifted import's imp__ZN..., which scripts/cpu_report.py reads) stays as it is, and so does every name when there is no demangler. crit.py, hotlist.py, vkprof.py and chain.py read maps through it too.
    • waiting() compares a frame's name without its parameter list (bare()).
    • WAIT_FRAMES gains sleep_away (host/hos_thread.c: where hos_sleep_ns and the guest's sleeps block), libc++'s __libcpp_condvar_wait / __libcpp_condvar_timedwait, and the renderer's wait for a GPU fence (mkwvk::(anonymous namespace)::wait, vk_replay.cpp).
  • scripts/profile_report.py: the log's thread names are written as the samplers write them (spaces and | made _), so a thread has one row.
  • tests/test_profile_report_names.py (new; the first commit, failing on main for each of the three): a made-up Windows map's names, the waits, and a made-up profile folder's Threads table. Without a demangler on the PC, the demangling part says so and is skipped.

Measurements

Verdict: the report's running shares now agree with the measured CPU.

Hardware: Windows 11, Ryzen 5 5600 (6 cores), 16 GB, RX 6700 XT, AMD 26.10.1
Scene: the menu after the start, headless, HOS_PROFILE=1, ~11 minutes (a game built from main @ 8e9cfc3a)
Method: one profile folder read by scripts/profile_report.py on each side (the report is deterministic: one run each); the log's CPU is its prof: threads in 5 s lines in play
How to reproduce: python -I scripts/profile_report.py <profile folder> --no-trace on main and on this PR, then the report's Threads table and its At a glance lines

Before: main @ 6d72ca93 After: this PR @ 5506d389
sead::MainThread running (the log: ~33 % of a core) 63 % 28 %
mkw_vk_replay running (the log: ~35 %) 99 % (its own row; CPU % on another) 41 % (one row)
mkw_vk_recorder running (the log: ~30-35 %) 100 % 37 %
main thread's hottest code longshot:sleep_away 27.6 %, _ZN5mkwvk12_GLOBAL__N_14waitEyy 5.9 % longshot:spin_until_set 3.2 %, mkwvk::guarded_copy(void*, void const*, unsigned long long) 3.0 %

Flags and settings

None new. To find a demangler, scripts/profile_report.py also reads LONGSHOT_MINGW, which already exists (docs/development.md, the llvm-mingw to build with); nothing sets or changes it.

Testing

Tested:

  • The profile above, read on main and on this PR (the table): one Threads row a thread, C++ names demangled, waits out of the running shares.
  • tests/test_profile_report_names.py fails on main for each of its three parts (demangling, with llvm-cxxfilt on the PATH; the waits; the Threads rows) and passes here, with a demangler and without one (LOCALAPPDATA hidden: the demangling part skipped). tests/test_prof_symbols.py and tests/test_profile_stream.py pass.
  • python -I -B tests/run_all.py on Windows: 589 passed, 42 failed, 90 skipped; main @ 6d72ca93 on the same PC at the
    same time: 588 passed and the same 42 failed (test_auto_handheld, test_bug_report, test_bug_report_defaults, test_cpu_report, test_crash_dialog, test_crash_groups, test_crt_options, test_damage_debug_menu, test_display_startup_failure, test_dlss_feature_size, test_failed_compile_keeps_objects, test_framegen_preset, test_guest_imported, test_hardware_report, test_hud_layout_row, test_invalidate_vcore, test_job_queue, test_linux_preflight, test_linux_pressure, test_mirror_budget, test_missing_nup, test_pads_after_menu, test_pending_guest_import, test_perf_report, test_pipeline_play_cache, test_portable_builder_home, test_portable_builder_startup, test_present_thread, test_profiling_mode, test_readback_timeout_source, test_rumble_menu, test_run_all_changed, test_slow_frames, test_switch_table, test_target_image_pool, test_tested_configurations, test_upscaler_round_trip, test_upscaler_rows, test_vcore_escape, test_vk_occlusion_queries, test_vk_swapchain_compat, test_walk_key).

Not tested: Linux, where nm -C names the frames already. The waits now match there too, so a Linux report's running shares drop the same way; that's the intent of WAIT_FRAMES. TrashPanda asked for that Linux read (review on 5506d389): the same profile folder read on main and on this PR, with the waits out of the running shares and one row a thread. It's still to do.

Must be tested:

  • Linux, any GPU (by whoever reads a Linux profile next; the Test Farm can't read a report): python -I scripts/profile_report.py <a HOS_PROFILE=1 folder> --no-trace on main and on this PR, expecting the C++ waits (std::...::condition_variable::wait, sleep_away) out of the running shares and one Threads row a thread

Risks

What could break: a script that matched a mangled _Z... name read through prof.load_map (none in the repository: cpu_report.py has its own map reader and matches imp__ZN...). Running shares in reports go down where threads were asleep.

Turning it off: revert the commit.

Compatibility: scripts only; no game, cache or file format change.

Issues

Found testing #1279 on Windows (its "Must be tested"). Prior work: none open (searched demangle, profile_report, GetThreadTimes, thread names).

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, RX 6700 XT · **Needs:** none ## Why On Windows, the profiling mode's report (#1279, `HOS_PROFILE=1`) mixes two kinds of names, and that puts wrong numbers in three places (found testing #1279 on Windows, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1279#issuecomment-40075): - **Unreadable C++ names:** the Windows build's linker maps (llvm-mingw) hold Itanium names, but `prof.demangle` only knew MSVC's. The report's hot code showed `_ZN5mkwvk12_GLOBAL__N_14waitEyy` where the Linux report shows `mkwvk::(anonymous namespace)::wait(...)`. - **Waits counted as work:** a sample counts as "running" unless its innermost Longshot frame ends with a name in `WAIT_FRAMES`. Mangled names never matched, and on either system a C++ name with its parameters (`...condition_variable::wait(std::__1::unique_lock<...>&)`, as `nm -C` gives it) didn't either. The host's sleeps now block in `sleep_away`, which wasn't in the list. So threads asleep all run long came out running 99-100 % of the time (the renderer's replay and recorder threads, the vibration thread), and the main thread 63 %. - **One thread, two rows:** the Threads table matched the log's `prof: threads in` names (`mkw_vk replay`) against the samplers' (`mkw_vk_replay`). Each thread with a space in its name got one row with its CPU % and another with its samples. ## How to reproduce **Seen on:** Windows 11, RX 6700 XT, main @ 8e9cfc3a (#1279's merge commit). **Steps:** run the game with `HOS_PROFILE=1` (a minute is enough), then `python -I scripts/profile_report.py <game folder>/logs/profile-<date>-<time> --no-trace`. **Expected:** the C++ frames named as on Linux (`mkwvk::...`), a thread asleep in a wait not counted as running, one Threads row a thread. **Actual:** `_ZN...` names; the renderer's replay and recorder threads running 99-100 % of their samples while the log gives them ~30-35 % of a core; `mkw_vk replay` and `mkw_vk_replay` on two rows. ## What changed The report now names a Windows build's C++ code the way the Linux report does, so its waits are recognised and a sleeping thread no longer counts as busy. Each thread also gets a single row in the Threads table. - `scripts/prof.py`: - `load_map` demangles a map's Itanium names (`_Z...`) as `nm -C` does on Linux (`demangle_all`). The demangler is `llvm-cxxfilt` or `c++filt` (`demangler()`): from the PATH, `LONGSHOT_MINGW`, or the llvm-mingw of a build tools folder on the PC (`scripts/dev_build.py`'s, the Longshot Launcher's). A name that isn't Itanium (C, MSVC's, a lifted import's `imp__ZN...`, which `scripts/cpu_report.py` reads) stays as it is, and so does every name when there is no demangler. `crit.py`, `hotlist.py`, `vkprof.py` and `chain.py` read maps through it too. - `waiting()` compares a frame's name without its parameter list (`bare()`). - `WAIT_FRAMES` gains `sleep_away` (host/hos_thread.c: where `hos_sleep_ns` and the guest's sleeps block), libc++'s `__libcpp_condvar_wait` / `__libcpp_condvar_timedwait`, and the renderer's wait for a GPU fence (`mkwvk::(anonymous namespace)::wait`, vk_replay.cpp). - `scripts/profile_report.py`: the log's thread names are written as the samplers write them (spaces and `|` made `_`), so a thread has one row. - `tests/test_profile_report_names.py` (new; the first commit, failing on main for each of the three): a made-up Windows map's names, the waits, and a made-up profile folder's Threads table. Without a demangler on the PC, the demangling part says so and is skipped. ## Measurements **Verdict:** the report's running shares now agree with the measured CPU. Hardware: Windows 11, Ryzen 5 5600 (6 cores), 16 GB, RX 6700 XT, AMD 26.10.1 Scene: the menu after the start, headless, `HOS_PROFILE=1`, ~11 minutes (a game built from main @ 8e9cfc3a) Method: one profile folder read by `scripts/profile_report.py` on each side (the report is deterministic: one run each); the log's CPU is its `prof: threads in 5 s` lines in play How to reproduce: `python -I scripts/profile_report.py <profile folder> --no-trace` on main and on this PR, then the report's Threads table and its At a glance lines | | Before: main @ 6d72ca93 | After: this PR @ 5506d389 | |---|---|---| | sead::MainThread running (the log: ~33 % of a core) | 63 % | 28 % | | mkw_vk_replay running (the log: ~35 %) | 99 % (its own row; CPU % on another) | 41 % (one row) | | mkw_vk_recorder running (the log: ~30-35 %) | 100 % | 37 % | | main thread's hottest code | `longshot:sleep_away` 27.6 %, `_ZN5mkwvk12_GLOBAL__N_14waitEyy` 5.9 % | `longshot:spin_until_set` 3.2 %, `mkwvk::guarded_copy(void*, void const*, unsigned long long)` 3.0 % | ### Flags and settings None new. To find a demangler, `scripts/profile_report.py` also reads `LONGSHOT_MINGW`, which already exists (`docs/development.md`, the llvm-mingw to build with); nothing sets or changes it. ## Testing **Tested:** - The profile above, read on main and on this PR (the table): one Threads row a thread, C++ names demangled, waits out of the running shares. - `tests/test_profile_report_names.py` fails on main for each of its three parts (demangling, with `llvm-cxxfilt` on the PATH; the waits; the Threads rows) and passes here, with a demangler and without one (`LOCALAPPDATA` hidden: the demangling part skipped). `tests/test_prof_symbols.py` and `tests/test_profile_stream.py` pass. - `python -I -B tests/run_all.py` on Windows: 589 passed, 42 failed, 90 skipped; main @ 6d72ca93 on the same PC at the same time: 588 passed and the same 42 failed (`test_auto_handheld`, `test_bug_report`, `test_bug_report_defaults`, `test_cpu_report`, `test_crash_dialog`, `test_crash_groups`, `test_crt_options`, `test_damage_debug_menu`, `test_display_startup_failure`, `test_dlss_feature_size`, `test_failed_compile_keeps_objects`, `test_framegen_preset`, `test_guest_imported`, `test_hardware_report`, `test_hud_layout_row`, `test_invalidate_vcore`, `test_job_queue`, `test_linux_preflight`, `test_linux_pressure`, `test_mirror_budget`, `test_missing_nup`, `test_pads_after_menu`, `test_pending_guest_import`, `test_perf_report`, `test_pipeline_play_cache`, `test_portable_builder_home`, `test_portable_builder_startup`, `test_present_thread`, `test_profiling_mode`, `test_readback_timeout_source`, `test_rumble_menu`, `test_run_all_changed`, `test_slow_frames`, `test_switch_table`, `test_target_image_pool`, `test_tested_configurations`, `test_upscaler_round_trip`, `test_upscaler_rows`, `test_vcore_escape`, `test_vk_occlusion_queries`, `test_vk_swapchain_compat`, `test_walk_key`). **Not tested:** Linux, where `nm -C` names the frames already. The waits now match there too, so a Linux report's running shares drop the same way; that's the intent of `WAIT_FRAMES`. TrashPanda asked for that Linux read (review on 5506d389): the same profile folder read on main and on this PR, with the waits out of the running shares and one row a thread. It's still to do. **Must be tested:** - Linux, any GPU (by whoever reads a Linux profile next; the Test Farm can't read a report): `python -I scripts/profile_report.py <a HOS_PROFILE=1 folder> --no-trace` on main and on this PR, expecting the C++ waits (`std::...::condition_variable::wait`, `sleep_away`) out of the running shares and one Threads row a thread ## Risks **What could break:** a script that matched a mangled `_Z...` name read through `prof.load_map` (none in the repository: `cpu_report.py` has its own map reader and matches `imp__ZN...`). Running shares in reports go down where threads were asleep. **Turning it off:** revert the commit. **Compatibility:** scripts only; no game, cache or file format change. ## Issues Found testing #1279 on Windows (its "Must be tested"). Prior work: none open (searched `demangle`, `profile_report`, `GetThreadTimes`, `thread names`). ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
tests/test_profile_report_names.py: a made-up Windows linker map's Itanium names (llvm-mingw) come out of prof.load_map
as nm -C gives them on Linux, a plain C name and a lifted import's name unchanged; a sample asleep in a wait with a C++
name's parameters (std::condition_variable::wait, libc++'s timed wait, the renderer's GPU wait) or in host/hos_thread.c's
sleep_away counts as waiting; and profile_report.py's Threads table puts the log's CPU % ("mkw_vk replay") on the
samples' row ("mkw_vk_replay"). Fails on main for each of the three.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(profiling): a Windows profile names its C++ code, counts its waits as waits, and gives a thread one row
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Successful in 12s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 3s
5506d38928
- scripts/prof.py: load_map demangles the Itanium names of a Windows build's linker maps (llvm-mingw), as nm -C does
  for the Linux build, with llvm-cxxfilt or c++filt (the PATH, LONGSHOT_MINGW, or the llvm-mingw of a build tools folder
  on the PC); a name that isn't one (C, MSVC's, a lifted import's imp__ZN...) stays, and so does everything when there is
  no demangler.
- waiting() compares a frame's name without its parameter list (bare()), so a C++ wait named with them, as nm -C and
  the demangling give it, matches WAIT_FRAMES; and WAIT_FRAMES has host/hos_thread.c's sleep_away (where hos_sleep_ns
  and the guest's sleeps block now), libc++'s condition-variable waits and the renderer's wait for a GPU fence.
- profile_report.py: the log's `prof: threads in` names (spaces) are put as the samplers write them (_), so a thread's
  CPU % and its samples share one row of Threads.

On an 11-minute Windows profile (#1279's mode): the main thread's running share 63 -> 28 % (the log: ~33 % CPU), the
replay thread's 99 -> 41 % (~35 %), the recorder's 100 -> 37 %, and the hot lists name mkwvk::... not _ZN5mkwvk...

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TrashPanda left a comment

A Windows profile names its C++ code, counts a wait as a wait, and gives each thread one row. I read head 5506d389. It applies onto current main, and this stays a comment.

A map's Itanium names (_Z...) are demangled the way Linux already names them. A name that is not Itanium is left as it is, and so is every name when there is no demangler. A wait is matched without the parameter list, and the list now includes sleep_away, libc++'s condition waits, and the renderer's fence wait. The log's thread names lose spaces and | the same way the sampler writes them (host/mkw_glue.c). On Windows, an RX 6700 XT, the replay thread's running share goes from 99 % on its own row to 41 % on the row that also has its CPU time, against about 35 % of a core in the log.

The Linux read this asks for is still open: the same profile folder on main and on this branch, with those waits out of the running shares and one row a thread.

A Windows profile names its C++ code, counts a wait as a wait, and gives each thread one row. I read head `5506d389`. It applies onto current main, and this stays a comment. A map's Itanium names (`_Z...`) are demangled the way Linux already names them. A name that is not Itanium is left as it is, and so is every name when there is no demangler. A wait is matched without the parameter list, and the list now includes `sleep_away`, libc++'s condition waits, and the renderer's fence wait. The log's thread names lose spaces and `|` the same way the sampler writes them (`host/mkw_glue.c`). On Windows, an RX 6700 XT, the replay thread's running share goes from 99 % on its own row to 41 % on the row that also has its CPU time, against about 35 % of a core in the log. The Linux read this asks for is still open: the same profile folder on main and on this branch, with those waits out of the running shares and one row a thread.
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Successful in 12s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 3s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/profile-report-names:madebyjordan-fix/profile-report-names'
git switch 'madebyjordan-fix/profile-report-names'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1321
No description provided.