MagicBeanDev/Longshot

Watch

Fork 99

perf(vk): the renderer is built with a committed play profile, about 3-5 % less frame time #569

Merged

HumanBean merged 1 commit from HumanBean/Longshot:perf/vk-profile into main

2026-11-14 20:21:40 +00:00

HumanBean commented

2026-11-14 20:04:38 +00:00

Owner

Why

The renderer's replay is the frame's limit in heavy places, and compiling it with a play profile makes it about 3-5 % faster per frame at almost no build cost. Most players build with DLSS or FSR 4, which compile the renderer on their own PC, so the profile is committed in the repo and used by every renderer build rather than only by a prebuilt one.

What changed

build: nx2/build.py, ported from ProjectReLink (GPL-3.0-or-later). RX_VK_PGO=gen builds the back end's own files (gpu/vk/*.cpp and suyu's texture decoders; not Eden's recompiler, sirit, Dear ImGui or zstd) with -fprofile-instr-generate and links the module's own copy of host/pgo_rt.c. RX_VK_PGO=<file> builds them with that profile; its contents go into the flags the object keys hash, so another profile rebuilds those 35 files and nothing else.
host: host/pgo_rt.c reads its variable through PGO_ENV (HOS_PGO for the game module, RX_VK_PGO_FILE for the renderer's copy). On Linux it finds the __llvm_prf_* ELF sections by their __start_/__stop_ symbols; ReLink only had Windows' COFF markers.
assets: assets/profiles/longshot_vk.profdata.xz (160 KB) and assets/profiles/README.md (where, how and on what it was recorded, and what it measured). One profile for every build for now: a Linux recording, Kokiri Forest standing and the Market walk, mesh shaders.
build: nx2/make_game.py vk_profile_for():
- the checkout's work/longshot_vk.profdata when there is one (a developer's recording), else the committed profile, unpacked into the work folder as longshot_vk.shipped.profdata (rewritten only when it changes);
- passed as RX_VK_PGO only when this compiler reads it, and unless RX_VK_PGO is set already;
- profile_readable() now checks on Windows too for this profile (any_os=True): it comes from LLVM 23.1.5, llvm-mingw is another LLVM, and an unreadable profile would stop the build;
- RX_VK_PGO and RX_VK_PGO_FILE are left out of the lift key.
build: scripts/linux/appimage.sh builds the AppImage's renderer with the same profile (the checkout's, else the committed one). The AppImage already carries assets/, so a player's own renderer there (DLSS, FSR 4) gets it too.
build: scripts/dev_packs.py builds a pack of the renderer alone with its base's profile.
docs: docs/development.md ("The renderer's play profile": the committed profile, recording a new one, shipping it; the switch rows), docs/design.md, RELEASING.md (step 0: record it again before a minor's first candidate when the renderer changed much).
tests: tests/test_vk_pgo.py (new: the choice between the checkout's and the committed profile, the unpacking, the opt-outs, the committed file and its note, the writer, the AppImage), tests/test_shader_stamp_cache.py (build.py's real branch with RX_VK_PGO: which files count, the writer, the object keys), tests/test_dlss_sdk_cache.py (the new name in its namespace).

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
`RX_VK_PGO=gen` / `RX_VK_PGO=<file>`	The renderer counting its branches for a profile, or built with that file in place of the committed one	unset: the committed profile, when the compiler reads it	A developer switch, for recording	-	-	Windows
`RX_VK_PGO_FILE=<file>`	Where a counting renderer writes, with `<file>.reset` / `<file>.dump`	unset	A developer switch, for recording	-	-	Windows
the committed profile	Every renderer build compiles with `assets/profiles/longshot_vk.profdata.xz`	on	On: about -3 to -5 % frame time, no build cost to speak of	RTX 3090, Linux, mesh shaders	not measured anywhere else	Windows (llvm-mingw), AMD and Intel GPUs, the compute fallback, the Steam Deck

Measurements

main: 0caf0618 (2026-11-14)
this PR: the profile's first version, build/vk-pgo @ a601a547 on 0caf0618 (the same code as this PR's, before the profile moved to assets/profiles/); rebased since onto main 7ff5256e with no conflict
Hardware: RTX 3090, Ryzen 9 7900X, 32 GB, CachyOS (Linux)
Scene: the Market from a save, standing, then running in the square; 2560x1440, default options
Method: full-length sessions at low priority on a busy PC, interleaved: control, profile, profile, control, control, profile, baseline x86-64, control. The median frame of each phase, from the frame log (RX_VK_FRAME_LOG). Both arms ran with the default reserved core and without ReplayPin in the base. A fourth control in a noisy stretch (14.63 / 17.95 ms) and the single baseline x86-64 session are left out of the table.
How to reproduce: build the renderer with and without RX_VK_PGO=<the profile> over one base (scripts/dev_packs.py), and play the same save.

Median frame, ms	main @ `0caf0618`	this PR's profile @ a601a547
Standing in the Market (sessions)	13.53, 13.49, 13.70	12.96, 13.37, 12.94
Running in the square	15.89, 16.13, 15.73	15.13, 15.67, 15.52
Median	13.53 / 15.89	12.96 / 15.52
Change		about -4 to -5 % standing, -3 % running

The profile was recorded on Kokiri Forest and the Market walk, so the Market numbers are on part of the route it was recorded on. A broader recording (the pipeline tour, PR 111) is to follow.

Testing

Tested: the sessions above, on Linux with the RTX 3090: the counting build, the recording (.reset/.dump), llvm-profdata merge, and the renderer built with the profile all ran, and the game played through the route each time. No local checks run (tests/run_all.py is left to CI).

Not tested: this PR's own path from the committed file: vk_profile_for() unpacking assets/profiles/longshot_vk.profdata.xz and appimage.sh's fallback have not run in a build. Windows: the counting renderer, its writer (ReLink's COFF code) and whether llvm-mingw's clang reads this profile. AMD and Intel GPUs, the compute fallback, the Steam Deck.

Needs other testers:

Windows, any GPU: a build of this PR, expecting the Vulkan back end: built with the profile ...longshot_vk.shipped.profdata in the build log (or this compiler cannot read the play profile, then built without), and the game running (request to open)
Windows: a recording of the pipeline tour with a counting renderer, for a Windows profile (request to open)
Linux AppImage: a build with DLSS or FSR 4, expecting the same log line (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A wrong profile can only make the renderer slower, not wrong: profile-guided optimisation changes layout and inlining, not what the code does.
A compiler that can't read the profile builds the renderer without it, now checked on Windows as well as Linux.
The first build after this rebuilds the renderer's 35 own files once; the lift and the game's objects are untouched.
To turn it off: delete assets/profiles/longshot_vk.profdata.xz, or revert.

Reviewer notes

Start with vk_profile_for() in nx2/make_game.py, then nx2/build.py's RX_VK_PGO branch and host/pgo_rt.c.
The profile's path is in the renderer objects' keys, next to its contents, so another work folder rebuilds those 35 files. The hot pieces have the same issue; #30's flag_text change fixes both.
Left for later: recording from the tour, a Windows profile merged in, and removing the Linux prebuilt renderer (a separate cleanup).

Issues

See #505

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). (not run here: left to CI)
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags noted, and the switch reference updated (docs/development.md, the README).
A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md; no appended row or assembled view).
A check under tests/ that reads changed source updated in this PR, not weakened.
Docs updated.

🤖 Generated with Claude Code

## Why The renderer's replay is the frame's limit in heavy places, and compiling it with a play profile makes it about 3-5 % faster per frame at almost no build cost. Most players build with DLSS or FSR 4, which compile the renderer on their own PC, so the profile is committed in the repo and used by every renderer build rather than only by a prebuilt one. ## What changed - build: `nx2/build.py`, ported from [ProjectReLink](https://github.com/pixl27/ProjectReLink) (GPL-3.0-or-later). `RX_VK_PGO=gen` builds the back end's own files (`gpu/vk/*.cpp` and suyu's texture decoders; not Eden's recompiler, sirit, Dear ImGui or zstd) with `-fprofile-instr-generate` and links the module's own copy of `host/pgo_rt.c`. `RX_VK_PGO=<file>` builds them with that profile; its contents go into the flags the object keys hash, so another profile rebuilds those 35 files and nothing else. - host: `host/pgo_rt.c` reads its variable through `PGO_ENV` (`HOS_PGO` for the game module, `RX_VK_PGO_FILE` for the renderer's copy). On Linux it finds the `__llvm_prf_*` ELF sections by their `__start_`/`__stop_` symbols; ReLink only had Windows' COFF markers. - assets: `assets/profiles/longshot_vk.profdata.xz` (160 KB) and `assets/profiles/README.md` (where, how and on what it was recorded, and what it measured). One profile for every build for now: a Linux recording, Kokiri Forest standing and the Market walk, mesh shaders. - build: `nx2/make_game.py` `vk_profile_for()`: - the checkout's `work/longshot_vk.profdata` when there is one (a developer's recording), else the committed profile, unpacked into the work folder as `longshot_vk.shipped.profdata` (rewritten only when it changes); - passed as `RX_VK_PGO` only when this compiler reads it, and unless `RX_VK_PGO` is set already; - `profile_readable()` now checks on Windows too for this profile (`any_os=True`): it comes from LLVM 23.1.5, llvm-mingw is another LLVM, and an unreadable profile would stop the build; - `RX_VK_PGO` and `RX_VK_PGO_FILE` are left out of the lift key. - build: `scripts/linux/appimage.sh` builds the AppImage's renderer with the same profile (the checkout's, else the committed one). The AppImage already carries `assets/`, so a player's own renderer there (DLSS, FSR 4) gets it too. - build: `scripts/dev_packs.py` builds a pack of the renderer alone with its base's profile. - docs: `docs/development.md` ("The renderer's play profile": the committed profile, recording a new one, shipping it; the switch rows), `docs/design.md`, `RELEASING.md` (step 0: record it again before a minor's first candidate when the renderer changed much). - tests: `tests/test_vk_pgo.py` (new: the choice between the checkout's and the committed profile, the unpacking, the opt-outs, the committed file and its note, the writer, the AppImage), `tests/test_shader_stamp_cache.py` (build.py's real branch with `RX_VK_PGO`: which files count, the writer, the object keys), `tests/test_dlss_sdk_cache.py` (the new name in its namespace). ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `RX_VK_PGO=gen` / `RX_VK_PGO=<file>` | The renderer counting its branches for a profile, or built with that file in place of the committed one | unset: the committed profile, when the compiler reads it | A developer switch, for recording | - | - | Windows | | `RX_VK_PGO_FILE=<file>` | Where a counting renderer writes, with `<file>.reset` / `<file>.dump` | unset | A developer switch, for recording | - | - | Windows | | the committed profile | Every renderer build compiles with `assets/profiles/longshot_vk.profdata.xz` | **on** | On: about -3 to -5 % frame time, no build cost to speak of | RTX 3090, Linux, mesh shaders | not measured anywhere else | Windows (llvm-mingw), AMD and Intel GPUs, the compute fallback, the Steam Deck | ## Measurements main: 0caf0618 (2026-11-14) this PR: the profile's first version, `build/vk-pgo` @ a601a547 on 0caf0618 (the same code as this PR's, before the profile moved to `assets/profiles/`); rebased since onto main 7ff5256e with no conflict Hardware: RTX 3090, Ryzen 9 7900X, 32 GB, CachyOS (Linux) Scene: the Market from a save, standing, then running in the square; 2560x1440, default options Method: full-length sessions at low priority on a busy PC, interleaved: control, profile, profile, control, control, profile, baseline x86-64, control. The median frame of each phase, from the frame log (`RX_VK_FRAME_LOG`). Both arms ran with the default reserved core and without ReplayPin in the base. A fourth control in a noisy stretch (14.63 / 17.95 ms) and the single baseline x86-64 session are left out of the table. How to reproduce: build the renderer with and without `RX_VK_PGO=<the profile>` over one base (`scripts/dev_packs.py`), and play the same save. | Median frame, ms | main @ 0caf0618 | this PR's profile @ a601a547 | |---|---|---| | Standing in the Market (sessions) | 13.53, 13.49, 13.70 | 12.96, 13.37, 12.94 | | Running in the square | 15.89, 16.13, 15.73 | 15.13, 15.67, 15.52 | | Median | 13.53 / 15.89 | 12.96 / 15.52 | | Change | | about -4 to -5 % standing, -3 % running | The profile was recorded on Kokiri Forest and the Market walk, so the Market numbers are on part of the route it was recorded on. A broader recording (the pipeline tour, [PR 111](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/111)) is to follow. ## Testing **Tested:** the sessions above, on Linux with the RTX 3090: the counting build, the recording (`.reset`/`.dump`), `llvm-profdata merge`, and the renderer built with the profile all ran, and the game played through the route each time. No local checks run (`tests/run_all.py` is left to CI). **Not tested:** this PR's own path from the committed file: `vk_profile_for()` unpacking `assets/profiles/longshot_vk.profdata.xz` and `appimage.sh`'s fallback have not run in a build. Windows: the counting renderer, its writer (ReLink's COFF code) and whether llvm-mingw's clang reads this profile. AMD and Intel GPUs, the compute fallback, the Steam Deck. **Needs other testers:** - Windows, any GPU: a build of this PR, expecting `the Vulkan back end: built with the profile ...longshot_vk.shipped.profdata` in the build log (or `this compiler cannot read the play profile`, then built without), and the game running (request to open) - Windows: a recording of the pipeline tour with a counting renderer, for a Windows profile (request to open) - Linux AppImage: a build with DLSS or FSR 4, expecting the same log line (request to open) ## Screenshots Nothing on screen. ## Risk and rollback - A wrong profile can only make the renderer slower, not wrong: profile-guided optimisation changes layout and inlining, not what the code does. - A compiler that can't read the profile builds the renderer without it, now checked on Windows as well as Linux. - The first build after this rebuilds the renderer's 35 own files once; the lift and the game's objects are untouched. - To turn it off: delete `assets/profiles/longshot_vk.profdata.xz`, or revert. ## Reviewer notes - Start with `vk_profile_for()` in `nx2/make_game.py`, then `nx2/build.py`'s `RX_VK_PGO` branch and `host/pgo_rt.c`. - The profile's path is in the renderer objects' keys, next to its contents, so another work folder rebuilds those 35 files. The hot pieces have the same issue; https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/30's `flag_text` change fixes both. - Left for later: recording from the tour, a Windows profile merged in, and removing the Linux prebuilt renderer (a separate cleanup). ## Issues See https://git.magicbean.dev/MagicBeanDev/Longshot/issues/505 ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). (not run here: left to CI) - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`; no appended row or assembled view). - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

HumanBean added 1 commit

2026-11-14 20:04:38 +00:00

perf(vk): the renderer is built with a committed play profile, about 3-5 % less frame time

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 5s

Details

Third-party parts / As their notes say (pull_request) Successful in 7s

Details

beanstalk/approval awaiting maintainer approval

Details

PR description / Description follows the template (pull_request_target) Failing after 16s

Details

66328eff50

The Vulkan back end gets profile-guided builds of its own, apart from the hot game code's (RX_PGO), ported from
ProjectReLink (https://github.com/pixl27/ProjectReLink, GPL-3.0-or-later), and a profile committed in
assets/profiles/ that every build compiling the renderer uses: a player's (with DLSS or FSR 4 too), the AppImage's,
a developer's. On an RTX 3090 / Ryzen 9 7900X from the Market save (3 sessions each, median frame): 13.53 / 13.49 /
13.70 ms standing without, 12.96 / 13.37 / 12.94 ms with; 15.89 / 16.13 / 15.73 ms running without, 15.13 / 15.67 /
15.52 ms with.

- nx2/build.py: RX_VK_PGO=gen builds the back end's own files (gpu/vk/*.cpp and suyu's texture decoders; not Eden's
  recompiler, sirit, Dear ImGui or zstd) with -fprofile-instr-generate and links the module's own copy of
  host/pgo_rt.c; RX_VK_PGO=<file> builds them with that profile, its contents in the flags the object keys hash.
- host/pgo_rt.c: the variable that drives a run is PGO_ENV (HOS_PGO for the game module, RX_VK_PGO_FILE for the
  renderer's copy), and Linux: the ELF sections __llvm_prf_cnts/_data/_names by their __start_/__stop_ symbols.
- assets/profiles/longshot_vk.profdata.xz (160 KB) and its note: one Linux recording (Kokiri Forest, the Market) for
  every build for now.
- nx2/make_game.py: the checkout's work/longshot_vk.profdata when there is one, else the committed profile unpacked
  into the work folder, passed as RX_VK_PGO when this compiler reads it (checked on Windows too now), unless
  RX_VK_PGO is set already; RX_VK_PGO and RX_VK_PGO_FILE are out of the lift key.
- scripts/linux/appimage.sh: the AppImage's renderer is built with the same profile.
- scripts/dev_packs.py: a pack of the renderer alone is built with its base's profile.
- docs/development.md (the switches, "The renderer's play profile"), docs/design.md, RELEASING.md (record it again
  before a minor's first candidate when the renderer changed much).
- tests: test_vk_pgo.py, test_shader_stamp_cache.py; test_dlss_sdk_cache.py's namespace has the new name.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

HumanBean referenced this pull request

2026-11-14 20:13:29 +00:00

chore(release): play profiles are committed in assets/profiles/, not a tools.lock archive: tools.py assemble profile removed #572

HumanBean referenced this pull request

2026-11-14 20:14:15 +00:00

build(linux): the AppImage's renderer is compiled by the player's build, as with DLSS and FSR 4; no prebuilt renderer #574

HumanBean changed title from ~~WIP: perf(vk): the renderer is built with a committed play profile, about 3-5 % less frame time~~ to perf(vk): the renderer is built with a committed play profile, about 3-5 % less frame time

2026-11-14 20:21:34 +00:00

HumanBean merged commit 4e80ea632c into main

2026-11-14 20:21:40 +00:00

HumanBean referenced this pull request from a commit

2026-11-14 20:21:40 +00:00

perf(vk): the renderer is built with a committed play profile, about 3-5 % less frame time (#569)

HumanBean deleted branch perf/vk-profile

2026-11-14 20:21:41 +00:00

HumanBean referenced this pull request

2026-11-14 20:22:23 +00:00

build(linux): the AppImage's renderer is compiled by the player's build, as with DLSS and FSR 4; no prebuilt renderer #574

HumanBean referenced this pull request