MagicBeanDev/Longshot

Watch

Fork 98

test(build): isolate shipped-profile lookups in game-free fixtures #592

Open

futondestroyer wants to merge 2 commits from futondestroyer/Longshot:test/hermetic-vk-profile-fixtures into main

futondestroyer commented

2026-11-14 21:39:52 +00:00

Member

Type: test · Risk: low · Tested on: Windows 11 build 26300, Python 3.14.7 (game-free) · Needs: none

Why

Two game-free check drivers fail on main after the renderer's committed PGO profile was added. Their authored checkouts are meant to keep real compiler probes out, but the profile helper still looks in the real repository. This restores those checks without changing the player's build or disabling their process-launch guards.

No existing tracker issue was found for vk_profile_for; this follows the shipped-profile change in #569.

How to reproduce

Seen on: Windows 11 build 26300, native Python 3.14.7; no GPU used.
Settings and switches: the check runner's ordinary tool environment; no game or dump.
Steps: On main 5d3ecc78, run python -I -B tests/run_all.py test_make_game_cache test_linux_preflight test_vk_pgo.
Expected: the authored fixtures reach their intended build boundaries without launching a profile-readability compiler probe.
Actual: the cache check stops with Unexpected process launch; the injected-preflight case records an unexpected compiler process.

What changed

Both fixtures now supply their authored checkout explicitly to the real vk_profile_for helper, whose default root was bound when the module was imported. functools.partial retains the helper's actual selection and readability behavior; the authored root has no profile to probe.

tests/test_make_game_cache.py: bind the profile lookup to the same temporary root already used by the cache fixture.
tests/test_linux_preflight.py: bind it to the injected preflight checkout.
Keep every existing assertion and subprocess/shell/file-launch guard. The PGO contract check remains enabled and passes. No production file changes.
Record each targeted and full-suite run in a separate docs/tested/ file.

Measurements

Verdict: the two affected check drivers pass in both interleaved comparisons; the full suite has exactly two fewer failures and no new failures.

Hardware: Windows 11 build 26300, native Python 3.14.7, llvm-mingw clang 23.1.5, i7-13700F, 32 GB RAM; no GPU work.
Scene: authored game-free fixtures; no game scene.
Method: main/PR/main/PR runs of the three selected drivers; median failed-driver count below. One complete game-free suite per arm, 12 workers. Same environment; optional NGX SDK variable unset. All logs retained locally.
How to reproduce: the targeted command above, followed by python -I -B tests/run_all.py --jobs 12 on each checkout.

	Before: main @ `5d3ecc78`	After: this PR @ `328c326b`
Targeted drivers passing, run 1 / run 2	1/3 / 1/3	3/3 / 3/3
Failed targeted drivers, median	2	0
Full suite passed / failed / skipped	533 / 9 / 34	535 / 7 / 34

328c326b is the measured code commit; the following run-record commit changes only documentation. Earlier checks of the same eight-line patch on main 414f56e2 are retained in six additional records: 532/7/34 before, 534/5/34 after. The final table above uses the refreshed base, including its newer upstream fixture changes. No game-performance claim.

main, targeted run 1 and run 2: 1 passed, 2 failed, 0 skipped
this PR, targeted run 1 and run 2: 3 passed, 0 failed, 0 skipped

Flags and settings

None. No production switch, default, profile or build behavior changes.

Testing

Tested: both interleaved targeted pairs and both refreshed full suites above on native Windows. The remaining seven failures are shared with the clean current-main control: tests/test_appimage_player_build.py, tests/test_cpm_discovery.py, tests/test_lift_cleanup_error.py, tests/test_oot_state_tap.py, tests/test_release.py, tests/test_tested_configurations.py, tests/test_walk_key.py. No PR-only failure. The original-base comparison also has no PR-only failure. The real tested-configurations renderer accepts all twelve new run fragments; the frozen source table is untouched. The source helpers, process guards and PGO contract check remain in use.

Not tested: native Linux execution or a game build. Only Python test-fixture root injection and authored run records change; the Linux-preflight cases use their existing injected Linux environment on Windows. No runtime, native host or profile-selection implementation changes.

Must be tested: None; existing game-free regressions and the unchanged PGO contract check cover this fixture-only change.

Risks

What could break: a future fixture that deliberately adds a profile will invoke the real readability check within its own root, and its existing process guards will still reject an unexpected compiler launch.
Turning it off: revert the fixture commit.
Compatibility: no player-visible behavior, game data, saves, caches, SDKs or renderer ABI changes.

Issues

See #569

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main (5d3ecc78 for this validation).
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md).
A check under tests/ that reads changed source updated in this PR, not weakened.
Docs updated.

**Type:** test · **Risk:** low · **Tested on:** Windows 11 build 26300, Python 3.14.7 (game-free) · **Needs:** none ## Why Two game-free check drivers fail on main after the renderer's committed PGO profile was added. Their authored checkouts are meant to keep real compiler probes out, but the profile helper still looks in the real repository. This restores those checks without changing the player's build or disabling their process-launch guards. No existing tracker issue was found for `vk_profile_for`; this follows the shipped-profile change in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/569. ## How to reproduce **Seen on:** Windows 11 build 26300, native Python 3.14.7; no GPU used. **Settings and switches:** the check runner's ordinary tool environment; no game or dump. **Steps:** On main `5d3ecc78`, run `python -I -B tests/run_all.py test_make_game_cache test_linux_preflight test_vk_pgo`. **Expected:** the authored fixtures reach their intended build boundaries without launching a profile-readability compiler probe. **Actual:** the cache check stops with `Unexpected process launch`; the injected-preflight case records an unexpected compiler process. ## What changed Both fixtures now supply their authored checkout explicitly to the real `vk_profile_for` helper, whose default root was bound when the module was imported. `functools.partial` retains the helper's actual selection and readability behavior; the authored root has no profile to probe. - `tests/test_make_game_cache.py`: bind the profile lookup to the same temporary root already used by the cache fixture. - `tests/test_linux_preflight.py`: bind it to the injected preflight checkout. - Keep every existing assertion and subprocess/shell/file-launch guard. The PGO contract check remains enabled and passes. No production file changes. - Record each targeted and full-suite run in a separate `docs/tested/` file. ## Measurements **Verdict:** the two affected check drivers pass in both interleaved comparisons; the full suite has exactly two fewer failures and no new failures. Hardware: Windows 11 build 26300, native Python 3.14.7, llvm-mingw clang 23.1.5, i7-13700F, 32 GB RAM; no GPU work. Scene: authored game-free fixtures; no game scene. Method: main/PR/main/PR runs of the three selected drivers; median failed-driver count below. One complete game-free suite per arm, 12 workers. Same environment; optional NGX SDK variable unset. All logs retained locally. How to reproduce: the targeted command above, followed by `python -I -B tests/run_all.py --jobs 12` on each checkout. | | Before: main @ 5d3ecc78 | After: this PR @ 328c326b | |---|---:|---:| | Targeted drivers passing, run 1 / run 2 | 1/3 / 1/3 | 3/3 / 3/3 | | Failed targeted drivers, median | 2 | 0 | | Full suite passed / failed / skipped | 533 / 9 / 34 | 535 / 7 / 34 | `328c326b` is the measured code commit; the following run-record commit changes only documentation. Earlier checks of the same eight-line patch on main `414f56e2` are retained in six additional records: 532/7/34 before, 534/5/34 after. The final table above uses the refreshed base, including its newer upstream fixture changes. No game-performance claim. ```text main, targeted run 1 and run 2: 1 passed, 2 failed, 0 skipped this PR, targeted run 1 and run 2: 3 passed, 0 failed, 0 skipped ``` ### Flags and settings None. No production switch, default, profile or build behavior changes. ## Testing **Tested:** both interleaved targeted pairs and both refreshed full suites above on native Windows. The remaining seven failures are shared with the clean current-main control: `tests/test_appimage_player_build.py`, `tests/test_cpm_discovery.py`, `tests/test_lift_cleanup_error.py`, `tests/test_oot_state_tap.py`, `tests/test_release.py`, `tests/test_tested_configurations.py`, `tests/test_walk_key.py`. No PR-only failure. The original-base comparison also has no PR-only failure. The real tested-configurations renderer accepts all twelve new run fragments; the frozen source table is untouched. The source helpers, process guards and PGO contract check remain in use. **Not tested:** native Linux execution or a game build. Only Python test-fixture root injection and authored run records change; the Linux-preflight cases use their existing injected Linux environment on Windows. No runtime, native host or profile-selection implementation changes. **Must be tested:** None; existing game-free regressions and the unchanged PGO contract check cover this fixture-only change. ## Risks **What could break:** a future fixture that deliberately adds a profile will invoke the real readability check within its own root, and its existing process guards will still reject an unexpected compiler launch. **Turning it off:** revert the fixture commit. **Compatibility:** no player-visible behavior, game data, saves, caches, SDKs or renderer ABI changes. ## Issues See https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/569 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main` (`5d3ecc78` for this validation). - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.

futondestroyer added 2 commits

2026-11-14 21:39:53 +00:00

test(build): isolate shipped-profile lookups in game-free fixtures 328c326be6

Keep vk_profile_for's real selection logic inside each authored checkout. Its import-time default root otherwise reads the repository's newly shipped renderer profile and launches a compiler where the cache and injected-preflight checks forbid subprocesses. Preserve all existing process guards and assertions.

Assisted-by: OpenAI Codex

docs(tested): record shipped-profile fixture comparisons

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

beanstalk/approval Test farm not requested

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

d953d4ae03

Preserve the initial and refreshed interleaved game-free checks and full-suite controls as individual run records. No game or performance claim.

Assisted-by: OpenAI Codex

futondestroyer changed title from ~~WIP: test(build): isolate shipped-profile lookups in game-free fixtures~~ to test(build): isolate shipped-profile lookups in game-free fixtures

2026-11-14 21:40:08 +00:00

futondestroyer commented

2026-11-14 21:41:31 +00:00

Author

Member

Ready for review with refreshed Windows main/PR controls: exactly two check-driver failures removed, no PR-only failures. All twelve game-free run records and the complete checklist are in the description.

~~zeldaootr referenced this pull request 2026-11-15 00:55:49 +00:00~~

test(vk): the env_on and late readback checks match the render thread's source #622

zeldaootr referenced this pull request

2026-11-15 00:55:52 +00:00

test(build): the lift clean-up check stands in for the shared lift store #623

zeldaootr referenced this pull request

2026-11-15 00:55:54 +00:00

test(linux): the AppImage check looks for the renderer's folder in the image, not among the build tools #624

zeldaootr referenced this pull request

2026-11-15 00:55:57 +00:00

docs(profiles): the renderer profile's note has its Recorded line again #625

~~zeldaootr referenced this pull request 2026-11-15 00:56:00 +00:00~~

test(host): the crash window's check runs through on Linux with g++ and clang #626

zeldaootr referenced this pull request

2026-11-15 00:56:03 +00:00

test(scripts): the pipeline tour check's stale snapshot is older than calm's tolerance #627

zeldaootr referenced this pull request

2026-11-15 00:56:05 +00:00

test: three checks run or skip by what the PC has, instead of failing without it #628

gmourao referenced this pull request

2026-11-15 05:51:24 +00:00

Game: Ganondorf initially renders completely black, then recovers after a long delay on RX 9060 XT #715

siantm referenced this pull request

2026-11-15 10:00:25 +00:00

WIP: fix(build): honor selected external hot-function profiles #754

d4nkn3ss referenced this pull request

2026-11-15 13:07:53 +00:00

test(build): the make_game checks let go of the work folder's lock, so Windows can delete their temporary folders #804

d4nkn3ss commented

2026-11-15 13:08:07 +00:00

Contributor

For the record from Windows: test_make_game_cache and test_linux_preflight also error on Windows since #630 (WinError 32 on work\make_game.lock, 14 and 17 errors). #804 fixes that part. With #804 and this PR's diff together on main aa95d35f, both checks pass on Windows 11 (17 and 19 tests OK). With either alone, they still fail.

For the record from Windows: test_make_game_cache and test_linux_preflight also error on Windows since #630 (WinError 32 on work\make_game.lock, 14 and 17 errors). https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/804 fixes that part. With #804 and this PR's diff together on main aa95d35f, both checks pass on Windows 11 (17 and 19 tests OK). With either alone, they still fail.

d4nkn3ss referenced this pull request

2026-11-15 13:13:37 +00:00

test(menu): the options check reads the keys save() owns on a line of their own, as gpu_device_id since #736 #806

d4nkn3ss referenced this pull request

2026-11-15 13:17:21 +00:00

docs(profiles): the play profile's note says Recorded: again, which test_vk_pgo looks for #807