fix(build): check Linux prerequisites before reading the dump #2793

Closed
github-sync wants to merge 3 commits from github/pr/2793 into main
Owner

@mooki000 opened this pull request on GitHub · 2026-11-12 13:31 UTC · original
Merged on GitHub in dae19300f2

What this is for: Linux source builders should see all missing build prerequisites before their game files are read or translated, rather than fixing separate errors minutes into the build. What is measured: real builder starts on Arch Linux (the missing-FFmpeg error appeared after 0.141 s, and a normal start passed preflight after 0.141 s), one complete source build with the preflight in place (493 files, 0 failed) whose game then reached the title screen, and authored dependency and ordering tests. Not measured: gameplay beyond the title, an installed AppImage, or Windows execution.

Refs #1000, specifically the source-build prerequisite report. This does not fix or close the tracker's other Linux/Steam Deck reports.

Problem

  • make_game.main() currently reaches extraction and translation before the Linux compiler check.
  • Missing Python modules can stop extraction. Renderer-header discovery and FFmpeg use happen later, so another missing prerequisite can stop a subsequent attempt.
  • The existing Windows compiler preflight already demonstrates the intended ordering: fail before opening the chosen game.

What changed

  • nx2/make_game.py: check Linux prerequisites before the read stage and report the complete missing set in one @@error message. Mandatory Python module names come from the top-level pins in requirements.txt; the included button-prompt requirements remain optional.
  • nx2/build.py: share renderer-header and prebuilt-renderer selection with the preflight. Keep the existing lookup precedence: CPM caches, explicit header roots, then system prefixes. Extend the existing FFmpeg resolver to explain missing pkg-config entries or files in its selected static prefix.
  • Compiler selection still uses the build's toolchain resolver. The C compile/link probe uses lld and the build's target/sysroot flags. A source checkout still needs clang++ for its native translator even with a prebuilt renderer; the packaged prebuilt path does not require renderer headers or a C++ compile.
  • Windows cross builds keep their Windows toolchain checks, rather than acquiring requirements for Linux FFmpeg development packages. NGX continues to select a source renderer instead of the prebuilt one.
  • Add authored prerequisite/ordering controls, preserve the existing NGX wiring assertion through the shared helper, and document the behavior in docs/development.md.

The check installs nothing and calls no package manager.

How it was tested

Environment: 2026-11-12, Arch-based Omarchy 4.5.0, Linux 7.2.15-1-omarchy, i9-13900K; GCC 16.2.1 and Clang 23.1.3. Base 51da0464b4cf0c7f56b81ebd6bb309325835e175; head 9fc8038b. The real builder starts below ran c8bccacedb9e5f5306aee0d8dac889086873fd1b; the only subsequent commit changes tests, not product code. The PC has an RTX 4090 with NVIDIA 615.78.08, and the builder starts below use only the CPU; the complete build's title run further down is the one part that used the card. Load averages at all three real starts were 34.45 / 56.47 / 51.74; these are startup observations under other work, not frame-rate benchmarks.

Real builder starts

Each start ran the unmodified make_game.py from a temporary archive of its named commit, with separate output/work folders and the existing extracted-game paths. For the missing case, HOOKSHOT_FFMPEG, PKG_CONFIG_LIBDIR and PKG_CONFIG_PATH pointed to an empty temporary directory. Nothing was removed from the system.

Build / setup Observed result Time from process start Exit
Base, FFmpeg hidden Reached Reading your game files without reporting the missing libraries 0.062 s Intentionally stopped at that stage
c8bccace, FFmpeg hidden One prerequisite error naming libavcodec, libswscale, libswresample and libavutil, with their resolution route 0.141 s; process ended at 0.150 s 1
c8bccace, normal dependencies Linux build prerequisites: OK, then the read stage 0.141 s; read stage at 0.157 s Intentionally stopped at the read stage

The failing head start emitted neither a read nor a lift stage and created neither the extracted-code directory nor translation output. No game data was read or translated on that failure path. The normal start's total time through successful preflight was below one second. The controls were deliberately stopped before doing a full build.

The error was:

@@error Missing Linux build prerequisites: FFmpeg: libavcodec, libswscale, libswresample, libavutil (HOOKSHOT_FFMPEG with lib/libavcodec.a first, else pkg-config --cflags/--libs for libavcodec libswscale libswresample libavutil; install the development libraries or set HOOKSHOT_FFMPEG). Command '['pkg-config', '--cflags', 'libavcodec', 'libswscale', 'libswresample', 'libavutil']' returned non-zero exit status 1.

A complete source build, and the game it made at the title

Built 9fc8038b completely from source into a fresh game folder (Clang 23.1.3, the renderer compiled too, eight lift and eight compile workers). The build log begins:

Linux build prerequisites: OK
@@stage read Reading your game files

The preflight passed 0.094 s after the start and the read stage began at 0.104 s. All 493 planned files compiled (compiled in 746s, 0 failed), both binaries linked, exit 0, 14 min 55 s in all. This is one build under other work on the PC (load average 11.5 falling to 11.0), not a build-speed figure.

That game then ran 90 seconds without a window on the RTX 4090 (NVIDIA 615.78.08) with a fresh profile and RX_VK_PRECOMPILE=0. It reached the title and was still there at the end; the log has 0 fatal, crash, device-loss or driver-fault lines. No frame rate is claimed.

The title screen of the game built from source with the prerequisite check in place: logo, New Game and Options

Game-free checks

  • New test: 17 cases pass with GCC and Clang. The production C probe is compiled without linking, so this game-free test needs no lld; the real builder runs above exercise linking. Controls cover valid CPM, precedence, explicit package/include roots, both system prefixes, the selected work-folder cache, prebuilt/NGX selection, optional prompt packages, bundled/partial static FFmpeg, linker failure and Windows cross-target selection. A new guard verifies that cache-style fixtures use their existing injected toolchain-check boundary without querying host dependencies; no production skip switch was added.
  • Fail-before: the same all-missing fixture against base source fails with both compiler selections: ('game', ['read', 'lift']) != (1, []). Main requests the authored extraction and lift before checking the deliberately missing module, headers, pkg-config entries and compilers. The head reports the missing set first.
  • Existing C++ preflight: 4 cases pass with g++ and clang++.
  • CI-like environment: pkg-config's search paths point to an empty directory, HOOKSHOT_FFMPEG, RX_CPM and the VMA override are unset, and only CI's pinned Vulkan/SPIR-V include overrides remain. The first strict whole-suite comparison had no preflight-caused failures: main 475 passed, 0 failed, 20 skipped; c8bccace 475 passed, 1 failed, 20 skipped, with the sole failure a shader-fixture compiler killed by SIGKILL in nx2/test_depth_sample_types.py, which never invokes the builder. Its cause is not established, and the failed output is retained.
  • Absent-linker control: with the real Clang directed at an absent linker in a temporary directory, the original positive fixture fails; the compile-only fixture passes. The separate authored linker-error test still requires the real preflight to reject a failed link before reading the dump.
  • Final strict suites on 9fc8038b, four workers: CI-like 476 passed, 0 failed, 20 skipped in 289 s; normal dependencies 476 passed, 0 failed, 20 skipped in 274 s. Neither final run has a failure, including the shader fixture that was killed in the earlier eight-worker run.
  • Windows-target syntax: the captured production C probe compiles for x86_64-w64-mingw32, preserving its compile flags: one translation unit, no diagnostics. Automatic Windows-selected fixture capture reached no compile because the real Linux probe is explicitly skipped there; the separate capture exercised that exact case. This is not Windows execution.
  • Branch/title checks pass. No core dumps were found after the checks.

Not tested

  • Gameplay beyond the title screen, and full shader preparation (the run above had RX_VK_PRECOMPILE=0); no game area or visual comparison is claimed.
  • An actual packaged AppImage build or a full Windows cross build. Those selection paths are covered with authored fixtures only.
  • Windows execution of the generated compiler probe or the new Python tests; the Windows-target result above is syntax only.
  • Other distributions, AMD/Intel runtime behavior, or physical input devices.

Risks

The new rejection happens earlier, so an incorrectly shared resolution rule could reject a working environment. The tests use the real build resolvers and cover their alternate roots and prebuilt/cross-build routes. Module presence is checked, not package-version compatibility; this does not attempt to certify every possible SDK or library incompatibility.

Overlaps

The open-PR comparison found one additional textual conflict: #1872, the macOS draft, changes FFmpeg's package list/static-library handling and renderer selection in nx2/build.py. Its thread and hunks were read; it is not a competing Linux prerequisite fix. An eventual integration must retain its platform-specific package choices. Other PRs sharing these code files merge cleanly, including the build-loop changes #2675, #2686, #2693 and #2706.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra [email protected]

> [@mooki000](https://github.com/mooki000) opened this pull request on GitHub · 2026-11-12 13:31 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2793) > **Merged on GitHub** in https://git.magicbean.dev/MagicBeanDev/Hookshot/commit/dae19300f2c373d4b9e9dd191b76b432618faf11 **What this is for:** Linux source builders should see all missing build prerequisites before their game files are read or translated, rather than fixing separate errors minutes into the build. **What is measured:** real builder starts on Arch Linux (the missing-FFmpeg error appeared after 0.141 s, and a normal start passed preflight after 0.141 s), one complete source build with the preflight in place (493 files, 0 failed) whose game then reached the title screen, and authored dependency and ordering tests. **Not measured:** gameplay beyond the title, an installed AppImage, or Windows execution. Refs #1000, specifically [the source-build prerequisite report](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/1000). This does not fix or close the tracker's other Linux/Steam Deck reports. ## Problem - `make_game.main()` currently reaches extraction and translation before the Linux compiler check. - Missing Python modules can stop extraction. Renderer-header discovery and FFmpeg use happen later, so another missing prerequisite can stop a subsequent attempt. - The existing Windows compiler preflight already demonstrates the intended ordering: fail before opening the chosen game. ## What changed - `nx2/make_game.py`: check Linux prerequisites before the read stage and report the complete missing set in one `@@error` message. Mandatory Python module names come from the top-level pins in `requirements.txt`; the included button-prompt requirements remain optional. - `nx2/build.py`: share renderer-header and prebuilt-renderer selection with the preflight. Keep the existing lookup precedence: CPM caches, explicit header roots, then system prefixes. Extend the existing FFmpeg resolver to explain missing pkg-config entries or files in its selected static prefix. - Compiler selection still uses the build's toolchain resolver. The C compile/link probe uses lld and the build's target/sysroot flags. A source checkout still needs clang++ for its native translator even with a prebuilt renderer; the packaged prebuilt path does not require renderer headers or a C++ compile. - Windows cross builds keep their Windows toolchain checks, rather than acquiring requirements for Linux FFmpeg development packages. NGX continues to select a source renderer instead of the prebuilt one. - Add authored prerequisite/ordering controls, preserve the existing NGX wiring assertion through the shared helper, and document the behavior in `docs/development.md`. The check installs nothing and calls no package manager. ## How it was tested Environment: 2026-11-12, Arch-based Omarchy 4.5.0, Linux 7.2.15-1-omarchy, i9-13900K; GCC 16.2.1 and Clang 23.1.3. Base `51da0464b4cf0c7f56b81ebd6bb309325835e175`; head `9fc8038b`. The real builder starts below ran `c8bccacedb9e5f5306aee0d8dac889086873fd1b`; the only subsequent commit changes tests, not product code. The PC has an RTX 4090 with NVIDIA 615.78.08, and the builder starts below use only the CPU; the complete build's title run further down is the one part that used the card. Load averages at all three real starts were 34.45 / 56.47 / 51.74; these are startup observations under other work, not frame-rate benchmarks. ### Real builder starts Each start ran the unmodified `make_game.py` from a temporary archive of its named commit, with separate output/work folders and the existing extracted-game paths. For the missing case, `HOOKSHOT_FFMPEG`, `PKG_CONFIG_LIBDIR` and `PKG_CONFIG_PATH` pointed to an empty temporary directory. Nothing was removed from the system. | Build / setup | Observed result | Time from process start | Exit | |---|---|---:|---| | Base, FFmpeg hidden | Reached `Reading your game files` without reporting the missing libraries | 0.062 s | Intentionally stopped at that stage | | `c8bccace`, FFmpeg hidden | One prerequisite error naming libavcodec, libswscale, libswresample and libavutil, with their resolution route | 0.141 s; process ended at 0.150 s | 1 | | `c8bccace`, normal dependencies | `Linux build prerequisites: OK`, then the read stage | 0.141 s; read stage at 0.157 s | Intentionally stopped at the read stage | The failing head start emitted neither a read nor a lift stage and created neither the extracted-code directory nor translation output. No game data was read or translated on that failure path. The normal start's total time through successful preflight was below one second. The controls were deliberately stopped before doing a full build. The error was: ```text @@error Missing Linux build prerequisites: FFmpeg: libavcodec, libswscale, libswresample, libavutil (HOOKSHOT_FFMPEG with lib/libavcodec.a first, else pkg-config --cflags/--libs for libavcodec libswscale libswresample libavutil; install the development libraries or set HOOKSHOT_FFMPEG). Command '['pkg-config', '--cflags', 'libavcodec', 'libswscale', 'libswresample', 'libavutil']' returned non-zero exit status 1. ``` ### A complete source build, and the game it made at the title Built `9fc8038b` completely from source into a fresh game folder (Clang 23.1.3, the renderer compiled too, eight lift and eight compile workers). The build log begins: ```text Linux build prerequisites: OK @@stage read Reading your game files ``` The preflight passed 0.094 s after the start and the read stage began at 0.104 s. All **493 planned files** compiled (`compiled in 746s, 0 failed`), both binaries linked, exit 0, 14 min 55 s in all. This is one build under other work on the PC (load average 11.5 falling to 11.0), not a build-speed figure. That game then ran 90 seconds without a window on the RTX 4090 (NVIDIA 615.78.08) with a fresh profile and `RX_VK_PRECOMPILE=0`. It reached the title and was still there at the end; the log has 0 fatal, crash, device-loss or driver-fault lines. No frame rate is claimed. ![The title screen of the game built from source with the prerequisite check in place: logo, New Game and Options](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/e722a9787439c2df4d2767ed2e70a174b76ca4cb/hookshot-2793-linux-preflight-title.jpg) ### Game-free checks - **New test: 17 cases pass with GCC and Clang.** The production C probe is compiled without linking, so this game-free test needs no lld; the real builder runs above exercise linking. Controls cover valid CPM, precedence, explicit package/include roots, both system prefixes, the selected work-folder cache, prebuilt/NGX selection, optional prompt packages, bundled/partial static FFmpeg, linker failure and Windows cross-target selection. A new guard verifies that cache-style fixtures use their existing injected toolchain-check boundary without querying host dependencies; no production skip switch was added. - **Fail-before:** the same all-missing fixture against base source fails with both compiler selections: `('game', ['read', 'lift']) != (1, [])`. Main requests the authored extraction and lift before checking the deliberately missing module, headers, pkg-config entries and compilers. The head reports the missing set first. - **Existing C++ preflight: 4 cases pass** with g++ and clang++. - **CI-like environment:** pkg-config's search paths point to an empty directory, `HOOKSHOT_FFMPEG`, `RX_CPM` and the VMA override are unset, and only CI's pinned Vulkan/SPIR-V include overrides remain. The first strict whole-suite comparison had **no preflight-caused failures**: main `475 passed, 0 failed, 20 skipped`; c8bccace `475 passed, 1 failed, 20 skipped`, with the sole failure a shader-fixture compiler killed by SIGKILL in `nx2/test_depth_sample_types.py`, which never invokes the builder. Its cause is not established, and the failed output is retained. - **Absent-linker control:** with the real Clang directed at an absent linker in a temporary directory, the original positive fixture fails; the compile-only fixture passes. The separate authored linker-error test still requires the real preflight to reject a failed link before reading the dump. - **Final strict suites on `9fc8038b`, four workers:** CI-like **`476 passed, 0 failed, 20 skipped in 289 s`**; normal dependencies **`476 passed, 0 failed, 20 skipped in 274 s`**. Neither final run has a failure, including the shader fixture that was killed in the earlier eight-worker run. - **Windows-target syntax:** the captured production C probe compiles for `x86_64-w64-mingw32`, preserving its compile flags: one translation unit, no diagnostics. Automatic Windows-selected fixture capture reached no compile because the real Linux probe is explicitly skipped there; the separate capture exercised that exact case. This is not Windows execution. - Branch/title checks pass. No core dumps were found after the checks. ## Not tested - Gameplay beyond the title screen, and full shader preparation (the run above had `RX_VK_PRECOMPILE=0`); no game area or visual comparison is claimed. - An actual packaged AppImage build or a full Windows cross build. Those selection paths are covered with authored fixtures only. - Windows execution of the generated compiler probe or the new Python tests; the Windows-target result above is syntax only. - Other distributions, AMD/Intel runtime behavior, or physical input devices. ## Risks The new rejection happens earlier, so an incorrectly shared resolution rule could reject a working environment. The tests use the real build resolvers and cover their alternate roots and prebuilt/cross-build routes. Module presence is checked, not package-version compatibility; this does not attempt to certify every possible SDK or library incompatibility. ## Overlaps The open-PR comparison found one additional textual conflict: [#1872](https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/1872), the macOS draft, changes FFmpeg's package list/static-library handling and renderer selection in `nx2/build.py`. Its thread and hunks were read; it is not a competing Linux prerequisite fix. An eventual integration must retain its platform-specific package choices. Other PRs sharing these code files merge cleanly, including the build-loop changes #2675, #2686, #2693 and #2706. Assisted-by: GPT-6 Astra Co-Authored-By: GPT-6 Astra <[email protected]> <!-- github-sync:item:2793 -->
Collect missing mandatory Python modules, compiler and linker, renderer headers and FFmpeg development libraries in one diagnostic. Reuse the build resolvers and keep packaged and Windows cross-build selection intact.

Refs #1000

Assisted-by: GPT-6 Astra

Co-Authored-By: GPT-6 Astra <[email protected]>
A source checkout builds a64emit with clang++ even when it uses a prebuilt renderer. Keep that prerequisite for source builds while the packaged prebuilt-renderer path needs neither C++ compile. Extend the authored controls and declare the compiler needed by the real probe.

Assisted-by: GPT-6 Astra

Co-Authored-By: GPT-6 Astra <[email protected]>
Assert that cache-style injected build steps use the existing toolchain-check boundary without querying host prerequisites. Compile the production C probe without linking so the game-free fixture does not require lld; authored linker failures and the real CLI starts retain link coverage. No production bypass switch is added.

Assisted-by: GPT-6 Astra

Co-Authored-By: GPT-6 Astra <[email protected]>
github-sync closed this pull request 2026-11-12 17:59:21 +00:00
Author
Owner

@hckrman101 approved on GitHub · 2026-11-12 17:24 UTC · original

Looks good to me. Failing in 0.14 s with one message instead of minutes into a build is a real improvement for source builders, and the evidence is the kind I'd want: the same hidden-FFmpeg start on base (goes straight to "Reading your game files") versus the head (one error, no read or lift stage), a normal start passing the preflight, a full source build from the head with the game reaching the title, and CI green with no conflicts.

What I checked in the code:

  • False positives: every check goes through the resolvers the compile itself uses (header_package, ffmpeg_linux, toolchain.find, target()), with the same precedence (CPM, then HOOKSHOT_*_HEADERS, then /usr/local and /usr), and the CPM lookup now takes the --work folder, so it matches what the compile subprocess sees. Anything the preflight refuses, the compile would have stopped on later, so Arch, Fedora, Ubuntu, SteamOS and NixOS (pkg-config, EMSDK, HOOKSHOT_FFMPEG and HOOKSHOT_SYSROOT all honored) behave as before, just earlier. The bundled build-linux/deps setup works through EMSDK, RX_CPM and HOOKSHOT_FFMPEG the same way. The AppImage ships no requirements.txt and sets a prebuilt renderer, so it skips the Python module and header checks. That is only tested through fixtures, as the description says.
  • NGX: moving fetch_ngx_sdk before the preflight is needed, since build.NGX_SDK is read when build is first imported, and the preflight now does that import. test_make_game_ngx only follows the refactor: it asserts the same NGX_SDK expression, now inside prebuilt_renderer(), plus the call site. Behavior is unchanged.
  • Windows: check_windows_sdk is now called unconditionally, but on a non-Linux target it takes exactly the old branches. RX_TARGET=windows has LINUX false, so the Linux checks don't apply.
  • Overrides: there is no skip switch. Since each requirement is something the compile needs anyway, I think that's right: a skip would only move the failure later.

Nits, none blocking:

  1. The error is one long line (joined with ; ), and the FFmpeg entry ends with the raw Command '[...]' returned non-zero exit status 1. text, which adds noise after an already clear message. The GUI protocol needs one line, so this is fine, just wordy.
  2. The module check uses the pin name as the import name. That holds for all five pins today (lz4, numpy, capstone, unicorn, zstandard), but a future pin such as Pillow or PyYAML would be reported missing while installed. A short comment or a name map would save the next person.
  3. check() (the --check --nup path) calls check_windows_sdk first, so on Linux checking that a dump is the right game now also requires FFmpeg and the renderer headers, not only a compiler. That is probably fine, since it can't be built without them, but worth knowing if the launcher uses --check on its own.

Evidence is Arch only (the description says so); I'd welcome a Fedora, Ubuntu or SteamOS start from a tester.

For ordering: #1872 (macOS draft) touches the same build.py FFmpeg code and will need to rebase on this.

> [@hckrman101](https://github.com/hckrman101) approved on GitHub · 2026-11-12 17:24 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2793#pullrequestreview-5878987006) Looks good to me. Failing in 0.14 s with one message instead of minutes into a build is a real improvement for source builders, and the evidence is the kind I'd want: the same hidden-FFmpeg start on base (goes straight to "Reading your game files") versus the head (one error, no read or lift stage), a normal start passing the preflight, a full source build from the head with the game reaching the title, and CI green with no conflicts. What I checked in the code: - **False positives:** every check goes through the resolvers the compile itself uses (`header_package`, `ffmpeg_linux`, `toolchain.find`, `target()`), with the same precedence (CPM, then `HOOKSHOT_*_HEADERS`, then `/usr/local` and `/usr`), and the CPM lookup now takes the `--work` folder, so it matches what the compile subprocess sees. Anything the preflight refuses, the compile would have stopped on later, so Arch, Fedora, Ubuntu, SteamOS and NixOS (pkg-config, `EMSDK`, `HOOKSHOT_FFMPEG` and `HOOKSHOT_SYSROOT` all honored) behave as before, just earlier. The bundled `build-linux/deps` setup works through `EMSDK`, `RX_CPM` and `HOOKSHOT_FFMPEG` the same way. The AppImage ships no `requirements.txt` and sets a prebuilt renderer, so it skips the Python module and header checks. That is only tested through fixtures, as the description says. - **NGX:** moving `fetch_ngx_sdk` before the preflight is needed, since `build.NGX_SDK` is read when `build` is first imported, and the preflight now does that import. `test_make_game_ngx` only follows the refactor: it asserts the same `NGX_SDK` expression, now inside `prebuilt_renderer()`, plus the call site. Behavior is unchanged. - **Windows:** `check_windows_sdk` is now called unconditionally, but on a non-Linux target it takes exactly the old branches. `RX_TARGET=windows` has `LINUX` false, so the Linux checks don't apply. - **Overrides:** there is no skip switch. Since each requirement is something the compile needs anyway, I think that's right: a skip would only move the failure later. Nits, none blocking: 1. The error is one long line (joined with `; `), and the FFmpeg entry ends with the raw `Command '[...]' returned non-zero exit status 1.` text, which adds noise after an already clear message. The GUI protocol needs one line, so this is fine, just wordy. 2. The module check uses the pin name as the import name. That holds for all five pins today (lz4, numpy, capstone, unicorn, zstandard), but a future pin such as Pillow or PyYAML would be reported missing while installed. A short comment or a name map would save the next person. 3. `check()` (the `--check --nup` path) calls `check_windows_sdk` first, so on Linux checking that a dump is the right game now also requires FFmpeg and the renderer headers, not only a compiler. That is probably fine, since it can't be built without them, but worth knowing if the launcher uses `--check` on its own. Evidence is Arch only (the description says so); I'd welcome a Fedora, Ubuntu or SteamOS start from a tester. For ordering: #1872 (macOS draft) touches the same `build.py` FFmpeg code and will need to rebase on this. <!-- github-sync:r:5878987006 -->
Commenting is not possible because the repository is archived.
No description provided.