[Tracking] Builder, setup and updates: errors, disk space, slow builds and shader preparation #2505
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
2 participants
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#2505
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Attach the Builder's log (or the error text) and say where the game folder is.
Reports
These reports were collected here so one fix closes them all and the open list stays readable. Add new details as a comment here (version, card and driver, where in the game, and the hookshot_log.txt from the game folder).
Another report collected here: #2512: Aesthetic fix: Builder top bar thickness
More reports of this, collected here:
A data point from the RX 6700 XT (i7-12700K, Windows, AMD proprietary): with current main's per-stage progress logging, the Builder's headless preparation completes on this machine for the first time. The same machine saw the 15-minute silence watchdog stop the preparation three times before (reported in #2242: the headless game would go quiet during the pipeline stages and be stopped, pushing everything onto first start).
This run: every 10 s a progress line kept the watchdog fed through all stages — the pipeline prediction (55,391 pipelines), the noted pipelines (28,347), and the precompile finished in one headless pass:
precompile: 10392 programs, 17955 graphics and 9691 compute pipelines, 701 mesh fallback programs in 5509705 ms— 92 minutes end to end, no stall, no kill. The line-per-10-seconds alone appears to be what lets an AMD-slow preparation survive; nothing else changed on this machine. One caveat worth noting for anyone reproducing: the resulting driver cache is keyed to the build folder's executable path — running the game from a different folder after copying it starts the interactive preparation over, so the bake pays off for the folder it ran in.
An update to #1388: I can build the game just fine, without it eating up all my remaining space, if I set the build/install path to anywhere on my C: drive. However, when I try and move the folder to my D: drive following the installation, I get somewhat of an odd error (see screenshot). For reference, the folder is only 33.6 GB.
First start on a card without mesh shaders: RX 5600 XT 6 GB (RDNA1), Adrenalin 26.10.1, Windows 11, Ryzen 5 5600X, 24 GB. Main
de866017.Times per part, from the log:
--prepare-shaders, shader programswarm_computeruns first. It builds every compute variant (9,691) in the driver, seconds each on AMD. The bar barely moves, and the log'swaiting: nothing ... has finished for 41 sreads like a hang79116b290d46542f,the branch at instruction 16404 goes outside the program)hookshot_prepared.bin.warmingwas left by the stopped builder run), so this is translation alonedriver shader cache2305 → 3393 MB. Next start:the driver's shader cache kept 31 of 32 predicted pipelines asked, so the AMD driver does keep themThree more things from the same runs:
the preparation did not stop within 10 s: closing anyway, and the process ended beforeprepare_with_screen'spipeline_cache_save(true).mkw_vk_pipelines.binstayed at its start size, and the next start began the noted pipelines from zero. On this card the part takes ~40 min, so a periodic save withinprecompile_run(or one at the 10 s limit) would keep most of it.waitanswers within the hour. A script waiting for frame 1 across a long first start has to wait again:scripts/hookshot_bench.pyin #2714 does that.Reporting startup pipeline-compilation OOM kills on v2.1.0 (source e561602e), with a confirmed startup workaround. Related to #775, but this is an Intel Arc Pro B50 discrete GPU and a newer Hookshot version.
Environment:
0x6402008.hookshotshell launcher, on a local disk.The launcher starts successfully, but the game is killed while preparing pipelines. The kernel recorded three Hookshot OOM kills across November 11–12. Selected kernel evidence (timestamps are local):
The latest failed run ended around
pipelines 1085 of 20463, 5% of all. The preceding run's log ended aroundpredicted pipelines 38475 of 67106, 58% of all. These are kernel OOM kills, rather than an exception reported by the shell launcher. The exact allocation/driver cause has not been isolated.Workaround tested: add this immediately after
set -euin the game-folder shell launcher, so its builder/game child inherits it:This defaults startup precompilation off while allowing
RX_VK_PRECOMPILE=1to restore it. No shader caches were deleted and no game binaries were patched. With this change the game reached the Zelda title / New Game screen, and the user confirmed it worked. An early process sample was about 2.5 GiB RSS; this is not a measured peak. Full gameplay stability has not been tested, and on-demand compilation may cause stutters.Could startup precompilation be bounded by available memory, or offer a persistent skip/fallback after OOM failures?
Latest failed hookshot_log.txt (local username/path redacted)
Follow-up to my report above: the workaround only gets through startup; gameplay is not fixed. The user reached the title and watched the intro, then the screen went black as the 3D scene began. This appears similar to the Intel reports in #999 / #366, but the underlying cause is not confirmed.
The same process remained alive (about 11.6 GiB RSS in one sample), with no new kernel OOM or GPU-reset message observed. The log shows skipped compute dispatches while pipelines compile, followed by slow draws, and then stops reporting presented frames while occasional game-thread messages continue. A debugger attach was denied, so I do not have stacks proving a driver/compiler stall.
Selected end of log: