WIP: feat(build): one shader that crashes or hangs the GPU driver costs the preparation that shader, not the whole run #1386
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1386
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:feat/prepare-supervisor"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Linux, RTX 3090 · Needs: Windows (the Builder there, and a real driver fault); Intel
Why
After a build the Builder runs
longshot --prepare-shaders, which has the GPU driver compile every shader and pipeline before the first start. It is all or nothing: one shader that crashes or hangs the driver's compiler loses the whole run, and the work falls to the game's own first start, where the same shader meets the same driver inside the game. The in-process guards catch such a fault but leave the driver's state behind it. Track E of the renderer rework plan, step 1. Stacked on #1309: review the last commit only.How much it helps, honestly: of 55 crash reports grouped for the plan, about 5 are in the shader phase or inside a driver. Most device-lost reports are during play, which this does not touch.
What changed
gpu/vk/prepare_journal.h): in a preparation run started with--prepare-journal <file>, each call into the driver's compiler writes a begin line before it and an end line after, flushed to the OS first, so the entry that was open survives the process dying. An entry is named exactly aslongshot_driver_faults.txtnames a program. Play pays one test of a flag.--prepare-entry-limit, 240 s from the Builder): an entry open longer ends the run with exit status 6 and a line naming it. The whole-run watchdog and exit codes 3 and 4 are as they were.nx2/prepare_supervisor.py, used bynx2/make_game.pyandscripts/dev_packs.py): when the run dies, hangs or exits 6, it reads the journal, names the program that was inside the driver, appends it tolongshot_driver_faults.txtfor that driver version, and starts again past it. With several programs open it starts again building those one at a time, so a neighbour is never blamed. Bounds: 8 programs, 20 starts; past them it gives up as today. It writescache/longshot_prepare_results.txt.RX_VK_TEST_PREPARE_FAULT(diagnostic) crashes or hangs a chosen program in a journalled preparation, to test all of this. Five rows ingpu/vk/switches.def;docs/development/shader-preparation.md(new).No file of the shader translation's stamp is edited: nobody's shaders translate again.
Measurements
Nothing to measure in play. The preparation itself: a clean run journalled 259,795 entries and ended
prepared; its duration with and without the journal was not compared.Testing
Tested: Linux, RTX 3090 (NVIDIA 620.70), a dev runtime:
RX_VK_TEST_PREPARE_FAULT=crash:5000, five programs): 11 runs, each culprit named (with nine programs open at once, the suspects were rebuilt one at a time and the right one found), five lines inlongshot_driver_faults.txt, and the preparation endedprepared.vk: preparation: STOPPED: entry 153032 (program 1f2e258ec71b4c98) has been inside the GPU driver for 20 s, exit 6, the program left out, the second runprepared.vk: 5 programs the driver faulted on in earlier runs are not asked of it, then Kokiri Forest and Hyrule Field at main's frame rates (67.1 and 92.8 fps), no error line.Known wrong: the result file's totals said
crashed 0andtimed-out 0while its rows for the programs left out were right.Not tested:
tests/run_all.py, and the two new checks.nx2/make_game.py): the supervisor was run directly and throughdev_build.py rt.Must be tested:
RX_VK_TEST_PREPARE_FAULT=crash:5000set, expecting the same supervisor lines and a game that starts; then the same withhang:5000.Risks
What could break: a slow but finite compile is left out (missing draws for that program until the driver updates); a program is blamed wrongly if the journal names the wrong one for a pipeline; the preparation takes longer when it restarts.
Turning it off:
LONGSHOT_PREPARE_ENTRY_S=0in the Builder's environment turns the per-entry limit off. Deletingcache/longshot_driver_faults.txttries every program again.Compatibility: none: no cache is rebuilt,
MKWVK_API_VERSIONstays. The preparation's exit code 6 is new.Issues
See #1300
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Not run on this PC; left to the forge's checks.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
https://claude.ai/code/session_017vw8dzCsAL3UdW7ft8ptL9
Tested in game: Linux, RTX 3090, NVIDIA 620.70: a clean supervised preparation (259,795 entries), forced crashes of five programs and a forced hang each isolated and quarantined, then a game start that logged the 5 programs left out and played Kokiri Forest and Hyrule Field at main's frame rates.
Possibly related, for context:
Folded into #1334, which is now the one pull request for tooling (trace replay and shader preparation). The commits are unchanged apart from being rebased into one stack and carrying the fixes the check suite asked for; nothing was dropped.
Pull request closed