fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compile #2114
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2114
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:fix/build-jobs-every-step"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The launcher's "Build workers" setting only reached the compile, so the translation, the button prompts and the shader preparation still used every CPU thread and the PC was unusable while an update ran (LongshotLauncher#148, LongshotLauncher#163). This makes every step
make_game.pyruns side by side keep to the count it is given, which LongshotLauncher#235 now always passes (Automatic leaves a quarter of the threads free).Summary
make_game.py --jobs N(Linux launcher) andLONGSHOT_JOBS=N(Windows launcher) set how many files compile at once, andnothing else. The other parallel steps sized themselves from the CPU count. Now one helper,
nx2/build_jobs.py, holds therule, and each step uses it:
nx2/make_game.pymain: when a count is given (--jobs, orLONGSHOT_JOBSalready set; not 0), it is put inLONGSHOT_JOBSfor every step it starts. When none is given nothing is set, so each step keeps today's count.nx2/chunks.pyfile_jobs: the lift's emit processes, its per-file passes, its side process andnx2/discover.py'swalks (
lift.pypassesfile_jobs()) take at most that many. The memory rule still applies below it, andLONGSHOT_LIFT_JOBSstill overrides it.nx2/prompt_assets.pyface_jobs: the prompts' face processes, the same way (LONGSHOT_PROMPT_JOBSstill overrides it).nx2/make_game.pyprepare_shaders: the game's--prepare-shadersstarts on N of the CPUs the builder may use(
build_jobs.children_on_cpus: the calling thread's mask on Linux, the process's on Windows, set while the child iscreated and put back straight after). The game's environment drops the
LONGSHOT_build variables (game_env), sothe affinity mask is how the count gets to it. Its preparation pools already follow the mask (
usable_cpusingpu/vk/vk_progcache.cpp,tests/test_prepare_cpus.py), so the game itself is unchanged. Without a count the gamegets every CPU, as before.
nx2/build.py: already takes--jobs. Unchanged.Left alone, and why:
nx2/nup.py(reading the.nup),unpack.py,text_assets.pyandshader_census.pyrun on one thread. The overlappedpackage read (
overlap_package_read) is one extra reader thread, and it is already off at--jobs 1.third_party_objects.stamp's five threads are five short compiler probes at the start of the compile (a few seconds).build.pynotes it), so I didn't add a--threadsflag. That flag would also change every link key.
third_party_objects.py makeandscripts/validate_spirv.pyare release and dev tools thatmake_game.pydoesn't run.Read first:
nx2/build_jobs.py, then the calls inchunks.file_jobs,prompt_assets.face_jobsandmake_game.main/prepare_shaders.Testing
Tested: Linux (CachyOS, 24 threads), game-free:
tests/test_build_jobs.py(new) checks four things. The count's parsing. Thatmake_game.mainexports a given count andexports none when no count is given. That the lift's per-file pool (
chunks._each, sized byfile_jobs) runs on atmost 4 distinct worker processes with
LONGSHOT_JOBS=4, and on 16 without it. Thatprepare_shadersstarts its childon 2 CPUs with
LONGSHOT_JOBS=2and on all of them without, and that the builder gets its own mask back afterwards.The child is a stand-in that prints
len(os.sched_getaffinity(0)), not the game.tests/test_lift_workers.pyandtests/test_prompt_speed.py(FaceJobs) gained the ceiling cases, and their setUp nowclears
LONGSHOT_JOBSso a value exported in the shell doesn't change them.tests/test_lift_cache.py:LONGSHOT_JOBSis listed inLIFT_ENV_NOT_OUTPUT, so it is not part of the lift key.python -I -B tests/run_all.py --jobs 8(not--strict, in a jj workspace): 730 passed, 88 failed, 116 skipped. I ranthe 88 failures again on main's sources (
1bd93a48) in the same setup, and 87 of them fail there too: missing tools,checks that need a git checkout, and source checks that already fail on main.
tests/test_lift_cache.pyis one ofthem; its one failure, "...and the others still by their code", happens on main as well. The 88th was
tests/test_prompt_skip.py, whose list of the prompts' local imports neededbuild_jobs. It is now on the list withkitandlong_paths: it decides how many processes draw the faces, not what they draw. That test passes now.No real build or game run: the game binary is unchanged, and the only change to the shader preparation is that it may
start on fewer CPUs, a case
usable_cpusalready handles (taskset, Process Lasso).Still to test:
Manager to show the translation and the shader preparation using about the chosen number of threads instead of all
of them, and the log's
discovery: ... in N processesto show at most that N (by hand).--jobs(by hand).Measurements
Verdict: a synthetic run (no game data) with
LONGSHOT_JOBS=4on a 24-thread PC uses 4 workers in each step, andthe same as before without it. This change limits CPU use on purpose. It doesn't make anything faster.
Setup: CachyOS, 24 CPU threads. A script calls the functions as
make_game.py's steps do.chunks._available_gbis setto 64 because this PC was busy and its free memory would otherwise have lowered the lift count further. 200 CPU-bound
items go through
chunks._each, with distinct worker pids counted, and the stand-in game runs throughmake_game.prepare_shaders.1bd93a48, no count1bd93a48,LONGSHOT_JOBS=44ff025d4, no count4ff025d4,LONGSHOT_JOBS=4file_jobs, distinct worker pids over 200 items)face_jobs(5))Risks
nx2/chunks.pyis one of the lift's sources, so the first update after this merges translates the game again (a fewminutes, once).
LONGSHOT_JOBSitself isn't in the lift key.prompt_assets.pyis part of the prompts' key, so they are made again once (seconds).player who set 1 or 2 for memory reasons will see slower updates than before.
processor groups), the preparation gets every CPU, as before.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.🤖 Generated with Claude Code
WIP: fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compileto fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compile