fix(dev): a renderer-only build compiles LONGSHOT_JOBS files at once, not one #1147
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1147
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:dev-packs-renderer-jobs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux x86-64, 16 threads (a renderer-only pack timed before and after; the game-free checks) · Needs: none
Why
A change to the renderer alone, built with
scripts/dev_packs.py packorscripts/dev_build.py vk, compiled one file at a time whateverLONGSHOT_JOBSsaid, so a change to a header every renderer file includes took minutes longer than it needs to. Developers and agents testing a renderer change against a base wait on this every time.The tracker was searched ("dev_packs jobs", "renderer-only pack", "LONGSHOT_JOBS", "dev_build vk"): no issue or open pull request covers it.
How to reproduce
Seen on: Linux x86-64, 16 threads, any GPU (no game is run)
Settings and switches:
LONGSHOT_JOBS=14Steps: on a base,
python -I scripts/dev_packs.py pack <base> <name> --tree <checkout>with a checkout that changes a renderer header (gpu/vk/vk_internal.h), orpython -I scripts/dev_build.py vkExpected:
nx2/build.pycompiles 14 files at onceActual: the command is
nx2/build.py --standalonewith no--jobs, and the build says207 files, 1 jobsWhat changed
The renderer-only build now passes
--jobstonx2/build.py:LONGSHOT_JOBSwhen it is set (0 or empty: automatic), else one compiler for every four CPU threads.nx2/build.pystill starts each compiler only when the free memory holds it.renderer_build_cmd()inscripts/dev_packs.pybuilds the command, andcmd_packruns it; aLONGSHOT_JOBSthat is not a whole number is refused with the messagemake_game.pygives.cmd_vkinscripts/dev_build.pybuilds its renderer with the same helper, run by the build's own Python, in its light queue slot.environment()inscripts/dev_build.pykeeps an exportedLONGSHOT_JOBS. It dropped every name starting withLONGSHOT_JOBto keep the job queue's ownLONGSHOT_JOB,LONGSHOT_JOB_NAMEandLONGSHOT_JOB_MODEout of a build, which droppedLONGSHOT_JOBStoo; it now dropsLONGSHOT_JOBand theLONGSHOT_JOB_prefix only (and an older queue'sOOT_STATUS_*, as before). Found in review. This also cures the same loss ondev_build.py base,packandbuild, which ranmake_game.pyanddev_packs.pywithout an exportedLONGSHOT_JOBSsincedev_build.pycame in (#1085).test_renderer_build_jobsintests/test_dev_packs.py(the count fromLONGSHOT_JOBS, the default, never 0, bad values refused);test_vk_compiles_several_filesintests/test_dev_build.py(the commandcmd_vkqueues carries--jobs);test_vk_takes_the_exported_jobsintests/test_dev_build.py, through the realenvironment()withLONGSHOT_JOBSexported (1, then 14, on 16 threads) and the queue's names set: the queued command carries that count, the build's environment keepsLONGSHOT_JOBS, andLONGSHOT_JOB,LONGSHOT_JOB_NAME,LONGSHOT_JOB_MODEandOOT_STATUS_LOCKare still dropped.docs/development.md, Testing a change against a base, says how many files a renderer-only pack compiles at once;dev_build.py's usage line says it forvk, and its notes sayLONGSHOT_JOBSstays.Measurements
Verdict: a renderer-only pack that compiles the whole renderer went from 402 s to 77 s with
LONGSHOT_JOBS=14, the same 207 files planned and about 200 compiled each time. That is one pair of runs on a shared PC: evidence that the files now compile in parallel, not a repeatable 5.2x. The game-free checks pass, apart from two that fail onmaintoo.Hardware: Linux x86-64 (Ubuntu, kernel 6.8), 16 CPU threads, 62 GB RAM; other agents' work running beside it (load average 4-16)
Scene: no game; a renderer-only pack on a shared development base of main
9c9d69ed, from a checkout that changes 30 files undergpu/vk/(renderer headers among them) andtests/Method: one pack with each
dev_packs.py, run one after the other from the same checkout and toolchain,LONGSHOT_JOBS=14; before each, the base's renderer work folder was emptied (linked again from the base's objects) and the object store pointed at an empty folder, so both compiled the same files from the same state. The time is the pack's ownpack ... in N sline; the load average was read every 10 sHow to reproduce:
LONGSHOT_JOBS=14 LONGSHOT_OBJ_STORE=<empty folder> python -I <dev_packs.py under test> pack <base> jobs-before --tree <checkout>, with the base'srenderer-workfolder moved aside first; then the same with this PR'sdev_packs.pyandjobs-afterBaseline: the base is an older main (
9c9d69ed) because it was the shared base free to use; thedev_packs.pyrun is the file under test in each column, main's and this PR's, andnx2/build.pyis the checkout's in both, so only the command line differs. The after column iscff4ef52: the commits since change onlydev_build.py'senvironment()and its test, which a directdev_packs.py packdoes not go through, so the numbers still hold71b1edd8cff4ef52(0856544bsince the rebase)One run each on a shared PC (the load average differs between the two), so the ratio is not a guarantee: how much faster a renderer-only build gets depends on the threads, the memory and what else runs. The plan line (
1 jobsagainst14 jobs) is the change itself. The two packs'longshot_vk.sodiffer in their bytes; they were not compared further (the renderer's version stamp is built in), and the code that runs in the game is not changed by this PR.Flags and settings
No new switch: the renderer-only build now reads
LONGSHOT_JOBS, which the game's build already reads, with the same rule for its values.LONGSHOT_JOBSmake_game.py's game build run directly; ignored by a renderer-only build, which compiled one file at a time; dropped by everydev_build.pycommand before its build randev_packs.py pack,dev_build.py vk); unset, 0 or empty: one compiler for every four CPU threads; kept bydev_build.py, sobase,pack,buildandvkhonour itdev_build.pygame build compiles the exported countTesting
Tested:
longshot_vk.sowith 0 failures.python -I -B tests/run_all.py --changed forge/mainon this head (Linux, the Vulkan and SPIR-V headers found): 17 passed, 2 failed. Both,test_vk_occlusion_queriesandtest_walk_key, fail the same onmain31f8d8berun the same way.tests/test_dev_packs.py,tests/test_dev_build.py(16 tests) andtests/test_job_queue.pypass.test_renderer_build_jobsfails at the test's commit (d18168d0, was3513fcb7:dev_packshas norenderer_build_cmd) and passes after the fix;test_vk_compiles_several_filesfails on the commandcmd_vkhad (no--jobs).test_vk_takes_the_exported_jobsfails at its commit (62ab0afb, on0856544b; before the rebase 9b6973cc oncff4ef52:LONGSHOT_JOBS=1queued--jobs 4) and passes with the fix. Called directly withLONGSHOT_JOBS=1and the queue's three names exported,environment()returned none of the four before the fix andLONGSHOT_JOBS=1alone after it.Not tested:
os.cpu_count()and a command line), andtests/test_dev_build.py'scmd_vkchecks were run on Linux only.nx2/build.pystarts each compiler only when the free memory holds it, as for a game build.dev_build.pyrun withLONGSHOT_JOBSexported (the check goes throughenvironment()andcmd_vkwith the build itself stubbed).Must be tested: None
Risks
What could break: a renderer-only build now runs several compilers at once, so it takes more CPU and memory beside the job queue's other jobs: at most a quarter of the CPU threads by default,
LONGSHOT_JOBSwhen set, andnx2/build.py's memory gate still applies. ALONGSHOT_JOBSthat is not a whole number now stops a renderer-only build with the message the game's build gives, where before it was ignored.Anyone who exports
LONGSHOT_JOBSand builds withdev_build.py(base,pack,build,vk) now gets that count: before,dev_build.pydropped it andmake_game.pyused its default (one compiler per CPU thread but one). A lower number now gives a slower, lighter build; a value that is not a whole number now stops the build withmake_game.py's message.Turning it off:
LONGSHOT_JOBS=1gives the old one-at-a-time build; or revert the commit.Compatibility: Nothing changes for a player: development scripts only. The objects are the same (the count is in no object key), so a base's objects are reused as before.
Checklist
Everyone
type(scope): description) of the status line's type.main. Onto main31f8d8be(head9cba4513) with no conflict.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.LONGSHOT_JOBSfiles at once, not one 8bc66bdb36dev_build.py vkcompilesLONGSHOT_JOBSfiles at once, not oneWIP: fix(dev): a renderer-only build compilesto fix(dev): a renderer-only build compilesLONGSHOT_JOBSfiles at once, not oneLONGSHOT_JOBSfiles at once, not oneCode-only review of
cff4ef52ec; informational findings, with final approval left to another reviewer.P2 —
dev_build.py vkdrops the exported worker limit before the new helper reads it (scripts/dev_build.py:621).environment()at line 359 excludes every name starting withLONGSHOT_JOB, which includesLONGSHOT_JOBS. Consequently, on a 16-thread PC,LONGSHOT_JOBS=1 python -I scripts/dev_build.py vkqueues--jobs 4, not--jobs 1; an exported14likewise becomes4. The new parallel default therefore cannot be reduced through the documented environment switch on this path. Directdev_packs.py packusesos.environand does not have this problem. PreserveLONGSHOT_JOBSwhen removing queue bookkeeping, for example by removing onlyLONGSHOT_JOBand theLONGSHOT_JOB_prefix. Add a test through the real environment assembly with an exported limit;test_vk_compiles_several_filescurrently replacesenvironment()with a mock that already contains the value, so it bypasses the loss. This is derived from source, not an observed build run.The remaining change is confined to worker-count selection and its callers; the existing compiler memory gate and object keys remain in place. The author reports successful Linux renderer packs and relevant game-free tests. The 402 s versus 77 s result is one pair on a shared PC, useful evidence of parallel compilation but not a repeatable 5.2x guarantee. Native Windows and low-memory operation remain untested, as disclosed; no unrelated gameplay run is needed for this script-only change.
Read the full patch, environment/queue handoff, compiler scheduling and memory gate, new fixtures, and current feedback. The
dev_packs.pyoverlaps with 1122, 737, and 28 concern cache lists, cache defaults, or documentation, not this worker-count path; no functional merge-order dependency found. No tests, builds, or PR code were run.cff4ef52ec9cba451380Two commits after
bibiburl's review:dev_build.py'senvironment()dropped every name starting withLONGSHOT_JOB,LONGSHOT_JOBSamong them, sodev_build.py vkqueued--jobs 4on 16 threads whatever was exported. It now drops only the queue'sLONGSHOT_JOBandLONGSHOT_JOB_*.test_vk_takes_the_exported_jobsgoes through the realenvironment()(fails before the fix, passes after). The same fix letsdev_build.py base,packandbuildpass an exportedLONGSHOT_JOBSon tomake_game.py, which they had dropped since #1085. Description: What changed, Measurements (one pair of runs: evidence of parallel compilation, not a fixed ratio), Testing, Risks, checklist. Also rebased onto main31f8d8be(head9cba4513), with no conflict.Thanks,
bibiburl: confirmed.environment()inscripts/dev_build.pydropped every name starting withLONGSHOT_JOBto keep the job queue's own variables out of a build (LONGSHOT_JOB,LONGSHOT_JOB_NAMEandLONGSHOT_JOB_MODE, the only onesjob_queue.py'sjob_envsets). That droppedLONGSHOT_JOBStoo. Called directly withLONGSHOT_JOBS=1exported, it returned noLONGSHOT_JOBS, andcmd_vkqueued--jobs 4on 16 threads, as you said.It was older than this PR: since #1085,
dev_build.py base,packandbuildalso ranmake_game.pyanddev_packs.pywithout an exportedLONGSHOT_JOBS, somake_game.pyfell back to its own default.Fixed as you suggested: it now drops
LONGSHOT_JOBand theLONGSHOT_JOB_prefix only (and an older queue'sOOT_STATUS_*, as before). The newtest_vk_takes_the_exported_jobsintests/test_dev_build.pygoes through the realenvironment()with only the tool lookup stubbed. It exportsLONGSHOT_JOBS(1, then 14, on 16 threads) together with the queue's names, then checks three things: the commandcmd_vkqueues carries that count, the build's environment keepsLONGSHOT_JOBS, andLONGSHOT_JOB,LONGSHOT_JOB_NAME,LONGSHOT_JOB_MODEandOOT_STATUS_LOCKare still dropped. It fails on the previous head (--jobs 4where 1 was asked) and passes with the fix.I took your point on the measurement too: the description now calls the 402 s to 77 s pair evidence of parallel compilation on a shared PC, not a repeatable 5.2x.
A renderer-only build now compiles several files at once. I read head
9cba4513. It applies onto current main. This can merge as it is.renderer_build_cmdpasses--jobstonx2/build.py.LONGSHOT_JOBSis that count when it is a whole number. Unset, empty or 0 is one compiler for every four CPU threads, and never zero. A value that is not a whole number stops the build with the messagemake_game.pygives.dev_packs.py packanddev_build.py vkboth use it.nx2/build.pystill starts each compiler only when the free memory holds it.environment()used to drop every name starting withLONGSHOT_JOB, sodev_build.pythrew awayLONGSHOT_JOBSbefore the build ran. It now dropsLONGSHOT_JOBand theLONGSHOT_JOB_prefix, and an older queue'sOOT_STATUS_names.LONGSHOT_JOBSstays, sobase,pack,buildandvkpass an exported count on. The check goes through that real filter: an exported 1 or 14 is the--jobson the command, and the queue's own names are still absent.Must be tested is None. The objects do not depend on the count. One pair of packs on Linux,
LONGSHOT_JOBS=14, planned 207 files at 14 jobs and finished in 77 s where the one-at-a-time pack took 402 s.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.