feat(scripts): the PC's job queue (job_queue.py) and dev_build.py, whose builds share one translation and one set of objects #1085
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1085
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:feat/job-queue"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Linux, RTX 3090 · Needs: none
Why
Several agents (and the person they work for) share one PC to build and test Longshot. Builds, game runs and measurements went through a queue kept outside the repository, and under it no build ever took another build's translation from the lift store. That queue gives each job
OOT_STATUS_LOCK=<job name>, andmake_game.py lift_keyhashes everyOOT_variable. So every job had its own lift key, and every new work folder translated the game again. The queue's rules (no build beside a timed run, the GPU shared or not, memory, nothing in/tmp) were also known only to whoever had that script. This puts the queue and a build wrapper in the repository, keeps lift keys clean, and gives every checkout the same build environment.What changed
Two scripts.
scripts/job_queue.pyruns builds and game runs when they fit and keeps measurements apart.scripts/dev_build.pybuilds bases, packs, runtimes and whole game folders through it, all with one environment, so the second build of anything takes its translation and its objects from the shared stores. Neither changes what players build.scripts/job_queue.py(run NAME --mode M -- CMD,status,history,cancel):build,light,shared,gpu,timing,timing-cpuandprofile, each a GPU use and a CPU use;blocker): what runs holds what it uses, and every ready ticket ahead keeps its claim, so nothing jumps ahead;TMPDIRmoved off a RAM file system, and the NVIDIA shader cache kept apart for GPU jobs.A job gets
LONGSHOT_JOB*, never a newOOT_,RX_orA64_variable, and the older queue'sOOT_STATUS_*is dropped. The older queue's running tickets count as running jobs, so the two can be used side by side while sessions move over.scripts/dev_build.py(env,base,pack,rt,build,vk,ls), Linux:tools.lockthroughbuilder/linux/linux_tools.py, installed once pertools.lockunder~/.cache/longshot-dev/builder/(no more tools borrowed from an old checkout), plus a Python with the build's packages.dev_packs.py's object and translation stores, withLONGSHOT_LIFT_STORE_KEEP=6. The launcher's NGX SDK is used for DLSS.A64_,OOT_orRX_variable you set that will enter the lift key.~/.cache/longshot-dev/bases/, built by whoever asks first while the others wait for it, from a temporary workspace of that commit.build-linux/, so they go when the checkout goes.docs/development.md: Sharing a PC: the job queue (the modes, the rules, why no
OOT_variable, nothing on tmpfs). AGENTS.md: builds and game runs go through the queue, never on a RAM file system.tests/test_job_queue.py: the rules above, the job environment, the RAM check, and two real queues (exclusive jobs one after the other, shared ones together).
Measurements
Verdict: the second build of the same main in a new work folder took 22 s instead of 6 min 48 s. It copied the first build's translation from the lift store and took 512 of 514 objects from the object store; only the two objects that carry the version were compiled. Under the older queue's variable, the lift share key changes with each job name, so that build would have translated the game again.
Hardware: Linux (CachyOS), RTX 3090, driver 620.70.06, Ryzen 9 7900X, 30 GB
Method: the build's own
where the time wentlines;make_game.lift_share_keycalled with and without the variables. One run of each: the difference is what was compiled and translated, not noise.How to reproduce:
dev_build.py base, thendev_build.py buildin another checkout.dev_build.py base(main @02903b04, an empty store for this toolchain)dev_build.py build, this PR's checkout, a new work folderFlags and settings
No game switch. The scripts' own settings are environment variables, documented in their notes and in docs/development.md:
LONGSHOT_QUEUE_*: the slots, the memory limits, the CPU sets (by default the first L3 domain for timed runs),LONGSHOT_QUEUE_QUIETand the folders.LONGSHOT_DEV_*:HOME,PYTHON,DEPS,ROMFSandEXEFS.The defaults are the queue's current values; the tools are for agents' and developers' builds only.
Testing
Tested: On Linux, RTX 3090:
gpujobs ran one after the other and twosharedjobs together; exit statuses pass through; the job's environment has noOOT_variable.statusnamed the base's shader preparation as a job's game, not someone playing.dev_build.py:envinstalled thetools.lockbuild tools (5 s) and found the NGX SDK;basebuilt and prepared the base of main02903b04in 12 min;buildreused it as in Measurements.blockerwith the cases in the new test.Not tested:
pack,rtandvkwere not run end to end; they calldev_packs.py pack/rtandnx2/build.pywith the environment above.job_queue.pyhas a Windows path (msvcrt lock, process start times) that has not run, anddev_build.pyrefuses Windows.tests/run_all.py, the newtests/test_job_queue.pyamong them) were not run on this PC, at the person's request (no test suites on this PC), and CI runs its Python source checks only by hand (workflow_dispatch), so they haven't run yet.f7922126(https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/29145): this PR changes none of the files it fingerprints.Must be tested:
python -I -B tests/run_all.py(or CI's Python source checks run by hand), expectingtest_job_queue.pyto pass with everything else (by hand, or the CI dispatch)Risks
What could break: nothing a player builds or runs: the scripts are new, and nothing calls them but people and agents. A rule of the queue could hold a job longer than needed (
statussays what it waits for).Turning it off: don't use it, or revert the commit.
Compatibility: Nothing changes for a player. Agents' builds move to the
tools.locktoolchain, so their first build compiles cold once (a new object store namespace).Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Not run on this PC at the person's request, and CI runs it only by hand: open.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.A companion Tooling PR, MagicBeanDev/Tooling#16, does the same for workspaces: making them, listing them and cleaning up the old ones.
🤖 Generated with Claude Code
CI: 10 passed. Guide screenshots fails as it does on main at
f7922126(https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/29145); this PR touches none of its fingerprinted files. The Python source checks run only on workflow_dispatch, so tests/run_all.py (with the new test_job_queue.py) is still to run: the PR stays WIP until it has.WIP: feat(scripts): the PC's job queue (job_queue.py) and dev_build.py, whose builds share one translation and one set of objectsto feat(scripts): the PC's job queue (job_queue.py) and dev_build.py, whose builds share one translation and one set of objectsLONGSHOT_JOBSfiles at once, not one #1147LONGSHOT_JOBSfiles at once, not one #1147LONGSHOT_JOBSfiles at once, not one #1147