feat(scripts): one workspace file for the development folders and settings #1808
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1808
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:feat/workspace-file"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Windows 11 · Needs: Linux, any GPU (a script check, no game run)
TL;DR
Setting up a PC to build Longshot means saving four or five folder variables for every user, and a terminal opened before a change keeps the old ones, so a build can quietly use another folder or another queue. This PR lets one small file kept with those folders say where everything is, with paths relative to the file, so the whole setup can move to another drive and every checkout beside it shares one queue. Without the file nothing changes. 13 new checks pass, and on a Windows PC with the saved variables removed the scripts found the same folders from the file alone.
Breakdown
dev_build.pyand the job queue, above all several agents sharing one PC, or a setup kept on a second drive.longshot-workspace.jsonabove the checkout, when there is one, gives the folders and the queue's anddev_build's settings, relative to itself.job_queue.jsonanddev_build.jsonare still read below it. The game is untouched.dev_build's own checks; every check againstmainwith the Builder's compiler; the scripts run on this PC with and without the saved variables.Why
dev_build.py,dev_packs.pyandjob_queue.pytake their folders (LONGSHOT_DEV_HOME, the object and translation stores, the queue's state) from environment variables, and their settings fromjob_queue.jsonanddev_build.jsonin the user's settings folder. So each PC needs those variables saved for each user, a terminal or editor started before they changed keeps the old values, and a build run with them silently makes a second store, or puts its ticket in another queue than everyone else's, so two timed runs can start at once. Moving the folders to a bigger drive means changing every variable again. Searched the tracker for a workspace file,LONGSHOT_WORKSPACE, portable folders andjob_queue.json: nothing open or closed.What changed
One file,
longshot-workspace.json, kept with the folders it describes, every path relative to it. The scripts look for it in the checkout's folder and each folder above it, so a main checkout and the worktrees beside it find the same file;LONGSHOT_WORKSPACEnames one elsewhere (%APPDATA%\Longshot,../..), or0turns it off. The order is the environment, then the file, thenjob_queue.json/dev_build.json, then the defaults.scripts/workspace.py(new):find(),load()andpath(); a file that is there but can't be read stops the script, naming it.job_queue.py'ssettings()reads the file'sjob_queueandpathsunder the environment;settings_file()says which file the settings come from, andstatus(andstatus --json) print it.home()andconfig()read the file (dev_build'sromfs,exefs,pythonanddepsrelative to it);workspace_env()sets the file's paths, absolute, in the environment of the builds it runs, so the tools under them see the same folders.dev_packs.py'sobject_store()andlift_store()take the file's stores.docs/development.md, Sharing a PC: the file, the order, an example.tests/test_workspace.py(13 checks);tests/test_dev_build.pyandtests/test_job_queue.pyrun withLONGSHOT_WORKSPACE=0, so a workspace file around a checkout doesn't reach them.Measurements
Verdict: no cost that shows. Loading the queue's settings takes the same time with the file as without it, within the spread between runs.
Hardware: Windows 11, the PC also running the full test suite (light jobs)
Scene: none: a development script, no game run
Method:
python -I -B -c "import job_queue; job_queue.settings()"from the checkout, 8 runs each, interleaved, the median; the file's search and read is the only difference (LONGSHOT_WORKSPACE=0turns it off)How to reproduce: a
longshot-workspace.jsonin the folder above the checkout; from the checkout'sscripts/, timepython -I -B -c "import job_queue; job_queue.settings()"withLONGSHOT_WORKSPACEunset and with it set to0, interleavedLONGSHOT_WORKSPACE=0) @a851d08da851d08dBaseline: the same commit with the file turned off rather than
main, because the file's search and read is the whole difference, and it isolates it from the rest ofmain.Flags and settings
One new variable for the development scripts, not the game (no row in
gpu/vk/switches.def, which lists the game's), and four existing ones that the file can now give too.LONGSHOT_WORKSPACE0: noneLONGSHOT_DEV_HOME,LONGSHOT_OBJ_STORE,LONGSHOT_LIFT_STORE,LONGSHOT_QUEUE_STATEpaths, else as beforeTesting
Tested:
tests/test_workspace.py: 13 of 13.tests/test_dev_build.py: OK.tests/test_job_queue.py: one failure,test_ram_file_system, which fails the same way onmain(on Windowson_ram()returns no before it reads the mount table the check gives it).tests/run_all.pywith the Builder's llvm-mingw first onPATH: 701 passed, 49 failed, 126 skipped; the failing ones rerun on this branch and on a cleanmainwith the same compiler fail the same on both, none on this branch only (they aremain's known Windows failures)dev_build.py env --jsongave the same dev home, object store and translation store as the saved variables, andjob_queue.py statusread its settings from the file (same slots); withLONGSHOT_WORKSPACE=0, the queue fell back tojob_queue.jsonas before.Not tested: Linux. The code is path handling with the standard library, and the new checks run on any system, but nobody has run them there.
Must be tested:
longshot-workspace.jsonwith"paths": {"LONGSHOT_DEV_HOME": "dev"}in the folder above a checkout and runpython -I -B tests/test_workspace.py, thenpython -I -B scripts/dev_build.py env --jsonandpython -I -B scripts/job_queue.py statusfrom the checkout, expecting the file'sdevfolder as the dev home andsettings:naming the file.Risks
What could break: a
longshot-workspace.jsonthat someone left above a checkout now decides that checkout's folders (job_queue.py statusnames the file it read). A file that can't be read stopsdev_build.py,dev_packs.pyandjob_queue.py, naming it, until it is fixed or turned off.Turning it off:
LONGSHOT_WORKSPACE=0, or delete the file.Compatibility: Nothing changes without a file: the environment still wins, and
job_queue.jsonanddev_build.jsonare read as before.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Possibly related, for context:
Description: Flags and settings now lists the four existing folder variables the workspace file can also give, and the checklist has the box for the queue's and dev_build's checks, which run with LONGSHOT_WORKSPACE=0.
Closing: superseded by #1802. Its one development folder (
dev.py paths,LONGSHOT_DEV_HOME) gives the builds, the stores and the queue their folders from one setting, which is what this did, and the scripts this changed (job_queue.py,dev_build.py,dev_packs.py) are gone. Nothing here is left to port.Pull request closed