feat(scripts): one workspace file for the development folders and settings #1808

Closed
dyst wants to merge 1 commit from dyst/Longshot:feat/workspace-file into main
Member

Type: feat · Risk: low · Tested on: Windows 11 · Needs: Linux, any GPU (a script check, no game run)

TL;DR

Setting up a PC to build Longshot means saving four or five folder variables for every user, and a terminal opened before a change keeps the old ones, so a build can quietly use another folder or another queue. This PR lets one small file kept with those folders say where everything is, with paths relative to the file, so the whole setup can move to another drive and every checkout beside it shares one queue. Without the file nothing changes. 13 new checks pass, and on a Windows PC with the saved variables removed the scripts found the same folders from the file alone.

Breakdown

  • Who it helps: anyone who builds Longshot with dev_build.py and the job queue, above all several agents sharing one PC, or a setup kept on a second drive.
  • What was wrong: the scripts find their folders only from environment variables or files in the user's settings folder, so every shell and every user needs them saved, and a stale shell uses other folders without a word.
  • What changes: a longshot-workspace.json above the checkout, when there is one, gives the folders and the queue's and dev_build's settings, relative to itself.
  • What does not change: without a file, nothing: the environment still wins over the file, and job_queue.json and dev_build.json are still read below it. The game is untouched.
  • How it was checked: 13 new checks; the queue's and dev_build's own checks; every check against main with the Builder's compiler; the scripts run on this PC with and without the saved variables.
  • What is still open: a run on Linux.
  • Risk: low: a file that can't be read stops the script and names it, rather than quietly using other folders.

Why

dev_build.py, dev_packs.py and job_queue.py take their folders (LONGSHOT_DEV_HOME, the object and translation stores, the queue's state) from environment variables, and their settings from job_queue.json and dev_build.json in the user's settings folder. So each PC needs those variables saved for each user, a terminal or editor started before they changed keeps the old values, and a build run with them silently makes a second store, or puts its ticket in another queue than everyone else's, so two timed runs can start at once. Moving the folders to a bigger drive means changing every variable again. Searched the tracker for a workspace file, LONGSHOT_WORKSPACE, portable folders and job_queue.json: nothing open or closed.

What changed

One file, longshot-workspace.json, kept with the folders it describes, every path relative to it. The scripts look for it in the checkout's folder and each folder above it, so a main checkout and the worktrees beside it find the same file; LONGSHOT_WORKSPACE names one elsewhere (%APPDATA%\Longshot, ../..), or 0 turns it off. The order is the environment, then the file, then job_queue.json / dev_build.json, then the defaults.

{
  "paths": {"LONGSHOT_DEV_HOME": "longshot/dev", "LONGSHOT_OBJ_STORE": "longshot/store/obj",
            "LONGSHOT_LIFT_STORE": "longshot/store/lift", "LONGSHOT_QUEUE_STATE": "longshot/dev/queue"},
  "job_queue": {"LONGSHOT_QUEUE_BUILD_SLOTS": 2},
  "dev_build": {"romfs": "games/romfs"}
}
  • scripts: scripts/workspace.py (new): find(), load() and path(); a file that is there but can't be read stops the script, naming it.
  • queue: job_queue.py's settings() reads the file's job_queue and paths under the environment; settings_file() says which file the settings come from, and status (and status --json) print it.
  • dev_build: home() and config() read the file (dev_build's romfs, exefs, python and deps relative to it); workspace_env() sets the file's paths, absolute, in the environment of the builds it runs, so the tools under them see the same folders.
  • packs: dev_packs.py's object_store() and lift_store() take the file's stores.
  • docs: docs/development.md, Sharing a PC: the file, the order, an example.
  • tests: tests/test_workspace.py (13 checks); tests/test_dev_build.py and tests/test_job_queue.py run with LONGSHOT_WORKSPACE=0, so a workspace file around a checkout doesn't reach them.

Measurements

Verdict: no cost that shows. Loading the queue's settings takes the same time with the file as without it, within the spread between runs.

Hardware: Windows 11, the PC also running the full test suite (light jobs)
Scene: none: a development script, no game run
Method: python -I -B -c "import job_queue; job_queue.settings()" from the checkout, 8 runs each, interleaved, the median; the file's search and read is the only difference (LONGSHOT_WORKSPACE=0 turns it off)
How to reproduce: a longshot-workspace.json in the folder above the checkout; from the checkout's scripts/, time python -I -B -c "import job_queue; job_queue.settings()" with LONGSHOT_WORKSPACE unset and with it set to 0, interleaved

Before: no file (LONGSHOT_WORKSPACE=0) @ a851d08d After: the workspace file @ a851d08d
Loading the queue's settings, median of 8 380 ms 394 ms
Range 318–488 ms 313–542 ms

Baseline: the same commit with the file turned off rather than main, because the file's search and read is the whole difference, and it isolates it from the rest of main.

Flags and settings

One new variable for the development scripts, not the game (no row in gpu/vk/switches.def, which lists the game's), and four existing ones that the file can now give too.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
LONGSHOT_WORKSPACE the workspace file, or its folder; 0: none – unset: the file above the checkout, if any which folders and settings the development scripts use helps a shared or moved setup; nothing without a file leave unset; set it to keep the file elsewhere
LONGSHOT_DEV_HOME, LONGSHOT_OBJ_STORE, LONGSHOT_LIFT_STORE, LONGSHOT_QUEUE_STATE the development folders, as before the environment, else the user's cache folder the environment, else the workspace file's paths, else as before none: a second place to give the same folders helps a shared or moved setup as before; in the file when there is one

Testing

Tested:

  • Windows 11: tests/test_workspace.py: 13 of 13. tests/test_dev_build.py: OK. tests/test_job_queue.py: one failure, test_ram_file_system, which fails the same way on main (on Windows on_ram() returns no before it reads the mount table the check gives it).
  • Every check, tests/run_all.py with the Builder's llvm-mingw first on PATH: 701 passed, 49 failed, 126 skipped; the failing ones rerun on this branch and on a clean main with the same compiler fail the same on both, none on this branch only (they are main's known Windows failures)
  • On this PC, with a workspace file at the top of the folder that holds the checkouts: with the saved variables removed from the environment, dev_build.py env --json gave the same dev home, object store and translation store as the saved variables, and job_queue.py status read its settings from the file (same slots); with LONGSHOT_WORKSPACE=0, the queue fell back to job_queue.json as before.

Not tested: Linux. The code is path handling with the standard library, and the new checks run on any system, but nobody has run them there.

Must be tested:

  • Linux, any GPU, by hand: put a longshot-workspace.json with "paths": {"LONGSHOT_DEV_HOME": "dev"} in the folder above a checkout and run python -I -B tests/test_workspace.py, then python -I -B scripts/dev_build.py env --json and python -I -B scripts/job_queue.py status from the checkout, expecting the file's dev folder as the dev home and settings: naming the file.

Risks

What could break: a longshot-workspace.json that someone left above a checkout now decides that checkout's folders (job_queue.py status names the file it read). A file that can't be read stops dev_build.py, dev_packs.py and job_queue.py, naming it, until it is fixed or turned off.

Turning it off: LONGSHOT_WORKSPACE=0, or delete the file.

Compatibility: Nothing changes without a file: the environment still wins, and job_queue.json and dev_build.json are read as before.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** low · **Tested on:** Windows 11 · **Needs:** Linux, any GPU (a script check, no game run) ## TL;DR Setting up a PC to build Longshot means saving four or five folder variables for every user, and a terminal opened before a change keeps the old ones, so a build can quietly use another folder or another queue. This PR lets one small file kept with those folders say where everything is, with paths relative to the file, so the whole setup can move to another drive and every checkout beside it shares one queue. Without the file nothing changes. 13 new checks pass, and on a Windows PC with the saved variables removed the scripts found the same folders from the file alone. ## Breakdown - **Who it helps:** anyone who builds Longshot with `dev_build.py` and the job queue, above all several agents sharing one PC, or a setup kept on a second drive. - **What was wrong:** the scripts find their folders only from environment variables or files in the user's settings folder, so every shell and every user needs them saved, and a stale shell uses other folders without a word. - **What changes:** a `longshot-workspace.json` above the checkout, when there is one, gives the folders and the queue's and `dev_build`'s settings, relative to itself. - **What does not change:** without a file, nothing: the environment still wins over the file, and `job_queue.json` and `dev_build.json` are still read below it. The game is untouched. - **How it was checked:** 13 new checks; the queue's and `dev_build`'s own checks; every check against `main` with the Builder's compiler; the scripts run on this PC with and without the saved variables. - **What is still open:** a run on Linux. - **Risk:** low: a file that can't be read stops the script and names it, rather than quietly using other folders. ## Why `dev_build.py`, `dev_packs.py` and `job_queue.py` take their folders (`LONGSHOT_DEV_HOME`, the object and translation stores, the queue's state) from environment variables, and their settings from `job_queue.json` and `dev_build.json` in the user's settings folder. So each PC needs those variables saved for each user, a terminal or editor started before they changed keeps the old values, and a build run with them silently makes a second store, or puts its ticket in another queue than everyone else's, so two timed runs can start at once. Moving the folders to a bigger drive means changing every variable again. Searched the tracker for a workspace file, `LONGSHOT_WORKSPACE`, portable folders and `job_queue.json`: nothing open or closed. ## What changed One file, `longshot-workspace.json`, kept with the folders it describes, every path relative to it. The scripts look for it in the checkout's folder and each folder above it, so a main checkout and the worktrees beside it find the same file; `LONGSHOT_WORKSPACE` names one elsewhere (`%APPDATA%\Longshot`, `../..`), or `0` turns it off. The order is the environment, then the file, then `job_queue.json` / `dev_build.json`, then the defaults. ```json { "paths": {"LONGSHOT_DEV_HOME": "longshot/dev", "LONGSHOT_OBJ_STORE": "longshot/store/obj", "LONGSHOT_LIFT_STORE": "longshot/store/lift", "LONGSHOT_QUEUE_STATE": "longshot/dev/queue"}, "job_queue": {"LONGSHOT_QUEUE_BUILD_SLOTS": 2}, "dev_build": {"romfs": "games/romfs"} } ``` - scripts: `scripts/workspace.py` (new): `find()`, `load()` and `path()`; a file that is there but can't be read stops the script, naming it. - queue: `job_queue.py`'s `settings()` reads the file's `job_queue` and `paths` under the environment; `settings_file()` says which file the settings come from, and `status` (and `status --json`) print it. - dev_build: `home()` and `config()` read the file (`dev_build`'s `romfs`, `exefs`, `python` and `deps` relative to it); `workspace_env()` sets the file's paths, absolute, in the environment of the builds it runs, so the tools under them see the same folders. - packs: `dev_packs.py`'s `object_store()` and `lift_store()` take the file's stores. - docs: `docs/development.md`, *Sharing a PC*: the file, the order, an example. - tests: `tests/test_workspace.py` (13 checks); `tests/test_dev_build.py` and `tests/test_job_queue.py` run with `LONGSHOT_WORKSPACE=0`, so a workspace file around a checkout doesn't reach them. ## Measurements **Verdict:** no cost that shows. Loading the queue's settings takes the same time with the file as without it, within the spread between runs. Hardware: Windows 11, the PC also running the full test suite (light jobs) Scene: none: a development script, no game run Method: `python -I -B -c "import job_queue; job_queue.settings()"` from the checkout, 8 runs each, interleaved, the median; the file's search and read is the only difference (`LONGSHOT_WORKSPACE=0` turns it off) How to reproduce: a `longshot-workspace.json` in the folder above the checkout; from the checkout's `scripts/`, time `python -I -B -c "import job_queue; job_queue.settings()"` with `LONGSHOT_WORKSPACE` unset and with it set to `0`, interleaved | | Before: no file (`LONGSHOT_WORKSPACE=0`) @ a851d08d | After: the workspace file @ a851d08d | |---|---|---| | Loading the queue's settings, median of 8 | 380 ms | 394 ms | | Range | 318–488 ms | 313–542 ms | Baseline: the same commit with the file turned off rather than `main`, because the file's search and read is the whole difference, and it isolates it from the rest of `main`. ### Flags and settings One new variable for the development scripts, not the game (no row in `gpu/vk/switches.def`, which lists the game's), and four existing ones that the file can now give too. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_WORKSPACE` | the workspace file, or its folder; `0`: none | – | unset: the file above the checkout, if any | which folders and settings the development scripts use | helps a shared or moved setup; nothing without a file | leave unset; set it to keep the file elsewhere | | `LONGSHOT_DEV_HOME`, `LONGSHOT_OBJ_STORE`, `LONGSHOT_LIFT_STORE`, `LONGSHOT_QUEUE_STATE` | the development folders, as before | the environment, else the user's cache folder | the environment, else the workspace file's `paths`, else as before | none: a second place to give the same folders | helps a shared or moved setup | as before; in the file when there is one | ## Testing **Tested:** - Windows 11: `tests/test_workspace.py`: 13 of 13. `tests/test_dev_build.py`: OK. `tests/test_job_queue.py`: one failure, `test_ram_file_system`, which fails the same way on `main` (on Windows `on_ram()` returns no before it reads the mount table the check gives it). - Every check, `tests/run_all.py` with the Builder's llvm-mingw first on `PATH`: 701 passed, 49 failed, 126 skipped; the failing ones rerun on this branch and on a clean `main` with the same compiler fail the same on both, none on this branch only (they are `main`'s known Windows failures) - On this PC, with a workspace file at the top of the folder that holds the checkouts: with the saved variables removed from the environment, `dev_build.py env --json` gave the same dev home, object store and translation store as the saved variables, and `job_queue.py status` read its settings from the file (same slots); with `LONGSHOT_WORKSPACE=0`, the queue fell back to `job_queue.json` as before. **Not tested:** Linux. The code is path handling with the standard library, and the new checks run on any system, but nobody has run them there. **Must be tested:** - Linux, any GPU, by hand: put a `longshot-workspace.json` with `"paths": {"LONGSHOT_DEV_HOME": "dev"}` in the folder above a checkout and run `python -I -B tests/test_workspace.py`, then `python -I -B scripts/dev_build.py env --json` and `python -I -B scripts/job_queue.py status` from the checkout, expecting the file's `dev` folder as the dev home and `settings:` naming the file. ## Risks **What could break:** a `longshot-workspace.json` that someone left above a checkout now decides that checkout's folders (`job_queue.py status` names the file it read). A file that can't be read stops `dev_build.py`, `dev_packs.py` and `job_queue.py`, naming it, until it is fixed or turned off. **Turning it off:** `LONGSHOT_WORKSPACE=0`, or delete the file. **Compatibility:** Nothing changes without a file: the environment still wins, and `job_queue.json` and `dev_build.json` are read as before. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(scripts): one workspace file for the development folders and settings
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 18s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 23s
CI / Workflow lint (actionlint) (pull_request) Successful in 24s
Third-party parts / As their notes say (pull_request) Successful in 27s
PR description / Description follows the template (pull_request_target) Successful in 11s
a851d08dc7
A longshot-workspace.json in the checkout's folder or any folder above it holds the development folders
(LONGSHOT_DEV_HOME, the object and translation stores, the queue's state) and the queue's and dev_build's
settings, every path relative to the file: a workspace moved to another folder or drive keeps working with
no variable saved anywhere, and the main checkout and the worktrees beside it share one queue.
LONGSHOT_WORKSPACE names another file or its folder (absolute or relative to the checkout, %VAR%, $VAR and ~
expanded); 0 turns it off. Without a file nothing changes.

The order: the environment, then the workspace file, then job_queue.json and dev_build.json, then the
defaults. dev_build.py also sets the file's paths, as absolute paths, in the environment of the builds it
runs. A file that is there but can't be read stops the script, naming it: one that skipped it would put its
tickets in another queue than everyone else's. job_queue.py status names the file it read.

scripts/workspace.py, read by job_queue.py, dev_build.py and dev_packs.py; tests/test_workspace.py (13
checks); the queue's and dev_build's own checks run with LONGSHOT_WORKSPACE=0 so a workspace file around the
checkout doesn't reach them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #1802 WIP: refactor(dev): docs, tools and process revamp: one dev.py, dev.py ab, one dev folder, a short PR template, faster dev builds — Overlapping developer tooling and shared workspace/layout changes
Possibly related, for context: - PR #1802 WIP: refactor(dev): docs, tools and process revamp: one dev.py, dev.py ab, one dev folder, a short PR template, faster dev builds — Overlapping developer tooling and shared workspace/layout changes
Author
Member

Description: Flags and settings now lists the four existing folder variables the workspace file can also give, and the checklist has the box for the queue's and dev_build's checks, which run with LONGSHOT_WORKSPACE=0.

Description: Flags and settings now lists the four existing folder variables the workspace file can also give, and the checklist has the box for the queue's and dev_build's checks, which run with LONGSHOT_WORKSPACE=0.
Author
Member

Closing: superseded by #1802. Its one development folder (dev.py paths, LONGSHOT_DEV_HOME) gives the builds, the stores and the queue their folders from one setting, which is what this did, and the scripts this changed (job_queue.py, dev_build.py, dev_packs.py) are gone. Nothing here is left to port.

Closing: superseded by #1802. Its one development folder (`dev.py paths`, `LONGSHOT_DEV_HOME`) gives the builds, the stores and the queue their folders from one setting, which is what this did, and the scripts this changed (`job_queue.py`, `dev_build.py`, `dev_packs.py`) are gone. Nothing here is left to port.
dyst closed this pull request 2026-11-20 05:29:29 +00:00
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 18s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 23s
CI / Workflow lint (actionlint) (pull_request) Successful in 24s
Third-party parts / As their notes say (pull_request) Successful in 27s
PR description / Description follows the template (pull_request_target) Successful in 11s

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1808
No description provided.