feat(queue): an interface for tools: status and history as JSON, move to reorder, settings read again while a job waits #1852

Open
dyst wants to merge 6 commits from dyst/Longshot:feat/queue-interface into main
Member

Summary

The PC's job queue (dev.py queue) had no interface a tool could use. A dashboard, a script or an agent wanting to show what runs and waits, and why, could only scrape status's text: #1802 left status --json and history --json behind with job_queue.py. And nobody could change the order of waiting jobs. A waiting job also read job_queue.json once, when it was queued, so a slot changed there reached only the jobs queued after the change.

This PR adds three things:

  • dev.py queue status --json and history --json [-n N]: one JSON object ("schema": 1) with what status prints and more:
    • each waiting job's place, why it waits, and whether it was placed by hand;
    • the slots, memory, disk, CPUs, games and builds outside the queue;
    • every setting with where it comes from (environment, job_queue.json or default) and the settings file's path.
  • dev.py queue move JOB --first | --last | --to N | --before JOB | --after JOB, and move --clear:
    • The jobs you place go first, in the order you set, before any priority.
    • A move writes the queue as it then stands, so a job queued later goes behind them until --clear gives the order back to the queue.
    • No rule is jumped: a timed run still goes alone, and timed runs of one group follow the order one at a time.
  • A waiting job reads job_queue.json again every 5 s. A slot opened there starts the jobs already waiting. The queue's folders are kept: a job never changes queue.

Builds on #1816: this branch's first four commits are #1816's, needed for the memory in status --json on Windows and for memory_wait()'s reason in it. Review the last commit, feat(queue): an interface for tools....

Read first: scripts/devtools/jobqueue.py: snapshot(), rank()/ahead()/placed(), move(), the settings re-read in run().

How the order is kept:

  • It lives in <queue>/.order, JSON, written under the queue's .lock, as runner pid and start time, so a pid used again by a later job is not the placed one.
  • It is not a *.json file because every version of the queue reads those in that folder as tickets, and drops one with no live runner. A first try named it order.json: the runners deleted it at once.
  • Older versions of the queue don't read it. They decide under the same lock, so no job starts twice, but one of their jobs may start ahead of the order.
  • The runners never write it, so their own ticket writes can't undo a move.

Prior work: searched the tracker for a queue order or reorder, queue priority, status --json, history --json, queue status, move, cancel and settings: nothing but #1523, which brought status --json's Windows fields that #1802 then dropped. No claims in the chat.

Testing

Tested: Windows 11, 64 GB, on a private queue and read-only on this PC's own.

  • New checks in tests/test_job_queue.py:
    • Order: placed jobs go first and before a high priority; a reused pid is not the placed job; a timed run's group follows the order.
    • Move: every way to place a job, by name and by id; a job queued after a move goes behind it; --clear; a running job, an unknown name and a doubled name are refused; .order survives the runners' cleanup of dead tickets; status's snapshot gives each job's place, its reason and whether it was placed.
    • StatusJson: through dev.py, the schema, the slots, and each setting's source (environment over file over default).
    • RealOrder, real jobs on any OS: a job moved first starts first; a waiting job with no light slot starts once job_queue.json opens one.
  • Broken versions: four, each caught by the checks: the order ignored in the rank, no settings re-read, the order file named *.json, and the order ignored for a timed group.
  • By hand:
    • dev.py queue status, status --json, history -n 2 --json and move --help on this PC's queue;
    • status --json read LONGSHOT_QUEUE_STATE as coming from the settings file, and the file's path.
  • Every check: tests/run_all.py (mise run test's checks), with the Builder's llvm-mingw first on PATH: 692 passed, 56 failed, 123 skipped. The failing checks, rerun here and on a clean main with the same compiler, fail the same on both, none here only (main's known Windows failures).

Still to test:

  • Linux: python -I -B tests/test_job_queue.py Order Move StatusJson RealOrder, expecting all to pass. Nothing in the order or the JSON is system-specific (by hand).

Measurements

Verdict: no cost. status takes as long as on main, and the settings re-read costs 236 µs every 5 s while a job waits.

Setup: Windows 11, 64 GB. No game run: a development script. Measured dev.py queue status wall time, 5 runs each, interleaved in two rounds, the median of the second (the first had a cold start on main), on this PC's queue with one job running; and settings() timed over 5 × 1000 calls, the median.

Before: main @ 3f2c9b3c After: this PR @ 13df2323
dev.py queue status 0.19 s 0.19 s
dev.py queue status --json – (no such option) 0.19 s
A waiting job's settings re-read – (read once) 236 µs every 5 s

Risks

  • What could break:
    • A job placed by hand goes before a high-priority one queued earlier: that is what a move asks for, and status marks it (placed).
    • Jobs queued after a move go behind the placed ones until move --clear.
    • An older version of the queue on the same PC doesn't follow the order, but can't start a job twice.
    • status's reason for a waiting job is now exact, where it said starting (or waiting for memory).
  • Turning it off: don't use move, or dev.py queue move --clear; revert the commit for the rest. Nothing changes until someone moves a job or edits job_queue.json.
  • Settings: no new setting. LONGSHOT_QUEUE_DIR, LONGSHOT_QUEUE_STATE and LONGSHOT_QUEUE_QUIET show in new lines only as the queue's existing settings, read as before (the re-read keeps the first two as the job started with), at their defaults.
  • For a player: nothing changes; the queue is for development.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A check under tests/ that reads changed source is updated, not weakened.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary The PC's job queue (`dev.py queue`) had no interface a tool could use. A dashboard, a script or an agent wanting to show what runs and waits, and why, could only scrape `status`'s text: #1802 left `status --json` and `history --json` behind with `job_queue.py`. And nobody could change the order of waiting jobs. A waiting job also read `job_queue.json` once, when it was queued, so a slot changed there reached only the jobs queued after the change. This PR adds three things: - **`dev.py queue status --json`** and **`history --json [-n N]`**: one JSON object (`"schema": 1`) with what `status` prints and more: - each waiting job's place, why it waits, and whether it was placed by hand; - the slots, memory, disk, CPUs, games and builds outside the queue; - every setting with where it comes from (environment, `job_queue.json` or default) and the settings file's path. - **`dev.py queue move JOB --first | --last | --to N | --before JOB | --after JOB`**, and **`move --clear`**: - The jobs you place go first, in the order you set, before any priority. - A move writes the queue as it then stands, so a job queued later goes behind them until `--clear` gives the order back to the queue. - No rule is jumped: a timed run still goes alone, and timed runs of one group follow the order one at a time. - **A waiting job reads `job_queue.json` again every 5 s.** A slot opened there starts the jobs already waiting. The queue's folders are kept: a job never changes queue. **Builds on #1816:** this branch's first four commits are #1816's, needed for the memory in `status --json` on Windows and for `memory_wait()`'s reason in it. Review the last commit, `feat(queue): an interface for tools...`. **Read first:** `scripts/devtools/jobqueue.py`: `snapshot()`, `rank()`/`ahead()`/`placed()`, `move()`, the settings re-read in `run()`. **How the order is kept:** - It lives in `<queue>/.order`, JSON, written under the queue's `.lock`, as runner pid and start time, so a pid used again by a later job is not the placed one. - It is not a `*.json` file because every version of the queue reads those in that folder as tickets, and drops one with no live runner. A first try named it `order.json`: the runners deleted it at once. - Older versions of the queue don't read it. They decide under the same lock, so no job starts twice, but one of their jobs may start ahead of the order. - The runners never write it, so their own ticket writes can't undo a move. **Prior work:** searched the tracker for a queue order or reorder, queue priority, `status --json`, `history --json`, queue status, move, cancel and settings: nothing but #1523, which brought `status --json`'s Windows fields that #1802 then dropped. No claims in the chat. ## Testing **Tested:** Windows 11, 64 GB, on a private queue and read-only on this PC's own. - **New checks in `tests/test_job_queue.py`:** - `Order`: placed jobs go first and before a high priority; a reused pid is not the placed job; a timed run's group follows the order. - `Move`: every way to place a job, by name and by id; a job queued after a move goes behind it; `--clear`; a running job, an unknown name and a doubled name are refused; `.order` survives the runners' cleanup of dead tickets; `status`'s snapshot gives each job's place, its reason and whether it was placed. - `StatusJson`: through `dev.py`, the schema, the slots, and each setting's source (environment over file over default). - `RealOrder`, real jobs on any OS: a job moved first starts first; a waiting job with no light slot starts once `job_queue.json` opens one. - **Broken versions:** four, each caught by the checks: the order ignored in the rank, no settings re-read, the order file named `*.json`, and the order ignored for a timed group. - **By hand:** - `dev.py queue status`, `status --json`, `history -n 2 --json` and `move --help` on this PC's queue; - `status --json` read `LONGSHOT_QUEUE_STATE` as coming from the settings file, and the file's path. - **Every check:** `tests/run_all.py` (`mise run test`'s checks), with the Builder's llvm-mingw first on `PATH`: 692 passed, 56 failed, 123 skipped. The failing checks, rerun here and on a clean `main` with the same compiler, fail the same on both, none here only (`main`'s known Windows failures). **Still to test:** - Linux: `python -I -B tests/test_job_queue.py Order Move StatusJson RealOrder`, expecting all to pass. Nothing in the order or the JSON is system-specific (by hand). ## Measurements **Verdict:** no cost. `status` takes as long as on `main`, and the settings re-read costs 236 µs every 5 s while a job waits. Setup: Windows 11, 64 GB. No game run: a development script. Measured `dev.py queue status` wall time, 5 runs each, interleaved in two rounds, the median of the second (the first had a cold start on `main`), on this PC's queue with one job running; and `settings()` timed over 5 × 1000 calls, the median. | | Before: main @ 3f2c9b3c | After: this PR @ 13df2323 | |---|---|---| | `dev.py queue status` | 0.19 s | 0.19 s | | `dev.py queue status --json` | – (no such option) | 0.19 s | | A waiting job's settings re-read | – (read once) | 236 µs every 5 s | ## Risks - **What could break:** - A job placed by hand goes before a high-priority one queued earlier: that is what a move asks for, and `status` marks it `(placed)`. - Jobs queued after a move go behind the placed ones until `move --clear`. - An older version of the queue on the same PC doesn't follow the order, but can't start a job twice. - `status`'s reason for a waiting job is now exact, where it said `starting (or waiting for memory)`. - **Turning it off:** don't use `move`, or `dev.py queue move --clear`; revert the commit for the rest. Nothing changes until someone moves a job or edits `job_queue.json`. - **Settings:** no new setting. `LONGSHOT_QUEUE_DIR`, `LONGSHOT_QUEUE_STATE` and `LONGSHOT_QUEUE_QUIET` show in new lines only as the queue's existing settings, read as before (the re-read keeps the first two as the job started with), at their defaults. - **For a player:** nothing changes; the queue is for development. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Three checks on the PC's own memory, on any system: meminfo() reads a total and what is available; with
LONGSHOT_QUEUE_MIN_FREE_GB above what any PC has a job waits, and with none it doesn't; with
LONGSHOT_QUEUE_ROOMY_GB at 0 the third build slot (LONGSHOT_QUEUE_BUILD_SLOTS_MAX) opens, and with more than
the PC has it doesn't. Memory pressure, read on Linux alone, is stood in for so a busy PC can't decide them.

On main they fail on Windows: meminfo() reads only /proc/meminfo, so there it is empty, no job waits for
memory and the third build slot never opens. They pass on Linux.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(queue): read the PC's memory on Windows: jobs wait for it, a third build needs room
Some checks failed
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 12s
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Website build and checks (pull_request) Has been cancelled
9839139546
meminfo() read /proc/meminfo alone, so on Windows it was empty: no job waited for free memory
(LONGSHOT_QUEUE_MIN_FREE_GB), the third build slot (LONGSHOT_QUEUE_BUILD_SLOTS_MAX) never opened, and status
showed no memory. #1523 brought Windows the player and outside-build checks; this is the memory.

On Windows it now reads GlobalMemoryStatusEx (ctypes): the physical memory's total and what is available, as
MemTotal and MemAvailable. The wait there is on the memory available alone: Windows has no PSI, and its page
file says nothing like Linux's swap in use, so LONGSHOT_QUEUE_MAX_PRESSURE and LONGSHOT_QUEUE_MAX_SWAP_GB stay
Linux's; status no longer shows a swap it didn't read. Linux is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
From review: with the memory read on Windows, a light job or a first build on a PC with less than
LONGSHOT_QUEUE_MIN_FREE_GB available waited even with nothing else in the queue, maybe for ever, since only
programs outside the queue could free memory for it. Checks, on Windows' reader standing in for a 16 GB PC: a
light, shared or build job alone below the floor starts; beside a running job it waits, naming the memory; it
starts once the memory comes back; the third build slot stays shut without LONGSHOT_QUEUE_ROOMY_GB free. On real
jobs: one alone starts though no PC has the memory asked, one beside it waits, says why, and starts once alone.

The first and the real-job check fail here: memory_wait doesn't exist yet, and a job alone waits.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
From review: a job waited for LONGSHOT_QUEUE_MIN_FREE_GB even with nothing else in the queue, so on a PC short
of memory (16 GB with a browser open, now that Windows' memory is read) a light check or a first build could wait
for ever: only programs outside the queue could free memory for it. memory_wait() asks for the floor only while
another job of the queue runs, where the queue causes the pressure and a job ending frees memory; a job alone
starts and says how little was free. Linux and Windows alike. status gives a waiting job's real reason (the
memory, or "starting"), where it said "starting (or waiting for memory)".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
test(queue): the real-job memory check says what the runners printed when it fails
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Failing after 4m54s
fbdbf85d31
It failed once on Windows while another branch's full run_all loaded the PC, and passed in every rerun (alone,
six at once, four whole files at once): its messages now carry what the runners said and their tickets' states.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(queue): an interface for tools: status and history as JSON, move to reorder, settings read again while a job waits
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Failing after 4m37s
13df23233a
A tool that shows or runs the queue (a dashboard, a script) had only status's text to read, and #1802 left
status --json and history --json behind with job_queue.py. Nobody could change the order of waiting jobs, and a
waiting job read job_queue.json once, so a slot changed there reached only jobs queued after.

- status --json: one object (schema 1) with what status prints and more: each waiting job's place, why it waits
  (memory too, where status said "starting (or waiting for memory)") and whether it was placed by hand; the
  slots, memory, disk, CPUs, games, builds outside the queue, and every setting with where it comes from
  (environment, job_queue.json, default). history --json [-n N] the same for the history.
- move JOB --first|--last|--to N|--before JOB|--after JOB, and move --clear: writes the queue as it stands, with
  the job where asked, to <queue>/.order under the queue's lock. Placed jobs go first, in that order, before any
  priority; a job queued later goes behind them; timed runs of a group follow it. Keys are pid and start time, so a
  pid used again is not the placed job. Not *.json: every version reads those as tickets and drops one with no
  live runner. Older versions don't read it; they decide under the same lock, so no job starts twice.
- A waiting job reads the settings again every 5 s (its queue's folders excepted).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Failing after 4m37s
This pull request has changes conflicting with the target branch.
  • docs/dev/developing.md
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/queue-interface:dyst-feat/queue-interface'
git switch 'dyst-feat/queue-interface'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1852
No description provided.