fix(queue): read the PC's memory on Windows: jobs wait for it, a third build needs room #1816

Open
dyst wants to merge 5 commits from dyst/Longshot:fix/queue-windows-memory into main
Member

Summary

On Windows the job queue (dev.py queue) never checked how much memory was free. So it started builds and test runs even when the PC was nearly out of memory, and it never opened the extra build slot meant for a PC with room to spare. It read memory from /proc/meminfo, which Windows doesn't have, so meminfo() was empty there. #1523 brought Windows the queue's player and outside-build checks; this is the memory, the part it left as Linux's.

This PR reads the free memory on Windows too (GlobalMemoryStatusEx), and makes the floor wait only where waiting can help:

  • A job beside a running one waits while less than LONGSHOT_QUEUE_MIN_FREE_GB (6 GB) is available, and says why. On main it starts at once.
  • A job alone in the queue starts however little is free, and says so. Only programs outside the queue could free memory for it, so on a short PC (16 GB, a browser open) it could wait forever. This holds on Linux too, where such a job used to wait.
  • The third build slot opens only with LONGSHOT_QUEUE_ROOMY_GB (14 GB) available, as on Linux.
  • status gives a waiting job's real reason (the memory, or starting), where it said starting (or waiting for memory).

The code is in #1802's scripts/devtools/jobqueue.py, which kept the /proc-only reader.

To see it on main:

  1. On Windows, use a queue of its own (LONGSHOT_QUEUE_DIR and LONGSHOT_QUEUE_STATE in a temporary folder) and set LONGSHOT_QUEUE_MIN_FREE_GB=1000, more than any PC has.
  2. Start a job that keeps running: python -I -B scripts/dev.py queue run hold --mode light -- python -c "import time; time.sleep(60)".
  3. Beside it, run python -I -B scripts/dev.py queue run demo --mode build -- python -c "print('the job ran')".

On main, demo runs at once. Here it waits (queue: demo memory: 29.5 GB free (waits for 1000), and the same in status), and it starts once hold ends.

Read first: scripts/devtools/jobqueue.py, meminfo(), win_meminfo() and memory_wait().

Why only the available memory on Windows:

  • Windows has no PSI, so LONGSHOT_QUEUE_MAX_PRESSURE stays Linux's.
  • Its page file says nothing like Linux's swap in use, so LONGSHOT_QUEUE_MAX_SWAP_GB stays Linux's too.
  • status shows swap only where it is read.

Prior work: searched the tracker for the queue's memory on Windows, meminfo, MIN_FREE_GB and BUILD_SLOTS_MAX: nothing open or closed.

Testing

Tested: Windows 11, 64 GB.

  • tests/test_job_queue.py's Memory checks:
    • The memory is read (it fails on main: 0 not greater than 0).
    • A job waits with a floor no PC has; the third build slot opens with room (2 != 3 on main).
    • With Windows' reader standing in for a 16 GB PC with 3 GB available: a light, shared or build job alone starts; beside a running job it waits, naming the memory; at 7 GB it starts; the third build slot stays shut at 10 GB and opens at 15.
    • On real jobs: one alone starts though no PC has the memory asked; one beside it waits, says why, and starts once it is alone (it waits forever before the floor's fix).
    • Broken versions of the fix each fail them: the total read as available, the page file read as available, nothing available, and waiting jobs counted as running ones.
  • The steps above: run from a clean main and from this branch. On main both jobs start at once. Here a job alone starts with queue: alone starts though 29.6 GB free (the floor is 1000): no other job of the queue runs, so it would wait for programs outside it, and one beside it waits as above.
  • status: memory: 29.5 GB available, and the waiting job's line ends in memory: 29.5 GB free (waits for 1000).
  • Every check: tests/run_all.py (mise run test's checks), with the Builder's llvm-mingw first on PATH: 692 passed, 56 failed, 123 skipped. The failing checks, rerun here and on a clean main with the same compiler, fail the same on both, none here only. These are main's known Windows failures, among them test_job_queue.py's test_ram_file_system (on Windows on_ram() answers no before it reads the table the check gives it).
  • A flaky run: the real-job check failed once while another branch's full run loaded the PC, and passed in every rerun (alone, six at once, four whole files at once). Its messages now say what the runners printed.

Still to test:

  • Linux: python -I -B tests/test_job_queue.py Memory, expecting 6 passes. The reader is unchanged there; the floor now lets a job alone start (by hand).

Measurements

Verdict: fixed. A job short of memory beside another now waits on Windows; a job alone starts on both systems; reading the memory costs 17 µs.

Setup: Windows 11, 64 GB. No game run: a development script. Measured with the Memory checks, then the steps in Summary on a private queue from a clean main and from this branch, then meminfo() timed over 5 × 1000 calls (the median).

Before: main @ 3f2c9b3c After: this PR @ fbdbf85d
tests/test_job_queue.py Memory (6) 6 of 6 fail (4 failures; 2 errors: no Windows reader) 6 pass
A job beside a running one, LONGSHOT_QUEUE_MIN_FREE_GB=1000 runs at once waits: memory: 29.5 GB free (waits for 1000)
A job alone, same floor runs at once (Linux: waits) runs, saying the memory was short
status's memory line none memory: 29.5 GB available
meminfo() – (empty) 16.6 µs a call

Risks

  • What could break:
    • On Windows, a job beside a running one on a PC with less than 6 GB available now waits until a job ends or memory frees, as on Linux. status says why.
    • On Linux, a job alone no longer waits for memory. It starts and says how little was free, where it waited (for programs outside the queue to free some).
  • Turning it off: LONGSHOT_QUEUE_MIN_FREE_GB=0 (no wait), or revert the commit.
  • Settings: no new setting. LONGSHOT_QUEUE_MIN_FREE_GB, LONGSHOT_QUEUE_ROOMY_GB and LONGSHOT_QUEUE_BUILD_SLOTS_MAX now take effect on Windows with the defaults they have.
  • For a player: nothing changes; the queue is for development.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary On Windows the job queue (`dev.py queue`) never checked how much memory was free. So it started builds and test runs even when the PC was nearly out of memory, and it never opened the extra build slot meant for a PC with room to spare. It read memory from `/proc/meminfo`, which Windows doesn't have, so `meminfo()` was empty there. #1523 brought Windows the queue's player and outside-build checks; this is the memory, the part it left as Linux's. This PR reads the free memory on Windows too (`GlobalMemoryStatusEx`), and makes the floor wait only where waiting can help: - **A job beside a running one** waits while less than `LONGSHOT_QUEUE_MIN_FREE_GB` (6 GB) is available, and says why. On `main` it starts at once. - **A job alone in the queue** starts however little is free, and says so. Only programs outside the queue could free memory for it, so on a short PC (16 GB, a browser open) it could wait forever. This holds on Linux too, where such a job used to wait. - **The third build slot** opens only with `LONGSHOT_QUEUE_ROOMY_GB` (14 GB) available, as on Linux. - **`status`** gives a waiting job's real reason (the memory, or `starting`), where it said `starting (or waiting for memory)`. The code is in #1802's `scripts/devtools/jobqueue.py`, which kept the `/proc`-only reader. **To see it on `main`:** 1. On Windows, use a queue of its own (`LONGSHOT_QUEUE_DIR` and `LONGSHOT_QUEUE_STATE` in a temporary folder) and set `LONGSHOT_QUEUE_MIN_FREE_GB=1000`, more than any PC has. 2. Start a job that keeps running: `python -I -B scripts/dev.py queue run hold --mode light -- python -c "import time; time.sleep(60)"`. 3. Beside it, run `python -I -B scripts/dev.py queue run demo --mode build -- python -c "print('the job ran')"`. On `main`, `demo` runs at once. Here it waits (`queue: demo memory: 29.5 GB free (waits for 1000)`, and the same in `status`), and it starts once `hold` ends. **Read first:** `scripts/devtools/jobqueue.py`, `meminfo()`, `win_meminfo()` and `memory_wait()`. **Why only the available memory on Windows:** - Windows has no PSI, so `LONGSHOT_QUEUE_MAX_PRESSURE` stays Linux's. - Its page file says nothing like Linux's swap in use, so `LONGSHOT_QUEUE_MAX_SWAP_GB` stays Linux's too. - `status` shows swap only where it is read. **Prior work:** searched the tracker for the queue's memory on Windows, `meminfo`, `MIN_FREE_GB` and `BUILD_SLOTS_MAX`: nothing open or closed. ## Testing **Tested:** Windows 11, 64 GB. - **`tests/test_job_queue.py`'s `Memory` checks:** - The memory is read (it fails on `main`: `0 not greater than 0`). - A job waits with a floor no PC has; the third build slot opens with room (`2 != 3` on `main`). - With Windows' reader standing in for a 16 GB PC with 3 GB available: a light, shared or build job alone starts; beside a running job it waits, naming the memory; at 7 GB it starts; the third build slot stays shut at 10 GB and opens at 15. - On real jobs: one alone starts though no PC has the memory asked; one beside it waits, says why, and starts once it is alone (it waits forever before the floor's fix). - Broken versions of the fix each fail them: the total read as available, the page file read as available, nothing available, and waiting jobs counted as running ones. - **The steps above:** run from a clean `main` and from this branch. On `main` both jobs start at once. Here a job alone starts with `queue: alone starts though 29.6 GB free (the floor is 1000): no other job of the queue runs, so it would wait for programs outside it`, and one beside it waits as above. - **`status`:** `memory: 29.5 GB available`, and the waiting job's line ends in `memory: 29.5 GB free (waits for 1000)`. - **Every check:** `tests/run_all.py` (`mise run test`'s checks), with the Builder's llvm-mingw first on `PATH`: 692 passed, 56 failed, 123 skipped. The failing checks, rerun here and on a clean `main` with the same compiler, fail the same on both, none here only. These are `main`'s known Windows failures, among them `test_job_queue.py`'s `test_ram_file_system` (on Windows `on_ram()` answers no before it reads the table the check gives it). - **A flaky run:** the real-job check failed once while another branch's full run loaded the PC, and passed in every rerun (alone, six at once, four whole files at once). Its messages now say what the runners printed. **Still to test:** - Linux: `python -I -B tests/test_job_queue.py Memory`, expecting 6 passes. The reader is unchanged there; the floor now lets a job alone start (by hand). ## Measurements **Verdict:** fixed. A job short of memory beside another now waits on Windows; a job alone starts on both systems; reading the memory costs 17 µs. Setup: Windows 11, 64 GB. No game run: a development script. Measured with the `Memory` checks, then the steps in Summary on a private queue from a clean `main` and from this branch, then `meminfo()` timed over 5 × 1000 calls (the median). | | Before: main @ 3f2c9b3c | After: this PR @ fbdbf85d | |---|---|---| | `tests/test_job_queue.py Memory` (6) | 6 of 6 fail (4 failures; 2 errors: no Windows reader) | 6 pass | | A job beside a running one, `LONGSHOT_QUEUE_MIN_FREE_GB=1000` | runs at once | waits: `memory: 29.5 GB free (waits for 1000)` | | A job alone, same floor | runs at once (Linux: waits) | runs, saying the memory was short | | `status`'s memory line | none | `memory: 29.5 GB available` | | `meminfo()` | – (empty) | 16.6 µs a call | ## Risks - **What could break:** - On Windows, a job beside a running one on a PC with less than 6 GB available now waits until a job ends or memory frees, as on Linux. `status` says why. - On Linux, a job alone no longer waits for memory. It starts and says how little was free, where it waited (for programs outside the queue to free some). - **Turning it off:** `LONGSHOT_QUEUE_MIN_FREE_GB=0` (no wait), or revert the commit. - **Settings:** no new setting. `LONGSHOT_QUEUE_MIN_FREE_GB`, `LONGSHOT_QUEUE_ROOMY_GB` and `LONGSHOT_QUEUE_BUILD_SLOTS_MAX` now take effect on Windows with the defaults they have. - **For a player:** nothing changes; the queue is for development. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Three checks on the PC's own memory, on any system: meminfo() reads a total and what is available; with
LONGSHOT_QUEUE_MIN_FREE_GB above what any PC has a job waits, and with none it doesn't; with
LONGSHOT_QUEUE_ROOMY_GB at 0 the third build slot (LONGSHOT_QUEUE_BUILD_SLOTS_MAX) opens, and with more than
the PC has it doesn't. Memory pressure, read on Linux alone, is stood in for so a busy PC can't decide them.

On main they fail on Windows: meminfo() reads only /proc/meminfo, so there it is empty, no job waits for
memory and the third build slot never opens. They pass on Linux.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(queue): read the PC's memory on Windows: jobs wait for it, a third build needs room
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
PR description / Description follows the template (pull_request_target) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 15s
747a35a04d
meminfo() read /proc/meminfo alone, so on Windows it was empty: no job waited for free memory
(LONGSHOT_QUEUE_MIN_FREE_GB), the third build slot (LONGSHOT_QUEUE_BUILD_SLOTS_MAX) never opened, and status
showed no memory. #1523 brought Windows the player and outside-build checks; this is the memory.

On Windows it now reads GlobalMemoryStatusEx (ctypes): the physical memory's total and what is available, as
MemTotal and MemAvailable. The wait there is on the memory available alone: Windows has no PSI, and its page
file says nothing like Linux's swap in use, so LONGSHOT_QUEUE_MAX_PRESSURE and LONGSHOT_QUEUE_MAX_SWAP_GB stay
Linux's; status no longer shows a swap it didn't read. Linux is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
futondestroyer left a comment

Verdict: Request changes. The Windows memory floor can stall both a light job and the first build below 6 GiB available.

Reviewed 747a35a0 against 0de25c3e. In scripts/job_queue.py:1057–1075, the gate applies outside the stated extra-build case. On a supported Windows machine below that floor, a lightweight job can wait even with no other job running. Scope the gate to the admission policy you intend, or explicitly document and justify any floor for light and single-build jobs. At scripts/job_queue.py:1177, make the status report describe the same active policy and why a job is waiting.

Please add Windows game-free cases for light and first build below the floor, extra-build refusal, and admission after memory recovers. Reconcile the checklist with the runs actually performed.

**Verdict: Request changes.** The Windows memory floor can stall both a light job and the first build below 6 GiB available. Reviewed `747a35a0` against `0de25c3e`. In `scripts/job_queue.py:1057–1075`, the gate applies outside the stated extra-build case. On a supported Windows machine below that floor, a lightweight job can wait even with no other job running. Scope the gate to the admission policy you intend, or explicitly document and justify any floor for light and single-build jobs. At `scripts/job_queue.py:1177`, make the status report describe the same active policy and why a job is waiting. Please add Windows game-free cases for light and first build below the floor, extra-build refusal, and admission after memory recovers. Reconcile the checklist with the runs actually performed.
dyst force-pushed fix/queue-windows-memory from 747a35a04d
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
PR description / Description follows the template (pull_request_target) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 15s
to 9839139546
Some checks failed
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 12s
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Website build and checks (pull_request) Has been cancelled
2026-11-20 05:40:49 +00:00
Compare
Author
Member

Moved onto #1802: the fix and its three checks are now in scripts/devtools/jobqueue.py and tests/test_job_queue.py (the same two commits, test then fix, on main @ 3f2c9b3c), and the description is on the new template. Retested on Windows 11: the checks fail on main and pass here; a job with LONGSHOT_QUEUE_MIN_FREE_GB=1000 runs at once on main and waits here; every check fails the same on both (57), none here only.

Moved onto #1802: the fix and its three checks are now in `scripts/devtools/jobqueue.py` and `tests/test_job_queue.py` (the same two commits, test then fix, on main @ 3f2c9b3c), and the description is on the new template. Retested on Windows 11: the checks fail on main and pass here; a job with LONGSHOT_QUEUE_MIN_FREE_GB=1000 runs at once on main and waits here; every check fails the same on both (57), none here only.
From review: with the memory read on Windows, a light job or a first build on a PC with less than
LONGSHOT_QUEUE_MIN_FREE_GB available waited even with nothing else in the queue, maybe for ever, since only
programs outside the queue could free memory for it. Checks, on Windows' reader standing in for a 16 GB PC: a
light, shared or build job alone below the floor starts; beside a running job it waits, naming the memory; it
starts once the memory comes back; the third build slot stays shut without LONGSHOT_QUEUE_ROOMY_GB free. On real
jobs: one alone starts though no PC has the memory asked, one beside it waits, says why, and starts once alone.

The first and the real-job check fail here: memory_wait doesn't exist yet, and a job alone waits.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
From review: a job waited for LONGSHOT_QUEUE_MIN_FREE_GB even with nothing else in the queue, so on a PC short
of memory (16 GB with a browser open, now that Windows' memory is read) a light check or a first build could wait
for ever: only programs outside the queue could free memory for it. memory_wait() asks for the floor only while
another job of the queue runs, where the queue causes the pressure and a job ending frees memory; a job alone
starts and says how little was free. Linux and Windows alike. status gives a waiting job's real reason (the
memory, or "starting"), where it said "starting (or waiting for memory)".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
test(queue): the real-job memory check says what the runners printed when it fails
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Failing after 4m54s
fbdbf85d31
It failed once on Windows while another branch's full run_all loaded the PC, and passed in every rerun (alone,
six at once, four whole files at once): its messages now carry what the runners said and their tickets' states.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

@futondestroyer thanks, the floor was too wide. Answered in ae6fc98f (checks) and afa3604d (fix), on the branch moved onto #1802's scripts/devtools/jobqueue.py.

  • Scope: the floor now holds a job only while another job of the queue runs (memory_wait()). There the queue causes the pressure, and a job ending frees memory. A job alone, light or the first build, starts however little is free, and says so (with a floor of 1000 GB: queue: alone starts though 29.6 GB free (the floor is 1000): no other job of the queue runs, so it would wait for programs outside it). This holds on Linux too, where a job alone used to wait; Risks says so.
  • Status: a waiting job's line gives the policy's real reason: the memory (memory: 29.5 GB free (waits for 1000)), or starting. It used to say starting (or waiting for memory).
  • Windows game-free checks (Memory, on Windows' reader standing in for a 16 GB PC with 3 GB available):
    • light, shared and a first build alone start;
    • beside a running build they wait;
    • at 7 GB they start, so a job is admitted once memory recovers;
    • the third build slot stays shut at 10 GB and opens at 15.
  • On real jobs: one alone starts under a floor no PC meets; one beside it waits and starts once it is alone.
  • Against main: all 6 fail on main's code and pass here.
  • Checklist: the description is on the new template, and Testing lists only the runs made.
@futondestroyer thanks, the floor was too wide. Answered in ae6fc98f (checks) and afa3604d (fix), on the branch moved onto #1802's `scripts/devtools/jobqueue.py`. - **Scope:** the floor now holds a job only while another job of the queue runs (`memory_wait()`). There the queue causes the pressure, and a job ending frees memory. A job alone, light or the first build, starts however little is free, and says so (with a floor of 1000 GB: `queue: alone starts though 29.6 GB free (the floor is 1000): no other job of the queue runs, so it would wait for programs outside it`). This holds on Linux too, where a job alone used to wait; Risks says so. - **Status:** a waiting job's line gives the policy's real reason: the memory (`memory: 29.5 GB free (waits for 1000)`), or `starting`. It used to say `starting (or waiting for memory)`. - **Windows game-free checks** (`Memory`, on Windows' reader standing in for a 16 GB PC with 3 GB available): - light, shared and a first build alone start; - beside a running build they wait; - at 7 GB they start, so a job is admitted once memory recovers; - the third build slot stays shut at 10 GB and opens at 15. - **On real jobs:** one alone starts under a floor no PC meets; one beside it waits and starts once it is alone. - **Against main:** all 6 fail on `main`'s code and pass here. - **Checklist:** the description is on the new template, and Testing lists only the runs made.
HumanBean left a comment

Recommendation: Request changes. The Windows reader is correct and the new floor fixes the old wait-for-ever, but a job that is waiting for memory can be passed again and again by jobs that arrive after it. When memory is short on a busy PC, a build can starve. The fix is a few lines.

On Windows this lets the queue see free memory, so a job started beside a running one waits while less than 6 GB is available, and it stops the jobs-alone wait on both systems. The code gets better (one small function, a clearer status), apart from the ordering hole below.

Does it help: not a perf change. The fix is shown working on the author's Windows 11 PC (64 GB). With a 1000 GB floor, a job beside another waits and a job alone starts. The 6 Memory checks fail on main and pass here. meminfo() costs about 17 µs a call.
Other systems: Linux keeps its reader, PSI and swap checks unchanged. Its status line is unchanged too, because /proc/meminfo always has SwapTotal. One behaviour does change on purpose: a job alone no longer waits for memory, pressure or swap. That includes a timed run, which now measures on a short PC instead of waiting (it says so). Risks states this. No Linux run of the new checks exists yet: they are in Still to test, and CI's Python source jobs were skipped on this head. The game, players and the Steam Deck can't be affected: this is development tooling.
Switches: no new one. LONGSHOT_QUEUE_MIN_FREE_GB (6) and LONGSHOT_QUEUE_ROOMY_GB (14) now take effect on Windows. Both defaults are right. LONGSHOT_QUEUE_BUILD_SLOTS_MAX stays at 2 by default, so the third build slot opens only on a PC whose owner set it to 3. Nothing changes for anyone else.

Reviewed: fbdbf85d31 against main 3f2c9b3c (the PR's base): all of scripts/devtools/jobqueue.py's diff, plus the code it relies on: memory_short, memory_roomy, slots_now, blocker/conflict/rank, run()'s admission loop and status(). Also all of tests/test_job_queue.py's diff. Nothing run (a code review).
Fixed in this review: nothing.
Checked and fine:

  • MEMORYSTATUSEX is correct: two DWORDs, then seven c_ulonglongs, 64 bytes, dwLength set from sizeof. The 64-bit fields mean no 4 GB truncation.
  • ullAvailPhys matches Linux's MemAvailable (free plus standby). The page file is rightly left out.
  • windll is guarded for a check that patches WINDOWS on Linux.
  • No deadlock: memory_wait holds a job only while another active ticket exists. That job's end, or a dead runner's ticket being cleaned, frees it. Two waiters released together are serialised by .lock: the second sees the first active and waits again.
  • memory_wait runs only after blocker(), so slots and timed-run rules come first.
  • The "starts though" line is said once, just before the break.
  • status now gives the same reason the runner uses.
  • futondestroyer's earlier points are answered on this head:
    • the floor is scoped to "beside another job";
    • status names the real reason;
    • Windows-reader cases exist for light, shared and first build alone, beside another, after recovery, and for the third slot at 10 and 15 GB;
    • Testing lists only the runs made.
  • The tests would fail with the fix undone: the reader returns {}, and memory_wait's active check is removed. The author also reports mutation runs.

Blocking

  1. scripts/devtools/jobqueue.py:317 (memory_wait): the "alone" exemption ignores rank.
    • Case: memory is under 6 GB. Build B waits beside a running light job L. L ends. Within B's 1 s poll, a newer job C (light, or a build while a build slot is free) calls blocker(). C passes, because B's claim doesn't conflict with it. memory_wait sees nothing active, so C starts. B then sees C active and waits again.
    • On a shared PC with agents queueing short jobs, this can repeat for as long as memory stays short, so B waits for ever in practice.
    • On main this couldn't happen: everyone waited until memory came back.
    • Fix: exempt only the ready waiting ticket that ranks first. Return memory_short(s) when any active ticket exists, or when a ready waiting ticket ranks ahead of me (by rank()). Add a check where a newer ticket asks while an older one is waiting for memory.

Suggestions

  • jobqueue.py:638-641: the "starts though" text is built by .replace() on memory_short's message. That breaks silently if its wording changes. Have memory_short return the reason without the "waits for" tail, or format the message here.
  • win_meminfo could use procs.py's kernel32 bindings (W.k, with argtypes) instead of a bare ctypes.windll call, so the module binds Win32 in one way.

Testing:

  • The author ran the Memory checks, the Summary's steps and the full tests/run_all.py on Windows 11 (64 GB). The same 56 failures occur on main and none occur only here.
  • Missing: a Linux run of tests/test_job_queue.py Memory (Still to test). The CI's Python jobs were skipped on this head.
  • The Website and Guide screenshots CI jobs fail on this head. This PR touches neither.

Overlaps and merge order:

  • 1852 (JSON interface, move/reorder) is stacked on this PR's five commits, so this one merges first. The rank fix above should go in here, not there.
  • 1916 (GPU heat) touches run() after the admission loop, and the history. No textual overlap; either order.
  • 1915 (RAM-fs rule) touches only common.py. No overlap.
**Recommendation: Request changes.** The Windows reader is correct and the new floor fixes the old wait-for-ever, but a job that is waiting for memory can be passed again and again by jobs that arrive after it. When memory is short on a busy PC, a build can starve. The fix is a few lines. On Windows this lets the queue see free memory, so a job started beside a running one waits while less than 6 GB is available, and it stops the jobs-alone wait on both systems. The code gets better (one small function, a clearer `status`), apart from the ordering hole below. **Does it help:** not a perf change. The fix is shown working on the author's Windows 11 PC (64 GB). With a 1000 GB floor, a job beside another waits and a job alone starts. The 6 `Memory` checks fail on `main` and pass here. `meminfo()` costs about 17 µs a call. **Other systems:** Linux keeps its reader, PSI and swap checks unchanged. Its `status` line is unchanged too, because `/proc/meminfo` always has `SwapTotal`. One behaviour does change on purpose: a job alone no longer waits for memory, pressure or swap. That includes a timed run, which now measures on a short PC instead of waiting (it says so). Risks states this. No Linux run of the new checks exists yet: they are in Still to test, and CI's Python source jobs were skipped on this head. The game, players and the Steam Deck can't be affected: this is development tooling. **Switches:** no new one. `LONGSHOT_QUEUE_MIN_FREE_GB` (6) and `LONGSHOT_QUEUE_ROOMY_GB` (14) now take effect on Windows. Both defaults are right. `LONGSHOT_QUEUE_BUILD_SLOTS_MAX` stays at 2 by default, so the third build slot opens only on a PC whose owner set it to 3. Nothing changes for anyone else. **Reviewed:** fbdbf85d31 against main 3f2c9b3c (the PR's base): all of `scripts/devtools/jobqueue.py`'s diff, plus the code it relies on: `memory_short`, `memory_roomy`, `slots_now`, `blocker`/`conflict`/`rank`, `run()`'s admission loop and `status()`. Also all of `tests/test_job_queue.py`'s diff. Nothing run (a code review). **Fixed in this review:** nothing. **Checked and fine:** - `MEMORYSTATUSEX` is correct: two `DWORD`s, then seven `c_ulonglong`s, 64 bytes, `dwLength` set from `sizeof`. The 64-bit fields mean no 4 GB truncation. - `ullAvailPhys` matches Linux's `MemAvailable` (free plus standby). The page file is rightly left out. - `windll` is guarded for a check that patches `WINDOWS` on Linux. - No deadlock: `memory_wait` holds a job only while another *active* ticket exists. That job's end, or a dead runner's ticket being cleaned, frees it. Two waiters released together are serialised by `.lock`: the second sees the first active and waits again. - `memory_wait` runs only after `blocker()`, so slots and timed-run rules come first. - The "starts though" line is said once, just before the break. - `status` now gives the same reason the runner uses. - futondestroyer's earlier points are answered on this head: - the floor is scoped to "beside another job"; - `status` names the real reason; - Windows-reader cases exist for light, shared and first build alone, beside another, after recovery, and for the third slot at 10 and 15 GB; - Testing lists only the runs made. - The tests would fail with the fix undone: the reader returns `{}`, and `memory_wait`'s active check is removed. The author also reports mutation runs. **Blocking** 1. `scripts/devtools/jobqueue.py:317` (`memory_wait`): the "alone" exemption ignores rank. - Case: memory is under 6 GB. Build B waits beside a running light job L. L ends. Within B's 1 s poll, a newer job C (light, or a build while a build slot is free) calls `blocker()`. C passes, because B's claim doesn't conflict with it. `memory_wait` sees nothing active, so C starts. B then sees C active and waits again. - On a shared PC with agents queueing short jobs, this can repeat for as long as memory stays short, so B waits for ever in practice. - On `main` this couldn't happen: everyone waited until memory came back. - Fix: exempt only the ready waiting ticket that ranks first. Return `memory_short(s)` when any active ticket exists, *or* when a ready waiting ticket ranks ahead of `me` (by `rank()`). Add a check where a newer ticket asks while an older one is waiting for memory. **Suggestions** - `jobqueue.py:638-641`: the "starts though" text is built by `.replace()` on `memory_short`'s message. That breaks silently if its wording changes. Have `memory_short` return the reason without the "waits for" tail, or format the message here. - `win_meminfo` could use `procs.py`'s kernel32 bindings (`W.k`, with argtypes) instead of a bare `ctypes.windll` call, so the module binds Win32 in one way. **Testing:** - The author ran the `Memory` checks, the Summary's steps and the full `tests/run_all.py` on Windows 11 (64 GB). The same 56 failures occur on `main` and none occur only here. - Missing: a Linux run of `tests/test_job_queue.py Memory` (Still to test). The CI's Python jobs were skipped on this head. - The Website and Guide screenshots CI jobs fail on this head. This PR touches neither. **Overlaps and merge order:** - [1852](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1852) (JSON interface, move/reorder) is stacked on this PR's five commits, so this one merges first. The rank fix above should go in here, not there. - [1916](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916) (GPU heat) touches `run()` after the admission loop, and the history. No textual overlap; either order. - [1915](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1915) (RAM-fs rule) touches only `common.py`. No overlap.
@ -287,0 +314,4 @@
"""Why the ticket `me` should wait for memory, or None. Only while another job of the queue runs: then the queue
causes the pressure, and a job ending frees memory. A job alone starts however little is free (waiting would only
wait for programs outside the queue, maybe for ever)"""
if not any(t["state"] == "active" and t["pid"] != me["pid"] for t in ts):
Owner

A job alone skips the floor whatever its rank. If a newer job polls first after the active one ends, it starts, and the older one, waiting for memory, waits again. This can repeat while memory stays short. Exempt only the top-ranked ready waiting ticket: also return memory_short(s) when a ready waiting ticket ranks ahead of me.

A job alone skips the floor whatever its rank. If a newer job polls first after the active one ends, it starts, and the older one, waiting for memory, waits again. This can repeat while memory stays short. Exempt only the top-ranked ready waiting ticket: also return memory_short(s) when a ready waiting ticket ranks ahead of me.
@ -603,0 +637,4 @@
why = memory_wait(s, ts, me)
short = None if why else memory_short(s)
if short:
say("%s starts though %s: no other job of the queue runs, so it would wait for programs "
Owner

Nit: this message is built by .replace() on memory_short's wording, so it breaks silently if that wording changes. Return the reason without its tail, or format it here.

Nit: this message is built by .replace() on memory_short's wording, so it breaks silently if that wording changes. Return the reason without its tail, or format it here.
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Failing after 4m54s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/queue-windows-memory:dyst-fix/queue-windows-memory'
git switch 'dyst-fix/queue-windows-memory'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1816
No description provided.