fix(queue): read the PC's memory on Windows: jobs wait for it, a third build needs room #1816
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1816
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:fix/queue-windows-memory"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
On Windows the job queue (
dev.py queue) never checked how much memory was free. So it started builds and test runs even when the PC was nearly out of memory, and it never opened the extra build slot meant for a PC with room to spare. It read memory from/proc/meminfo, which Windows doesn't have, someminfo()was empty there. #1523 brought Windows the queue's player and outside-build checks; this is the memory, the part it left as Linux's.This PR reads the free memory on Windows too (
GlobalMemoryStatusEx), and makes the floor wait only where waiting can help:LONGSHOT_QUEUE_MIN_FREE_GB(6 GB) is available, and says why. Onmainit starts at once.LONGSHOT_QUEUE_ROOMY_GB(14 GB) available, as on Linux.statusgives a waiting job's real reason (the memory, orstarting), where it saidstarting (or waiting for memory).The code is in #1802's
scripts/devtools/jobqueue.py, which kept the/proc-only reader.To see it on
main:LONGSHOT_QUEUE_DIRandLONGSHOT_QUEUE_STATEin a temporary folder) and setLONGSHOT_QUEUE_MIN_FREE_GB=1000, more than any PC has.python -I -B scripts/dev.py queue run hold --mode light -- python -c "import time; time.sleep(60)".python -I -B scripts/dev.py queue run demo --mode build -- python -c "print('the job ran')".On
main,demoruns at once. Here it waits (queue: demo memory: 29.5 GB free (waits for 1000), and the same instatus), and it starts onceholdends.Read first:
scripts/devtools/jobqueue.py,meminfo(),win_meminfo()andmemory_wait().Why only the available memory on Windows:
LONGSHOT_QUEUE_MAX_PRESSUREstays Linux's.LONGSHOT_QUEUE_MAX_SWAP_GBstays Linux's too.statusshows swap only where it is read.Prior work: searched the tracker for the queue's memory on Windows,
meminfo,MIN_FREE_GBandBUILD_SLOTS_MAX: nothing open or closed.Testing
Tested: Windows 11, 64 GB.
tests/test_job_queue.py'sMemorychecks:main:0 not greater than 0).2 != 3onmain).mainand from this branch. Onmainboth jobs start at once. Here a job alone starts withqueue: alone starts though 29.6 GB free (the floor is 1000): no other job of the queue runs, so it would wait for programs outside it, and one beside it waits as above.status:memory: 29.5 GB available, and the waiting job's line ends inmemory: 29.5 GB free (waits for 1000).tests/run_all.py(mise run test's checks), with the Builder's llvm-mingw first onPATH: 692 passed, 56 failed, 123 skipped. The failing checks, rerun here and on a cleanmainwith the same compiler, fail the same on both, none here only. These aremain's known Windows failures, among themtest_job_queue.py'stest_ram_file_system(on Windowson_ram()answers no before it reads the table the check gives it).Still to test:
python -I -B tests/test_job_queue.py Memory, expecting 6 passes. The reader is unchanged there; the floor now lets a job alone start (by hand).Measurements
Verdict: fixed. A job short of memory beside another now waits on Windows; a job alone starts on both systems; reading the memory costs 17 µs.
Setup: Windows 11, 64 GB. No game run: a development script. Measured with the
Memorychecks, then the steps in Summary on a private queue from a cleanmainand from this branch, thenmeminfo()timed over 5 × 1000 calls (the median).3f2c9b3cfbdbf85dtests/test_job_queue.py Memory(6)LONGSHOT_QUEUE_MIN_FREE_GB=1000memory: 29.5 GB free (waits for 1000)status's memory linememory: 29.5 GB availablememinfo()Risks
statussays why.LONGSHOT_QUEUE_MIN_FREE_GB=0(no wait), or revert the commit.LONGSHOT_QUEUE_MIN_FREE_GB,LONGSHOT_QUEUE_ROOMY_GBandLONGSHOT_QUEUE_BUILD_SLOTS_MAXnow take effect on Windows with the defaults they have.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Verdict: Request changes. The Windows memory floor can stall both a light job and the first build below 6 GiB available.
Reviewed
747a35a0against0de25c3e. Inscripts/job_queue.py:1057–1075, the gate applies outside the stated extra-build case. On a supported Windows machine below that floor, a lightweight job can wait even with no other job running. Scope the gate to the admission policy you intend, or explicitly document and justify any floor for light and single-build jobs. Atscripts/job_queue.py:1177, make the status report describe the same active policy and why a job is waiting.Please add Windows game-free cases for light and first build below the floor, extra-build refusal, and admission after memory recovers. Reconcile the checklist with the runs actually performed.
747a35a04d9839139546Moved onto #1802: the fix and its three checks are now in
scripts/devtools/jobqueue.pyandtests/test_job_queue.py(the same two commits, test then fix, on main @3f2c9b3c), and the description is on the new template. Retested on Windows 11: the checks fail on main and pass here; a job with LONGSHOT_QUEUE_MIN_FREE_GB=1000 runs at once on main and waits here; every check fails the same on both (57), none here only.@futondestroyer thanks, the floor was too wide. Answered in
ae6fc98f(checks) andafa3604d(fix), on the branch moved onto #1802'sscripts/devtools/jobqueue.py.memory_wait()). There the queue causes the pressure, and a job ending frees memory. A job alone, light or the first build, starts however little is free, and says so (with a floor of 1000 GB:queue: alone starts though 29.6 GB free (the floor is 1000): no other job of the queue runs, so it would wait for programs outside it). This holds on Linux too, where a job alone used to wait; Risks says so.memory: 29.5 GB free (waits for 1000)), orstarting. It used to saystarting (or waiting for memory).Memory, on Windows' reader standing in for a 16 GB PC with 3 GB available):main's code and pass here.Recommendation: Request changes. The Windows reader is correct and the new floor fixes the old wait-for-ever, but a job that is waiting for memory can be passed again and again by jobs that arrive after it. When memory is short on a busy PC, a build can starve. The fix is a few lines.
On Windows this lets the queue see free memory, so a job started beside a running one waits while less than 6 GB is available, and it stops the jobs-alone wait on both systems. The code gets better (one small function, a clearer
status), apart from the ordering hole below.Does it help: not a perf change. The fix is shown working on the author's Windows 11 PC (64 GB). With a 1000 GB floor, a job beside another waits and a job alone starts. The 6
Memorychecks fail onmainand pass here.meminfo()costs about 17 µs a call.Other systems: Linux keeps its reader, PSI and swap checks unchanged. Its
statusline is unchanged too, because/proc/meminfoalways hasSwapTotal. One behaviour does change on purpose: a job alone no longer waits for memory, pressure or swap. That includes a timed run, which now measures on a short PC instead of waiting (it says so). Risks states this. No Linux run of the new checks exists yet: they are in Still to test, and CI's Python source jobs were skipped on this head. The game, players and the Steam Deck can't be affected: this is development tooling.Switches: no new one.
LONGSHOT_QUEUE_MIN_FREE_GB(6) andLONGSHOT_QUEUE_ROOMY_GB(14) now take effect on Windows. Both defaults are right.LONGSHOT_QUEUE_BUILD_SLOTS_MAXstays at 2 by default, so the third build slot opens only on a PC whose owner set it to 3. Nothing changes for anyone else.Reviewed:
fbdbf85d31against main3f2c9b3c(the PR's base): all ofscripts/devtools/jobqueue.py's diff, plus the code it relies on:memory_short,memory_roomy,slots_now,blocker/conflict/rank,run()'s admission loop andstatus(). Also all oftests/test_job_queue.py's diff. Nothing run (a code review).Fixed in this review: nothing.
Checked and fine:
MEMORYSTATUSEXis correct: twoDWORDs, then sevenc_ulonglongs, 64 bytes,dwLengthset fromsizeof. The 64-bit fields mean no 4 GB truncation.ullAvailPhysmatches Linux'sMemAvailable(free plus standby). The page file is rightly left out.windllis guarded for a check that patchesWINDOWSon Linux.memory_waitholds a job only while another active ticket exists. That job's end, or a dead runner's ticket being cleaned, frees it. Two waiters released together are serialised by.lock: the second sees the first active and waits again.memory_waitruns only afterblocker(), so slots and timed-run rules come first.statusnow gives the same reason the runner uses.statusnames the real reason;{}, andmemory_wait's active check is removed. The author also reports mutation runs.Blocking
scripts/devtools/jobqueue.py:317(memory_wait): the "alone" exemption ignores rank.blocker(). C passes, because B's claim doesn't conflict with it.memory_waitsees nothing active, so C starts. B then sees C active and waits again.mainthis couldn't happen: everyone waited until memory came back.memory_short(s)when any active ticket exists, or when a ready waiting ticket ranks ahead ofme(byrank()). Add a check where a newer ticket asks while an older one is waiting for memory.Suggestions
jobqueue.py:638-641: the "starts though" text is built by.replace()onmemory_short's message. That breaks silently if its wording changes. Havememory_shortreturn the reason without the "waits for" tail, or format the message here.win_meminfocould useprocs.py's kernel32 bindings (W.k, with argtypes) instead of a barectypes.windllcall, so the module binds Win32 in one way.Testing:
Memorychecks, the Summary's steps and the fulltests/run_all.pyon Windows 11 (64 GB). The same 56 failures occur onmainand none occur only here.tests/test_job_queue.py Memory(Still to test). The CI's Python jobs were skipped on this head.Overlaps and merge order:
run()after the admission loop, and the history. No textual overlap; either order.common.py. No overlap.@ -287,0 +314,4 @@"""Why the ticket `me` should wait for memory, or None. Only while another job of the queue runs: then the queuecauses the pressure, and a job ending frees memory. A job alone starts however little is free (waiting would onlywait for programs outside the queue, maybe for ever)"""if not any(t["state"] == "active" and t["pid"] != me["pid"] for t in ts):A job alone skips the floor whatever its rank. If a newer job polls first after the active one ends, it starts, and the older one, waiting for memory, waits again. This can repeat while memory stays short. Exempt only the top-ranked ready waiting ticket: also return memory_short(s) when a ready waiting ticket ranks ahead of me.
@ -603,0 +637,4 @@why = memory_wait(s, ts, me)short = None if why else memory_short(s)if short:say("%s starts though %s: no other job of the queue runs, so it would wait for programs "Nit: this message is built by .replace() on memory_short's wording, so it breaks silently if that wording changes. Return the reason without its tail, or format it here.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.