WIP: feat(queue): a timed run's history says how hot the GPU got and whether it was slowed for heat #1916

Draft
siantm wants to merge 7 commits from siantm/Longshot:feat/queue-gpu-heat into main
Contributor

Summary

A timed run's line in dev.py queue history says how busy the rest of the PC was while it measured, so a noisy
measurement shows. It said nothing of the GPU: a card at its temperature limit runs its clocks slower, and a timed run's
frame rate then follows the card's heat and the order the runs came in, not the build it measures. On Windows the
history's noise had no reading at all (no PSI, no load average).

A timed run that uses the GPU (timing, profile) now reads each NVIDIA adapter's temperature and its clocks' reasons
for running slower, and its noise gains gpu_temp_max, gpu_samples (the readings taken), gpu_slowed_for_heat (how
many of them found an adapter slowed for heat: NVML's software or hardware thermal slowdown among the clock reasons, a
count so one slowed reading in a long run still shows) and gpu_incomplete_samples (readings where an adapter's
temperature or reasons were unavailable). The run says so when it ends with an adapter slowed. The adapters read are
the host's NVIDIA ones and may not include the rendering adapter.

The readings are taken by a worker process (Python's multiprocessing, spawned) that the queue's runner starts once
the run's command has started, so a driver call that hangs or crashes is not in the runner. The worker loads NVIDIA's
management library (nvml.dll, libnvidia-ml.so.1); its first reading is about 5 s after it has started and loaded
the library, then one about every 5 s (each wait follows the reading before it), so a shorter run has none. Without
the library or an adapter it ends at once and nothing is added. The worker is optional to the run: if it cannot start,
the command runs on without readings (gpu_monitor: "not started"). When the command ends the runner asks it to stop,
then terminates and kills it if it does not (once), keeps its last complete readings, and names a worker that did not
run to the end in gpu_monitor (failed (exit code N), terminated, killed, or not stopped when it is still
alive after the kill), so partial readings do not pass for a whole run's; the command's end and the ticket's removal
do not depend on it. timing-cpu (a measured build), builds and shared runs start no worker.

Read first: nvml_reader, gpu_heat, _watch_heat, _HeatWatcher and the heat lines of run() in
scripts/devtools/jobqueue.py; the GpuHeat and GpuHeatRun tests in tests/test_job_queue.py. Other functions of
the same files than the open #1816 and #1852.

Testing

Tested: Windows 11, RTX 4060 (NVIDIA 610.88), i7-9700, Python 3.14.7 (the Builder's), each run in a hidden
console of its own through the PC's job queue, on this branch rebased onto main @ 6e1521a7 (head 5f8fe594):

  • python -I -B tests/test_job_queue.py -v @ 5f8fe594: 33 tests, 31 passed, 1 failed, 1 skipped. The failure,
    WindowsPlayersReal.test_a_build_outside_the_queue, fails on main @ 6e1521a7 the same way in the same job
    (main: 21 tests, 19 passed, that 1 failed, 1 skipped). All GpuHeat and GpuHeatRun tests pass, among them
    GpuHeatRun.test_real_worker: the real worker spawned and stopped around a 3 s timed command through run()
    (status 0, the ticket removed, a normal end so no gpu_monitor, no reading before 5 s).
  • The worker's failure tests at the test commit 9c6c5d32 (before the fix): 5 of them fail there
    (test_worker_that_cannot_start, test_cancel_with_a_worker_whose_stop_fails, test_worker_that_survives_the_kill,
    test_worker_that_failed, test_worker_stops_before_publishing) and pass @ 5f8fe594.
  • tests/run_all.py --strict --changed origin/main @ 5f8fe594: 4 passed, 2 failed: test_job_queue (the failure
    above, shared with main) and test_dev_packs (an unclassified longshot_fsr4_pipelines.bin beside this PC's game;
    it fails the same way on main @ 6e1521a7 in the queue).
  • CI's "Guide screenshots" steps @ 5f8fe594 and on main @ 6e1521a7: scripts/check_guide_screenshots.py --review-sources fails the same way on both (37 issues, review hashes of files this PR does not touch), its two
    test files pass on both. The CI job fails the same way on main's own push runs for 80bb1ccc and 6e1521a7.
  • Before the worker (at 05eaf6f8, the reader then in the runner's process): NVIDIA's library read from Python on this
    PC with nvml_reader: loaded in 45.8 ms, then 2,000 reads at 0.0033 ms each.

Not run: ruff (mise run lint; ruff is not installed on this PC). No game run under this branch's own queue yet: it is
a developer tool (the queue's runner), which the Test Farm does not run.

Still to test:

  • Windows 11, this PC: one real timed job of this branch's queue (dev.py queue run ... --mode timing around a game
    run), then dev.py queue history, expecting gpu_temp_max and gpu_slowed_for_heat in that job's noise, and the
    same with the worker made to fail (expecting the command's own status, its ticket removed and gpu_monitor in the
    history) (by hand, pending)
  • Windows 11, this PC: the worker's cost on main's build: main's pack, timed game runs, at least 4 a side
    interleaved after a warm-up, wrapped by main's dev.py queue run --mode timing and by this branch's (only the
    latter starts the worker), comparing each run's median frame, p99, 1 % low and frames over 50 ms, expecting no
    difference beyond the run-to-run spread (by hand, pending)
  • A hybrid laptop (NVIDIA adapter beside the one that renders): whether the worker's readings change the NVIDIA
    adapter's power state while the game renders on the other (by hand)
  • Windows 11, this PC: the whole suite (tests/run_all.py --strict) here and on main, expecting no failure here that
    main does not have (by hand, pending)
  • Linux, an NVIDIA card: python -I scripts/dev.py queue run heat --mode timing -- sleep 20, then
    python -I scripts/dev.py queue history, expecting gpu_temp_max and gpu_slowed_for_heat in the job's noise
    (by hand: the Test Farm runs game scenarios only, not the queue)
  • A PC with an AMD or Intel GPU only: the same, expecting the noise as before, with no GPU fields (by hand: the
    Test Farm runs game scenarios only, not the queue)

Measurements

Verdict: not measured at this head. A reading cost 0.0033 ms and loading the library 45.8 ms when the reader ran
in the queue's own process (05eaf6f8); the reader now runs in a worker process started beside the timed run, whose
start, NVIDIA library load and shutdown have not been timed, and whose effect on a game run has not been measured
(Still to test). In the game, on the branch this was ported from (the reader in the runner's process, no worker), 4
interleaved timed runs with the reader and 4 without showed no difference: with the reader, the runs' median frame,
CPU time a frame and frames a second each lie within the range of the runs without it, and p99 varies more between
runs of one arm than between the arms.

Setup: Windows 11, RTX 4060 (NVIDIA 610.88), i7-9700, Python 3.14.7; nvml_reader() timed from Python, 2,000 reads, at
05eaf6f8; tests/test_job_queue.py, one run each side, in one job of the PC's job queue

Before: main @ 6e1521a7 After: this PR @ 5f8fe594
tests/test_job_queue.py 21 tests: 19 passed, 1 failed, 1 skipped 33 tests: 31 passed, the same 1 failed, 1 skipped
processes started while a timed run measures none one: the heat worker (its sampling loop starts none: GpuHeat.test_watch_starts_no_process)
a sample's cost no sample 0.0033 ms (2,000 reads, at 05eaf6f8, in-process); the worker's start and shutdown not timed
nvml_reader: loaded in 45.8 ms
a read: 0.0033 ms (2000 reads), cards [(52, 1)]

The watch in the game, measured before this PR (2026-11-20, for e49c733f on the user's performance branch, the
commit this PR's nvml_reader and _watch_heat were ported from, reading the same and as often): that branch's game
build (v3.0.0-rc.7-292-g8c5ecbe9) and its own scripts/job_queue.py, not main's; Kokiri Forest, 800x600 window
(CPU-paced), RTX 4060; one warm-up run, then 8 timed runs interleaved P N N P P N N P through the PC's job queue: N
with the reader (each N job's history line has gpu_temp_max, 78-82 °C, never slowed for heat), P under that queue
script as it was before the heat record (no GPU fields). Each run's numbers from its metrics.json:

run  median frame  p90    p99    frames > 50 ms  fps    CPU time a frame
P2   13.27 ms      15.31  45.87  2               72.32  60.20 ms
N3   13.20         15.34  44.90  1               72.51  58.77
N4   13.06         15.56  45.13  0               72.49  60.02
P5   13.06         15.50  45.86  2               72.70  57.40
P6   12.82         15.23  43.96  0               74.26  58.70
N7   12.87         15.18  30.29  1               74.55  58.94
N8   13.08         15.38  22.03  1               73.39  58.43
P9   12.75         15.04  19.52  3               74.95  57.31
N (with the reader), mean of 4:  median frame 13.05 ms (12.87-13.20), CPU 59.04 ms (58.43-60.02), 73.23 fps
P (without), mean of 4:          median frame 12.97 ms (12.75-13.27), CPU 58.40 ms (57.31-60.20), 73.56 fps

Four runs an arm do not show a difference under about 1 %. The form before it, which started nvidia-smi at each
sample, did cost the run (median frame 13.24 against 13.01 ms, the ranges apart), which is why this PR reads through
the library.

Risks

The worker is a process the queue's runner starts beside every GPU timed run (timing, profile), on hosts without
NVIDIA too (it ends at once there). Its start, NVIDIA library load and reads overlap the measured run: their cost in
a game run is not measured yet (Still to test), and on a hybrid laptop reading the NVIDIA adapter could change its
power state. If it cannot start, the run goes on without readings; if it does not stop, it is terminated, then killed;
one still alive after the kill (a driver call it cannot leave) is marked not stopped and left to end with that call:
the run's history is written and its ticket removed, and the runner's own exit waits for it (Python joins its child
processes). Nothing is read for builds, shared runs or timing-cpu. No switch: a revert is enough. Players are not
affected: scripts/devtools/ is not part of the game or its build.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A check under tests/ that reads changed source is updated, not weakened.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary A timed run's line in `dev.py queue history` says how busy the rest of the PC was while it measured, so a noisy measurement shows. It said nothing of the GPU: a card at its temperature limit runs its clocks slower, and a timed run's frame rate then follows the card's heat and the order the runs came in, not the build it measures. On Windows the history's noise had no reading at all (no PSI, no load average). A timed run that uses the GPU (`timing`, `profile`) now reads each NVIDIA adapter's temperature and its clocks' reasons for running slower, and its noise gains `gpu_temp_max`, `gpu_samples` (the readings taken), `gpu_slowed_for_heat` (how many of them found an adapter slowed for heat: NVML's software or hardware thermal slowdown among the clock reasons, a count so one slowed reading in a long run still shows) and `gpu_incomplete_samples` (readings where an adapter's temperature or reasons were unavailable). The run says so when it ends with an adapter slowed. The adapters read are the host's NVIDIA ones and may not include the rendering adapter. The readings are taken by a worker process (Python's `multiprocessing`, spawned) that the queue's runner starts once the run's command has started, so a driver call that hangs or crashes is not in the runner. The worker loads NVIDIA's management library (`nvml.dll`, `libnvidia-ml.so.1`); its first reading is about 5 s after it has started and loaded the library, then one about every 5 s (each wait follows the reading before it), so a shorter run has none. Without the library or an adapter it ends at once and nothing is added. The worker is optional to the run: if it cannot start, the command runs on without readings (`gpu_monitor: "not started"`). When the command ends the runner asks it to stop, then terminates and kills it if it does not (once), keeps its last complete readings, and names a worker that did not run to the end in `gpu_monitor` (`failed (exit code N)`, `terminated`, `killed`, or `not stopped` when it is still alive after the kill), so partial readings do not pass for a whole run's; the command's end and the ticket's removal do not depend on it. `timing-cpu` (a measured build), builds and shared runs start no worker. Read first: `nvml_reader`, `gpu_heat`, `_watch_heat`, `_HeatWatcher` and the heat lines of `run()` in `scripts/devtools/jobqueue.py`; the `GpuHeat` and `GpuHeatRun` tests in `tests/test_job_queue.py`. Other functions of the same files than the open #1816 and #1852. ## Testing **Tested:** Windows 11, RTX 4060 (NVIDIA 610.88), i7-9700, Python 3.14.7 (the Builder's), each run in a hidden console of its own through the PC's job queue, on this branch rebased onto `main` @ 6e1521a7 (head 5f8fe594): - `python -I -B tests/test_job_queue.py -v` @ 5f8fe594: 33 tests, 31 passed, 1 failed, 1 skipped. The failure, `WindowsPlayersReal.test_a_build_outside_the_queue`, fails on `main` @ 6e1521a7 the same way in the same job (`main`: 21 tests, 19 passed, that 1 failed, 1 skipped). All `GpuHeat` and `GpuHeatRun` tests pass, among them `GpuHeatRun.test_real_worker`: the real worker spawned and stopped around a 3 s timed command through `run()` (status 0, the ticket removed, a normal end so no `gpu_monitor`, no reading before 5 s). - The worker's failure tests at the test commit 9c6c5d32 (before the fix): 5 of them fail there (`test_worker_that_cannot_start`, `test_cancel_with_a_worker_whose_stop_fails`, `test_worker_that_survives_the_kill`, `test_worker_that_failed`, `test_worker_stops_before_publishing`) and pass @ 5f8fe594. - `tests/run_all.py --strict --changed origin/main` @ 5f8fe594: 4 passed, 2 failed: `test_job_queue` (the failure above, shared with `main`) and `test_dev_packs` (an unclassified `longshot_fsr4_pipelines.bin` beside this PC's game; it fails the same way on `main` @ 6e1521a7 in the queue). - CI's "Guide screenshots" steps @ 5f8fe594 and on `main` @ 6e1521a7: `scripts/check_guide_screenshots.py --review-sources` fails the same way on both (37 issues, review hashes of files this PR does not touch), its two test files pass on both. The CI job fails the same way on `main`'s own push runs for 80bb1ccc and 6e1521a7. - Before the worker (at 05eaf6f8, the reader then in the runner's process): NVIDIA's library read from Python on this PC with `nvml_reader`: loaded in 45.8 ms, then 2,000 reads at 0.0033 ms each. Not run: ruff (`mise run lint`; ruff is not installed on this PC). No game run under this branch's own queue yet: it is a developer tool (the queue's runner), which the Test Farm does not run. **Still to test:** - Windows 11, this PC: one real timed job of this branch's queue (`dev.py queue run ... --mode timing` around a game run), then `dev.py queue history`, expecting `gpu_temp_max` and `gpu_slowed_for_heat` in that job's noise, and the same with the worker made to fail (expecting the command's own status, its ticket removed and `gpu_monitor` in the history) (by hand, pending) - Windows 11, this PC: the worker's cost on `main`'s build: `main`'s pack, timed game runs, at least 4 a side interleaved after a warm-up, wrapped by `main`'s `dev.py queue run --mode timing` and by this branch's (only the latter starts the worker), comparing each run's median frame, p99, 1 % low and frames over 50 ms, expecting no difference beyond the run-to-run spread (by hand, pending) - A hybrid laptop (NVIDIA adapter beside the one that renders): whether the worker's readings change the NVIDIA adapter's power state while the game renders on the other (by hand) - Windows 11, this PC: the whole suite (`tests/run_all.py --strict`) here and on `main`, expecting no failure here that `main` does not have (by hand, pending) - Linux, an NVIDIA card: `python -I scripts/dev.py queue run heat --mode timing -- sleep 20`, then `python -I scripts/dev.py queue history`, expecting `gpu_temp_max` and `gpu_slowed_for_heat` in the job's noise (by hand: the Test Farm runs game scenarios only, not the queue) - A PC with an AMD or Intel GPU only: the same, expecting the noise as before, with no GPU fields (by hand: the Test Farm runs game scenarios only, not the queue) ## Measurements **Verdict:** not measured at this head. A reading cost 0.0033 ms and loading the library 45.8 ms when the reader ran in the queue's own process (05eaf6f8); the reader now runs in a worker process started beside the timed run, whose start, NVIDIA library load and shutdown have not been timed, and whose effect on a game run has not been measured (Still to test). In the game, on the branch this was ported from (the reader in the runner's process, no worker), 4 interleaved timed runs with the reader and 4 without showed no difference: with the reader, the runs' median frame, CPU time a frame and frames a second each lie within the range of the runs without it, and p99 varies more between runs of one arm than between the arms. Setup: Windows 11, RTX 4060 (NVIDIA 610.88), i7-9700, Python 3.14.7; `nvml_reader()` timed from Python, 2,000 reads, at 05eaf6f8; `tests/test_job_queue.py`, one run each side, in one job of the PC's job queue | | Before: main @ 6e1521a7 | After: this PR @ 5f8fe594 | |---|---|---| | `tests/test_job_queue.py` | 21 tests: 19 passed, 1 failed, 1 skipped | 33 tests: 31 passed, the same 1 failed, 1 skipped | | processes started while a timed run measures | none | one: the heat worker (its sampling loop starts none: `GpuHeat.test_watch_starts_no_process`) | | a sample's cost | no sample | 0.0033 ms (2,000 reads, at 05eaf6f8, in-process); the worker's start and shutdown not timed | ``` nvml_reader: loaded in 45.8 ms a read: 0.0033 ms (2000 reads), cards [(52, 1)] ``` **The watch in the game, measured before this PR** (2026-11-20, for e49c733f on the user's performance branch, the commit this PR's `nvml_reader` and `_watch_heat` were ported from, reading the same and as often): that branch's game build (`v3.0.0-rc.7-292-g8c5ecbe9`) and its own `scripts/job_queue.py`, not `main`'s; Kokiri Forest, 800x600 window (CPU-paced), RTX 4060; one warm-up run, then 8 timed runs interleaved P N N P P N N P through the PC's job queue: N with the reader (each N job's history line has `gpu_temp_max`, 78-82 °C, never slowed for heat), P under that queue script as it was before the heat record (no GPU fields). Each run's numbers from its `metrics.json`: ``` run median frame p90 p99 frames > 50 ms fps CPU time a frame P2 13.27 ms 15.31 45.87 2 72.32 60.20 ms N3 13.20 15.34 44.90 1 72.51 58.77 N4 13.06 15.56 45.13 0 72.49 60.02 P5 13.06 15.50 45.86 2 72.70 57.40 P6 12.82 15.23 43.96 0 74.26 58.70 N7 12.87 15.18 30.29 1 74.55 58.94 N8 13.08 15.38 22.03 1 73.39 58.43 P9 12.75 15.04 19.52 3 74.95 57.31 N (with the reader), mean of 4: median frame 13.05 ms (12.87-13.20), CPU 59.04 ms (58.43-60.02), 73.23 fps P (without), mean of 4: median frame 12.97 ms (12.75-13.27), CPU 58.40 ms (57.31-60.20), 73.56 fps ``` Four runs an arm do not show a difference under about 1 %. The form before it, which started `nvidia-smi` at each sample, did cost the run (median frame 13.24 against 13.01 ms, the ranges apart), which is why this PR reads through the library. ## Risks The worker is a process the queue's runner starts beside every GPU timed run (`timing`, `profile`), on hosts without NVIDIA too (it ends at once there). Its start, NVIDIA library load and reads overlap the measured run: their cost in a game run is not measured yet (Still to test), and on a hybrid laptop reading the NVIDIA adapter could change its power state. If it cannot start, the run goes on without readings; if it does not stop, it is terminated, then killed; one still alive after the kill (a driver call it cannot leave) is marked `not stopped` and left to end with that call: the run's history is written and its ticket removed, and the runner's own exit waits for it (Python joins its child processes). Nothing is read for builds, shared runs or `timing-cpu`. No switch: a revert is enough. Players are not affected: `scripts/devtools/` is not part of the game or its build. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [ ] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
feat(queue): a timed run's history says how hot the GPU got and whether it was slowed for heat
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
05eaf6f884
A timed run on the GPU reads each NVIDIA card's temperature and its clocks' reasons for running slower every 5 s,
through NVIDIA's management library loaded into the queue's own process (nvml.dll, libnvidia-ml.so.1): nothing is
started while the run measures. Its noise in `dev.py queue history` gains gpu_temp_max and gpu_slowed_for_heat (the
share of the samples with a card slowed for heat, the software's or the hardware's thermal slowdown), and the run says
so when it ends with a card slowed. Nothing is added without the library or a card, or for timing-cpu.

A card at its temperature limit runs its clocks slower, so a timed run's frame rate there follows the card's heat and
the order the runs came in, and the history could not show it.

Assisted-by: Claude Code
BeanBot left a comment

Recommendation: Request changes. The history can report no thermal slowdown even after observing one, and the new monitoring runs during measurements without an end-to-end comparison of its effect on frame times.

The queue records sampled NVIDIA temperatures and thermal slowdown reasons for GPU timing and profile jobs, which could help explain noisy comparisons. The aggregation and stand-in reader tests are useful, but rounding can hide a real warning and the cost of querying the driver during a game measurement remains unknown.

Does it help: Unproven. The author reports a 45.8 ms library load and 0.0033 ms per read over 2,000 reads on Windows 11 with an RTX 4060; no real timed job or before-and-after frame-time comparison is reported. The Windows queue tests report 21 passed, two failures also seen on main, and one skip.
Other systems: The watcher runs for timing and profile jobs on Windows and Linux, including machines with multiple NVIDIA cards. Linux NVIDIA and AMD/Intel-only fallback runs are still pending. It does not run for timing-cpu or shared jobs; the game, launcher and build outputs are unchanged. The effect of NVML calls on GPU work and frame pacing has not been measured.
Switches: None added, changed or removed.

Reviewed: 05eaf6f884 against 8859673e4b: Read AGENTS.md, CONTRIBUTING.md, the pull request template, the applicable standards, the diff and commit, and the changed code and its queue and history paths. Read the changed documentation and tests; no generated files or translations changed. docs/review-checklist.md and the adjacent Tooling checkout were absent. The network was unavailable, so open PRs and Tooling's rules could not be independently checked.; nothing run (a code review).
Checked and fine:

  • The thermal reason mask and multi-card sample aggregation match the stated definition.
  • Missing library or successfully enumerated cards yields no GPU fields; timing-cpu does not start the watcher.
  • The change is confined to the queue, its tests and developer documentation, with no new dependency or bundled vendor SDK.

Blocking (to change before merge)

  1. scripts/devtools/jobqueue.py:573: Rounding the fraction before checking it can hide an observed slowdown. For example, one thermal sample among 201 rounds to 0.0, so history reports zero and the end-of-run warning is suppressed. Preserve enough precision and base the warning on whether any sample was slowed; add a test for this case.
  2. scripts/devtools/jobqueue.py:697: NVML initialization and polling begin after the game starts, but the reported read time measures only the reader, not its effect on the game or driver. An expensive driver query could alter frame pacing during the measurement this feature is meant to assess. Compare repeated interleaved timed game runs against the base, including frame-time tails, on affected hardware and address any meaningful cost.

Suggestions (the author may take or leave)

  • Sample at the start or explicitly document the coverage gap: a job shorter than five seconds has no GPU reading even when NVML is available.
  • Add an integration test that exercises watcher startup, shutdown and the history record through a timed queue job.

Testing: Read-only review; no code or tests were run. The PR still needs its real timed job, full strict suite and listed Linux and non-NVIDIA checks, plus applicable Tier 1 coverage and passing CI.
Overlaps and merge order: The description identifies #1816 and #1852 as touching other functions in the same files and states that #1852 applies on top. Their contents and merge order could not be independently checked offline.


Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The history can report no thermal slowdown even after observing one, and the new monitoring runs during measurements without an end-to-end comparison of its effect on frame times. The queue records sampled NVIDIA temperatures and thermal slowdown reasons for GPU timing and profile jobs, which could help explain noisy comparisons. The aggregation and stand-in reader tests are useful, but rounding can hide a real warning and the cost of querying the driver during a game measurement remains unknown. **Does it help:** Unproven. The author reports a 45.8 ms library load and 0.0033 ms per read over 2,000 reads on Windows 11 with an RTX 4060; no real timed job or before-and-after frame-time comparison is reported. The Windows queue tests report 21 passed, two failures also seen on main, and one skip. **Other systems:** The watcher runs for timing and profile jobs on Windows and Linux, including machines with multiple NVIDIA cards. Linux NVIDIA and AMD/Intel-only fallback runs are still pending. It does not run for timing-cpu or shared jobs; the game, launcher and build outputs are unchanged. The effect of NVML calls on GPU work and frame pacing has not been measured. **Switches:** None added, changed or removed. **Reviewed:** 05eaf6f884f5 against 8859673e4bf5: Read AGENTS.md, CONTRIBUTING.md, the pull request template, the applicable standards, the diff and commit, and the changed code and its queue and history paths. Read the changed documentation and tests; no generated files or translations changed. docs/review-checklist.md and the adjacent Tooling checkout were absent. The network was unavailable, so open PRs and Tooling's rules could not be independently checked.; nothing run (a code review). **Checked and fine:** - The thermal reason mask and multi-card sample aggregation match the stated definition. - Missing library or successfully enumerated cards yields no GPU fields; timing-cpu does not start the watcher. - The change is confined to the queue, its tests and developer documentation, with no new dependency or bundled vendor SDK. **Blocking** (to change before merge) 1. scripts/devtools/jobqueue.py:573: Rounding the fraction before checking it can hide an observed slowdown. For example, one thermal sample among 201 rounds to 0.0, so history reports zero and the end-of-run warning is suppressed. Preserve enough precision and base the warning on whether any sample was slowed; add a test for this case. 2. scripts/devtools/jobqueue.py:697: NVML initialization and polling begin after the game starts, but the reported read time measures only the reader, not its effect on the game or driver. An expensive driver query could alter frame pacing during the measurement this feature is meant to assess. Compare repeated interleaved timed game runs against the base, including frame-time tails, on affected hardware and address any meaningful cost. **Suggestions** (the author may take or leave) - Sample at the start or explicitly document the coverage gap: a job shorter than five seconds has no GPU reading even when NVML is available. - Add an integration test that exercises watcher startup, shutdown and the history record through a timed queue job. **Testing:** Read-only review; no code or tests were run. The PR still needs its real timed job, full strict suite and listed Linux and non-NVIDIA checks, plus applicable Tier 1 coverage and passing CI. **Overlaps and merge order:** The description identifies #1816 and #1852 as touching other functions in the same files and states that #1852 applies on top. Their contents and merge order could not be independently checked offline. --- <sub>Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -519,0 +570,4 @@
return None
slowed = sum(1 for cards in samples if any(why & NVML_SLOWED_FOR_HEAT for _, why in cards))
return {"gpu_temp_max": max(t for cards in samples for t, _ in cards),
"gpu_slowed_for_heat": round(slowed / len(samples), 2)}
Member

One slowed sample among 201 rounds to 0.0, so history says zero and the warning is skipped despite observing thermal slowdown. Keep sufficient precision and trigger the warning from the unrounded count; please cover this case in a test.

One slowed sample among 201 rounds to 0.0, so history says zero and the warning is skipped despite observing thermal slowdown. Keep sufficient precision and trigger the warning from the unrounded count; please cover this case in a test.
@ -519,0 +577,4 @@
"""Every GPU_HEAT_EVERY seconds while a timed run uses the GPU: the cards' heat into its noise. Read in this process:
nothing is started while the run measures"""
read = nvml_reader()
while read and not stop.wait(GPU_HEAT_EVERY):
Member

The first reading waits five seconds, so shorter GPU timing jobs always have no heat data even with a working reader. Consider sampling at the start or documenting this coverage gap.

The first reading waits five seconds, so shorter GPU timing jobs always have no heat data even with a working reader. Consider sampling at the start or documenting this coverage gap.
@ -632,6 +695,8 @@ def run(name, mode, cmd, environ=None, s=None, cwd=None, desc=None):
if timing_cpus and not common.WINDOWS:
threading.Thread(target=_keep_apart, args=(s, qdir, me, child.pid, cpu, timing_cpus, other_cpus, noise, stop),
daemon=True).start()
Member

This loads and queries NVML while the game is running. The reported reader timing does not show whether driver calls affect frame pacing. Please compare repeated interleaved timed game runs against the base, including frame-time tails, and address any meaningful cost.

This loads and queries NVML while the game is running. The reported reader timing does not show whether driver calls affect frame pacing. Please compare repeated interleaved timed game runs against the base, including frame-time tails, and address any meaningful cost.
GpuHeat.test_one_slowed_sample_is_said: one sample with a card slowed for heat among 201 (a run of about 17 minutes).
It fails here: gpu_slowed_for_heat, the share rounded to two places, reads 0.0, so the history says the card was never
slowed and the run's end says nothing.

Assisted-by: Claude Code
gpu_slowed_for_heat was the share of the samples with a card slowed for heat, rounded to two places: one slowed sample
among 201 read as 0.0, so the history said the card was never slowed and the run's end said nothing. It is now the
number of such samples, beside gpu_samples, the number with a reading; the run's end says so when it is not 0.

tests/test_job_queue.py's GpuHeat tests pass, test_one_slowed_sample_is_said among them.

Assisted-by: Claude Code
docs(queue): a timed run shorter than 5 s has no GPU heat reading
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
5a5a638ddb
The cards are read every 5 s from the run's start, the first reading 5 s in, so a run that ends sooner has none and its
history line no GPU fields. docs/dev/developing.md and _watch_heat say so.

Assisted-by: Claude Code
Author
Contributor

Pushed 6f4da2dd, 92724a75 and 5a5a638d for BeanBot's review; the description is updated to match.

  • Blocking 1 (the rounded share hides one slowed sample): confirmed, round(1/201, 2) is 0.0 and the run's warning read that field. 6f4da2dd adds GpuHeat.test_one_slowed_sample_is_said (one slowed sample among 201), which fails there; 92724a75 makes gpu_slowed_for_heat the number of slowed samples, beside gpu_samples, so the history and the warning see any slowed sample, and the test passes. tests/test_job_queue.py @ 5a5a638d: 25 tests, 22 passed, the same 2 failures as main, 1 skipped, run through this PC's job queue.
  • Suggestion, first reading at 5 s: documented rather than sampled at the start (a reading at the start would be taken before the game has loaded anything and would count in gpu_samples). 5a5a638d says it in docs/dev/developing.md and in _watch_heat's docstring.
  • Blocking 2 (NVML polled while the game runs): agreed that the reader's own timing does not show its effect on frame pacing. Interleaved timed game runs with main's queue and this branch's (same pack, at least 4 a side, frame-trace median, p99 and 1 % low) are now in Still to test and pending; nothing about it is measured yet.
  • Suggestion, an integration test through a timed queue job: the watch's start and stop are covered by GpuHeat.test_watch_starts_no_process and the record by test_record; what run() adds is the thread's start (scripts/devtools/jobqueue.py:699-700) and the record and warning (:708-714). A real timed job through this branch's queue and its history line are already in Still to test, so I've left it there.
Pushed 6f4da2dd, 92724a75 and 5a5a638d for BeanBot's review; the description is updated to match. - [Blocking 1](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-60337) (the rounded share hides one slowed sample): confirmed, `round(1/201, 2)` is 0.0 and the run's warning read that field. 6f4da2dd adds `GpuHeat.test_one_slowed_sample_is_said` (one slowed sample among 201), which fails there; 92724a75 makes `gpu_slowed_for_heat` the number of slowed samples, beside `gpu_samples`, so the history and the warning see any slowed sample, and the test passes. `tests/test_job_queue.py` @ 5a5a638d: 25 tests, 22 passed, the same 2 failures as `main`, 1 skipped, run through this PC's job queue. - [Suggestion, first reading at 5 s](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-60339): documented rather than sampled at the start (a reading at the start would be taken before the game has loaded anything and would count in `gpu_samples`). 5a5a638d says it in `docs/dev/developing.md` and in `_watch_heat`'s docstring. - [Blocking 2](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-60338) (NVML polled while the game runs): agreed that the reader's own timing does not show its effect on frame pacing. Interleaved timed game runs with `main`'s queue and this branch's (same pack, at least 4 a side, frame-trace median, p99 and 1 % low) are now in Still to test and pending; nothing about it is measured yet. - Suggestion, an integration test through a timed queue job: the watch's start and stop are covered by `GpuHeat.test_watch_starts_no_process` and the record by `test_record`; what `run()` adds is the thread's start (`scripts/devtools/jobqueue.py:699-700`) and the record and warning (`:708-714`). A real timed job through this branch's queue and its history line are already in Still to test, so I've left it there.
Author
Contributor

On Blocking 2 (NVML polled while the game runs): this was measured on 2026-11-20 for e49c733f, the commit on the user's performance branch this PR's nvml_reader and _watch_heat were ported from. Kokiri Forest, 8 timed runs interleaved P N N P P N N P after a warm-up, 4 with the reader and 4 without: median frame 13.05 ms (12.87-13.20) with against 12.97 (12.75-13.27) without, CPU time a frame 59.04 against 58.40 ms, the reader's ranges inside the others; p99 ran from 22.0 to 45.1 ms with and from 19.5 to 45.9 ms without, so it varies more between runs of one arm than between arms. The Measurements section now has each run's numbers and their limits: that branch's game build and queue script, not main's, and four runs an arm resolve nothing under about 1 %. A repeat on main's build with this branch's queue stays under Still to test, pending.

On [Blocking 2](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-60338) (NVML polled while the game runs): this was measured on 2026-11-20 for e49c733f, the commit on the user's performance branch this PR's `nvml_reader` and `_watch_heat` were ported from. Kokiri Forest, 8 timed runs interleaved P N N P P N N P after a warm-up, 4 with the reader and 4 without: median frame 13.05 ms (12.87-13.20) with against 12.97 (12.75-13.27) without, CPU time a frame 59.04 against 58.40 ms, the reader's ranges inside the others; p99 ran from 22.0 to 45.1 ms with and from 19.5 to 45.9 ms without, so it varies more between runs of one arm than between arms. The Measurements section now has each run's numbers and their limits: that branch's game build and queue script, not `main`'s, and four runs an arm resolve nothing under about 1 %. A repeat on `main`'s build with this branch's queue stays under Still to test, pending.
Author
Contributor

Still to test: the items for hardware this PC lacks go to the Test Farm (@Beanstalk form).

Still to test: the items for hardware this PC lacks go to the Test Farm (@Beanstalk form).
Member

@siantm only members of BigBeans can start the farm.

One of them can approve this request by reacting 👍 to it or to this comment. This request was for 5a5a638, and the PR has changed since (3df0421), so a 👍 no longer approves it. Comment @Beanstalk again to ask for the new commit.

@siantm only members of `BigBeans` can start the farm. ~~One of them can approve this request by reacting 👍 to it or to this comment.~~ This request was for `5a5a638`, and the PR has changed since (`3df0421`), so a 👍 no longer approves it. Comment `@Beanstalk` again to ask for the new commit.
Author
Contributor

The Test Farm request in my previous comment was not meant: the farm plays game scenarios and times builds, comparing this PR's game against main's, and this change is in a developer tool that the game does not run (no plan of the farm's runs a test file or the job queue). The Linux and other-GPU items in Still to test are back to by hand. Nothing for a BigBean to approve here.

The Test Farm request in my previous comment was not meant: the farm plays game scenarios and times builds, comparing this PR's game against main's, and this change is in a developer tool that the game does not run (no plan of the farm's runs a test file or the job queue). The Linux and other-GPU items in Still to test are back to by hand. Nothing for a BigBean to approve here.
Owner
@fix
Member

I changed this pull request by bringing it up to date: the new head is 3df042149f. I rebased it onto main (e9e2edc627).

Left for the author

  • GPU heat reporting and measurement cost: The precision fix and 201-sample regression test already exist in scripts/devtools/jobqueue.py:564 and tests/test_job_queue.py:386; the warning checks the unrounded count at scripts/devtools/jobqueue.py:712. The author must compare repeated interleaved timed game runs against the base on affected hardware, including frame-time tails, and address any meaningful cost. The existing reader timing does not measure the effect on the game.

Nothing was built or run: check the change. It will be reviewed again.


Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.

I changed this pull request by bringing it up to date: the new head is 3df042149f3a. I rebased it onto `main` (e9e2edc62776). **Left for the author** - **GPU heat reporting and measurement cost**: The precision fix and 201-sample regression test already exist in scripts/devtools/jobqueue.py:564 and tests/test_job_queue.py:386; the warning checks the unrounded count at scripts/devtools/jobqueue.py:712. The author must compare repeated interleaved timed game runs against the base on affected hardware, including frame-time tails, and address any meaningful cost. The existing reader timing does not measure the effect on the game. Nothing was built or run: check the change. It will be reviewed again. --- <sub>Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.</sub>
BeanBot force-pushed feat/queue-gpu-heat from 5a5a638ddb
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 3df042149f
Some checks failed
beanstalk/approval changed since the request: ask again
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 22s
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 15s
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Workflow lint (actionlint) (pull_request) Successful in 24s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 26s
Third-party parts / As their notes say (pull_request) Successful in 38s
CI / Website build and checks (pull_request) Has been cancelled
2026-11-21 00:05:58 +00:00
Compare
BeanBot left a comment

Recommendation: Request changes. GPU heat is useful context for a timed job, but the current watcher can misattribute throttling, discard valid readings, and outlive the queue slot. The cost of initializing and polling NVML during measurements still needs a comparison at this head against its base.

Adds sampled NVIDIA temperatures and thermal slowdown counts to queue history and warns at job completion. The additive history format and integer count are useful. The cost is native driver work during a measurement, with incomplete handling of unsupported queries and watcher shutdown.

Does it help: Unproven end to end. The author measured 45.8 ms to load and initialize the reader and 0.0033 ms per read over 2,000 reads on a Windows 11 RTX 4060 at 05eaf6f8. Four interleaved runs per side on the source branch gave 13.05 vs 12.97 ms median frame, 73.23 vs 73.56 fps, and overlapping p99 ranges of 22.03–45.13 vs 19.52–45.87 ms. That CPU-paced scene shows no clear cost within the spread. The rebased queue still needs a real job and a same-build comparison against e9e2edc.
Other systems: Timing and profile use the watcher on Windows and Linux. Linux NVIDIA remains untested. Hybrid systems can query a sleeping or unused NVIDIA adapter even when the game uses another GPU. AMD/Intel-only and Steam Deck should omit the GPU fields when no usable NVIDIA device is found, with fallback coverage pending. Shared, GPU-only, build, light and timing-cpu jobs skip the watcher. History renders the noise object directly and accepts older records. The diff touches developer tooling, tests and docs, with no game, launcher, save, audio, input or shipped build changes.
Switches: None.

Reviewed: 3df042149f against e9e2edc627: Read base AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards, the full diff and commit messages, and both changed Python files. Read the changed documentation and surrounding queue section, traced CLI dispatch, job shutdown and history readers, and reviewed the supplied comments, earlier bot review and reported runs. The rest of developing.md was outside the reviewed section. The review checklist and adjacent Tooling checkout were absent. No translations or generated files changed. Forgejo overlap and CI status could not be verified offline.; nothing run (a code review).
Checked and fine:

  • The integer count preserves one thermal sample among 201. The explicit software and hardware thermal masks exclude idle and power-cap reasons.
  • The mode gate selects timing and profile. Existing history records remain readable and the new keys are optional.
  • The change stays on one topic, follows the surrounding Python style, uses the standard library and records AI assistance in the commit trailers. The diff contains no game data, generated output or bundled SDK. Renderer, shader, menu and screenshot checks do not apply.
  • The sampling delay is documented for short runs. The watcher waits five seconds after reader initialization, so actual first coverage can be later.

Blocking (to change before merge)

  1. scripts/devtools/jobqueue.py:543: A successful nvmlInit_v2 has no matching nvmlShutdown. A count error, zero cards or a failed handle enumeration can return immediately with an outstanding initialization. Successful watches also leave it outstanding when an in-process caller returns. Give the watcher ownership of the NVML session and pair every successful init with shutdown, including setup failures and normal stop.
  2. scripts/devtools/jobqueue.py:704: The stop event interrupts the sleep only. If the child exits during a slow query, history can snapshot before that sample finishes and the queue can release its slot while the query is still running. Coordinate sample publication with the history snapshot and finish watcher cleanup before releasing the slot. If a hung native call must be tolerated, use a worker that can be stopped within a bounded time.
  3. scripts/devtools/jobqueue.py:556: The reader keeps a card only when both queries succeed. A card that reports temperature but returns NOT_SUPPORTED for clock reasons loses its valid temperature. If a second card succeeds, history can report zero thermal samples as though both cards were checked. Preserve successful readings independently and expose incomplete thermal coverage. Test one card succeeding on only one query.
  4. scripts/devtools/jobqueue.py:713: The warning attributes any enumerated card's slowdown to the timed run's frame rate. On a hybrid laptop measuring on the iGPU, or a workstation with a hot second NVIDIA card, that conclusion can be false. Identify the rendering adapter before attributing an effect, or state that an NVIDIA adapter on the host reported thermal slowdown and remove the causal claim from the warning and docs.
  5. tests/test_job_queue.py:404: The no-process test replaces nvml_reader entirely. A future change that starts nvidia-smi from that function would pass this test unchanged. Keep the production reader in the test, substitute a fake ctypes library and verify both the reader's observable output and the absence of subprocess calls.

Waiting on the author (a run, a measurement or a decision; no edit does it)

  • scripts/devtools/jobqueue.py:699: Initialization and polling overlap the timed workload. The per-read average misses startup, driver stalls and power effects, including querying a dormant NVIDIA adapter on a hybrid system. Compare this head with the merge base using repeated interleaved runs of the same game build, scene and settings. Report median, tail frame times and hitch counts, and check whether polling changes adapter power state on an affected hybrid setup.
  • The rebased head has no reported end-to-end queue job, applicable Tier 1 platform coverage, full strict check results or passing CI. Run the Windows and Linux NVIDIA queue checks and the AMD/Intel-only fallback checks, record the actual history and warning output, and report lint, strict suite and CI results. Compare failures against the current base.

Suggestions (the author may take or leave)

  • scripts/devtools/jobqueue.py:704: A query already in flight can outlive both the history snapshot and the queue slot. Coordinate publication and finish cleanup before releasing the slot.

Testing: Read-only review. The author reported 22 passed, two failures also seen on the older main, and one skip at 5a5a638d. The 201-sample test failed at 6f4da2dd and passed at 5a5a638d. The changed strict suite also reported the pre-existing local pack classification failure. No run at 3df04214 is reported.
Overlaps and merge order: The author identifies #1816 and #1852 as touching other functions in the same files and reports that #1852 applies on top. Neither overlap was independently verified and no merge order is established.


Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** GPU heat is useful context for a timed job, but the current watcher can misattribute throttling, discard valid readings, and outlive the queue slot. The cost of initializing and polling NVML during measurements still needs a comparison at this head against its base. Adds sampled NVIDIA temperatures and thermal slowdown counts to queue history and warns at job completion. The additive history format and integer count are useful. The cost is native driver work during a measurement, with incomplete handling of unsupported queries and watcher shutdown. **Does it help:** Unproven end to end. The author measured 45.8 ms to load and initialize the reader and 0.0033 ms per read over 2,000 reads on a Windows 11 RTX 4060 at 05eaf6f8. Four interleaved runs per side on the source branch gave 13.05 vs 12.97 ms median frame, 73.23 vs 73.56 fps, and overlapping p99 ranges of 22.03–45.13 vs 19.52–45.87 ms. That CPU-paced scene shows no clear cost within the spread. The rebased queue still needs a real job and a same-build comparison against e9e2edc. **Other systems:** Timing and profile use the watcher on Windows and Linux. Linux NVIDIA remains untested. Hybrid systems can query a sleeping or unused NVIDIA adapter even when the game uses another GPU. AMD/Intel-only and Steam Deck should omit the GPU fields when no usable NVIDIA device is found, with fallback coverage pending. Shared, GPU-only, build, light and timing-cpu jobs skip the watcher. History renders the noise object directly and accepts older records. The diff touches developer tooling, tests and docs, with no game, launcher, save, audio, input or shipped build changes. **Switches:** None. **Reviewed:** 3df042149f3a against e9e2edc62776: Read base AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards, the full diff and commit messages, and both changed Python files. Read the changed documentation and surrounding queue section, traced CLI dispatch, job shutdown and history readers, and reviewed the supplied comments, earlier bot review and reported runs. The rest of developing.md was outside the reviewed section. The review checklist and adjacent Tooling checkout were absent. No translations or generated files changed. Forgejo overlap and CI status could not be verified offline.; nothing run (a code review). **Checked and fine:** - The integer count preserves one thermal sample among 201. The explicit software and hardware thermal masks exclude idle and power-cap reasons. - The mode gate selects timing and profile. Existing history records remain readable and the new keys are optional. - The change stays on one topic, follows the surrounding Python style, uses the standard library and records AI assistance in the commit trailers. The diff contains no game data, generated output or bundled SDK. Renderer, shader, menu and screenshot checks do not apply. - The sampling delay is documented for short runs. The watcher waits five seconds after reader initialization, so actual first coverage can be later. **Blocking** (to change before merge) 1. scripts/devtools/jobqueue.py:543: A successful nvmlInit_v2 has no matching nvmlShutdown. A count error, zero cards or a failed handle enumeration can return immediately with an outstanding initialization. Successful watches also leave it outstanding when an in-process caller returns. Give the watcher ownership of the NVML session and pair every successful init with shutdown, including setup failures and normal stop. 2. scripts/devtools/jobqueue.py:704: The stop event interrupts the sleep only. If the child exits during a slow query, history can snapshot before that sample finishes and the queue can release its slot while the query is still running. Coordinate sample publication with the history snapshot and finish watcher cleanup before releasing the slot. If a hung native call must be tolerated, use a worker that can be stopped within a bounded time. 3. scripts/devtools/jobqueue.py:556: The reader keeps a card only when both queries succeed. A card that reports temperature but returns NOT_SUPPORTED for clock reasons loses its valid temperature. If a second card succeeds, history can report zero thermal samples as though both cards were checked. Preserve successful readings independently and expose incomplete thermal coverage. Test one card succeeding on only one query. 4. scripts/devtools/jobqueue.py:713: The warning attributes any enumerated card's slowdown to the timed run's frame rate. On a hybrid laptop measuring on the iGPU, or a workstation with a hot second NVIDIA card, that conclusion can be false. Identify the rendering adapter before attributing an effect, or state that an NVIDIA adapter on the host reported thermal slowdown and remove the causal claim from the warning and docs. 5. tests/test_job_queue.py:404: The no-process test replaces nvml_reader entirely. A future change that starts nvidia-smi from that function would pass this test unchanged. Keep the production reader in the test, substitute a fake ctypes library and verify both the reader's observable output and the absence of subprocess calls. **Waiting on the author** (a run, a measurement or a decision; no edit does it) - scripts/devtools/jobqueue.py:699: Initialization and polling overlap the timed workload. The per-read average misses startup, driver stalls and power effects, including querying a dormant NVIDIA adapter on a hybrid system. Compare this head with the merge base using repeated interleaved runs of the same game build, scene and settings. Report median, tail frame times and hitch counts, and check whether polling changes adapter power state on an affected hybrid setup. - The rebased head has no reported end-to-end queue job, applicable Tier 1 platform coverage, full strict check results or passing CI. Run the Windows and Linux NVIDIA queue checks and the AMD/Intel-only fallback checks, record the actual history and warning output, and report lint, strict suite and CI results. Compare failures against the current base. **Suggestions** (the author may take or leave) - `scripts/devtools/jobqueue.py:704`: A query already in flight can outlive both the history snapshot and the queue slot. Coordinate publication and finish cleanup before releasing the slot. **Testing:** Read-only review. The author reported 22 passed, two failures also seen on the older main, and one skip at 5a5a638d. The 201-sample test failed at 6f4da2dd and passed at 5a5a638d. The changed strict suite also reported the pre-existing local pack classification failure. No run at 3df04214 is reported. **Overlaps and merge order:** The author identifies #1816 and #1852 as touching other functions in the same files and reports that #1852 applies on top. Neither overlap was independently verified and no merge order is established. --- <sub>Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.</sub>
@ -174,1 +174,4 @@
- **Measure with `timing`, run everything else with `shared`.** Keep each GPU job short: a long batch holds the queue.
A timed run's line in `dev.py queue history` says how busy the rest of the PC was and, on an NVIDIA card, how hot the
GPU got and in how many of the run's readings a card was slowed for heat (`gpu_temp_max`, `gpu_slowed_for_heat` of
`gpu_samples`; the run says so when it ends). The cards are read every 5 s from the run's start: a shorter run has no
Member

The first wait starts after NVML setup, and each subsequent wait starts after the preceding read. Describe the cadence as approximate.

The first wait starts after NVML setup, and each subsequent wait starts after the preceding read. Describe the cadence as approximate.
@ -519,0 +540,4 @@
lib.nvmlDeviceGetTemperature.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(ctypes.c_uint)]
lib.nvmlDeviceGetCurrentClocksThrottleReasons.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulonglong)]
n = ctypes.c_uint()
if lib.nvmlInit_v2() or lib.nvmlDeviceGetCount_v2(ctypes.byref(n)):
Member

Pair each successful NVML init with shutdown, including count or enumeration failures and normal watcher exit.

Pair each successful NVML init with shutdown, including count or enumeration failures and normal watcher exit.
@ -519,0 +553,4 @@
def read():
out = []
for h in cards:
t, why = ctypes.c_uint(), ctypes.c_ulonglong()
Member

A successful temperature read is discarded if clock reasons are unsupported. Keep valid data and show when thermal coverage is incomplete.

A successful temperature read is discarded if clock reasons are unsupported. Keep valid data and show when thermal coverage is incomplete.
@ -632,6 +696,8 @@ def run(name, mode, cmd, environ=None, s=None, cwd=None, desc=None):
if timing_cpus and not common.WINDOWS:
threading.Thread(target=_keep_apart, args=(s, qdir, me, child.pid, cpu, timing_cpus, other_cpus, noise, stop),
daemon=True).start()
if gpu and cpu == "quiet":
Member

Measure startup and polling against the base with the same game build and interleaved runs. Include frame-time tails and check hybrid adapter wakeups.

Measure startup and polling against the base with the same game build and interleaved runs. Include frame-time tails and check hybrid adapter wakeups.
@ -644,2 +710,4 @@
if noise.strays or noise.cpu_pressure > 20:
say("%s: the PC was busy while it measured (%s): check the numbers" % (name, json.dumps(rec["noise"])))
if rec["noise"].get("gpu_slowed_for_heat"):
say("%s: a GPU was slowed for heat while it measured (%s): its frame rate follows the card's temperature, "
Member

On a multi-GPU system this may be a card the game never used. Report a host adapter observation unless the rendering adapter is known.

On a multi-GPU system this may be a card the game never used. Report a host adapter observation unless the rendering adapter is known.
@ -368,0 +401,4 @@
n, stop = q.Noise(), threading.Event()
started = AssertionError("a process was started while a timed run measures")
with mock.patch.object(q, "nvml_reader", return_value=lambda: [(90, 0x20)]), \
mock.patch.object(q, "GPU_HEAT_EVERY", 0.05), \
Member

Mock the ctypes library instead of nvml_reader so this test would catch a subprocess introduced inside the reader.

Mock the ctypes library instead of nvml_reader so this test would catch a subprocess introduced inside the reader.
Owner
@fix
Member

I changed this pull request as the review asked: the new head is e1b71543b6. I rebased it onto main (80bb1ccc18).

Changed

  • NVML session ownership: scripts/devtools/jobqueue.py:529 pairs successful initialization with shutdown on setup failure and watcher exit; tests cover setup failures.
  • Watcher and history synchronization: scripts/devtools/jobqueue.py:717 joins watchers before recording history or releasing the ticket. A hung native call will hold the slot until it returns.
  • Partial GPU readings: scripts/devtools/jobqueue.py:565 retains each successful reading and counts incomplete samples; tests/test_job_queue.py:437 covers cards with only one successful query.
  • Thermal slowdown attribution: scripts/devtools/jobqueue.py:740 and docs/dev/developing.md:175 describe host NVIDIA adapters without attributing frame-rate effects to the rendering adapter.
  • No-process reader test: tests/test_job_queue.py:409 exercises the production reader with a fake ctypes library and checks its output and subprocess calls. Tests were not run as instructed.
  • Bound NVML watcher cleanup: scripts/devtools/jobqueue.py:649: Moved NVML work to a process with bounded shutdown, termination and kill waits; completed samples are published before history is recorded. Added a lifecycle test at tests/test_job_queue.py:465. Tests were not run as instructed.

Still for the author (from the review)

  • scripts/devtools/jobqueue.py:699: Initialization and polling overlap the timed workload. The per-read average misses startup, driver stalls and power effects, including querying a dormant NVIDIA adapter on a hybrid system. Compare this head with the merge base using repeated interleaved runs of the same game build, scene and settings. Report median, tail frame times and hitch counts, and check whether polling changes adapter power state on an affected hybrid setup.
  • The rebased head has no reported end-to-end queue job, applicable Tier 1 platform coverage, full strict check results or passing CI. Run the Windows and Linux NVIDIA queue checks and the AMD/Intel-only fallback checks, record the actual history and warning output, and report lint, strict suite and CI results. Compare failures against the current base.

Nothing was built or run: check the change. It will be reviewed again.


Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.

I changed this pull request as the review asked: the new head is e1b71543b606. I rebased it onto `main` (80bb1ccc182d). **Changed** - **NVML session ownership**: scripts/devtools/jobqueue.py:529 pairs successful initialization with shutdown on setup failure and watcher exit; tests cover setup failures. - **Watcher and history synchronization**: scripts/devtools/jobqueue.py:717 joins watchers before recording history or releasing the ticket. A hung native call will hold the slot until it returns. - **Partial GPU readings**: scripts/devtools/jobqueue.py:565 retains each successful reading and counts incomplete samples; tests/test_job_queue.py:437 covers cards with only one successful query. - **Thermal slowdown attribution**: scripts/devtools/jobqueue.py:740 and docs/dev/developing.md:175 describe host NVIDIA adapters without attributing frame-rate effects to the rendering adapter. - **No-process reader test**: tests/test_job_queue.py:409 exercises the production reader with a fake ctypes library and checks its output and subprocess calls. Tests were not run as instructed. - **Bound NVML watcher cleanup: scripts/devtools/jobqueue.py:649**: Moved NVML work to a process with bounded shutdown, termination and kill waits; completed samples are published before history is recorded. Added a lifecycle test at tests/test_job_queue.py:465. Tests were not run as instructed. **Still for the author** (from the review) - scripts/devtools/jobqueue.py:699: Initialization and polling overlap the timed workload. The per-read average misses startup, driver stalls and power effects, including querying a dormant NVIDIA adapter on a hybrid system. Compare this head with the merge base using repeated interleaved runs of the same game build, scene and settings. Report median, tail frame times and hitch counts, and check whether polling changes adapter power state on an affected hybrid setup. - The rebased head has no reported end-to-end queue job, applicable Tier 1 platform coverage, full strict check results or passing CI. Run the Windows and Linux NVIDIA queue checks and the AMD/Intel-only fallback checks, record the actual history and warning output, and report lint, strict suite and CI results. Compare failures against the current base. Nothing was built or run: check the change. It will be reviewed again. --- <sub>Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.</sub>
BeanBot force-pushed feat/queue-gpu-heat from 3df042149f
Some checks failed
beanstalk/approval changed since the request: ask again
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 22s
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 15s
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Workflow lint (actionlint) (pull_request) Successful in 24s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 26s
Third-party parts / As their notes say (pull_request) Successful in 38s
CI / Website build and checks (pull_request) Has been cancelled
to e1b71543b6
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
beanstalk/approval changed since the request: ask again
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
PR description / Description follows the template (pull_request_target) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 15s
CI / Website build and checks (pull_request) Has been cancelled
2026-11-21 00:50:52 +00:00
Compare
BeanBot left a comment

Recommendation: Request changes. The heat readings are useful context for timed runs, but failure to start or stop the monitoring worker can still turn a successful command into a queue failure. The new worker’s effect on frame pacing and hybrid GPU power state remains unmeasured.

GPU timing and profile jobs now record the highest sampled NVIDIA temperature, the number of samples with thermal slowdown, and incomplete coverage. The worker isolates native driver calls from the queue runner, at the cost of starting another process during each monitored job and adding shutdown logic.

Since the last review (on 3df042149f)
Resolved:

  • scripts/devtools/jobqueue.py:543: A successful nvmlInit_v2 has no matching nvmlShutdown. A count error, zero cards or a failed handle enumeration can return immediately with an outstanding initialization. Successful watches also leave it outstanding when an in-process caller returns. Give the watche (nvml_reader pairs successful initialization with shutdown on setup failure. _watch_heat calls shutdown in finally on normal worker exit.)
  • scripts/devtools/jobqueue.py:556: The reader keeps a card only when both queries succeed. A card that reports temperature but returns NOT_SUPPORTED for clock reasons loses its valid temperature. If a second card succeeds, history can report zero thermal samples as though both cards were checked. Pre (nvml_reader preserves each successful query independently and gpu_heat counts incomplete samples. test_partial_card_readings covers both kinds of partial result.)
  • scripts/devtools/jobqueue.py:713: The warning attributes any enumerated card's slowdown to the timed run's frame rate. On a hybrid laptop measuring on the iGPU, or a workstation with a hot second NVIDIA card, that conclusion can be false. Identify the rendering adapter before attributing an effect, (The warning names an NVIDIA adapter on the host, and developing.md says the sampled adapters may exclude the rendering adapter.)
  • tests/test_job_queue.py:404: The no-process test replaces nvml_reader entirely. A future change that starts nvidia-smi from that function would pass this test unchanged. Keep the production reader in the test, substitute a fake ctypes library and verify both the reader's observable output and the ab (test_watch_starts_no_process exercises the production reader with a fake ctypes library and verifies its readings and shutdown. It checks the sampling loop inside the worker.)
    Still open:
  • scripts/devtools/jobqueue.py:704: The stop event interrupts the sleep only. If the child exits during a slow query, history can snapshot before that sample finishes and the queue can release its slot while the query is still running. Coordinate sample publication with the history snapshot and finish (The worker publishes complete aggregates and is joined on normal exit. If it survives the kill timeout, stop raises and the outer cleanup calls it again before reaching command, job-object or ticket cleanup.)
  • scripts/devtools/jobqueue.py:699: Initialization and polling overlap the timed workload. The per-read average misses startup, driver stalls and power effects, including querying a dormant NVIDIA adapter on a hybrid system. Compare this head with the merge base using repeated interleaved runs of the (The worker starts after the measured command launches. The reported comparisons predate process startup and do not cover hybrid adapter power.)
  • The rebased head has no reported end-to-end queue job, applicable Tier 1 platform coverage, full strict check results or passing CI. Run the Windows and Linux NVIDIA queue checks and the AMD/Intel-only fallback checks, record the actual history and warning output, and report lint, strict suite and C (No end-to-end queue run, applicable Tier 1 coverage, full strict and lint results, or passing CI was reported for HEAD.)
    New:
  • scripts/devtools/jobqueue.py:805: Worker startup is mandatory after the command has already launched. If the host rejects process creation at its process limit, _HeatWatcher raises and the outer cleanup terminates the command. Handle expected worker setup failures at this boundary, issue a diagnostic, and let the timed command finish without GPU readings.
  • scripts/devtools/jobqueue.py:656: The description explicitly says nothing starts during measurement, but every top-level timing or profile job now spawns a Python worker after the command starts, even on a host without NVIDIA. Update Summary, Risks and Measurements to describe the process and its shutdown behavior, and scope the no-process test wording to the sampling loop.

Does it help: Unproven end to end. The author measured 45.8 ms for NVML startup and 0.0033 ms per read over 2,000 reads on an RTX 4060 under Windows 11. Four interleaved game runs per side on the source branch showed 13.05 vs 12.97 ms median frame time and overlapping p99 ranges of 22.03–45.13 vs 19.52–45.87 ms. Those runs predate the worker process. HEAD still needs a real history result and repeated same-build comparisons against 80bb1ccc18.
Other systems: The worker starts for timing and profile on Windows and Linux, including AMD/Intel-only hosts and Steam Deck, where history should omit GPU fields if NVML finds no usable NVIDIA adapter. That fallback is pending. Hybrid systems can query an idle NVIDIA adapter while the game renders elsewhere. Build, light, shared, GPU-only and timing-cpu jobs skip the worker. History accepts older records. No game, launcher, save, audio, input or shipped build code changed.
Switches: None.

Reviewed: e1b71543b6 against 80bb1ccc18: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, developer standards, the full base-to-HEAD diff and commit log, and both changed Python files in full. Compared the earlier and current diffs and traced CLI dispatch, A/B invocation, history, and Windows job cleanup. Read the changed developer docs and surrounding queue section. The rest of developing.md was outside that read. Reviewed the supplied comments, reviews and recorded runs. The review checklist and adjacent Tooling checkout were absent. Forgejo CI and overlap status were unavailable offline.; nothing run (a code review).
Checked and fine:

  • One thermal sample among 201 still produces a nonzero count, and the warning uses that count. The mask selects the two thermal reasons.
  • Alternating slots keep an interrupted write from replacing the last complete aggregate. The normal path reads the aggregate after the worker exits.
  • The optional history keys preserve compatibility. The change stays on one topic and adds no third-party dependencies, bundled SDK, game data or generated output.

Blocking (to change before merge)

  1. scripts/devtools/jobqueue.py:833: If a Linux driver call leaves the worker in uninterruptible sleep, it can survive the kill timeout. stop then raises, and the second stop call in finally can raise before cancellation terminates the measured command or removes its ticket. Make queue cleanup run independently of watcher errors, perform stop escalation once, and define how the queue accounts for a worker that remains alive.
  2. scripts/devtools/jobqueue.py:805: Worker startup is mandatory after the command has already launched. If the host rejects process creation at its process limit, _HeatWatcher raises and the outer cleanup terminates the command. Handle expected worker setup failures at this boundary, issue a diagnostic, and let the timed command finish without GPU readings.

Waiting on the author (a run, a measurement or a decision; no edit does it)

  • scripts/devtools/jobqueue.py:656: The description explicitly says nothing starts during measurement, but every top-level timing or profile job now spawns a Python worker after the command starts, even on a host without NVIDIA. Update Summary, Risks and Measurements to describe the process and its shutdown behavior, and scope the no-process test wording to the sampling loop.
  • scripts/devtools/jobqueue.py:805: Worker startup and NVML calls overlap the workload. On a hybrid laptop, querying a dormant NVIDIA adapter can change its power state while the iGPU renders. Compare HEAD and the merge base using repeated interleaved runs of the same build, scene and settings. Report median and tail frame times, hitch counts, and hybrid adapter power state.
  • The reported queue tests predate the worker, and the mocked lifecycle test cannot exercise real process startup or history publication. Report real Windows and Linux NVIDIA queue runs, history and warning output, applicable Tier 1 and AMD/Intel-only fallback coverage, lint, the full strict suite, and CI for HEAD. Compare failures with the current base.

Suggestions (the author may take or leave)

  • Describe the first sample as five seconds after worker startup and NVML initialization. Later waits also follow the preceding query, so the documented interval from job start is approximate.

Testing: Read-only review. git diff --check passed. No project code or tests were executed. The author’s earlier Windows run at 5a5a638d reported 22 passed, two failures shared with its comparison main, and one skip.
Overlaps and merge order: The description identifies #1816 and #1852 as touching other functions in the same files and reports that #1852 applies on top. Current compatibility and merge order were unverified offline.


Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The heat readings are useful context for timed runs, but failure to start or stop the monitoring worker can still turn a successful command into a queue failure. The new worker’s effect on frame pacing and hybrid GPU power state remains unmeasured. GPU timing and profile jobs now record the highest sampled NVIDIA temperature, the number of samples with thermal slowdown, and incomplete coverage. The worker isolates native driver calls from the queue runner, at the cost of starting another process during each monitored job and adding shutdown logic. **Since the last review** (on 3df042149f3a) Resolved: - scripts/devtools/jobqueue.py:543: A successful nvmlInit_v2 has no matching nvmlShutdown. A count error, zero cards or a failed handle enumeration can return immediately with an outstanding initialization. Successful watches also leave it outstanding when an in-process caller returns. Give the watche (nvml_reader pairs successful initialization with shutdown on setup failure. _watch_heat calls shutdown in finally on normal worker exit.) - scripts/devtools/jobqueue.py:556: The reader keeps a card only when both queries succeed. A card that reports temperature but returns NOT_SUPPORTED for clock reasons loses its valid temperature. If a second card succeeds, history can report zero thermal samples as though both cards were checked. Pre (nvml_reader preserves each successful query independently and gpu_heat counts incomplete samples. test_partial_card_readings covers both kinds of partial result.) - scripts/devtools/jobqueue.py:713: The warning attributes any enumerated card's slowdown to the timed run's frame rate. On a hybrid laptop measuring on the iGPU, or a workstation with a hot second NVIDIA card, that conclusion can be false. Identify the rendering adapter before attributing an effect, (The warning names an NVIDIA adapter on the host, and developing.md says the sampled adapters may exclude the rendering adapter.) - tests/test_job_queue.py:404: The no-process test replaces nvml_reader entirely. A future change that starts nvidia-smi from that function would pass this test unchanged. Keep the production reader in the test, substitute a fake ctypes library and verify both the reader's observable output and the ab (test_watch_starts_no_process exercises the production reader with a fake ctypes library and verifies its readings and shutdown. It checks the sampling loop inside the worker.) Still open: - scripts/devtools/jobqueue.py:704: The stop event interrupts the sleep only. If the child exits during a slow query, history can snapshot before that sample finishes and the queue can release its slot while the query is still running. Coordinate sample publication with the history snapshot and finish (The worker publishes complete aggregates and is joined on normal exit. If it survives the kill timeout, stop raises and the outer cleanup calls it again before reaching command, job-object or ticket cleanup.) - scripts/devtools/jobqueue.py:699: Initialization and polling overlap the timed workload. The per-read average misses startup, driver stalls and power effects, including querying a dormant NVIDIA adapter on a hybrid system. Compare this head with the merge base using repeated interleaved runs of the (The worker starts after the measured command launches. The reported comparisons predate process startup and do not cover hybrid adapter power.) - The rebased head has no reported end-to-end queue job, applicable Tier 1 platform coverage, full strict check results or passing CI. Run the Windows and Linux NVIDIA queue checks and the AMD/Intel-only fallback checks, record the actual history and warning output, and report lint, strict suite and C (No end-to-end queue run, applicable Tier 1 coverage, full strict and lint results, or passing CI was reported for HEAD.) New: - scripts/devtools/jobqueue.py:805: Worker startup is mandatory after the command has already launched. If the host rejects process creation at its process limit, _HeatWatcher raises and the outer cleanup terminates the command. Handle expected worker setup failures at this boundary, issue a diagnostic, and let the timed command finish without GPU readings. - scripts/devtools/jobqueue.py:656: The description explicitly says nothing starts during measurement, but every top-level timing or profile job now spawns a Python worker after the command starts, even on a host without NVIDIA. Update Summary, Risks and Measurements to describe the process and its shutdown behavior, and scope the no-process test wording to the sampling loop. **Does it help:** Unproven end to end. The author measured 45.8 ms for NVML startup and 0.0033 ms per read over 2,000 reads on an RTX 4060 under Windows 11. Four interleaved game runs per side on the source branch showed 13.05 vs 12.97 ms median frame time and overlapping p99 ranges of 22.03–45.13 vs 19.52–45.87 ms. Those runs predate the worker process. HEAD still needs a real history result and repeated same-build comparisons against 80bb1ccc182d. **Other systems:** The worker starts for timing and profile on Windows and Linux, including AMD/Intel-only hosts and Steam Deck, where history should omit GPU fields if NVML finds no usable NVIDIA adapter. That fallback is pending. Hybrid systems can query an idle NVIDIA adapter while the game renders elsewhere. Build, light, shared, GPU-only and timing-cpu jobs skip the worker. History accepts older records. No game, launcher, save, audio, input or shipped build code changed. **Switches:** None. **Reviewed:** e1b71543b606 against 80bb1ccc182d: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, developer standards, the full base-to-HEAD diff and commit log, and both changed Python files in full. Compared the earlier and current diffs and traced CLI dispatch, A/B invocation, history, and Windows job cleanup. Read the changed developer docs and surrounding queue section. The rest of developing.md was outside that read. Reviewed the supplied comments, reviews and recorded runs. The review checklist and adjacent Tooling checkout were absent. Forgejo CI and overlap status were unavailable offline.; nothing run (a code review). **Checked and fine:** - One thermal sample among 201 still produces a nonzero count, and the warning uses that count. The mask selects the two thermal reasons. - Alternating slots keep an interrupted write from replacing the last complete aggregate. The normal path reads the aggregate after the worker exits. - The optional history keys preserve compatibility. The change stays on one topic and adds no third-party dependencies, bundled SDK, game data or generated output. **Blocking** (to change before merge) 1. scripts/devtools/jobqueue.py:833: If a Linux driver call leaves the worker in uninterruptible sleep, it can survive the kill timeout. stop then raises, and the second stop call in finally can raise before cancellation terminates the measured command or removes its ticket. Make queue cleanup run independently of watcher errors, perform stop escalation once, and define how the queue accounts for a worker that remains alive. 2. scripts/devtools/jobqueue.py:805: Worker startup is mandatory after the command has already launched. If the host rejects process creation at its process limit, _HeatWatcher raises and the outer cleanup terminates the command. Handle expected worker setup failures at this boundary, issue a diagnostic, and let the timed command finish without GPU readings. **Waiting on the author** (a run, a measurement or a decision; no edit does it) - scripts/devtools/jobqueue.py:656: The description explicitly says nothing starts during measurement, but every top-level timing or profile job now spawns a Python worker after the command starts, even on a host without NVIDIA. Update Summary, Risks and Measurements to describe the process and its shutdown behavior, and scope the no-process test wording to the sampling loop. - scripts/devtools/jobqueue.py:805: Worker startup and NVML calls overlap the workload. On a hybrid laptop, querying a dormant NVIDIA adapter can change its power state while the iGPU renders. Compare HEAD and the merge base using repeated interleaved runs of the same build, scene and settings. Report median and tail frame times, hitch counts, and hybrid adapter power state. - The reported queue tests predate the worker, and the mocked lifecycle test cannot exercise real process startup or history publication. Report real Windows and Linux NVIDIA queue runs, history and warning output, applicable Tier 1 and AMD/Intel-only fallback coverage, lint, the full strict suite, and CI for HEAD. Compare failures with the current base. **Suggestions** (the author may take or leave) - Describe the first sample as five seconds after worker startup and NVML initialization. Later waits also follow the preceding query, so the documented interval from job start is approximate. **Testing:** Read-only review. git diff --check passed. No project code or tests were executed. The author’s earlier Windows run at 5a5a638d reported 22 passed, two failures shared with its comparison main, and one skip. **Overlaps and merge order:** The description identifies #1816 and #1852 as touching other functions in the same files and reports that #1852 applies on top. Current compatibility and merge order were unverified offline. --- <sub>Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.</sub>
@ -175,0 +176,4 @@
the maximum reported temperature and how many readings reported thermal slowdown (`gpu_temp_max`, `gpu_slowed_for_heat`
of `gpu_samples`; the run says so when it ends). `gpu_incomplete_samples` counts readings where any adapter's temperature
or slowdown reasons were unavailable, so a zero slowdown count need not mean all adapters were checked. The adapters are
read every 5 s from the run's start: a shorter run has no reading. They may not include the rendering adapter.
Member

The first five-second wait begins after worker startup and NVML initialization. Describe that delay so missing readings on short jobs are expected.

The first five-second wait begins after worker startup and NVML initialization. Describe that delay so missing readings on short jobs are expected.
@ -519,0 +653,4 @@
receiver, self.control = ctx.Pipe(duplex=False)
self.process = ctx.Process(target=_heat_worker, args=(self.shared, receiver), daemon=True)
try:
self.process.start()
Member

This spawns a process during measurement. Update the PR description and scope the no-process test wording to sampling inside that worker.

This spawns a process during measurement. Update the PR description and scope the no-process test wording to sampling inside that worker.
@ -630,2 +802,4 @@
with common.Lock(lock):
write_ticket(qdir, me)
if gpu and cpu == "quiet":
heat = _HeatWatcher()
Member

A worker startup error can terminate an already-running timed command. Report the monitoring failure and let the command finish without GPU readings.

A worker startup error can terminate an already-running timed command. Report the monitoring failure and let the command finish without GPU readings.
Member

Measure the spawned worker and NVML calls against the current base, including frame-time tails, hitches and hybrid adapter power state.

Measure the spawned worker and NVML calls against the current base, including frame-time tails, hitches and hybrid adapter power state.
@ -648,1 +830,4 @@
finally:
stop.set()
if heat is not None:
heat.stop(noise)
Member

If stop raises again here after a kill timeout, the measured command and ticket cleanup are skipped. Run queue cleanup independently and escalate worker shutdown only once.

If stop raises again here after a kill timeout, the measured command and ticket cleanup are skipped. Run queue cleanup independently and escalate worker shutdown only once.
futondestroyer left a comment

Recommendation: Request changes. The heat monitor needs to be optional even when its worker fails. The game or profile command should keep its own result and the PC queue must release its ticket.

I independently confirm the worker start and shutdown blockers in the earlier exact-head review. Please also distinguish a dead or forced-stopped monitor from a normal run with no NVIDIA support. The parent reads the last complete aggregate without checking the worker's exit status. A process that dies after publishing one reading can leave a long run looking like it was monitored successfully, even though the rest was unsampled. Preserve the partial counts with an explicit monitor failure or incomplete-coverage marker.

Does it help: Unproven at this head. The earlier paired Windows results and the 2,000-query microcheck predate the spawned worker. The new mocks exercise aggregation and escalation but do not run the real spawn lifecycle. Verify a normal run and a forced worker failure on each OS, including that the command still exits and its queue ticket is removed. Compare paired tail frame times on NVIDIA and verify the fallback on AMD/Intel-only hardware and the adapter power behavior on a hybrid machine.

Testing: Read-only review of e1b71543b6 against 80bb1ccc18. Forgejo reported a failed guide-screenshot job, two pending statuses, five skipped and four successful. The author withdrew the old farm request. The normal in-game farm runner cannot establish a change to the developer queue's worker and history path.

**Recommendation: Request changes.** The heat monitor needs to be optional even when its worker fails. The game or profile command should keep its own result and the PC queue must release its ticket. I independently confirm the worker start and shutdown blockers in [the earlier exact-head review](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61603). Please also distinguish a dead or forced-stopped monitor from a normal run with no NVIDIA support. The parent reads the last complete aggregate without checking the worker's exit status. A process that dies after publishing one reading can leave a long run looking like it was monitored successfully, even though the rest was unsampled. Preserve the partial counts with an explicit monitor failure or incomplete-coverage marker. **Does it help:** Unproven at this head. The earlier paired Windows results and the 2,000-query microcheck predate the spawned worker. The new mocks exercise aggregation and escalation but do not run the real spawn lifecycle. Verify a normal run and a forced worker failure on each OS, including that the command still exits and its queue ticket is removed. Compare paired tail frame times on NVIDIA and verify the fallback on AMD/Intel-only hardware and the adapter power behavior on a hybrid machine. **Testing:** Read-only review of e1b71543b6064d2e357aa70aef1b72eb1a31f6ee against 80bb1ccc182da2400aecd1ed585d47cbe9746c88. Forgejo reported a failed guide-screenshot job, two pending statuses, five skipped and four successful. The author withdrew the old farm request. The normal in-game farm runner cannot establish a change to the developer queue's worker and history path.
@ -519,0 +678,4 @@
if process.is_alive():
raise RuntimeError("GPU heat worker did not stop")
start = 1 + 4 * self.shared[0]
count, slowed, temp, incomplete = self.shared[start:start + 4]
Member

The worker can exit abnormally after publishing some samples, or be terminated during shutdown. This reads the previous aggregate and records it like an ordinary run. Keep the available readings but record abnormal/forced termination separately, so partial sampling cannot look like the monitor ran for the whole command.

The worker can exit abnormally after publishing some samples, or be terminated during shutdown. This reads the previous aggregate and records it like an ordinary run. Keep the available readings but record abnormal/forced termination separately, so partial sampling cannot look like the monitor ran for the whole command.
siantm force-pushed feat/queue-gpu-heat from e1b71543b6
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
beanstalk/approval changed since the request: ask again
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
PR description / Description follows the template (pull_request_target) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 15s
CI / Website build and checks (pull_request) Has been cancelled
to 5f8fe59455
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
beanstalk/approval changed since the request: ask again
PR description / Description follows the template (pull_request_target) Successful in 7s
2026-11-21 03:24:26 +00:00
Compare
Author
Contributor

Pushed 9c6c5d32 (tests) and 5f8fe594 (fix) for BeanBot's review of e1b71543, with the branch rebased onto main @ 6e1521a7 (from 80bb1ccc; git range-diff shows the five earlier commits unchanged). The description is updated.

  • Blocking 1, jobqueue.py:833 (a worker surviving the kill makes the second stop raise before cleanup): fixed in 5f8fe594. _HeatWatcher.stop escalates once (stop, terminate, kill), drops its process reference first so a second call does nothing, and never raises for a worker that will not end: it is marked gpu_monitor: "not stopped" and left to end with its driver call. run()'s outer cleanup is now a try/finally, so the command is ended and the ticket removed even if a stop raises. The runner's own exit still waits for such a worker (Python joins its child processes); the Risks section says so. Tests: GpuHeat.test_worker_that_survives_the_kill, GpuHeatRun.test_cancel_with_a_worker_whose_stop_fails (a cancel during a 60 s command, a stop that raises: the command ended, the ticket gone).
  • Blocking 2, jobqueue.py:805 (worker startup failure ends the command): fixed in 5f8fe594. A start failure is reported ("the GPU heat worker did not start (...): this run has no GPU readings"), the command runs to its end, and the history has gpu_monitor: "not started". Test: GpuHeatRun.test_worker_that_cannot_start.
  • Waiting, jobqueue.py:656 (the description says nothing is started): Summary, Risks and Measurements now describe the worker process, when it starts, its shutdown and its failure marks; the tests' no-process wording is scoped to the sampling loop inside the worker (5f8fe594).
  • Suggestion, developing.md:179 (first sample after worker start and NVML init, cadence approximate): taken in 5f8fe594, in docs/dev/developing.md and _watch_heat's docstring.
  • Waiting, jobqueue.py:805 (measure the worker and NVML calls against the base, frame-time tails, hybrid adapter power): pending, for the owner. Nothing about the worker's cost is measured yet; the earlier in-game comparison predates the worker, and the description now says so. It is in Still to test.
  • The review's other waiting items (a real timed queue job, Windows/Linux/AMD-Intel runs, the full strict suite, lint, CI): pending. Run at 5f8fe594 through this PC's job queue: tests/test_job_queue.py 33 tests, 31 passed, 1 failed, 1 skipped; the failure (WindowsPlayersReal.test_a_build_outside_the_queue) fails the same way on main @ 6e1521a7 (21 tests, 19 passed). The five new failure tests fail at 9c6c5d32 and pass at 5f8fe594; GpuHeatRun.test_real_worker runs the real spawned worker around a 3 s timed command. Ruff was not run (not installed on this PC).
  • CI's failed "Guide screenshots" on e1b71543: it is not this PR's. scripts/check_guide_screenshots.py --review-sources fails the same way on main @ 6e1521a7 and on 5f8fe594 here (37 review-hash issues for files this PR does not touch); its two test files pass on both. The same job failed on main's own push runs for 80bb1ccc and 6e1521a7 and on every open PR with a finished run I checked (#1977, #1959, #1958, #1945, #1927, #1921, #1911, #1910).

This is a developer tool (the queue's runner): the Test Farm plays game scenarios and cannot run it.

Pushed 9c6c5d32 (tests) and 5f8fe594 (fix) for [BeanBot's review of e1b71543](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61603), with the branch rebased onto `main` @ 6e1521a7 (from 80bb1ccc; `git range-diff` shows the five earlier commits unchanged). The description is updated. - [Blocking 1, `jobqueue.py:833`](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61598) (a worker surviving the kill makes the second stop raise before cleanup): fixed in 5f8fe594. `_HeatWatcher.stop` escalates once (stop, terminate, kill), drops its process reference first so a second call does nothing, and never raises for a worker that will not end: it is marked `gpu_monitor: "not stopped"` and left to end with its driver call. `run()`'s outer cleanup is now a `try`/`finally`, so the command is ended and the ticket removed even if a stop raises. The runner's own exit still waits for such a worker (Python joins its child processes); the Risks section says so. Tests: `GpuHeat.test_worker_that_survives_the_kill`, `GpuHeatRun.test_cancel_with_a_worker_whose_stop_fails` (a cancel during a 60 s command, a stop that raises: the command ended, the ticket gone). - [Blocking 2, `jobqueue.py:805`](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61599) (worker startup failure ends the command): fixed in 5f8fe594. A start failure is reported ("the GPU heat worker did not start (...): this run has no GPU readings"), the command runs to its end, and the history has `gpu_monitor: "not started"`. Test: `GpuHeatRun.test_worker_that_cannot_start`. - [Waiting, `jobqueue.py:656`](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61600) (the description says nothing is started): Summary, Risks and Measurements now describe the worker process, when it starts, its shutdown and its failure marks; the tests' no-process wording is scoped to the sampling loop inside the worker (5f8fe594). - [Suggestion, `developing.md:179`](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61602) (first sample after worker start and NVML init, cadence approximate): taken in 5f8fe594, in `docs/dev/developing.md` and `_watch_heat`'s docstring. - [Waiting, `jobqueue.py:805`](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-61601) (measure the worker and NVML calls against the base, frame-time tails, hybrid adapter power): pending, for the owner. Nothing about the worker's cost is measured yet; the earlier in-game comparison predates the worker, and the description now says so. It is in Still to test. - The review's other waiting items (a real timed queue job, Windows/Linux/AMD-Intel runs, the full strict suite, lint, CI): pending. Run at 5f8fe594 through this PC's job queue: `tests/test_job_queue.py` 33 tests, 31 passed, 1 failed, 1 skipped; the failure (`WindowsPlayersReal.test_a_build_outside_the_queue`) fails the same way on `main` @ 6e1521a7 (21 tests, 19 passed). The five new failure tests fail at 9c6c5d32 and pass at 5f8fe594; `GpuHeatRun.test_real_worker` runs the real spawned worker around a 3 s timed command. Ruff was not run (not installed on this PC). - CI's failed "Guide screenshots" on e1b71543: it is not this PR's. `scripts/check_guide_screenshots.py --review-sources` fails the same way on `main` @ 6e1521a7 and on 5f8fe594 here (37 review-hash issues for files this PR does not touch); its two test files pass on both. The same job failed on `main`'s own push runs for 80bb1ccc and 6e1521a7 and on every open PR with a finished run I checked (#1977, #1959, #1958, #1945, #1927, #1921, #1911, #1910). This is a developer tool (the queue's runner): the Test Farm plays game scenarios and cannot run it.
Author
Contributor

@futondestroyer, on your review and the worker's exit status: done in 5f8fe594 (test 9c6c5d32 first).

  • The monitor is optional: a worker that cannot start leaves the command running to its own status, without readings (gpu_monitor: "not started"); the command's end and the ticket's removal no longer depend on the worker's stop.
  • A worker that did not finish normally is recorded separately from a run without NVIDIA: its last complete counts are kept and gpu_monitor says failed (exit code N), terminated, killed or not stopped; the run prints that its readings cover part of it. A normal end (status 0) adds nothing, as a host without NVIDIA does.
  • Covered in tests/test_job_queue.py (Windows, this PC's queue): start failure and a cancel with a failing stop through run() with a real command and ticket, the stop's marks over a stand-in process, and the real spawned worker around a 3 s command. A real timed game job, a forced failure on Linux, the paired tail-frame runs and the hybrid/AMD/Intel checks have not run; they are in Still to test.
@futondestroyer, on [your review](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-62716) and [the worker's exit status](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1916#issuecomment-62715): done in 5f8fe594 (test 9c6c5d32 first). - The monitor is optional: a worker that cannot start leaves the command running to its own status, without readings (`gpu_monitor: "not started"`); the command's end and the ticket's removal no longer depend on the worker's stop. - A worker that did not finish normally is recorded separately from a run without NVIDIA: its last complete counts are kept and `gpu_monitor` says `failed (exit code N)`, `terminated`, `killed` or `not stopped`; the run prints that its readings cover part of it. A normal end (status 0) adds nothing, as a host without NVIDIA does. - Covered in `tests/test_job_queue.py` (Windows, this PC's queue): start failure and a cancel with a failing stop through `run()` with a real command and ticket, the stop's marks over a stand-in process, and the real spawned worker around a 3 s command. A real timed game job, a forced failure on Linux, the paired tail-frame runs and the hybrid/AMD/Intel checks have not run; they are in Still to test.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
beanstalk/approval changed since the request: ask again
PR description / Description follows the template (pull_request_target) Successful in 7s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/queue-gpu-heat:siantm-feat/queue-gpu-heat'
git switch 'siantm-feat/queue-gpu-heat'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1916
No description provided.