WIP: feat(queue): a timed run's history says how hot the GPU got and whether it was slowed for heat #1916
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1916
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:feat/queue-gpu-heat"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A timed run's line in
dev.py queue historysays how busy the rest of the PC was while it measured, so a noisymeasurement shows. It said nothing of the GPU: a card at its temperature limit runs its clocks slower, and a timed run's
frame rate then follows the card's heat and the order the runs came in, not the build it measures. On Windows the
history's noise had no reading at all (no PSI, no load average).
A timed run that uses the GPU (
timing,profile) now reads each NVIDIA adapter's temperature and its clocks' reasonsfor running slower, and its noise gains
gpu_temp_max,gpu_samples(the readings taken),gpu_slowed_for_heat(howmany of them found an adapter slowed for heat: NVML's software or hardware thermal slowdown among the clock reasons, a
count so one slowed reading in a long run still shows) and
gpu_incomplete_samples(readings where an adapter'stemperature or reasons were unavailable). The run says so when it ends with an adapter slowed. The adapters read are
the host's NVIDIA ones and may not include the rendering adapter.
The readings are taken by a worker process (Python's
multiprocessing, spawned) that the queue's runner starts oncethe run's command has started, so a driver call that hangs or crashes is not in the runner. The worker loads NVIDIA's
management library (
nvml.dll,libnvidia-ml.so.1); its first reading is about 5 s after it has started and loadedthe library, then one about every 5 s (each wait follows the reading before it), so a shorter run has none. Without
the library or an adapter it ends at once and nothing is added. The worker is optional to the run: if it cannot start,
the command runs on without readings (
gpu_monitor: "not started"). When the command ends the runner asks it to stop,then terminates and kills it if it does not (once), keeps its last complete readings, and names a worker that did not
run to the end in
gpu_monitor(failed (exit code N),terminated,killed, ornot stoppedwhen it is stillalive after the kill), so partial readings do not pass for a whole run's; the command's end and the ticket's removal
do not depend on it.
timing-cpu(a measured build), builds and shared runs start no worker.Read first:
nvml_reader,gpu_heat,_watch_heat,_HeatWatcherand the heat lines ofrun()inscripts/devtools/jobqueue.py; theGpuHeatandGpuHeatRuntests intests/test_job_queue.py. Other functions ofthe same files than the open #1816 and #1852.
Testing
Tested: Windows 11, RTX 4060 (NVIDIA 610.88), i7-9700, Python 3.14.7 (the Builder's), each run in a hidden
console of its own through the PC's job queue, on this branch rebased onto
main@6e1521a7(head5f8fe594):python -I -B tests/test_job_queue.py -v@5f8fe594: 33 tests, 31 passed, 1 failed, 1 skipped. The failure,WindowsPlayersReal.test_a_build_outside_the_queue, fails onmain@6e1521a7the same way in the same job(
main: 21 tests, 19 passed, that 1 failed, 1 skipped). AllGpuHeatandGpuHeatRuntests pass, among themGpuHeatRun.test_real_worker: the real worker spawned and stopped around a 3 s timed command throughrun()(status 0, the ticket removed, a normal end so no
gpu_monitor, no reading before 5 s).9c6c5d32(before the fix): 5 of them fail there(
test_worker_that_cannot_start,test_cancel_with_a_worker_whose_stop_fails,test_worker_that_survives_the_kill,test_worker_that_failed,test_worker_stops_before_publishing) and pass @5f8fe594.tests/run_all.py --strict --changed origin/main@5f8fe594: 4 passed, 2 failed:test_job_queue(the failureabove, shared with
main) andtest_dev_packs(an unclassifiedlongshot_fsr4_pipelines.binbeside this PC's game;it fails the same way on
main@6e1521a7in the queue).5f8fe594and onmain@6e1521a7:scripts/check_guide_screenshots.py --review-sourcesfails the same way on both (37 issues, review hashes of files this PR does not touch), its twotest files pass on both. The CI job fails the same way on
main's own push runs for80bb1cccand6e1521a7.05eaf6f8, the reader then in the runner's process): NVIDIA's library read from Python on thisPC with
nvml_reader: loaded in 45.8 ms, then 2,000 reads at 0.0033 ms each.Not run: ruff (
mise run lint; ruff is not installed on this PC). No game run under this branch's own queue yet: it isa developer tool (the queue's runner), which the Test Farm does not run.
Still to test:
dev.py queue run ... --mode timingaround a gamerun), then
dev.py queue history, expectinggpu_temp_maxandgpu_slowed_for_heatin that job's noise, and thesame with the worker made to fail (expecting the command's own status, its ticket removed and
gpu_monitorin thehistory) (by hand, pending)
main's build:main's pack, timed game runs, at least 4 a sideinterleaved after a warm-up, wrapped by
main'sdev.py queue run --mode timingand by this branch's (only thelatter starts the worker), comparing each run's median frame, p99, 1 % low and frames over 50 ms, expecting no
difference beyond the run-to-run spread (by hand, pending)
adapter's power state while the game renders on the other (by hand)
tests/run_all.py --strict) here and onmain, expecting no failure here thatmaindoes not have (by hand, pending)python -I scripts/dev.py queue run heat --mode timing -- sleep 20, thenpython -I scripts/dev.py queue history, expectinggpu_temp_maxandgpu_slowed_for_heatin the job's noise(by hand: the Test Farm runs game scenarios only, not the queue)
Test Farm runs game scenarios only, not the queue)
Measurements
Verdict: not measured at this head. A reading cost 0.0033 ms and loading the library 45.8 ms when the reader ran
in the queue's own process (
05eaf6f8); the reader now runs in a worker process started beside the timed run, whosestart, NVIDIA library load and shutdown have not been timed, and whose effect on a game run has not been measured
(Still to test). In the game, on the branch this was ported from (the reader in the runner's process, no worker), 4
interleaved timed runs with the reader and 4 without showed no difference: with the reader, the runs' median frame,
CPU time a frame and frames a second each lie within the range of the runs without it, and p99 varies more between
runs of one arm than between the arms.
Setup: Windows 11, RTX 4060 (NVIDIA 610.88), i7-9700, Python 3.14.7;
nvml_reader()timed from Python, 2,000 reads, at05eaf6f8;tests/test_job_queue.py, one run each side, in one job of the PC's job queue6e1521a75f8fe594tests/test_job_queue.pyGpuHeat.test_watch_starts_no_process)05eaf6f8, in-process); the worker's start and shutdown not timedThe watch in the game, measured before this PR (2026-11-20, for e49c733f on the user's performance branch, the
commit this PR's
nvml_readerand_watch_heatwere ported from, reading the same and as often): that branch's gamebuild (
v3.0.0-rc.7-292-g8c5ecbe9) and its ownscripts/job_queue.py, notmain's; Kokiri Forest, 800x600 window(CPU-paced), RTX 4060; one warm-up run, then 8 timed runs interleaved P N N P P N N P through the PC's job queue: N
with the reader (each N job's history line has
gpu_temp_max, 78-82 °C, never slowed for heat), P under that queuescript as it was before the heat record (no GPU fields). Each run's numbers from its
metrics.json:Four runs an arm do not show a difference under about 1 %. The form before it, which started
nvidia-smiat eachsample, did cost the run (median frame 13.24 against 13.01 ms, the ranges apart), which is why this PR reads through
the library.
Risks
The worker is a process the queue's runner starts beside every GPU timed run (
timing,profile), on hosts withoutNVIDIA too (it ends at once there). Its start, NVIDIA library load and reads overlap the measured run: their cost in
a game run is not measured yet (Still to test), and on a hybrid laptop reading the NVIDIA adapter could change its
power state. If it cannot start, the run goes on without readings; if it does not stop, it is terminated, then killed;
one still alive after the kill (a driver call it cannot leave) is marked
not stoppedand left to end with that call:the run's history is written and its ticket removed, and the runner's own exit waits for it (Python joins its child
processes). Nothing is read for builds, shared runs or
timing-cpu. No switch: a revert is enough. Players are notaffected:
scripts/devtools/is not part of the game or its build.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Recommendation: Request changes. The history can report no thermal slowdown even after observing one, and the new monitoring runs during measurements without an end-to-end comparison of its effect on frame times.
The queue records sampled NVIDIA temperatures and thermal slowdown reasons for GPU timing and profile jobs, which could help explain noisy comparisons. The aggregation and stand-in reader tests are useful, but rounding can hide a real warning and the cost of querying the driver during a game measurement remains unknown.
Does it help: Unproven. The author reports a 45.8 ms library load and 0.0033 ms per read over 2,000 reads on Windows 11 with an RTX 4060; no real timed job or before-and-after frame-time comparison is reported. The Windows queue tests report 21 passed, two failures also seen on main, and one skip.
Other systems: The watcher runs for timing and profile jobs on Windows and Linux, including machines with multiple NVIDIA cards. Linux NVIDIA and AMD/Intel-only fallback runs are still pending. It does not run for timing-cpu or shared jobs; the game, launcher and build outputs are unchanged. The effect of NVML calls on GPU work and frame pacing has not been measured.
Switches: None added, changed or removed.
Reviewed:
05eaf6f884against8859673e4b: Read AGENTS.md, CONTRIBUTING.md, the pull request template, the applicable standards, the diff and commit, and the changed code and its queue and history paths. Read the changed documentation and tests; no generated files or translations changed. docs/review-checklist.md and the adjacent Tooling checkout were absent. The network was unavailable, so open PRs and Tooling's rules could not be independently checked.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Suggestions (the author may take or leave)
Testing: Read-only review; no code or tests were run. The PR still needs its real timed job, full strict suite and listed Linux and non-NVIDIA checks, plus applicable Tier 1 coverage and passing CI.
Overlaps and merge order: The description identifies #1816 and #1852 as touching other functions in the same files and states that #1852 applies on top. Their contents and merge order could not be independently checked offline.
Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.
@ -519,0 +570,4 @@return Noneslowed = sum(1 for cards in samples if any(why & NVML_SLOWED_FOR_HEAT for _, why in cards))return {"gpu_temp_max": max(t for cards in samples for t, _ in cards),"gpu_slowed_for_heat": round(slowed / len(samples), 2)}One slowed sample among 201 rounds to 0.0, so history says zero and the warning is skipped despite observing thermal slowdown. Keep sufficient precision and trigger the warning from the unrounded count; please cover this case in a test.
@ -519,0 +577,4 @@"""Every GPU_HEAT_EVERY seconds while a timed run uses the GPU: the cards' heat into its noise. Read in this process:nothing is started while the run measures"""read = nvml_reader()while read and not stop.wait(GPU_HEAT_EVERY):The first reading waits five seconds, so shorter GPU timing jobs always have no heat data even with a working reader. Consider sampling at the start or documenting this coverage gap.
@ -632,6 +695,8 @@ def run(name, mode, cmd, environ=None, s=None, cwd=None, desc=None):if timing_cpus and not common.WINDOWS:threading.Thread(target=_keep_apart, args=(s, qdir, me, child.pid, cpu, timing_cpus, other_cpus, noise, stop),daemon=True).start()This loads and queries NVML while the game is running. The reported reader timing does not show whether driver calls affect frame pacing. Please compare repeated interleaved timed game runs against the base, including frame-time tails, and address any meaningful cost.
Pushed
6f4da2dd,92724a75and5a5a638dfor BeanBot's review; the description is updated to match.round(1/201, 2)is 0.0 and the run's warning read that field.6f4da2ddaddsGpuHeat.test_one_slowed_sample_is_said(one slowed sample among 201), which fails there;92724a75makesgpu_slowed_for_heatthe number of slowed samples, besidegpu_samples, so the history and the warning see any slowed sample, and the test passes.tests/test_job_queue.py@5a5a638d: 25 tests, 22 passed, the same 2 failures asmain, 1 skipped, run through this PC's job queue.gpu_samples).5a5a638dsays it indocs/dev/developing.mdand in_watch_heat's docstring.main's queue and this branch's (same pack, at least 4 a side, frame-trace median, p99 and 1 % low) are now in Still to test and pending; nothing about it is measured yet.GpuHeat.test_watch_starts_no_processand the record bytest_record; whatrun()adds is the thread's start (scripts/devtools/jobqueue.py:699-700) and the record and warning (:708-714). A real timed job through this branch's queue and its history line are already in Still to test, so I've left it there.On Blocking 2 (NVML polled while the game runs): this was measured on 2026-11-20 for e49c733f, the commit on the user's performance branch this PR's
nvml_readerand_watch_heatwere ported from. Kokiri Forest, 8 timed runs interleaved P N N P P N N P after a warm-up, 4 with the reader and 4 without: median frame 13.05 ms (12.87-13.20) with against 12.97 (12.75-13.27) without, CPU time a frame 59.04 against 58.40 ms, the reader's ranges inside the others; p99 ran from 22.0 to 45.1 ms with and from 19.5 to 45.9 ms without, so it varies more between runs of one arm than between arms. The Measurements section now has each run's numbers and their limits: that branch's game build and queue script, notmain's, and four runs an arm resolve nothing under about 1 %. A repeat onmain's build with this branch's queue stays under Still to test, pending.Still to test: the items for hardware this PC lacks go to the Test Farm (@Beanstalk form).
@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for5a5a638, and the PR has changed since (3df0421), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.The Test Farm request in my previous comment was not meant: the farm plays game scenarios and times builds, comparing this PR's game against main's, and this change is in a developer tool that the game does not run (no plan of the farm's runs a test file or the job queue). The Linux and other-GPU items in Still to test are back to by hand. Nothing for a BigBean to approve here.
@fix
I changed this pull request by bringing it up to date: the new head is
3df042149f. I rebased it ontomain(e9e2edc627).Left for the author
Nothing was built or run: check the change. It will be reviewed again.
Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.
5a5a638ddb3df042149fRecommendation: Request changes. GPU heat is useful context for a timed job, but the current watcher can misattribute throttling, discard valid readings, and outlive the queue slot. The cost of initializing and polling NVML during measurements still needs a comparison at this head against its base.
Adds sampled NVIDIA temperatures and thermal slowdown counts to queue history and warns at job completion. The additive history format and integer count are useful. The cost is native driver work during a measurement, with incomplete handling of unsupported queries and watcher shutdown.
Does it help: Unproven end to end. The author measured 45.8 ms to load and initialize the reader and 0.0033 ms per read over 2,000 reads on a Windows 11 RTX 4060 at
05eaf6f8. Four interleaved runs per side on the source branch gave 13.05 vs 12.97 ms median frame, 73.23 vs 73.56 fps, and overlapping p99 ranges of 22.03–45.13 vs 19.52–45.87 ms. That CPU-paced scene shows no clear cost within the spread. The rebased queue still needs a real job and a same-build comparison againste9e2edc.Other systems: Timing and profile use the watcher on Windows and Linux. Linux NVIDIA remains untested. Hybrid systems can query a sleeping or unused NVIDIA adapter even when the game uses another GPU. AMD/Intel-only and Steam Deck should omit the GPU fields when no usable NVIDIA device is found, with fallback coverage pending. Shared, GPU-only, build, light and timing-cpu jobs skip the watcher. History renders the noise object directly and accepts older records. The diff touches developer tooling, tests and docs, with no game, launcher, save, audio, input or shipped build changes.
Switches: None.
Reviewed:
3df042149fagainste9e2edc627: Read base AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards, the full diff and commit messages, and both changed Python files. Read the changed documentation and surrounding queue section, traced CLI dispatch, job shutdown and history readers, and reviewed the supplied comments, earlier bot review and reported runs. The rest of developing.md was outside the reviewed section. The review checklist and adjacent Tooling checkout were absent. No translations or generated files changed. Forgejo overlap and CI status could not be verified offline.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Waiting on the author (a run, a measurement or a decision; no edit does it)
Suggestions (the author may take or leave)
scripts/devtools/jobqueue.py:704: A query already in flight can outlive both the history snapshot and the queue slot. Coordinate publication and finish cleanup before releasing the slot.Testing: Read-only review. The author reported 22 passed, two failures also seen on the older main, and one skip at
5a5a638d. The 201-sample test failed at6f4da2ddand passed at5a5a638d. The changed strict suite also reported the pre-existing local pack classification failure. No run at3df04214is reported.Overlaps and merge order: The author identifies #1816 and #1852 as touching other functions in the same files and reports that #1852 applies on top. Neither overlap was independently verified and no merge order is established.
Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.
@ -174,1 +174,4 @@- **Measure with `timing`, run everything else with `shared`.** Keep each GPU job short: a long batch holds the queue.A timed run's line in `dev.py queue history` says how busy the rest of the PC was and, on an NVIDIA card, how hot theGPU got and in how many of the run's readings a card was slowed for heat (`gpu_temp_max`, `gpu_slowed_for_heat` of`gpu_samples`; the run says so when it ends). The cards are read every 5 s from the run's start: a shorter run has noThe first wait starts after NVML setup, and each subsequent wait starts after the preceding read. Describe the cadence as approximate.
@ -519,0 +540,4 @@lib.nvmlDeviceGetTemperature.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(ctypes.c_uint)]lib.nvmlDeviceGetCurrentClocksThrottleReasons.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulonglong)]n = ctypes.c_uint()if lib.nvmlInit_v2() or lib.nvmlDeviceGetCount_v2(ctypes.byref(n)):Pair each successful NVML init with shutdown, including count or enumeration failures and normal watcher exit.
@ -519,0 +553,4 @@def read():out = []for h in cards:t, why = ctypes.c_uint(), ctypes.c_ulonglong()A successful temperature read is discarded if clock reasons are unsupported. Keep valid data and show when thermal coverage is incomplete.
@ -632,6 +696,8 @@ def run(name, mode, cmd, environ=None, s=None, cwd=None, desc=None):if timing_cpus and not common.WINDOWS:threading.Thread(target=_keep_apart, args=(s, qdir, me, child.pid, cpu, timing_cpus, other_cpus, noise, stop),daemon=True).start()if gpu and cpu == "quiet":Measure startup and polling against the base with the same game build and interleaved runs. Include frame-time tails and check hybrid adapter wakeups.
@ -644,2 +710,4 @@if noise.strays or noise.cpu_pressure > 20:say("%s: the PC was busy while it measured (%s): check the numbers" % (name, json.dumps(rec["noise"])))if rec["noise"].get("gpu_slowed_for_heat"):say("%s: a GPU was slowed for heat while it measured (%s): its frame rate follows the card's temperature, "On a multi-GPU system this may be a card the game never used. Report a host adapter observation unless the rendering adapter is known.
@ -368,0 +401,4 @@n, stop = q.Noise(), threading.Event()started = AssertionError("a process was started while a timed run measures")with mock.patch.object(q, "nvml_reader", return_value=lambda: [(90, 0x20)]), \mock.patch.object(q, "GPU_HEAT_EVERY", 0.05), \Mock the ctypes library instead of nvml_reader so this test would catch a subprocess introduced inside the reader.
@fix
I changed this pull request as the review asked: the new head is
e1b71543b6. I rebased it ontomain(80bb1ccc18).Changed
Still for the author (from the review)
Nothing was built or run: check the change. It will be reviewed again.
Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.
3df042149fe1b71543b6Recommendation: Request changes. The heat readings are useful context for timed runs, but failure to start or stop the monitoring worker can still turn a successful command into a queue failure. The new worker’s effect on frame pacing and hybrid GPU power state remains unmeasured.
GPU timing and profile jobs now record the highest sampled NVIDIA temperature, the number of samples with thermal slowdown, and incomplete coverage. The worker isolates native driver calls from the queue runner, at the cost of starting another process during each monitored job and adding shutdown logic.
Since the last review (on
3df042149f)Resolved:
Still open:
New:
Does it help: Unproven end to end. The author measured 45.8 ms for NVML startup and 0.0033 ms per read over 2,000 reads on an RTX 4060 under Windows 11. Four interleaved game runs per side on the source branch showed 13.05 vs 12.97 ms median frame time and overlapping p99 ranges of 22.03–45.13 vs 19.52–45.87 ms. Those runs predate the worker process. HEAD still needs a real history result and repeated same-build comparisons against
80bb1ccc18.Other systems: The worker starts for timing and profile on Windows and Linux, including AMD/Intel-only hosts and Steam Deck, where history should omit GPU fields if NVML finds no usable NVIDIA adapter. That fallback is pending. Hybrid systems can query an idle NVIDIA adapter while the game renders elsewhere. Build, light, shared, GPU-only and timing-cpu jobs skip the worker. History accepts older records. No game, launcher, save, audio, input or shipped build code changed.
Switches: None.
Reviewed:
e1b71543b6against80bb1ccc18: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, developer standards, the full base-to-HEAD diff and commit log, and both changed Python files in full. Compared the earlier and current diffs and traced CLI dispatch, A/B invocation, history, and Windows job cleanup. Read the changed developer docs and surrounding queue section. The rest of developing.md was outside that read. Reviewed the supplied comments, reviews and recorded runs. The review checklist and adjacent Tooling checkout were absent. Forgejo CI and overlap status were unavailable offline.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Waiting on the author (a run, a measurement or a decision; no edit does it)
Suggestions (the author may take or leave)
Testing: Read-only review. git diff --check passed. No project code or tests were executed. The author’s earlier Windows run at
5a5a638dreported 22 passed, two failures shared with its comparison main, and one skip.Overlaps and merge order: The description identifies #1816 and #1852 as touching other functions in the same files and reports that #1852 applies on top. Current compatibility and merge order were unverified offline.
Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.
@ -175,0 +176,4 @@the maximum reported temperature and how many readings reported thermal slowdown (`gpu_temp_max`, `gpu_slowed_for_heat`of `gpu_samples`; the run says so when it ends). `gpu_incomplete_samples` counts readings where any adapter's temperatureor slowdown reasons were unavailable, so a zero slowdown count need not mean all adapters were checked. The adapters areread every 5 s from the run's start: a shorter run has no reading. They may not include the rendering adapter.The first five-second wait begins after worker startup and NVML initialization. Describe that delay so missing readings on short jobs are expected.
@ -519,0 +653,4 @@receiver, self.control = ctx.Pipe(duplex=False)self.process = ctx.Process(target=_heat_worker, args=(self.shared, receiver), daemon=True)try:self.process.start()This spawns a process during measurement. Update the PR description and scope the no-process test wording to sampling inside that worker.
@ -630,2 +802,4 @@with common.Lock(lock):write_ticket(qdir, me)if gpu and cpu == "quiet":heat = _HeatWatcher()A worker startup error can terminate an already-running timed command. Report the monitoring failure and let the command finish without GPU readings.
Measure the spawned worker and NVML calls against the current base, including frame-time tails, hitches and hybrid adapter power state.
@ -648,1 +830,4 @@finally:stop.set()if heat is not None:heat.stop(noise)If stop raises again here after a kill timeout, the measured command and ticket cleanup are skipped. Run queue cleanup independently and escalate worker shutdown only once.
Recommendation: Request changes. The heat monitor needs to be optional even when its worker fails. The game or profile command should keep its own result and the PC queue must release its ticket.
I independently confirm the worker start and shutdown blockers in the earlier exact-head review. Please also distinguish a dead or forced-stopped monitor from a normal run with no NVIDIA support. The parent reads the last complete aggregate without checking the worker's exit status. A process that dies after publishing one reading can leave a long run looking like it was monitored successfully, even though the rest was unsampled. Preserve the partial counts with an explicit monitor failure or incomplete-coverage marker.
Does it help: Unproven at this head. The earlier paired Windows results and the 2,000-query microcheck predate the spawned worker. The new mocks exercise aggregation and escalation but do not run the real spawn lifecycle. Verify a normal run and a forced worker failure on each OS, including that the command still exits and its queue ticket is removed. Compare paired tail frame times on NVIDIA and verify the fallback on AMD/Intel-only hardware and the adapter power behavior on a hybrid machine.
Testing: Read-only review of
e1b71543b6against80bb1ccc18. Forgejo reported a failed guide-screenshot job, two pending statuses, five skipped and four successful. The author withdrew the old farm request. The normal in-game farm runner cannot establish a change to the developer queue's worker and history path.@ -519,0 +678,4 @@if process.is_alive():raise RuntimeError("GPU heat worker did not stop")start = 1 + 4 * self.shared[0]count, slowed, temp, incomplete = self.shared[start:start + 4]The worker can exit abnormally after publishing some samples, or be terminated during shutdown. This reads the previous aggregate and records it like an ordinary run. Keep the available readings but record abnormal/forced termination separately, so partial sampling cannot look like the monitor ran for the whole command.
e1b71543b65f8fe59455Pushed
9c6c5d32(tests) and5f8fe594(fix) for BeanBot's review of e1b71543, with the branch rebased ontomain@6e1521a7(from80bb1ccc;git range-diffshows the five earlier commits unchanged). The description is updated.jobqueue.py:833(a worker surviving the kill makes the second stop raise before cleanup): fixed in5f8fe594._HeatWatcher.stopescalates once (stop, terminate, kill), drops its process reference first so a second call does nothing, and never raises for a worker that will not end: it is markedgpu_monitor: "not stopped"and left to end with its driver call.run()'s outer cleanup is now atry/finally, so the command is ended and the ticket removed even if a stop raises. The runner's own exit still waits for such a worker (Python joins its child processes); the Risks section says so. Tests:GpuHeat.test_worker_that_survives_the_kill,GpuHeatRun.test_cancel_with_a_worker_whose_stop_fails(a cancel during a 60 s command, a stop that raises: the command ended, the ticket gone).jobqueue.py:805(worker startup failure ends the command): fixed in5f8fe594. A start failure is reported ("the GPU heat worker did not start (...): this run has no GPU readings"), the command runs to its end, and the history hasgpu_monitor: "not started". Test:GpuHeatRun.test_worker_that_cannot_start.jobqueue.py:656(the description says nothing is started): Summary, Risks and Measurements now describe the worker process, when it starts, its shutdown and its failure marks; the tests' no-process wording is scoped to the sampling loop inside the worker (5f8fe594).developing.md:179(first sample after worker start and NVML init, cadence approximate): taken in5f8fe594, indocs/dev/developing.mdand_watch_heat's docstring.jobqueue.py:805(measure the worker and NVML calls against the base, frame-time tails, hybrid adapter power): pending, for the owner. Nothing about the worker's cost is measured yet; the earlier in-game comparison predates the worker, and the description now says so. It is in Still to test.5f8fe594through this PC's job queue:tests/test_job_queue.py33 tests, 31 passed, 1 failed, 1 skipped; the failure (WindowsPlayersReal.test_a_build_outside_the_queue) fails the same way onmain@6e1521a7(21 tests, 19 passed). The five new failure tests fail at9c6c5d32and pass at5f8fe594;GpuHeatRun.test_real_workerruns the real spawned worker around a 3 s timed command. Ruff was not run (not installed on this PC).e1b71543: it is not this PR's.scripts/check_guide_screenshots.py --review-sourcesfails the same way onmain@6e1521a7and on5f8fe594here (37 review-hash issues for files this PR does not touch); its two test files pass on both. The same job failed onmain's own push runs for80bb1cccand6e1521a7and on every open PR with a finished run I checked (#1977, #1959, #1958, #1945, #1927, #1921, #1911, #1910).This is a developer tool (the queue's runner): the Test Farm plays game scenarios and cannot run it.
@futondestroyer, on your review and the worker's exit status: done in
5f8fe594(test9c6c5d32first).gpu_monitor: "not started"); the command's end and the ticket's removal no longer depend on the worker's stop.gpu_monitorsaysfailed (exit code N),terminated,killedornot stopped; the run prints that its readings cover part of it. A normal end (status 0) adds nothing, as a host without NVIDIA does.tests/test_job_queue.py(Windows, this PC's queue): start failure and a cancel with a failing stop throughrun()with a real command and ticket, the stop's marks over a stand-in process, and the real spawned worker around a 3 s command. A real timed game job, a forced failure on Linux, the paired tail-frame runs and the hybrid/AMD/Intel checks have not run; they are in Still to test.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.