fix(queue): cancel ends every process of the job on Linux, its game too #2155

Merged
HumanBean merged 2 commits from HumanBean/Longshot:fix/queue-cancel-ends-game into main 2026-11-21 23:23:27 +00:00
Owner

Summary

On Linux, dev.py queue cancel <job> said "stopped " but left the job's game running. A cancelled game run
(dev.py ab, or any longshot_bridge.start_session(..., wrapper=[... dev.py queue run ...])) kept its game and the
game's --crash-watch process alive, holding the GPU and several GB of memory, so the next timed run waited behind a
game nobody owned. queue status still listed it as game: longshot pid N under job M for a job that was gone.

Why: cancel sent SIGTERM to the runner only; the runner's finally terminated its direct child only. Under a game
run, that child is the bridge's supervisor (scripts/longshot_bridge.py, supervise()), which has no SIGTERM handler,
so it died at once without stopping the game it started. The game and its crash watcher were orphaned, out of the
runner's process tree, with nothing stopping their systemd scope. Windows was fine: its job object ends the tree.

What changes (scripts/devtools/jobqueue.py, procs.running() in scripts/devtools/procs.py), Linux only:

  • job_procs(t): a job's processes, as {pid: start time}: its command's tree, plus every process whose environment
    carries this job's LONGSHOT_JOB (the runner's pid) and LONGSHOT_JOB_NAME, however it detached.
  • end_job(): SIGTERM to all of them and to the job's systemd scope, rescanning for late children (the crash watcher),
    then SIGKILL to what is left after 10 s. A pid is signalled only while its start time matches, never a reused one.
  • The runner, stopped while its command runs, calls end_job instead of child.terminate().
  • cancel snapshots job_procs before it stops the runner and then runs end_job itself, so a runner of an older
    queue version is covered too. It refuses a ticket from another pid namespace, whose pids name other processes here.
  • A command that ends by itself still leaves what it started running, as before. Windows, the ticket format and
    longshot_bridge.py are unchanged.

Read first: job_procs, end_job and cancel in scripts/devtools/jobqueue.py.

Testing

Tested: CachyOS Linux. The bug as seen on this PC on main: a dev.py queue run --mode timing game run started
through longshot_bridge.start_session, then dev.py queue cancel on it: the runner ended, and the game and its crash
watcher kept running until killed by hand. New game-free checks in tests/test_job_queue.py (PosixTree, skipped on
Windows): a real queue in a temporary folder, the command standing in for the supervisor, starting a grandchild "game".
They have not been run on this PC (its owner runs no test suites here); CI runs them.

  • test_cancel_ends_the_game_its_supervisor_started, test_cancel_ends_a_game_in_a_session_of_its_own (fail on main:
    the grandchild is still running 15 s after the cancel)
  • test_cancel_leaves_another_jobs_process (another job's process survives)
  • test_a_command_that_ends_leaves_what_it_started (passes on main too: behaviour that must not change)

Still to test:

  • Linux, this PC: start a game run through the queue, dev.py queue cancel it, expecting queue status to show no
    game left under that job and the line "stopped NAME (pid N) and K processes of its job" (by hand)
  • Linux CI: the four PosixTree checks pass, and the three cancel ones fail on main (CI)

Measurements

Nothing to measure in the game: a developer tool's cancel path; no game file changes. Its own cost: a cancel can now
take up to 10 s while the game shuts down, scanning /proc every 0.2 s meanwhile.

Risks

  • A process is matched only by this runner's pid and name in its environment; another job's processes carry another
    runner's pid. A process that drops LONGSHOT_JOB is still caught through the tree snapshot or the systemd scope.
  • A runner ended by SIGHUP (its terminal closed) now ends its game too, where before only its command got the signal.
  • The game gets SIGTERM then SIGKILL, not a bridge quit: a cancelled run's game does not drain its saves.
  • A revert is enough. Players notice nothing: no game file changes.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. (Not yet: the by-hand cancel of a
    game run on this PC is still to do.)
  • mise run test passes (any failure that also fails on main is named). (CI runs it; not run on this PC.)
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary On Linux, `dev.py queue cancel <job>` said "stopped <job>" but left the job's game running. A cancelled game run (`dev.py ab`, or any `longshot_bridge.start_session(..., wrapper=[... dev.py queue run ...])`) kept its game and the game's `--crash-watch` process alive, holding the GPU and several GB of memory, so the next timed run waited behind a game nobody owned. `queue status` still listed it as `game: longshot pid N under job M` for a job that was gone. Why: `cancel` sent SIGTERM to the runner only; the runner's `finally` terminated its direct child only. Under a game run, that child is the bridge's supervisor (`scripts/longshot_bridge.py`, `supervise()`), which has no SIGTERM handler, so it died at once without stopping the game it started. The game and its crash watcher were orphaned, out of the runner's process tree, with nothing stopping their systemd scope. Windows was fine: its job object ends the tree. What changes (`scripts/devtools/jobqueue.py`, `procs.running()` in `scripts/devtools/procs.py`), Linux only: - `job_procs(t)`: a job's processes, as {pid: start time}: its command's tree, plus every process whose environment carries this job's `LONGSHOT_JOB` (the runner's pid) and `LONGSHOT_JOB_NAME`, however it detached. - `end_job()`: SIGTERM to all of them and to the job's systemd scope, rescanning for late children (the crash watcher), then SIGKILL to what is left after 10 s. A pid is signalled only while its start time matches, never a reused one. - The runner, stopped while its command runs, calls `end_job` instead of `child.terminate()`. - `cancel` snapshots `job_procs` before it stops the runner and then runs `end_job` itself, so a runner of an older queue version is covered too. It refuses a ticket from another pid namespace, whose pids name other processes here. - A command that ends by itself still leaves what it started running, as before. Windows, the ticket format and `longshot_bridge.py` are unchanged. Read first: `job_procs`, `end_job` and `cancel` in `scripts/devtools/jobqueue.py`. ## Testing **Tested:** CachyOS Linux. The bug as seen on this PC on main: a `dev.py queue run --mode timing` game run started through `longshot_bridge.start_session`, then `dev.py queue cancel` on it: the runner ended, and the game and its crash watcher kept running until killed by hand. New game-free checks in `tests/test_job_queue.py` (`PosixTree`, skipped on Windows): a real queue in a temporary folder, the command standing in for the supervisor, starting a grandchild "game". They have not been run on this PC (its owner runs no test suites here); CI runs them. - `test_cancel_ends_the_game_its_supervisor_started`, `test_cancel_ends_a_game_in_a_session_of_its_own` (fail on main: the grandchild is still running 15 s after the cancel) - `test_cancel_leaves_another_jobs_process` (another job's process survives) - `test_a_command_that_ends_leaves_what_it_started` (passes on main too: behaviour that must not change) **Still to test:** - Linux, this PC: start a game run through the queue, `dev.py queue cancel` it, expecting `queue status` to show no game left under that job and the line "stopped NAME (pid N) and K processes of its job" (by hand) - Linux CI: the four `PosixTree` checks pass, and the three cancel ones fail on main (CI) ## Measurements Nothing to measure in the game: a developer tool's cancel path; no game file changes. Its own cost: a cancel can now take up to 10 s while the game shuts down, scanning `/proc` every 0.2 s meanwhile. ## Risks - A process is matched only by this runner's pid and name in its environment; another job's processes carry another runner's pid. A process that drops `LONGSHOT_JOB` is still caught through the tree snapshot or the systemd scope. - A runner ended by SIGHUP (its terminal closed) now ends its game too, where before only its command got the signal. - The game gets SIGTERM then SIGKILL, not a bridge quit: a cancelled run's game does not drain its saves. - A revert is enough. Players notice nothing: no game file changes. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. (Not yet: the by-hand cancel of a game run on this PC is still to do.) - [x] `mise run test` passes (any failure that also fails on `main` is named). (CI runs it; not run on this PC.) - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
dev.py queue cancel stopped the runner, and the runner its command only. A
command that a SIGTERM ends before it stops what it started (longshot_bridge.py's
supervisor, under dev.py ab and every start_session with a queue wrapper) left
its game running as an orphan, the job gone from the queue. The check runs a
real queue in a temporary folder: a command that starts a game and waits for
it, cancelled; the game must end, also from a session of its own, and another
job's process must not.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(queue): cancel ends every process of the job on Linux, its game too
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 13s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Website build and checks (pull_request) Failing after 28s
PR description / Description follows the template (pull_request_target) Failing after 7s
a3c7adb20a
dev.py queue cancel sent SIGTERM to the runner, and the runner's clean-up sent
it to its command alone. Under start_session's queue wrapper (dev.py ab, every
bridge launch through the queue) that command is longshot_bridge.py's
supervisor: a python with no SIGTERM handler, which ends at once and leaves the
game it started (and the game's crash watcher) running as orphans, outside its
tree and the job gone from the queue, holding the GPU and memory the next job
waits for.

The runner, when it is stopped, and cancel, for a runner of any version, now end
the whole job: its command's tree, every process carrying the job's
LONGSHOT_JOB and LONGSHOT_JOB_NAME wherever it went (a session of its own, a
parent that ended), and its systemd scope. SIGTERM first, SIGKILL after 10 s;
each pid checked against its start time, so a pid used again is never hit.
Another job's processes carry another runner's pid and are left alone; a command
that ends by itself still leaves what it started running, as on Windows. Cancel
also refuses a ticket from another pid namespace, whose pids name other
processes here. Windows is unchanged (its job object already ends the tree).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #1100 fix(dev): a job cancelled on Windows ends with everything it started — Merged Windows counterpart for ending cancelled job process trees
  • PR #1495 fix(tests): run_all ends every process a check started, on Linux — Merged Linux process discovery and cleanup approach in test runner
Possibly related, for context: - PR #1100 fix(dev): a job cancelled on Windows ends with everything it started — Merged Windows counterpart for ending cancelled job process trees - PR #1495 fix(tests): run_all ends every process a check started, on Linux — Merged Linux process discovery and cleanup approach in test runner
HumanBean changed title from WIP: fix(queue): cancel ends every process of the job on Linux, its game too to fix(queue): cancel ends every process of the job on Linux, its game too 2026-11-21 23:13:18 +00:00
HumanBean deleted branch fix/queue-cancel-ends-game 2026-11-21 23:23:27 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2155
No description provided.