fix(tests): the GPU hang watch's check times its fixture from main(), not from outside #1246

Open
Amateurwizrd wants to merge 1 commit from Amateurwizrd/Longshot:fix/hang-watch-test-time into main
Contributor

Type: fix · Risk: low · Tested on: Windows 11, llvm-mingw · Needs: none

Why

tests/test_gpu_hang_watch.py fails on main on Windows most of the time, though the GPU hang watch it checks works.
The check times the fixture program from outside, so Windows' first look at the freshly built program counts as the
watch's time: the stuck run took 3.4-8.1 s against a 3.4 s ceiling, while the watch itself ends it at about 3.0 s. A
red check that is noise hides real failures in everyone's runs and gates. enum noted it as flaky in #1032; no issue or
PR fixes it. See #937: mooki saw it fail on a Mac under heavy load (3.4-3.7 s, load average 21-32).

How to reproduce

Seen on: Windows 11 26200, llvm-mingw clang++ (Ryzen 7 7800X3D)
Settings and switches: defaults
Steps: python -I -B tests/test_gpu_hang_watch.py, five times; or python -I -B tests/run_all.py
Expected: PASS
Actual: on main @ 172598b8, 4 of 5 runs alone fail: FAIL: a stuck timeline did not end the process after the limit (exit 3, 3.6 s) (3.6, 8.1, 4.0, 3.7 s). The exit code and the report are right each time; only the time misses.

What changed

The fixture times itself: device_lost() prints how many milliseconds after main() it ran, and the check reads that
instead of the wall time around the process. The bounds stay as they were (the stuck run between 1.5 and 3.4 s, the
lost device under 2.5 s).

  • tests: tests/test_gpu_hang_watch.py: the fixture's device_lost() prints ended <ms> ms after main; run()
    returns that time (None when the fixture never reached device_lost(), which fails the stuck and lost checks).

Measurements

Verdict: the check passes on this PC every time, alone and under run_all's load, and still fails when the watch is
broken.

Hardware: Windows 11 26200, Ryzen 7 7800X3D, 32 GB; llvm-mingw clang++
Scene: none, a game-free check (its fixture program, no game)
Method: the check run alone 5 times on each side; python -I -B tests/run_all.py twice on this branch; the fixture's
own time read from its output. A first launch and repeat launches of one fixture were timed from outside to find the
cause.
How to reproduce: python -I -B tests/test_gpu_hang_watch.py five times on main and on this branch (Windows,
llvm-mingw's clang++ on PATH); python -I -B tests/run_all.py on this branch.

Before: main @ 172598b8 After: this PR @ 7ad10a1d
alone, 5 runs 1 pass, 4 fail (3.6, 8.1, 4.0, 3.7 s) 5 pass (3.016-3.033 s after main)
in run_all (16 at once) failed in this PC's earlier runs pass, pass

Where the time went (main's check, one fixture launched again and again, timed from outside): first launch 3.70 and
3.72 s; the next three 3.03-3.05 s; the lost device 1.01-1.02 s.

Flags and settings

None.

Testing

Tested: the check against a broken watch, each change made to a copy of gpu/vk/vk_replay.cpp and put back:
still > limit for still >= limit and still += 0 for still++ both fail ("did not end the process after the
limit (exit 0, None s)"); the lost-device test taken out of the watch fails ("a lost device did not end the process at
once"). python -I -B tests/run_all.py on Windows: 609 passed, 35 failed, 52 skipped, twice; all 35 fail on main @
172598b8 too: test_auto_handheld, test_bug_report_defaults, test_crash_dialog, test_crt_options,
test_damage_debug_menu, test_dlss_feature_size, test_failed_compile_keeps_objects, test_fastfp_fma, test_fpx_sticky,
test_framegen_preset, test_fsr_uniform_ownership, test_guest_imported, test_hud_layout_row, test_import_reach,
test_job_queue, test_lift_lookups, test_linux_preflight, test_make_game_cache, test_menu_i18n, test_missing_nup,
test_pads_after_menu, test_parallel_init_order, test_pending_guest_import, test_portable_builder_home,
test_portable_builder_startup, test_replay_write_order, test_rumble_menu, test_run_all_changed, test_spawn_emit,
test_tested_configurations, test_upscaler_round_trip, test_upscaler_rows, test_vk_occlusion_queries, test_walk,
test_walk_key.
Not tested: Linux (g++): the fixture's new lines are standard C++17 <chrono>, already included. A machine far
busier than its cores (the load averages of 21-32 in #937): the watch's own one-second sleeps can still run late there,
and the check still counts that, as it should.
Must be tested: None.

Risks

What could break: only this check. A watch that ends a run late now fails the check the same way as before; a slow
start of the program no longer does.
Turning it off: revert the commit.
Compatibility: Nothing changes: a test-only change.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
**Type:** fix · **Risk:** low · **Tested on:** Windows 11, llvm-mingw · **Needs:** none ## Why `tests/test_gpu_hang_watch.py` fails on main on Windows most of the time, though the GPU hang watch it checks works. The check times the fixture program from outside, so Windows' first look at the freshly built program counts as the watch's time: the stuck run took 3.4-8.1 s against a 3.4 s ceiling, while the watch itself ends it at about 3.0 s. A red check that is noise hides real failures in everyone's runs and gates. enum noted it as flaky in #1032; no issue or PR fixes it. See #937: mooki saw it fail on a Mac under heavy load (3.4-3.7 s, load average 21-32). ## How to reproduce **Seen on:** Windows 11 26200, llvm-mingw clang++ (Ryzen 7 7800X3D) **Settings and switches:** defaults **Steps:** `python -I -B tests/test_gpu_hang_watch.py`, five times; or `python -I -B tests/run_all.py` **Expected:** PASS **Actual:** on main @ 172598b8, 4 of 5 runs alone fail: `FAIL: a stuck timeline did not end the process after the limit (exit 3, 3.6 s)` (3.6, 8.1, 4.0, 3.7 s). The exit code and the report are right each time; only the time misses. ## What changed The fixture times itself: `device_lost()` prints how many milliseconds after `main()` it ran, and the check reads that instead of the wall time around the process. The bounds stay as they were (the stuck run between 1.5 and 3.4 s, the lost device under 2.5 s). - tests: `tests/test_gpu_hang_watch.py`: the fixture's `device_lost()` prints `ended <ms> ms after main`; `run()` returns that time (None when the fixture never reached `device_lost()`, which fails the stuck and lost checks). ## Measurements **Verdict:** the check passes on this PC every time, alone and under run_all's load, and still fails when the watch is broken. Hardware: Windows 11 26200, Ryzen 7 7800X3D, 32 GB; llvm-mingw clang++ Scene: none, a game-free check (its fixture program, no game) Method: the check run alone 5 times on each side; `python -I -B tests/run_all.py` twice on this branch; the fixture's own time read from its output. A first launch and repeat launches of one fixture were timed from outside to find the cause. How to reproduce: `python -I -B tests/test_gpu_hang_watch.py` five times on main and on this branch (Windows, llvm-mingw's clang++ on PATH); `python -I -B tests/run_all.py` on this branch. | | Before: main @ 172598b8 | After: this PR @ 7ad10a1d | |---|---|---| | alone, 5 runs | 1 pass, 4 fail (3.6, 8.1, 4.0, 3.7 s) | 5 pass (3.016-3.033 s after main) | | in run_all (16 at once) | failed in this PC's earlier runs | pass, pass | Where the time went (main's check, one fixture launched again and again, timed from outside): first launch 3.70 and 3.72 s; the next three 3.03-3.05 s; the lost device 1.01-1.02 s. ### Flags and settings None. ## Testing **Tested:** the check against a broken watch, each change made to a copy of `gpu/vk/vk_replay.cpp` and put back: `still > limit` for `still >= limit` and `still += 0` for `still++` both fail ("did not end the process after the limit (exit 0, None s)"); the lost-device test taken out of the watch fails ("a lost device did not end the process at once"). `python -I -B tests/run_all.py` on Windows: 609 passed, 35 failed, 52 skipped, twice; all 35 fail on main @ 172598b8 too: test_auto_handheld, test_bug_report_defaults, test_crash_dialog, test_crt_options, test_damage_debug_menu, test_dlss_feature_size, test_failed_compile_keeps_objects, test_fastfp_fma, test_fpx_sticky, test_framegen_preset, test_fsr_uniform_ownership, test_guest_imported, test_hud_layout_row, test_import_reach, test_job_queue, test_lift_lookups, test_linux_preflight, test_make_game_cache, test_menu_i18n, test_missing_nup, test_pads_after_menu, test_parallel_init_order, test_pending_guest_import, test_portable_builder_home, test_portable_builder_startup, test_replay_write_order, test_rumble_menu, test_run_all_changed, test_spawn_emit, test_tested_configurations, test_upscaler_round_trip, test_upscaler_rows, test_vk_occlusion_queries, test_walk, test_walk_key. **Not tested:** Linux (g++): the fixture's new lines are standard C++17 `<chrono>`, already included. A machine far busier than its cores (the load averages of 21-32 in #937): the watch's own one-second sleeps can still run late there, and the check still counts that, as it should. **Must be tested:** None. ## Risks **What could break:** only this check. A watch that ends a run late now fails the check the same way as before; a slow start of the program no longer does. **Turning it off:** revert the commit. **Compatibility:** Nothing changes: a test-only change. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.
fix(tests): the GPU hang watch's check times its fixture from main(), not from outside
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
7ad10a1dea
- Timed from outside, Windows' first look at the freshly built fixture counted as the watch's time: 3.7 s alone
  and up to 8.1 s under run_all's load, against a 3.4 s ceiling, so the check failed on main most runs here while
  the watch ended a stuck run at 3.02-3.03 s and a lost device at 1.01 s.
- The fixture prints how long after main() it reached device_lost(); the stuck and lost runs check that, with the
  bounds as they were.

Assisted-by: Claude
Collaborator

Possibly related, for context:

  • PR #937 test(vk): the replay write-order fixture cuts main's payload_short with payload (#231) — Its description reports test_gpu_hang_watch timing out under load; this PR fixes the timing measurement, though #937 changes a different fixture.
Possibly related, for context: - PR #937 test(vk): the replay write-order fixture cuts main's payload_short with payload (#231) — Its description reports test_gpu_hang_watch timing out under load; this PR fixes the timing measurement, though #937 changes a different fixture.
Amateurwizrd changed title from WIP: fix(tests): the GPU hang watch's check times its fixture from main(), not from outside to fix(tests): the GPU hang watch's check times its fixture from main(), not from outside 2026-11-16 22:33:14 +00:00
TrashPanda approved these changes 2026-11-17 01:02:09 +00:00
TrashPanda left a comment

The GPU hang watch's check times the watch from inside the fixture, so Windows' first look at the new program no longer counts as a hang. I read head 7ad10a1d. Approved.

device_lost() prints the milliseconds since main() set the clock, and the check reads that line. The bounds are the same: a stuck timeline between 1.5 and 3.4 seconds, a lost device under 2.5. A run that never reaches device_lost() has no time, and the stuck and lost cases fail. The clock is set before the watch thread starts. <chrono> was already included. %lld matches the long long they print, and the five passing runs were on Windows with llvm-mingw.

A watch that does not end the process still fails: the author changed still >= limit and the increment, and each copy failed. A machine busy enough that the watch's own sleeps run late can still fail, and that is the watch being late. This file is the only change.

The GPU hang watch's check times the watch from inside the fixture, so Windows' first look at the new program no longer counts as a hang. I read head `7ad10a1d`. Approved. `device_lost()` prints the milliseconds since `main()` set the clock, and the check reads that line. The bounds are the same: a stuck timeline between 1.5 and 3.4 seconds, a lost device under 2.5. A run that never reaches `device_lost()` has no time, and the stuck and lost cases fail. The clock is set before the watch thread starts. `<chrono>` was already included. `%lld` matches the `long long` they print, and the five passing runs were on Windows with llvm-mingw. A watch that does not end the process still fails: the author changed `still >= limit` and the increment, and each copy failed. A machine busy enough that the watch's own sleeps run late can still fail, and that is the watch being late. This file is the only change.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/hang-watch-test-time:Amateurwizrd-fix/hang-watch-test-time'
git switch 'Amateurwizrd-fix/hang-watch-test-time'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1246
No description provided.