fix(tests): the GPU hang watch's check times its fixture from main(), not from outside #1246
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1246
Loading…
Reference in a new issue
No description provided.
Delete branch "Amateurwizrd/Longshot:fix/hang-watch-test-time"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11, llvm-mingw · Needs: none
Why
tests/test_gpu_hang_watch.pyfails on main on Windows most of the time, though the GPU hang watch it checks works.The check times the fixture program from outside, so Windows' first look at the freshly built program counts as the
watch's time: the stuck run took 3.4-8.1 s against a 3.4 s ceiling, while the watch itself ends it at about 3.0 s. A
red check that is noise hides real failures in everyone's runs and gates. enum noted it as flaky in #1032; no issue or
PR fixes it. See #937: mooki saw it fail on a Mac under heavy load (3.4-3.7 s, load average 21-32).
How to reproduce
Seen on: Windows 11 26200, llvm-mingw clang++ (Ryzen 7 7800X3D)
Settings and switches: defaults
Steps:
python -I -B tests/test_gpu_hang_watch.py, five times; orpython -I -B tests/run_all.pyExpected: PASS
Actual: on main @
172598b8, 4 of 5 runs alone fail:FAIL: a stuck timeline did not end the process after the limit (exit 3, 3.6 s)(3.6, 8.1, 4.0, 3.7 s). The exit code and the report are right each time; only the time misses.What changed
The fixture times itself:
device_lost()prints how many milliseconds aftermain()it ran, and the check reads thatinstead of the wall time around the process. The bounds stay as they were (the stuck run between 1.5 and 3.4 s, the
lost device under 2.5 s).
tests/test_gpu_hang_watch.py: the fixture'sdevice_lost()printsended <ms> ms after main;run()returns that time (None when the fixture never reached
device_lost(), which fails the stuck and lost checks).Measurements
Verdict: the check passes on this PC every time, alone and under run_all's load, and still fails when the watch is
broken.
Hardware: Windows 11 26200, Ryzen 7 7800X3D, 32 GB; llvm-mingw clang++
Scene: none, a game-free check (its fixture program, no game)
Method: the check run alone 5 times on each side;
python -I -B tests/run_all.pytwice on this branch; the fixture'sown time read from its output. A first launch and repeat launches of one fixture were timed from outside to find the
cause.
How to reproduce:
python -I -B tests/test_gpu_hang_watch.pyfive times on main and on this branch (Windows,llvm-mingw's clang++ on PATH);
python -I -B tests/run_all.pyon this branch.172598b87ad10a1dWhere the time went (main's check, one fixture launched again and again, timed from outside): first launch 3.70 and
3.72 s; the next three 3.03-3.05 s; the lost device 1.01-1.02 s.
Flags and settings
None.
Testing
Tested: the check against a broken watch, each change made to a copy of
gpu/vk/vk_replay.cppand put back:still > limitforstill >= limitandstill += 0forstill++both fail ("did not end the process after thelimit (exit 0, None s)"); the lost-device test taken out of the watch fails ("a lost device did not end the process at
once").
python -I -B tests/run_all.pyon Windows: 609 passed, 35 failed, 52 skipped, twice; all 35 fail on main @172598b8too: test_auto_handheld, test_bug_report_defaults, test_crash_dialog, test_crt_options,test_damage_debug_menu, test_dlss_feature_size, test_failed_compile_keeps_objects, test_fastfp_fma, test_fpx_sticky,
test_framegen_preset, test_fsr_uniform_ownership, test_guest_imported, test_hud_layout_row, test_import_reach,
test_job_queue, test_lift_lookups, test_linux_preflight, test_make_game_cache, test_menu_i18n, test_missing_nup,
test_pads_after_menu, test_parallel_init_order, test_pending_guest_import, test_portable_builder_home,
test_portable_builder_startup, test_replay_write_order, test_rumble_menu, test_run_all_changed, test_spawn_emit,
test_tested_configurations, test_upscaler_round_trip, test_upscaler_rows, test_vk_occlusion_queries, test_walk,
test_walk_key.
Not tested: Linux (g++): the fixture's new lines are standard C++17
<chrono>, already included. A machine farbusier than its cores (the load averages of 21-32 in #937): the watch's own one-second sleeps can still run late there,
and the check still counts that, as it should.
Must be tested: None.
Risks
What could break: only this check. A watch that ends a run late now fails the check the same way as before; a slow
start of the program no longer does.
Turning it off: revert the commit.
Compatibility: Nothing changes: a test-only change.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.Possibly related, for context:
WIP: fix(tests): the GPU hang watch's check times its fixture from main(), not from outsideto fix(tests): the GPU hang watch's check times its fixture from main(), not from outsideThe GPU hang watch's check times the watch from inside the fixture, so Windows' first look at the new program no longer counts as a hang. I read head
7ad10a1d. Approved.device_lost()prints the milliseconds sincemain()set the clock, and the check reads that line. The bounds are the same: a stuck timeline between 1.5 and 3.4 seconds, a lost device under 2.5. A run that never reachesdevice_lost()has no time, and the stuck and lost cases fail. The clock is set before the watch thread starts.<chrono>was already included.%lldmatches thelong longthey print, and the five passing runs were on Windows with llvm-mingw.A watch that does not end the process still fails: the author changed
still >= limitand the increment, and each copy failed. A machine busy enough that the watch's own sleeps run late can still fail, and that is the watch being late. This file is the only change.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.