test(vk): serialize recording admission in its parallel fixture #2074

Merged
HumanBean merged 1 commit from PirateKingIsh/Longshot:fix/recording-admission-parallel into main 2026-11-21 17:30:36 +00:00
Contributor

Summary

Repeated runs of tests/test_recording_admission.py --case parallel can fail with worker slot ownership or pool ownership/double free even when no renderer code changed. The fixture starts two workers and calls the extracted recording function directly. It omitted the caller lock that current main holds around recording admission, so both workers could resize the same resource deque. Its fake Vulkan handles, maps and counters also had unsynchronized writes.

This restores the caller contract in the fixture, gives the fake driver atomic handles and locked bookkeeping, and makes its barrier flag thread-local as in the renderer. The workers still copy through their own recording state after admission. Renderer code, the source extraction, all assertions, failure injection and the parallel case are unchanged.

Read tests/recording_admission_test.cpp first. For the caller contract, gpu/vk/vk_replay.cpp holds replay_mu in run_submit and do_present before begin_recording calls draw_begin_recording. Current main has no internal ParLock in that extracted function: the fixture's own lock went with the renderer's parallel replay in #1763 (a3565ac5), while this two-worker case stayed, which is when it started failing intermittently.

Flags and settings: none.

Testing

Tested: Windows 11 build 26200, Python 3.14.10, LLVM-MinGW clang++ 23.1.5, normal user permissions. No game build or game run: only a game-free test fixture changes, with no Vulkan driver involved. Main was 9dd41326; the tested fixture is byte-identical to commit e4f47fb47bb973d51010c0f1e8caaad1448c2272.

  • Parallel case: main failed 33 of 50 runs. The patch passed all 100 runs without added load and all 100 beside six continuously looping suite-check workers.
  • All 22 cases passed in each of 20 complete test executions.
  • Reverting only this change restored 33 failures in 50 parallel runs, then the patch was restored and rehashed. Every failure in these two controls said worker slot ownership; the earlier investigation also saw pool ownership/double free. The individual failing interleavings were not instrumented, so no diagnostic is assigned to just one of the races.
  • The suite's declared compiler/header arguments were used, rebuilding the fixture for every invocation. The focused tests/run_all.py --strict --exact tests/test_recording_admission.py passed with no skip.
  • Six background workers repeatedly ran the HUD layout, HUD layout row, wide UI, lower, register-map marker and LDSM address checks: 921 background jobs passed. Six workers were sampled active at 97 foreground starts and five during turnover at 3 starts. This is a six-worker compilation/test load, not a replay of the earlier full suite's exact schedule.
  • An independent Codex review checked the locks, lifetime, TLS and unchanged assertions, with no actionable findings.
  • The complete unmodified Windows runner was also run on this commit with six jobs, strict missing-tool handling and a 600-second limit per check: 778 passed, 88 failed, 48 skipped. The earlier complete run of the exact same main 9dd41326, using the same compiler, arguments and permissions, had 777 passed, 89 failed, 48 skipped. 0 new failed checks. This was tests/run_all.py directly, not an all-green mise run test claim. Platform skips remain skips. The earlier long-path check rejected its temporary root as too deep; it passed here with a shorter temporary root, with its source unchanged. The other long-path test still rejects its root (250 against a limit of 248, previously 257). The portable-builder check now passes its native assertions before the missing .NET v8.0 reference assemblies stop it; an exact-main recheck under the same temporary root confirmed that same failure. Shared failures are named below; test_vbytes.py times out on both sides and test_job_queue.py passes on both.
Checks that also fail on the exact main baseline
  • tests/test_appimage_player_build.py
  • tests/test_auto_handheld.py
  • tests/test_bug_report_defaults.py
  • tests/test_capture_read_once.py
  • tests/test_capture_readback_visibility.py
  • tests/test_compile_retry_jobs.py
  • tests/test_compile_stops_on_exception.py
  • tests/test_crash_dialog.py
  • tests/test_crash_record.py
  • tests/test_crt_options.py
  • tests/test_damage_debug_menu.py
  • tests/test_dev_ab.py
  • tests/test_dev_build.py
  • tests/test_dev_migrate.py
  • tests/test_dev_packs.py
  • tests/test_device_lost_exit.py
  • tests/test_display_startup_failure.py
  • tests/test_dlss_feature_size.py
  • tests/test_draw_binds.py
  • tests/test_draw_stream_visibility.py
  • tests/test_failed_compile_keeps_objects.py
  • tests/test_flush_nothing_held.py
  • tests/test_frame_overlap.py
  • tests/test_fsr3fi_hud_layer.py
  • tests/test_fsr3fi_hudless_colour.py
  • tests/test_fsr_image_retirement.py
  • tests/test_fsr_uniform_ownership.py
  • tests/test_game_menu_mouse_buttons.py
  • tests/test_gpu_wait_stats.py
  • tests/test_guest_imported.py
  • tests/test_high_fps_fixes.py
  • tests/test_host_guest_writes.py
  • tests/test_input_background.py
  • tests/test_libc_hooks.py
  • tests/test_lift_cache.py
  • tests/test_lift_cleanup_error.py
  • tests/test_linux_preflight.py
  • tests/test_linux_pressure.py
  • tests/test_long_paths.py
  • tests/test_low_vram_texture_mips.py
  • tests/test_make_game_ngx.py
  • tests/test_menu_i18n.py
  • tests/test_menu_keys.py
  • tests/test_menu_text_field.py
  • tests/test_mesh_conv.py
  • tests/test_mise_build.py
  • tests/test_missing_nup.py
  • tests/test_mods_list.py
  • tests/test_nfc_model_info.py
  • tests/test_ngx_sdk.py
  • tests/test_oot_shadow_fit.py
  • tests/test_pads_after_menu.py
  • tests/test_pass_barrier.py
  • tests/test_pending_guest_import.py
  • tests/test_perf_report.py
  • tests/test_pipeline_play_cache.py
  • tests/test_portable_builder_home.py
  • tests/test_portable_builder_startup.py
  • tests/test_prepare_cache_save.py
  • tests/test_present_queue_default.py
  • tests/test_present_source_crop.py
  • tests/test_profiling_mode.py
  • tests/test_psychov35_fsr3_fg.py
  • tests/test_readback_timeout_source.py
  • tests/test_recorder_own_idle.py
  • tests/test_recorder_sync.py
  • tests/test_reflex_swapchain_lifetime.py
  • tests/test_renderer_new_handler.py
  • tests/test_renhdrum.py
  • tests/test_replay_write_order.py
  • tests/test_run_all_mingw.py
  • tests/test_run_all_output.py
  • tests/test_sampler_admission.py
  • tests/test_sdiv_overflow.py
  • tests/test_slow_frames.py
  • tests/test_song_note_cue.py
  • tests/test_sparse_mirror.py
  • tests/test_staging_burst.py
  • tests/test_target_reuse_memory.py
  • tests/test_texel_buffer_ownership.py
  • tests/test_ub_update_batch.py
  • tests/test_ui_record.py
  • tests/test_upscaler_rows.py
  • tests/test_vbytes.py
  • tests/test_vcore_escape.py
  • tests/test_vcore_mask0.py
  • tests/test_vk_occlusion_queries.py
  • tests/test_work_lock.py

Still to test:

  • Linux, clang and GCC: run this game-free test repeatedly, expecting all cases to pass (by hand, by a contributor with Linux). No Linux or independent GCC result is claimed here.

Measurements

Verdict: the fixture's intermittent ownership failures stopped in the requested patched repetitions and returned when only the fix was reverted. No game performance change is claimed.

All counts are measured. Setup is the Windows test environment above; each invocation compiled a fresh fixture. The two 50-run unpatched controls were before and after the patched tests, not interleaved performance samples.

Failed invocations / invocations Before: main @ 9dd41326 After: this PR @ e4f47fb4
Parallel, no added load 33 / 50 0 / 100
Parallel, six background workers Not repeated in this check 0 / 100
Complete test, 22 cases per invocation Not repeated in this check 0 / 20
Parallel, only this fix reverted afterward 33 / 50 Not applicable

Risks

A test-only synchronization change can hide a problem if it supplies a lock absent from production. Here the admission lock models the actual caller's replay_mu, and the fake-driver lock protects only fake handles, maps and counters. The fixture still does not model concurrent stream growth, concurrent descriptor binding, GPU execution or GPU resource lifetime. The observed failures alone do not prove a renderer race, nor does this test establish general renderer safety.

No game behavior, switch, save, cache or player setting changes. Reverting the commit is enough to return to the original fixture.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. This test-only change is not run by the game.
  • mise run test passes (any failure that also fails on main is named). Direct strict Windows runner results and shared failures are disclosed above; no all-green claim.
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A check under tests/ that reads changed source is updated, not weakened.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested.
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary Repeated runs of `tests/test_recording_admission.py --case parallel` can fail with `worker slot ownership` or `pool ownership/double free` even when no renderer code changed. The fixture starts two workers and calls the extracted recording function directly. It omitted the caller lock that current main holds around recording admission, so both workers could resize the same resource deque. Its fake Vulkan handles, maps and counters also had unsynchronized writes. This restores the caller contract in the fixture, gives the fake driver atomic handles and locked bookkeeping, and makes its barrier flag thread-local as in the renderer. The workers still copy through their own recording state after admission. Renderer code, the source extraction, all assertions, failure injection and the parallel case are unchanged. Read `tests/recording_admission_test.cpp` first. For the caller contract, `gpu/vk/vk_replay.cpp` holds `replay_mu` in `run_submit` and `do_present` before `begin_recording` calls `draw_begin_recording`. Current main has no internal `ParLock` in that extracted function: the fixture's own lock went with the renderer's parallel replay in #1763 (a3565ac5), while this two-worker case stayed, which is when it started failing intermittently. Flags and settings: none. ## Testing **Tested:** Windows 11 build 26200, Python 3.14.10, LLVM-MinGW clang++ 23.1.5, normal user permissions. No game build or game run: only a game-free test fixture changes, with no Vulkan driver involved. Main was `9dd41326`; the tested fixture is byte-identical to commit `e4f47fb47bb973d51010c0f1e8caaad1448c2272`. - Parallel case: main failed 33 of 50 runs. The patch passed all 100 runs without added load and all 100 beside six continuously looping suite-check workers. - All 22 cases passed in each of 20 complete test executions. - Reverting only this change restored 33 failures in 50 parallel runs, then the patch was restored and rehashed. Every failure in these two controls said `worker slot ownership`; the earlier investigation also saw `pool ownership/double free`. The individual failing interleavings were not instrumented, so no diagnostic is assigned to just one of the races. - The suite's declared compiler/header arguments were used, rebuilding the fixture for every invocation. The focused `tests/run_all.py --strict --exact tests/test_recording_admission.py` passed with no skip. - Six background workers repeatedly ran the HUD layout, HUD layout row, wide UI, lower, register-map marker and LDSM address checks: 921 background jobs passed. Six workers were sampled active at 97 foreground starts and five during turnover at 3 starts. This is a six-worker compilation/test load, not a replay of the earlier full suite's exact schedule. - An independent Codex review checked the locks, lifetime, TLS and unchanged assertions, with no actionable findings. - The complete unmodified Windows runner was also run on this commit with six jobs, strict missing-tool handling and a 600-second limit per check: 778 passed, 88 failed, 48 skipped. The earlier complete run of the exact same main `9dd41326`, using the same compiler, arguments and permissions, had 777 passed, 89 failed, 48 skipped. 0 new failed checks. This was `tests/run_all.py` directly, not an all-green `mise run test` claim. Platform skips remain skips. The earlier long-path check rejected its temporary root as too deep; it passed here with a shorter temporary root, with its source unchanged. The other long-path test still rejects its root (250 against a limit of 248, previously 257). The portable-builder check now passes its native assertions before the missing .NET v8.0 reference assemblies stop it; an exact-main recheck under the same temporary root confirmed that same failure. Shared failures are named below; `test_vbytes.py` times out on both sides and `test_job_queue.py` passes on both. <details> <summary>Checks that also fail on the exact main baseline</summary> - `tests/test_appimage_player_build.py` - `tests/test_auto_handheld.py` - `tests/test_bug_report_defaults.py` - `tests/test_capture_read_once.py` - `tests/test_capture_readback_visibility.py` - `tests/test_compile_retry_jobs.py` - `tests/test_compile_stops_on_exception.py` - `tests/test_crash_dialog.py` - `tests/test_crash_record.py` - `tests/test_crt_options.py` - `tests/test_damage_debug_menu.py` - `tests/test_dev_ab.py` - `tests/test_dev_build.py` - `tests/test_dev_migrate.py` - `tests/test_dev_packs.py` - `tests/test_device_lost_exit.py` - `tests/test_display_startup_failure.py` - `tests/test_dlss_feature_size.py` - `tests/test_draw_binds.py` - `tests/test_draw_stream_visibility.py` - `tests/test_failed_compile_keeps_objects.py` - `tests/test_flush_nothing_held.py` - `tests/test_frame_overlap.py` - `tests/test_fsr3fi_hud_layer.py` - `tests/test_fsr3fi_hudless_colour.py` - `tests/test_fsr_image_retirement.py` - `tests/test_fsr_uniform_ownership.py` - `tests/test_game_menu_mouse_buttons.py` - `tests/test_gpu_wait_stats.py` - `tests/test_guest_imported.py` - `tests/test_high_fps_fixes.py` - `tests/test_host_guest_writes.py` - `tests/test_input_background.py` - `tests/test_libc_hooks.py` - `tests/test_lift_cache.py` - `tests/test_lift_cleanup_error.py` - `tests/test_linux_preflight.py` - `tests/test_linux_pressure.py` - `tests/test_long_paths.py` - `tests/test_low_vram_texture_mips.py` - `tests/test_make_game_ngx.py` - `tests/test_menu_i18n.py` - `tests/test_menu_keys.py` - `tests/test_menu_text_field.py` - `tests/test_mesh_conv.py` - `tests/test_mise_build.py` - `tests/test_missing_nup.py` - `tests/test_mods_list.py` - `tests/test_nfc_model_info.py` - `tests/test_ngx_sdk.py` - `tests/test_oot_shadow_fit.py` - `tests/test_pads_after_menu.py` - `tests/test_pass_barrier.py` - `tests/test_pending_guest_import.py` - `tests/test_perf_report.py` - `tests/test_pipeline_play_cache.py` - `tests/test_portable_builder_home.py` - `tests/test_portable_builder_startup.py` - `tests/test_prepare_cache_save.py` - `tests/test_present_queue_default.py` - `tests/test_present_source_crop.py` - `tests/test_profiling_mode.py` - `tests/test_psychov35_fsr3_fg.py` - `tests/test_readback_timeout_source.py` - `tests/test_recorder_own_idle.py` - `tests/test_recorder_sync.py` - `tests/test_reflex_swapchain_lifetime.py` - `tests/test_renderer_new_handler.py` - `tests/test_renhdrum.py` - `tests/test_replay_write_order.py` - `tests/test_run_all_mingw.py` - `tests/test_run_all_output.py` - `tests/test_sampler_admission.py` - `tests/test_sdiv_overflow.py` - `tests/test_slow_frames.py` - `tests/test_song_note_cue.py` - `tests/test_sparse_mirror.py` - `tests/test_staging_burst.py` - `tests/test_target_reuse_memory.py` - `tests/test_texel_buffer_ownership.py` - `tests/test_ub_update_batch.py` - `tests/test_ui_record.py` - `tests/test_upscaler_rows.py` - `tests/test_vbytes.py` - `tests/test_vcore_escape.py` - `tests/test_vcore_mask0.py` - `tests/test_vk_occlusion_queries.py` - `tests/test_work_lock.py` </details> **Still to test:** - Linux, clang and GCC: run this game-free test repeatedly, expecting all cases to pass (by hand, by a contributor with Linux). No Linux or independent GCC result is claimed here. ## Measurements **Verdict:** the fixture's intermittent ownership failures stopped in the requested patched repetitions and returned when only the fix was reverted. No game performance change is claimed. All counts are measured. Setup is the Windows test environment above; each invocation compiled a fresh fixture. The two 50-run unpatched controls were before and after the patched tests, not interleaved performance samples. | Failed invocations / invocations | Before: main @ 9dd41326 | After: this PR @ e4f47fb4 | |---|---|---| | Parallel, no added load | 33 / 50 | 0 / 100 | | Parallel, six background workers | Not repeated in this check | 0 / 100 | | Complete test, 22 cases per invocation | Not repeated in this check | 0 / 20 | | Parallel, only this fix reverted afterward | 33 / 50 | Not applicable | ## Risks A test-only synchronization change can hide a problem if it supplies a lock absent from production. Here the admission lock models the actual caller's `replay_mu`, and the fake-driver lock protects only fake handles, maps and counters. The fixture still does not model concurrent stream growth, concurrent descriptor binding, GPU execution or GPU resource lifetime. The observed failures alone do not prove a renderer race, nor does this test establish general renderer safety. No game behavior, switch, save, cache or player setting changes. Reverting the commit is enough to return to the original fixture. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. This test-only change is not run by the game. - [x] `mise run test` passes (any failure that also fails on `main` is named). Direct strict Windows runner results and shared failures are disclosed above; no all-green claim. - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested. - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
test(vk): serialize recording admission in its parallel fixture
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
e4f47fb47b
Model the replay caller's admission lock when invoking the extracted
recording function. Keep fake Vulkan handles unique and protect its shared
bookkeeping; make the fake barrier flag thread-local as in the renderer.

Both workers still copy through their own recording state after admission.
Leave the extracted renderer code, failure injection and every assertion
unchanged.

Assisted-by: Codex (implementation, tests and independent review)
Assisted-by: Claude (investigation plan)
Co-Authored-By: Codex <[email protected]>
Author
Contributor

No game run: this commit changes only a game-free test fixture. Windows 11 build 26200, Python 3.14.10 and LLVM-MinGW clang++ 23.1.5. Main 9dd41326 failed 33/50 parallel repetitions. Commit e4f47fb47b passed 100/100 without added load, 100/100 beside six suite-check workers, and 20/20 complete executions (22 cases each). Reverting only the fixture change restored 33/50 failures. The focused strict suite passed. Renderer code, extraction, assertions and the parallel case are unchanged. Flags and settings: none.

No game run: this commit changes only a game-free test fixture. Windows 11 build 26200, Python 3.14.10 and LLVM-MinGW clang++ 23.1.5. Main 9dd41326 failed 33/50 parallel repetitions. Commit e4f47fb47bb973d51010c0f1e8caaad1448c2272 passed 100/100 without added load, 100/100 beside six suite-check workers, and 20/20 complete executions (22 cases each). Reverting only the fixture change restored 33/50 failures. The focused strict suite passed. Renderer code, extraction, assertions and the parallel case are unchanged. Flags and settings: none.
HumanBean deleted branch fix/recording-admission-parallel 2026-11-21 17:30:39 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Some workflows are waiting to be reviewed.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2074
No description provided.