WIP: perf(vk): the written-page scan outside the captures can start on a helper during the capture (RX_VK_COLLECT_AHEAD=1 to try) #2028

Draft
siantm wants to merge 2 commits from siantm/Longshot:perf/vk-collect-ahead into main
Contributor

Summary

With the replay on the submitting thread, that thread captures the submit's pages, takes the capture, and then
scans for the pages written outside the captures' ranges (memory_collect_written), one after the other.

With RX_VK_COLLECT_AHEAD=1, the submit (memory_capture_submit_collecting) arms its capture, which starts that scan
(collect_scan, under collect_mu, its marks put back as before) on a thread of its own (mkw_vk collect) as soon as
the capture's own scan is done, while the submitting thread copies the capture's pages and takes it;
memory_collect_written then takes that scan's pages (collect_ahead_take) and stamps them as before. Every write the
submit is ordered after is found by its scan; only a write racing the submit from another thread can fall to the next
collection, as it already could. Not armed with the render thread. A log line every 600 collections says what the scan
reads, by what the back end does with each pool. The result expected: the scan's time moved off the submitting
thread, the total CPU time unchanged.

The switch: RX_VK_COLLECT_AHEAD (new, an attempt, bool, its row in gpu/vk/switches.def), off by default as the
branch shipped it; unset or =0 is the code before it.

Depends on #2027 (the submit's collection leaves the capture's ranges to the captures): its commit
shows in this diff until it merges.

From c1dae730 on exp/x86-asm-sync.

Read first: collect_scan, collect_ahead_take and memory_capture_submit_collecting in gpu/vk/vk_memory.cpp, then
tests/test_collect_ahead.py.

Testing

Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head 7240ccbf on 48e5a97e (its parent's head, on main 75317e1f). Build jobs of this PC's job queue (batch CapCut3-b3), the checks in a hidden console: tests/run_all.py --strict --exact on tests/test_collect_ahead.py and every check --changed picks for the files this changes: 215 checks, 184 passed, 31 failed, 0 skipped; its own check passes, the switch's read included (fresh processes); the failing ones fail on their base too, run from a base worktree in the same batch: tests/test_capture_read_once.py (fails on main @ 75317e1f too), tests/test_capture_readback_visibility.py (fails on main @ 75317e1f too), tests/test_crash_dialog.py (fails on main @ 75317e1f too), tests/test_crash_record.py (fails on main @ 75317e1f too), tests/test_dev_ab.py (fails on main @ 75317e1f too), tests/test_dev_packs.py (fails on main @ 75317e1f too), tests/test_device_lost_exit.py (fails on main @ 75317e1f too), tests/test_draw_stream_visibility.py (fails on main @ 75317e1f too), tests/test_flush_nothing_held.py (fails on main @ 75317e1f too), tests/test_frame_overlap.py (fails on main @ 75317e1f too), tests/test_fsr_image_retirement.py (fails on main @ 75317e1f too), tests/test_fsr_uniform_ownership.py (fails on main @ 75317e1f too), tests/test_gpu_wait_stats.py (fails on main @ 75317e1f too), tests/test_guest_imported.py (fails on main @ 75317e1f too), tests/test_linux_pressure.py (fails on main @ 75317e1f too), tests/test_mesh_conv.py (fails on main @ 75317e1f too), tests/test_pending_guest_import.py (fails on main @ 75317e1f too), tests/test_perf_report.py (fails on main @ 75317e1f too), tests/test_present_queue_default.py (fails on main @ 75317e1f too), tests/test_present_source_crop.py (fails on main @ 75317e1f too), tests/test_profiling_mode.py (fails on main @ 75317e1f too), tests/test_recorder_own_idle.py (fails on main @ 75317e1f too), tests/test_renderer_new_handler.py (fails on main @ 75317e1f too), tests/test_replay_write_order.py (fails on main @ 75317e1f too), tests/test_slow_frames.py (fails on main @ 75317e1f too), tests/test_sparse_mirror.py (fails on main @ 75317e1f too), tests/test_target_reuse_memory.py (fails on main @ 75317e1f too), tests/test_texel_buffer_ownership.py (fails on main @ 75317e1f too), tests/test_ub_update_batch.py (fails on main @ 75317e1f too), tests/test_upscaler_rows.py (fails on main @ 75317e1f too), tests/test_vk_occlusion_queries.py (fails on main @ 75317e1f too).

It has NOT run in the game on this PC yet.

Still to test:

  • Windows 11, i7-9700, RTX 4060 (this PC): Kokiri Forest, Threaded rendering off, play three minutes turning the camera and walking into Link's house and out: main @ 75317e1f, this branch with RX_VK_COLLECT_AHEAD unset (off) and this branch with RX_VK_COLLECT_AHEAD=1, each warmed up a minute in the scene first, interleaved (ABBA), at least two runs each, expecting the same picture, no FATAL, the mkw_vk collect thread busy in a profile, and a median frame time with RX_VK_COLLECT_AHEAD=1 lower than unset and than main, and unset the same as main (by hand)
  • Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: Kokiri Forest, Threaded rendering off, play three minutes: main @ 75317e1f, this branch with RX_VK_COLLECT_AHEAD unset (off) and this branch with RX_VK_COLLECT_AHEAD=1, warmed up, ABBA, expecting the same picture and no FATAL, and a median frame time with RX_VK_COLLECT_AHEAD=1 lower than unset and than main, and unset the same as main (@Beanstalk linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest three minutes, Threaded rendering off: main @ 75317e1f, this branch with RX_VK_COLLECT_AHEAD unset (off) and this branch with RX_VK_COLLECT_AHEAD=1, warmed up, ABBA, the log, frame times and a screenshot)
  • Linux: build the renderer and run python3 -I -B tests/test_collect_ahead.py, expecting PASS (by hand)

Measurements

Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is
in Still to test. RX_VK_COLLECT_AHEAD ships off (an attempt) until the in-game A/B in Still to test shows it helps. RX_VK_COLLECT_ONCE comes with #2027, which says its default.

Risks

What could break, with the switch on only: a write missed by the helper's scan would reach the GPU a submit late (a
stale texture, a wrong value read back), or a hang if the take waited for a scan never started. The test runs the scan's
thread and hand-over cut from the source through 3,000 submits with three writer threads: every write found once, a
submit's own writes always in its scan. Off by default; unset or =0 turns it off. Renderer code: the runtime is
rebuilt; nothing changes for players' saves, options or caches.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary With the replay on the submitting thread, that thread captures the submit's pages, takes the capture, and then scans for the pages written outside the captures' ranges (`memory_collect_written`), one after the other. With `RX_VK_COLLECT_AHEAD=1`, the submit (`memory_capture_submit_collecting`) arms its capture, which starts that scan (`collect_scan`, under `collect_mu`, its marks put back as before) on a thread of its own (`mkw_vk collect`) as soon as the capture's own scan is done, while the submitting thread copies the capture's pages and takes it; `memory_collect_written` then takes that scan's pages (`collect_ahead_take`) and stamps them as before. Every write the submit is ordered after is found by its scan; only a write racing the submit from another thread can fall to the next collection, as it already could. Not armed with the render thread. A log line every 600 collections says what the scan reads, by what the back end does with each pool. The result expected: the scan's time moved off the submitting thread, the total CPU time unchanged. The switch: `RX_VK_COLLECT_AHEAD` (new, an attempt, bool, its row in `gpu/vk/switches.def`), off by default as the branch shipped it; unset or `=0` is the code before it. Depends on #2027 (the submit's collection leaves the capture's ranges to the captures): its commit shows in this diff until it merges. From c1dae730 on `exp/x86-asm-sync`. Read first: `collect_scan`, `collect_ahead_take` and `memory_capture_submit_collecting` in `gpu/vk/vk_memory.cpp`, then `tests/test_collect_ahead.py`. ## Testing **Tested:** Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head `7240ccbf` on `48e5a97e` (its parent's head, on main `75317e1f`). Build jobs of this PC's job queue (batch `CapCut3-b3`), the checks in a hidden console: `tests/run_all.py --strict --exact` on `tests/test_collect_ahead.py` and every check `--changed` picks for the files this changes: 215 checks, 184 passed, 31 failed, 0 skipped; its own check passes, the switch's read included (fresh processes); the failing ones fail on their base too, run from a base worktree in the same batch: `tests/test_capture_read_once.py` (fails on main @ 75317e1f too), `tests/test_capture_readback_visibility.py` (fails on main @ 75317e1f too), `tests/test_crash_dialog.py` (fails on main @ 75317e1f too), `tests/test_crash_record.py` (fails on main @ 75317e1f too), `tests/test_dev_ab.py` (fails on main @ 75317e1f too), `tests/test_dev_packs.py` (fails on main @ 75317e1f too), `tests/test_device_lost_exit.py` (fails on main @ 75317e1f too), `tests/test_draw_stream_visibility.py` (fails on main @ 75317e1f too), `tests/test_flush_nothing_held.py` (fails on main @ 75317e1f too), `tests/test_frame_overlap.py` (fails on main @ 75317e1f too), `tests/test_fsr_image_retirement.py` (fails on main @ 75317e1f too), `tests/test_fsr_uniform_ownership.py` (fails on main @ 75317e1f too), `tests/test_gpu_wait_stats.py` (fails on main @ 75317e1f too), `tests/test_guest_imported.py` (fails on main @ 75317e1f too), `tests/test_linux_pressure.py` (fails on main @ 75317e1f too), `tests/test_mesh_conv.py` (fails on main @ 75317e1f too), `tests/test_pending_guest_import.py` (fails on main @ 75317e1f too), `tests/test_perf_report.py` (fails on main @ 75317e1f too), `tests/test_present_queue_default.py` (fails on main @ 75317e1f too), `tests/test_present_source_crop.py` (fails on main @ 75317e1f too), `tests/test_profiling_mode.py` (fails on main @ 75317e1f too), `tests/test_recorder_own_idle.py` (fails on main @ 75317e1f too), `tests/test_renderer_new_handler.py` (fails on main @ 75317e1f too), `tests/test_replay_write_order.py` (fails on main @ 75317e1f too), `tests/test_slow_frames.py` (fails on main @ 75317e1f too), `tests/test_sparse_mirror.py` (fails on main @ 75317e1f too), `tests/test_target_reuse_memory.py` (fails on main @ 75317e1f too), `tests/test_texel_buffer_ownership.py` (fails on main @ 75317e1f too), `tests/test_ub_update_batch.py` (fails on main @ 75317e1f too), `tests/test_upscaler_rows.py` (fails on main @ 75317e1f too), `tests/test_vk_occlusion_queries.py` (fails on main @ 75317e1f too). **It has NOT run in the game on this PC yet.** **Still to test:** - Windows 11, i7-9700, RTX 4060 (this PC): Kokiri Forest, Threaded rendering off, play three minutes turning the camera and walking into Link's house and out: main @ 75317e1f, this branch with `RX_VK_COLLECT_AHEAD` unset (off) and this branch with `RX_VK_COLLECT_AHEAD=1`, each warmed up a minute in the scene first, interleaved (ABBA), at least two runs each, expecting the same picture, no FATAL, the `mkw_vk collect` thread busy in a profile, and a median frame time with `RX_VK_COLLECT_AHEAD=1` lower than unset and than main, and unset the same as main (by hand) - Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: Kokiri Forest, Threaded rendering off, play three minutes: main @ 75317e1f, this branch with `RX_VK_COLLECT_AHEAD` unset (off) and this branch with `RX_VK_COLLECT_AHEAD=1`, warmed up, ABBA, expecting the same picture and no FATAL, and a median frame time with `RX_VK_COLLECT_AHEAD=1` lower than unset and than main, and unset the same as main (@Beanstalk linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest three minutes, Threaded rendering off: main @ 75317e1f, this branch with RX_VK_COLLECT_AHEAD unset (off) and this branch with RX_VK_COLLECT_AHEAD=1, warmed up, ABBA, the log, frame times and a screenshot) - Linux: build the renderer and run `python3 -I -B tests/test_collect_ahead.py`, expecting PASS (by hand) ## Measurements **Verdict:** not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is in Still to test. `RX_VK_COLLECT_AHEAD` ships off (an attempt) until the in-game A/B in Still to test shows it helps. `RX_VK_COLLECT_ONCE` comes with #2027, which says its default. ## Risks What could break, with the switch on only: a write missed by the helper's scan would reach the GPU a submit late (a stale texture, a wrong value read back), or a hang if the take waited for a scan never started. The test runs the scan's thread and hand-over cut from the source through 3,000 submits with three writer threads: every write found once, a submit's own writes always in its scan. Off by default; unset or `=0` turns it off. Renderer code: the runtime is rebuilt; nothing changes for players' saves, options or caches. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [ ] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
perf(vk): the submit's collection leaves the capture's ranges to the captures
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 14s
Website / Website on Cloudflare (pull_request_target) Has been skipped
48e5a97e97
- A submit's capture reads and clears the write marks of its ranges (the mirrored pools, the texture storage, the pools
  up to 64 MB), and the collection that follows read every GPU-visible range again, those included, only to put back
  the marks it found there for the next capture. memory_collect_written now asks the host for the marks outside those
  ranges (collect_written_in with a count below 0, host/rx_backend.c host_collect_outside; the smallest int asks
  whether the host knows the form), and their marks stay as they are, which is what putting them back came to. The
  pools with a write-back shadow are still scanned (collect_skip_ranges, rebuilt when a shadow is made or freed:
  shadow_ver), so the shadows are told of the same pages as before. The full scan stays before the first capture,
  with RX_VK_CAPTURE=0 and with a host without the outside form.
- Switch RX_VK_COLLECT_ONCE (kill, on); =0 scans every range as before. A log line every 600 collections says what
  was found and how much was left to the captures.
- Unlike the branch, it is not turned off with the render thread: the branch read RX_VK_ASYNC alone (so a run with the
  Threaded rendering row's render thread had it on there too), and the ranges left out are the ones the full scan's
  put-back reads, capture_ranges_merged under the same collect_mu, whichever thread collects.
- Branch's numbers (d1e33769, profile, i7-9700, Kokiri Forest): collect_in was 5.1% of a core on the two threads the
  frame ran on, the capture's ranges 2.9 GB in ~1,465 ranges; at most the capture's own scan again, ~0.11 ms a submit,
  is saved. Not measured on the branch beyond that, and not measured in play on main.
- Tests: new tests/test_collect_outside.py (host_collect_in and host_collect_outside cut from rx_backend.c against a
  plain walk: the whole pages of the given ranges left out and nothing else, the question answered, the renderer's
  numbers the host's); test_wb_diff_stale_shadow runs collect_skip_ranges; test_wb_diff_cpu_writes checks the
  shadowed pools stay scanned; test_vk_collect_window runs the capture's window with the outside form too;
  test_wb_shadow_remake gets shadow_ver.

Switch reads: the test also runs RX_VK_COLLECT_ONCE as the source reads it, in fresh processes (unset, 0, 1, 2, off, each twice, the variable flipped after the first read: read once). The host's collection outside ranges falls back to the whole collection when its range list cannot be allocated.

Assisted-by: Claude Code
perf(vk): RX_VK_COLLECT_AHEAD=1 starts the written-page scan outside the captures on a helper during the capture
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 17s
Website / Website on Cloudflare (pull_request_target) Has been skipped
7240ccbf5a
Needs "perf(vk): the submit's collection leaves the capture's ranges to the captures" first.

- An experiment, off: with the replay on the submitting thread, that thread's submit (memory_capture_submit_collecting)
  arms its capture, which starts the collection's scan outside the captures' ranges (collect_scan, under collect_mu,
  marks put back there) on a thread of its own ("mkw_vk collect") as soon as its own scan is done, while the
  submitting thread copies the capture's pages and takes it; memory_collect_written then takes that scan's pages
  (collect_ahead_take) and stamps them as before. The scan and its result are split out of memory_collect_written
  (CollectScan, kPutBack: the stamps stay the collecting thread's). The same pages for every write the submit is
  ordered after (the helper is woken through a mutex); only a write racing the submit from another thread can fall to
  the next collection, as it could already. Not armed with the render thread.
- A log line every 600 collections says what the scan outside the captures' ranges reads, by what the back end does
  with each pool (a write-back shadow, a mirror, read in place, never bound), and where the pages it found lie (every
  16th collection's).
- Switch RX_VK_COLLECT_AHEAD (attempt, off).
- Branch's numbers (c1dae730): the scan was ~4 GB of marks a collection outside the captures' ranges in Kokiri Forest,
  5.2% of a core on the serial threads; the helper moves ~0.05-0.1 ms a submit off them, the total CPU unchanged. Not
  measured in play on main.
- Tests: new tests/test_collect_ahead.py (the scan's thread and handover cut from the source: a scan once per ask, a
  take waiting for it, 3,000 submits with three writer threads: every write found once, a submit's own always in its
  scan; and where the source arms and starts it). test_vk_collect_window, test_vk_range_walk, test_readback_early,
  test_wb_diff_cpu_writes and test_collect_outside read collect_scan where they read memory_collect_written.

Switch reads: the test also runs RX_VK_COLLECT_AHEAD as the source reads it, in fresh processes (unset, 0, 1, 2, off, each twice, the variable flipped after the first read: read once). If its helper thread cannot be started, the collecting thread scans, as with the switch off.

Assisted-by: Claude Code
Member

Possibly related, for context:

  • PR #1083 fix(vk): a submit's capture no longer misses the pages the replay's collection is marking again — Merged capture/collection synchronization this optimization builds on
  • PR #1848 perf(vk): arrivals' captures: retired capture buffers destroyed on a thread of their own, and a late page's pool given its stamps at once — Merged Vulkan capture-path performance work in same memory subsystem
Possibly related, for context: - PR #1083 fix(vk): a submit's capture no longer misses the pages the replay's collection is marking again — Merged capture/collection synchronization this optimization builds on - PR #1848 perf(vk): arrivals' captures: retired capture buffers destroyed on a thread of their own, and a late page's pool given its stamps at once — Merged Vulkan capture-path performance work in same memory subsystem
BeanBot left a comment

Recommendation: Request changes. The renderer fails to compile on Linux, and the new collection paths can delay dirty-page stamps or leave write-back shadows stale. The performance gain and the costs of the helper are still unmeasured in game.

The prerequisite skips a second scan of pages owned by captures. This PR adds an opt-in worker that overlaps the remaining scan with capture copying when replay runs on the submitting thread. It could shorten CPU-bound submits, but it adds a worker handoff, a second range cache, and periodic reporting to a path that determines when CPU writes reach the GPU.

@fix will do (14)

  1. L1 gpu/vk/vk_memory.cpp:1074: This std::min fails template deduction on LP64 Linux. (more on the line)
  2. L2 gpu/vk/vk_memory.cpp:963: Range exclusion and put-back use two different ownership snapshots. (more on the line)
  3. L3 gpu/vk/vk_memory.cpp:2567: Moving the shadowed-pool scan before the capture copy also moves its invalidation cutoff. (more on the line)
  4. L4 gpu/vk/vk_memory.cpp:1126: The early-read checker still counts each outside scan as a full collection. (more on the line)
  5. L5 host/rx_backend.c:325: The new scratch allocation falls back to host_collect, which allocates the same-sized range list and passes it to hos_gpu_guest_ranges without checking it. (more on the line)
  6. L6 gpu/vk/vk_memory.cpp:875: The negative-count extension changes the host callback contract while vk_api.h still documents an inside-only operation and the ABI version stays unchanged. (more on the line)
  7. L7 (minor) host/rx_backend.c:327: The host accepts decoded counts up to INT_MAX-1, but n + nskip and the capacity plus 64 are calculated as int. (more on the line)
  8. L8 (minor) gpu/vk/vk_memory.cpp:955: The new callback call treats an array of std::pair objects as an array of uint64_t[2]. (more on the line)
  9. L9 (minor) gpu/vk/vk_memory.cpp:1028: The detached worker has no stop or join path and waits on a static condition variable. (more on the line)
  10. L10 (minor) gpu/vk/vk_memory.cpp:1082: The report classifies each span by its starting address, so a shadowed import starting partway through a 1 MiB span can be counted as unbound or excluded. (more on the line)
  11. L11 tests/test_collect_ahead.py:96: The completion wait can exit before the second scan runs. (more on the line)
  12. L12 tests/test_collect_ahead.py:68: The test reimplements the consumer and checks the production wiring by substring. (more on the line)
  13. L13 (minor) tests/test_collect_outside.py:223: --cc defaults through CXX and is then used for both the C fixture and C++ switch probe. (more on the line)
  14. L14 (its part an edit can do): Gate the new handoff mutex on AHEAD, keep byte accounting out of the ordinary collection path, and time worker scan, handoff wait, and first-use startup separately.

For the author (1) (a run, a measurement or a decision)

  • L14 (minor): Run and report the required in-game comparison after the code fixes. The proposed head with AHEAD unset still enables COLLECT_ONCE, and the plan omits threaded rendering, where COLLECT_ONCE also ships on. Compare main, the head with both switches off, COLLECT_ONCE alone in both render modes, and AHEAD enabled in submitting-thread mode. Use matched interleaved runs on Tier 1 systems and report median and p99 frame time, average FPS and 1% lows, scan and wait times, total CPU, and memory. Use those results to choose the defaults.

Checks: Still running: PR description / Description follows the template (pull_request_target), CI / Conventional Commit title (pull_request), CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request), CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request), CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request), CI / Workflow lint (actionlint) (pull_request), CI / Website build and checks (pull_request), Guide screenshots / Guide images and local UI source review (pull_request), Third-party parts / As their notes say (pull_request), Third-party parts / The same as upstream (pull_request).

More: does it help, other systems, switches, what was read

Does it help: Unproven. The commit messages cite exploratory i7-9700 Kokiri Forest figures of 5.1–5.2% of one core in scans and an estimated 0.05–0.1 ms per submit to overlap at six submits per frame. The PR reports no game runs or matched frame-time and CPU measurements on the review commits.

Other systems: The helper is gated to submitting-thread replay. The default-on prerequisite also changes render-thread replay, including capture ownership, mirror and texture freshness, write-back diffs, and early-read diagnostics. Host range collection and the renderer build affect both OSes and every GPU vendor. The helper adds a thread and cached ranges, and the new reporting takes the pool lock on replay. No save, audio, input, options, or shader-translator files changed. No in-game evidence covers NVIDIA, AMD, Intel, or Steam Deck.

Switches: RX_VK_COLLECT_AHEAD is an A/B attempt, off by default, and requires captures, submitting-thread replay, and a host with RX_VK_COLLECT_ONCE support. RX_VK_COLLECT_ONCE is a kill switch, on by default with a supporting host, and also affects threaded rendering. Both have sorted table rows. The head has no measurements that select either shipped default. Disabling AHEAD leaves the prerequisite and some new collection overhead active.

Reviewed: 7240ccbf5a against 75317e1f31: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, and docs/dev/standards.md. The checkout has no docs/review-checklist.md. Read both commits, every hunk and enclosing changed function or fixture across all 13 files, and the new tests in full. Traced replay callers, capture ownership and locks, host range enumeration and write marks, write-back invalidation, early-read checks, shutdown, the renderer ABI, and the shader stamp. Skimmed unrelated portions of the large renderer and host files and the reported Windows batch. No recorded runs, translations, generated files, comments, or reviews were supplied.; nothing run (a code review).

Checked and fine:

  • Both commits carry Assisted-by trailers. The title has a Conventional Commit subject after the WIP marker.
  • No game-derived files, generated artifacts, shader changes, SDKs, or third-party edits are present in the diff.
  • The worker's request, busy, ready, and result fields are published under one mutex. Normal submissions use sync_order_mu and replay_mu to serialize the consumer, and generation stamping stays on the replay caller.
  • The outside-range walk preserves ascending page order for the host's sorted, disjoint page-aligned ranges. The ordinary nonnegative-count callback path remains in place.
  • The switch probes use fresh processes and scrub the switch being tested from the inherited environment. The changed tests write fixtures to temporary directories and invoke local compilers and fixtures.

Testing: Read-only review. The author's Windows batch reports 184 of 215 checks passing and 31 failures also present on main. The new helper fixture reportedly passed on Windows. The PR opened without the repository-required author game run. Linux renderer compilation, game runs, Tier 1 results, full lint and test status, and the performance comparison remain outstanding. Supplied CI is still running and lists no head-only failure.

Overlaps and merge order: #2027 is the prerequisite and appears in this diff as 48e5a97e. Resolve the prerequisite findings there and land it first, then review and measure #2028 against the updated main. No other overlap was identifiable from the checkout and supplied review context.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The renderer fails to compile on Linux, and the new collection paths can delay dirty-page stamps or leave write-back shadows stale. The performance gain and the costs of the helper are still unmeasured in game. The prerequisite skips a second scan of pages owned by captures. This PR adds an opt-in worker that overlaps the remaining scan with capture copying when replay runs on the submitting thread. It could shorten CPU-bound submits, but it adds a worker handoff, a second range cache, and periodic reporting to a path that determines when CPU writes reach the GPU. **`@fix` will do (14)** 1. **L1** `gpu/vk/vk_memory.cpp:1074`: This std::min fails template deduction on LP64 Linux. (more on the line) 2. **L2** `gpu/vk/vk_memory.cpp:963`: Range exclusion and put-back use two different ownership snapshots. (more on the line) 3. **L3** `gpu/vk/vk_memory.cpp:2567`: Moving the shadowed-pool scan before the capture copy also moves its invalidation cutoff. (more on the line) 4. **L4** `gpu/vk/vk_memory.cpp:1126`: The early-read checker still counts each outside scan as a full collection. (more on the line) 5. **L5** `host/rx_backend.c:325`: The new scratch allocation falls back to host_collect, which allocates the same-sized range list and passes it to hos_gpu_guest_ranges without checking it. (more on the line) 6. **L6** `gpu/vk/vk_memory.cpp:875`: The negative-count extension changes the host callback contract while vk_api.h still documents an inside-only operation and the ABI version stays unchanged. (more on the line) 7. **L7** (minor) `host/rx_backend.c:327`: The host accepts decoded counts up to INT_MAX-1, but n + nskip and the capacity plus 64 are calculated as int. (more on the line) 8. **L8** (minor) `gpu/vk/vk_memory.cpp:955`: The new callback call treats an array of std::pair objects as an array of uint64_t[2]. (more on the line) 9. **L9** (minor) `gpu/vk/vk_memory.cpp:1028`: The detached worker has no stop or join path and waits on a static condition variable. (more on the line) 10. **L10** (minor) `gpu/vk/vk_memory.cpp:1082`: The report classifies each span by its starting address, so a shadowed import starting partway through a 1 MiB span can be counted as unbound or excluded. (more on the line) 11. **L11** `tests/test_collect_ahead.py:96`: The completion wait can exit before the second scan runs. (more on the line) 12. **L12** `tests/test_collect_ahead.py:68`: The test reimplements the consumer and checks the production wiring by substring. (more on the line) 13. **L13** (minor) `tests/test_collect_outside.py:223`: --cc defaults through CXX and is then used for both the C fixture and C++ switch probe. (more on the line) 14. **L14** (its part an edit can do): Gate the new handoff mutex on AHEAD, keep byte accounting out of the ordinary collection path, and time worker scan, handoff wait, and first-use startup separately. **For the author (1)** (a run, a measurement or a decision) - **L14** (minor): Run and report the required in-game comparison after the code fixes. The proposed head with AHEAD unset still enables COLLECT_ONCE, and the plan omits threaded rendering, where COLLECT_ONCE also ships on. Compare main, the head with both switches off, COLLECT_ONCE alone in both render modes, and AHEAD enabled in submitting-thread mode. Use matched interleaved runs on Tier 1 systems and report median and p99 frame time, average FPS and 1% lows, scan and wait times, total CPU, and memory. Use those results to choose the defaults. **Checks:** Still running: `PR description / Description follows the template (pull_request_target)`, `CI / Conventional Commit title (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request)`, `CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request)`, `CI / Workflow lint (actionlint) (pull_request)`, `CI / Website build and checks (pull_request)`, `Guide screenshots / Guide images and local UI source review (pull_request)`, `Third-party parts / As their notes say (pull_request)`, `Third-party parts / The same as upstream (pull_request)`. <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Unproven. The commit messages cite exploratory i7-9700 Kokiri Forest figures of 5.1–5.2% of one core in scans and an estimated 0.05–0.1 ms per submit to overlap at six submits per frame. The PR reports no game runs or matched frame-time and CPU measurements on the review commits. **Other systems:** The helper is gated to submitting-thread replay. The default-on prerequisite also changes render-thread replay, including capture ownership, mirror and texture freshness, write-back diffs, and early-read diagnostics. Host range collection and the renderer build affect both OSes and every GPU vendor. The helper adds a thread and cached ranges, and the new reporting takes the pool lock on replay. No save, audio, input, options, or shader-translator files changed. No in-game evidence covers NVIDIA, AMD, Intel, or Steam Deck. **Switches:** RX_VK_COLLECT_AHEAD is an A/B attempt, off by default, and requires captures, submitting-thread replay, and a host with RX_VK_COLLECT_ONCE support. RX_VK_COLLECT_ONCE is a kill switch, on by default with a supporting host, and also affects threaded rendering. Both have sorted table rows. The head has no measurements that select either shipped default. Disabling AHEAD leaves the prerequisite and some new collection overhead active. **Reviewed:** 7240ccbf5af6 against 75317e1f31ff: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, and docs/dev/standards.md. The checkout has no docs/review-checklist.md. Read both commits, every hunk and enclosing changed function or fixture across all 13 files, and the new tests in full. Traced replay callers, capture ownership and locks, host range enumeration and write marks, write-back invalidation, early-read checks, shutdown, the renderer ABI, and the shader stamp. Skimmed unrelated portions of the large renderer and host files and the reported Windows batch. No recorded runs, translations, generated files, comments, or reviews were supplied.; nothing run (a code review). **Checked and fine:** - Both commits carry Assisted-by trailers. The title has a Conventional Commit subject after the WIP marker. - No game-derived files, generated artifacts, shader changes, SDKs, or third-party edits are present in the diff. - The worker's request, busy, ready, and result fields are published under one mutex. Normal submissions use sync_order_mu and replay_mu to serialize the consumer, and generation stamping stays on the replay caller. - The outside-range walk preserves ascending page order for the host's sorted, disjoint page-aligned ranges. The ordinary nonnegative-count callback path remains in place. - The switch probes use fresh processes and scrub the switch being tested from the inherited environment. The changed tests write fixtures to temporary directories and invoke local compilers and fixtures. **Testing:** Read-only review. The author's Windows batch reports 184 of 215 checks passing and 31 failures also present on main. The new helper fixture reportedly passed on Windows. The PR opened without the repository-required author game run. Linux renderer compilation, game runs, Tier 1 results, full lint and test status, and the performance comparison remain outstanding. Supplied CI is still running and lists no head-only failure. **Overlaps and merge order:** #2027 is the prerequisite and appears in this diff as 48e5a97e. Resolve the prerequisite findings there and land it first, then review and measure #2028 against the updated main. No other overlap was identifiable from the checkout and supplied review context. </details> --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -854,0 +872,4 @@
// before the outside form.
// The host's side is collect_written_in with a count below 0 (host/rx_backend.c host_collect_in): -k - 1, every GPU-visible
// range but the k given (sorted, apart, whole pages); the smallest int asks whether it knows that form, answered -1 with no
// pages. (No callback of its own in gpu/vk/vk_api.h: that file is in the shaders' stamp, and a change of it translates every
Member

L6: The negative-count extension changes the host callback contract while vk_api.h still documents an inside-only operation and the ABI version stays unchanged. A same-version host that rejects a negative count or reports -1 for invalid input can be misidentified by this probe. Add an explicit versioned capability or callback and document the contract in vk_api.h. The stated cache cost does not apply on this base because nx2/translator_files.py excludes vk_api.h.

**L6**: The negative-count extension changes the host callback contract while vk_api.h still documents an inside-only operation and the ABI version stays unchanged. A same-version host that rejects a negative count or reports -1 for invalid input can be misidentified by this probe. Add an explicit versioned capability or callback and document the contract in vk_api.h. The stated cache cost does not apply on this base because nx2/translator_files.py excludes vk_api.h.
@ -875,0 +952,4 @@
if (s.once) {
s.left_ranges = skip.size();
for (const auto& r : skip) s.left_bytes += r.second - r.first;
s.n = host->collect_written_in(reinterpret_cast<const uint64_t(*)[2]>(skip.data()), -(int)skip.size() - 1, &s.pages);
Member

L8 (minor): The new callback call treats an array of std::pair objects as an array of uint64_t[2]. C++ provides no such storage or aliasing guarantee for std::pair. Build the outgoing list in the callback's actual range type, or define a common C-compatible range type at the versioned boundary.

**L8** (minor): The new callback call treats an array of std::pair objects as an array of uint64_t[2]. C++ provides no such storage or aliasing guarantee for std::pair. Build the outgoing list in the callback's actual range type, or define a common C-compatible range type at the versioned boundary.
@ -879,0 +960,4 @@
if (s.n > 0) {
// (the captured ranges once a call: a lock, a pool-list refresh and three searches for every dirty page cost the
// replay thousands of lock cycles a submit, the cost that retired gpu_pages' lock below)
RangeWalk in_capture{capture_ranges_merged()};
Member

L2: Range exclusion and put-back use two different ownership snapshots. In threaded replay, capture_make can refresh cap_ranges_ver before trying collect_mu. If an RX_VK_CAPTURE_EAGER pool expires in that window, the outside scan can skip its marked page using the old ownership while the next capture excludes it using the new ownership. Main would stamp that page in the current collection. Reconcile skipped ranges that lose ownership before finalizing the result, and add a test that changes ownership during the scan.

**L2**: Range exclusion and put-back use two different ownership snapshots. In threaded replay, capture_make can refresh cap_ranges_ver before trying collect_mu. If an RX_VK_CAPTURE_EAGER pool expires in that window, the outside scan can skip its marked page using the old ownership while the next capture excludes it using the new ownership. Main would stamp that page in the current collection. Reconcile skipped ranges that lose ownership before finalizing the result, and add a test that changes ownership during the scan.
@ -879,0 +1025,4 @@
static bool helper = false; // (its thread running: set once, under the once_flag, read after it)
std::call_once(started, [] {
try {
std::thread(collect_ahead_main).detach();
Member

L9 (minor): The detached worker has no stop or join path and waits on a static condition variable. The game's current hard process exit masks this, and the fixture uses std::_Exit for the same reason. A normal exit or library teardown after an armed submit can destroy the condition variable with a waiter still using it. Give the worker a stop state and join it before destroying its state, then exercise that path in the fixture.

**L9** (minor): The detached worker has no stop or join path and waits on a static condition variable. The game's current hard process exit masks this, and the fixture uses std::_Exit for the same reason. A normal exit or library teardown after an armed submit can destroy the condition variable with a waiter still using it. Give the worker a stop state and join it before destroying its state, then exercise that path in the fixture.
@ -879,0 +1071,4 @@
all += g.hi - g.lo;
while (o < owned.size() && owned[o].second <= g.lo) o++;
for (uint64_t a = g.lo; a < g.hi;) {
uint64_t b = std::min(g.hi, (a | ((1ull << 20) - 1)) + 1);
Member

L1: This std::min fails template deduction on LP64 Linux. g.hi is uint64_t, which is unsigned long there, while the rounded boundary is unsigned long long. Use std::min<uint64_t> or a uint64_t mask.

**L1**: This std::min fails template deduction on LP64 Linux. g.hi is uint64_t, which is unsigned long there, while the rounded boundary is unsigned long long. Use std::min<uint64_t> or a uint64_t mask.
@ -879,0 +1079,4 @@
else in_owned = true, b = std::min(b, owned[t].second);
break;
}
const int k = collect_kind_of(a);
Member

L10 (minor): The report classifies each span by its starting address, so a shadowed import starting partway through a 1 MiB span can be counted as unbound or excluded. It also omits coalesced host scan gaps, labels allocation fallback as an outside scan, and prints 37 samples even though a 600-call window can have 38 or include full scans. Report actual scanned intervals and count actual samples, or label and gate the estimates as diagnostics.

**L10** (minor): The report classifies each span by its starting address, so a shadowed import starting partway through a 1 MiB span can be counted as unbound or excluded. It also omits coalesced host scan gaps, labels allocation fallback as an outside scan, and prints 37 samples even though a 600-call window can have 38 or include full scans. Report actual scanned intervals and count actual samples, or label and gate the estimates as diagnostics.
@ -879,0 +1123,4 @@
}
if (sample && s.once) collect_found_note(pages, n);
}
if (s.ec_gen) early_check_cpu_pages(pages, n, true, s.ec_gen); // (outside collect_mu: it takes ec_mu)
Member

L4: The early-read checker still counts each outside scan as a full collection. If a CPU write lands in an excluded small pool after two queued submits took their captures, both replays can advance the check to closed without ever reporting the mark. The checker then attributes a CPU change to the GPU. Keep full scans when RX_VK_LOG_EARLY_CHECK is active, or track when the checked range was actually scanned before closing it.

**L4**: The early-read checker still counts each outside scan as a full collection. If a CPU write lands in an excluded small pool after two queued submits took their captures, both replays can advance the check to closed without ever reporting the mark. The checker then attributes a CPU change to the GPU. Keep full scans when RX_VK_LOG_EARLY_CHECK is active, or track when the checked range was actually scanned before closing it.
@ -2302,6 +2564,7 @@ std::unique_ptr<Capture> capture_make() {
const int64_t n = capture_scan(mirrored, &pages);
if (n > 0) early_check_cpu_pages(pages, n, false, 0); // (RX_VK_LOG_EARLY_CHECK: the CPU's writes)
const auto t_scan = std::chrono::steady_clock::now();
if (ahead_armed) collect_ahead_start(), ahead_armed = false; // (longshot, RX_VK_COLLECT_AHEAD: the collection's scan, from now)
Member

L3: Moving the shadowed-pool scan before the capture copy also moves its invalidation cutoff. If a CPU thread writes 1 after the helper passes a page whose shadow holds 0, and this recording writes 0 back, the diff can see mirror 0 and shadow 0 and leave the CPU's 1 in guest memory. The old scan at submit_memory_work would have reported that write. Keep the shadowed-pool scan at the original collection point, or treat its pages conservatively at write-back. Test a write between helper completion and collection.

**L3**: Moving the shadowed-pool scan before the capture copy also moves its invalidation cutoff. If a CPU thread writes 1 after the helper passes a page whose shadow holds 0, and this recording writes 0 back, the diff can see mirror 0 and shadow 0 and leave the CPU's 1 in guest memory. The old scan at submit_memory_work would have reported that write. Keep the shadowed-pool scan at the original collection point, or treat its pages conservatively at write-back. Test a write between helper completion and collection.
@ -307,0 +322,4 @@
static uint64_t (*ranges)[2], (*parts)[2];
static int parts_cap;
if (!ranges) ranges = malloc(sizeof *ranges * 4096);
if (!ranges) return host_collect(pages); /* (no memory for the list: the whole collection, as before) */
Member

L5: The new scratch allocation falls back to host_collect, which allocates the same-sized range list and passes it to hos_gpu_guest_ranges without checking it. If both buffers are still cold and allocation fails under memory pressure, the fallback writes through null. Reuse checked scratch storage and handle page-output allocation failure before clearing marks, or use the host's controlled out-of-memory path. Cover allocation refusal in the outside-scan fixture.

**L5**: The new scratch allocation falls back to host_collect, which allocates the same-sized range list and passes it to hos_gpu_guest_ranges without checking it. If both buffers are still cold and allocation fails under memory pressure, the fallback writes through null. Reuse checked scratch storage and handle page-output allocation failure before clearing marks, or use the host's controlled out-of-memory path. Cover allocation refusal in the outside-scan fixture.
@ -307,0 +324,4 @@
if (!ranges) ranges = malloc(sizeof *ranges * 4096);
if (!ranges) return host_collect(pages); /* (no memory for the list: the whole collection, as before) */
const int n = hos_gpu_guest_ranges(ranges, 4096);
if (parts_cap < n + nskip) { /* (a range's rest, and at most one part before each range left out) */
Member

L7 (minor): The host accepts decoded counts up to INT_MAX-1, but n + nskip and the capacity plus 64 are calculated as int. A count near that limit can overflow before the allocation check and leave the parts array undersized. The current renderer has a tighter cap, but the callback needs its own bound. Validate counts at the boundary, use checked size arithmetic, and intersect endpoints with the guest window before page rounding so rounding a high uint64_t endpoint cannot wrap.

**L7** (minor): The host accepts decoded counts up to INT_MAX-1, but n + nskip and the capacity plus 64 are calculated as int. A count near that limit can overflow before the allocation check and leave the parts array undersized. The current renderer has a tighter cap, but the callback needs its own bound. Validate counts at the boundary, use checked size arithmetic, and intersect endpoints with the guest window before page rounding so rounding a high uint64_t endpoint cannot wrap.
@ -0,0 +65,4 @@
return s;
}
@AHEAD@
static CollectScan collect_now() { // (memory_collect_written's choice)
Member

L12: The test reimplements the consumer and checks the production wiring by substring. For example, short-circuiting the production take with false leaves the expected text in place and the detached-worker fixture still passes, even though real submits would stop consuming helper results. Drive the real consumer with the worker and assert that its own result arrives once, including the fallback path.

**L12**: The test reimplements the consumer and checks the production wiring by substring. For example, short-circuiting the production take with false leaves the expected text in place and the detached-worker fixture still passes, even though real submits would stop consuming helper results. Drive the real consumer with the worker and assert that its own result arrives once, including the fallback path.
@ -0,0 +93,4 @@
// (done and not taken yet: an ask is not a second scan, and the take gets the first)
marks[11] = 1;
collect_ahead_start();
while (scanning.load() || scans.load() == before + 1) std::this_thread::yield();
Member

L11: The completion wait can exit before the second scan runs. It can read scanning as zero, then observe the worker's increment of scans while that worker is still before the mark walk. Page 12 then joins the scan that the test expects to contain only page 11. Wait on a single explicit completion event or on ahead.ready under its mutex, and bound the fixture's wait.

**L11**: The completion wait can exit before the second scan runs. It can read scanning as zero, then observe the worker's increment of scans while that worker is still before the mark walk. Page 12 then joins the scan that the test expects to contain only page 11. Wait on a single explicit completion event or on ahead.ready under its mutex, and bound the fixture's wait.
@ -0,0 +220,4 @@
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--cc", default=None)
a = p.parse_args()
a.cc = a.cc or default_cxx()
Member

L13 (minor): --cc defaults through CXX and is then used for both the C fixture and C++ switch probe. CC=clang is ignored, and an ordinary --cc gcc can fail to link the C++ local-static initialization guard. Use separate CC and CXX selections and pass both through the test runner.

**L13** (minor): --cc defaults through CXX and is then used for both the C fixture and C++ switch probe. CC=clang is ignored, and an ordinary --cc gcc can fail to link the C++ local-static initialization guard. Use separate CC and CXX selections and pass both through the test runner.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 17s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/vk-collect-ahead:siantm-perf/vk-collect-ahead'
git switch 'siantm-perf/vk-collect-ahead'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2028
No description provided.