fix(vk): scan the import table as far back as its largest live import

mooki commented

2026-11-14 13:42:33 +00:00

Member

Type: fix · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (game-free checks, arm64 and x86_64 object compiles; no game run) · Needs: Windows or Linux, any GPU

Why

Once an import of game memory is larger than 1 GiB, the renderer can lose track of it. It can then import the same bytes
a second time, read them through a stale second copy, or let the game's freed pages under the large import be
decommitted while the GPU still maps them (a device fault).

vk_memory.cpp keeps its imports in a map by start address. Every scan down that map stopped once an import ended more
than 1 GiB below the address ("imports are at most 1 GB"). That is not true. pool_lookup's merge escape keeps a wanted
range larger than 1 GiB as one import, over the smaller imports inside it ("would take in … pools: left overlapping"), and
import_pool allows imports up to the device's max_import_size. A scan that met a small import inside the large one
stopped there and never saw the large one. That is reachable whenever a binding (or, with RX_VK_IMPORT_CHUNK_MB=0, a game
pool) still spans more than 1 GiB once rounded out to whole chunks. How often the game does that is not measured.

How to reproduce

Seen on: found by reading the code; not seen in a game. The game-free test shows it on macOS 26.7.1, Apple M4 Max,
against main 6c9edffe's vk_memory.cpp. The import table is the same code on every OS.
Settings and switches: defaults: a binding that still spans more than 1 GiB once rounded out to whole chunks (or,
with RX_VK_IMPORT_CHUNK_MB=0, such a game pool).
Steps: game-free, from this branch's checkout: python -I -B tests/test_import_reach.py --cxx clang++ --headers <Vulkan headers> --source <a tree of main>. A binding at 1 GiB imports its 16 MiB chunk C, then a 4 GiB binding from 0
goes through the merge escape and leaves the 4 GiB import A over C. Each case then works 2 GiB past C, inside A.
Expected: every scan finds A: no second import of A's bytes, memory_guest_imported = 1, A marked gpu_only, a
background import inside A dropped, nothing sent to the background importer or prefetched.
Actual: the scans stop at C and miss A: the source check and 6 of 7 cases fail on main (lookup, query, note, drain,
background, prefetch; the lines under Measurements).

What changed

The import table now counts its live imports by size, and every scan down it stops only where no live import can still
reach the address, instead of 1 GiB past an import's end. While every live import is smaller than 1 GiB the scans stop no
later than before; they go further only while an import of 1 GiB or more is live.

Read first: import_reach.h, then imported_add / imported_take and the eight import_reach::beyond sites in
vk_memory.cpp.
vk: gpu/vk/import_reach.h (new, no Vulkan). Reach counts the live imports by span class (std::bit_width).
bound() is above every live span, within twice the largest. beyond(r, start, a) says whether a scan going down from
an import starting at start can stop: nothing further down reaches a.
vk: gpu/vk/vk_memory.cpp.
- live_reach, with imported_add and imported_take. They are now the table's only ways in (pool_lookup,
  bg_drain, mirrors_trim) and out (retire_pool, imports_give_back), and keep the count.
- The six scans stop at import_reach::beyond instead of 1 GiB past an import's end: pool_lookup's merge plan,
  bg_drain, prefetch_chunks (imported_over), import_in_background, memory_note_gpu_write and
  memory_guest_imported.
- The two covering checks, pool_lookup's covering and import_in_background's "covered", looked only at the last
  import starting at or below the range. They now walk down until beyond too, so an import further down that holds the
  whole range is found.
- While every live import is smaller than 1 GiB, the bound is at most 1 GiB, so the six scans stop no later than
  before, and sooner when the imports are small (chunk-sized imports of 4 or 16 MiB, up to 64 MiB with import_span,
  give a bound of at most 128 MiB). They go further only while an import of 1 GiB or more is live.
- Unchanged: the merge escape itself (no merge above 1 GiB) and every place that binds or mirrors an import.
tests: tests/test_import_reach.py with tests/import_reach_test.cpp (new).
- It cuts struct Pool, retire_pool, bg_drain, prefetch_chunks, span_game_pools, pool_lookup, pool_of,
  import_in_background, memory_note_gpu_write, memory_guest_imported and, where they exist, the two helpers out of
  vk_memory.cpp, and runs them against recorded imports: ranges, no payload, no Vulkan, no game.
- Setup: a binding at 1 GiB imports its 16 MiB chunk C, then a 4 GiB binding from 0 goes through the merge escape and
  leaves the 4 GiB import A over C. Each case then works 2 GiB past C, inside A.
- lookup, query, note, drain, background and prefetch each expect the scan to find A: no second import of
  A's bytes, memory_guest_imported = 1, A marked gpu_only, a background import inside A dropped, nothing sent to the
  background importer or prefetched. retire checks that the bound comes back down after A is retired. header checks
  the classes, the bound and the stop rule.
- A source check: nothing but imported_add / imported_take changes the table, and no scan keeps the 1 GiB stop.
- Every case runs in its own process. --source <tree> runs another tree's vk_memory.cpp.
tests: fixtures that cut code which now reads live_reach also cut the two helpers and include the header:
nx2/test_import_windows.py (pool_lookup, prefetch_chunks) and tests/test_parallel_rb_refresh.py with
tests/parallel_rb_refresh_test.cpp (memory_note_gpu_write). Both put their own pools in the table through
imported_add, and test_import_windows's stand-in retire_pool takes them out through imported_take.
tests/test_mirror_gpu_only.py finds the new pool's insertion as imported_add(p). Their checks are otherwise the same.
Left out: three lookups that take the last import starting at or below an address and look no further,
late_fix_build's page lookup, memory_rb_refresh's page lookup and trims_settle's start. None of them imports, so
none can make a second copy. With a large import over a smaller one, they can miss the large one: a page's fix is not
built or a readback page is not refreshed. That is a follow-up, not part of this fix.

Measurements

Verdict: fixed in the game-free test: it fails on main (the source check and 6 of 7 cases) and passes here (the
source check and 8 of 8 cases). The full game-free suite changes only by the new test and the load-sensitive
test_gpu_hang_watch. A bookkeeping fix: nothing timed (speed is under Not tested).

Commits: main 6c9edffe5d (2026-11-14); this PR fix/import-lookback @
693acd941b.
Hardware: Apple M4 Max (16 CPUs), macOS 26.7.1, Apple clang 21. Game-free: no GPU, no game
Scene: none (a renderer bookkeeping fix; its proof is the test, as CONTRIBUTING's fix row allows)
Method: tests/test_import_reach.py against main's gpu/vk/vk_memory.cpp (6c9edffe) and against this PR;
the full game-free suite on both
How to reproduce: python -I -B tests/test_import_reach.py --cxx clang++ --headers <Vulkan headers> (add
--source <a tree of main> for main's side)

	Before: main @ `6c9edffe`	After: this PR @ `693acd94`
`tests/test_import_reach.py`	FAIL: the source check and 6 of 7 cases (lookup, query, note, drain, background, prefetch)	PASS: the source check and 8 of 8 cases
full game-free suite on macOS (`tests/run_all.py --jobs 6`)	531 passed, 32 failed, 17 skipped	531 passed, 33 failed, 17 skipped

main @ 6c9edffe:
FAIL source: vk_memory.cpp has no imported_add / imported_take: nothing counts the live imports' spans
FAIL: a lookup behind a small import missed the 4 GiB import covering it
FAIL: memory_guest_imported missed a live import above 1 GiB
FAIL: memory_note_gpu_write missed a live import above 1 GiB (its mirror could be given back)
FAIL: a background import inside a live import above 1 GiB was published
FAIL: import_in_background sent an import for bytes a live import above 1 GiB holds
FAIL: prefetch_chunks sent a chunk a live import above 1 GiB holds
PASS retire: 5 import-reach checks
FAILED: source: vk_memory.cpp has no imported_add / imported_take: nothing counts the live imports' spans, lookup, query, note, drain, background, prefetch

this PR @ 693acd94:
PASS lookup: 7 import-reach checks
PASS query: 5 import-reach checks
PASS note: 6 import-reach checks
PASS drain: 7 import-reach checks
PASS background: 5 import-reach checks
PASS prefetch: 6 import-reach checks
PASS retire: 9 import-reach checks
PASS header: 7 import-reach checks
PASS: every scan finds the import above 1 GiB, and only imported_add / imported_take change the table

Screenshots: none; nothing on screen in the game-free tests.

Flags and settings

None.

Testing

Tested:

On an Apple M4 Max with macOS 26.7.1, game-free only.
tests/test_import_reach.py fails on main and passes here (above).
Before the rebase (on 5e072fc8), two deliberate breakages of this PR's vk_memory.cpp (bg_drain putting its import
in with imported.emplace; memory_guest_imported with the 1 GiB stop again) each made it fail: the first in the
source check, the second in the source check and query. vk_memory.cpp rebased without a change to its text.
The checks this PR touches or that cut the changed functions all pass: nx2/test_import_windows.py,
tests/test_parallel_rb_refresh.py, tests/test_mirror_gpu_only.py, tests/test_sparse_mirror.py,
tests/test_retired_give_back.py, tests/test_import_chunk_default.py, tests/test_import_buffer_binding.py,
tests/test_texel_buffer_ownership.py and tests/test_direct_buffer_writes.py.
- tests/test_mirror_trim.py fails on a Mac on both sides before it reaches anything this PR changes (its own
  test_mirror_trim.cpp: "call to 'wait' is ambiguous"). Its source check and its trims_settle run, called on their
  own, pass here.
The full suite on both sides, compared by test name.
- By test name, against main 6c9edffe's run on the same Mac: 2 statuses changed: test_import_reach, absent on main,
  passes here; test_gpu_hang_watch: pass on main, FAIL here.
- test_gpu_hang_watch is load-sensitive (a time bound, 3.5 s against its 3.4 s; load averages at the run's start
  22.57 31.52 27.73) and is accepted without a rerun. This PR does not touch vk_replay.cpp.
- These 32 checks fail here and on main on this Mac, by name (not triaged here: some need the macOS port's other pull
  requests, some are checks main added or changed since 5e072fc8): test_appimage_package_retry,
  test_appimage_player_build, test_collect_wide, test_control, test_cpm_discovery, test_crash_dialog,
  test_crash_record, test_env_on, test_game_layout, test_lift_cleanup_error, test_linux_preflight,
  test_longshot_ctl, test_low_memory_notice, test_make_game_cache, test_mem_gate, test_mem_gate_children,
  test_mirror_trim, test_oot_state_tap, test_pcache_header_read, test_pipeline_cache_file,
  test_pipeline_cache_save, test_pipeline_play_cache, test_pipeline_tour, test_prof_symbols,
  test_program_cache_cleanup, test_program_cache_one_variant, test_readback_late, test_report_run,
  test_split_giants, test_vk_pgo, test_vram_budget_row, test_walk_key.
- The same 17 checks skipped as on main.
- Crash reports during the run: recorder_sync_test.exe-2026-11-14-185200.ips (main's run:
  fault-arm64-2026-11-14-180704.ips, recorder_sync_test.exe-2026-11-14-180744.ips).
Object compiles of gpu/vk/vk_memory.cpp with the renderer's flags (nx2/build.py), Apple clang 21, arm64 and x86_64.
- Main's renderer does not build on macOS as it is (environ in vk_shader.cpp, posix_fallocate in
  pipeline_cache_file.h). These compiles use a compile-only prelude that declares those two names.
- With it, all 4 objects build: 2 on main and 2 here.
After the rebase onto main aa95d35f, at 497be2d4 (693acd94 before the rebases onto e090cc48 and aa95d35f):
Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
apply) gave the same results as before; python -I -B tests/run_all.py on wave B's merge db0e2e16, which carries
this head: 569/32/18 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
(reports/2026-11-15-rebase-sweep-3.md).

Not tested:

In the game, on any platform. While every live import is smaller than 1 GiB, the six scans find what they found
before, only stopping sooner. The two covering checks now also find an older import that holds the range under a newer,
smaller one the merge escape left over it, and use it rather than import again. With an import of 1 GiB or more live,
the scans and the covering checks find it, where main imported its bytes again.
Speed. The two covering checks were one step of the map. They now walk down while an import could still reach the
range: a few steps with chunk-sized imports, more while a large import is live. They run only when pool_of's caches
miss and, with RX_VK_ASYNC_IMPORT=1, for a read binding in a chunk not imported yet. Not measured.
A build on Windows (llvm-mingw) or Linux: object compiles on macOS only. The change uses std::bit_width (<bit>, as
vk_bank.cpp does) and the map's node handles, which vk_memory.cpp already used.

Must be tested:

Windows or Linux, any GPU, any driver: build this PR (head 497be2d4) and main aa95d35f the same way (llvm-mingw on
Windows), run python -I -B tests/run_all.py in each checkout, then play a few minutes with this PR, expecting the
same suite results as main plus test_import_reach passing here, and no change in longshot_log.txt or on screen; if
a line vk: pool … would take in … pools, … MB: left overlapping appears, no later vk: pool … imported … as lo..hi
inside that range, where main can import those bytes again (by hand: test request T-4R7H).

Risks

What could break:

Low. A scan now stops only where no live import can reach its address. That is further than before only while an import
of 1 GiB or more is live, and no later than before otherwise.
If an import ever entered the table without imported_add, the bound could be too low and a scan could stop too early.
tests/test_import_reach.py's source check fails on that.
Landing order:
- After both rebases, this PR still conflicts with #491 (macOS renderer correctness) in vk_memory.cpp,
  tests/test_mirror_gpu_only.py and tests/test_parallel_rb_refresh.py (git merge-tree): whichever lands second
  needs a rebase. The macOS port carries the same fix in its renderer branch; it rebases onto this one.
- Needed when both this PR and #491 land: #658 (mac/renderer-core)'s 0030e14a (imported_add(Pool&&) and the
  test_import_reach / test_darwin_renderer fixtures) makes #431 and #491 build together on macOS.
- Rebased on main 6c9edffe (2026-11-14) without a conflict: main had not changed any file this PR touches since its
  base.

Turning it off: revert the commit. There is no switch.

Compatibility: Nothing changes: vk_memory.cpp and import_reach.h are not in the translator stamp
(nx2/translator_files.py). Saves, the options file, the shader and pipeline caches and the vk ABI are unchanged.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main: on aa95d35f (#761, 2026-11-15), head 497be2d4, with one conflict:
gpu/vk/vk_memory.cpp's includes met #735's reworded mirror_budget.h line (both kept); this PR's own commits, old →
new: 07501d53 → e968b6d5, 693acd94 → 497be2d4; the commit names elsewhere in this description are from before
the rebase.
python -I -B tests/run_all.py passes (any failure that also fails on main is named). On macOS: 531 passed, 17
skipped, and 32 failures that also fail on main 6c9edffe on this Mac, named under Testing, plus the load-sensitive
test_gpu_hang_watch timing (Testing). After the rebase onto main aa95d35f (head 497be2d4): wave B's merge
db0e2e16, 569/32/18 against main aa95d35f's 543/33/17, no new failure.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.

**Type:** fix · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (game-free checks, arm64 and x86_64 object compiles; no game run) · **Needs:** Windows or Linux, any GPU ## Why Once an import of game memory is larger than 1 GiB, the renderer can lose track of it. It can then import the same bytes a second time, read them through a stale second copy, or let the game's freed pages under the large import be decommitted while the GPU still maps them (a device fault). `vk_memory.cpp` keeps its imports in a map by start address. Every scan down that map stopped once an import ended more than 1 GiB below the address ("imports are at most 1 GB"). That is not true. `pool_lookup`'s merge escape keeps a wanted range larger than 1 GiB as one import, over the smaller imports inside it ("would take in … pools: left overlapping"), and `import_pool` allows imports up to the device's `max_import_size`. A scan that met a small import inside the large one stopped there and never saw the large one. That is reachable whenever a binding (or, with `RX_VK_IMPORT_CHUNK_MB=0`, a game pool) still spans more than 1 GiB once rounded out to whole chunks. How often the game does that is not measured. ## How to reproduce **Seen on:** found by reading the code; not seen in a game. The game-free test shows it on macOS 26.7.1, Apple M4 Max, against main 6c9edffe's `vk_memory.cpp`. The import table is the same code on every OS. **Settings and switches:** defaults: a binding that still spans more than 1 GiB once rounded out to whole chunks (or, with `RX_VK_IMPORT_CHUNK_MB=0`, such a game pool). **Steps:** game-free, from this branch's checkout: `python -I -B tests/test_import_reach.py --cxx clang++ --headers <Vulkan headers> --source <a tree of main>`. A binding at 1 GiB imports its 16 MiB chunk C, then a 4 GiB binding from 0 goes through the merge escape and leaves the 4 GiB import A over C. Each case then works 2 GiB past C, inside A. **Expected:** every scan finds A: no second import of A's bytes, `memory_guest_imported` = 1, A marked `gpu_only`, a background import inside A dropped, nothing sent to the background importer or prefetched. **Actual:** the scans stop at C and miss A: the source check and 6 of 7 cases fail on main (lookup, query, note, drain, background, prefetch; the lines under Measurements). ## What changed The import table now counts its live imports by size, and every scan down it stops only where no live import can still reach the address, instead of 1 GiB past an import's end. While every live import is smaller than 1 GiB the scans stop no later than before; they go further only while an import of 1 GiB or more is live. - Read first: `import_reach.h`, then `imported_add` / `imported_take` and the eight `import_reach::beyond` sites in `vk_memory.cpp`. - vk: `gpu/vk/import_reach.h` (new, no Vulkan). `Reach` counts the live imports by span class (`std::bit_width`). `bound()` is above every live span, within twice the largest. `beyond(r, start, a)` says whether a scan going down from an import starting at `start` can stop: nothing further down reaches `a`. - vk: `gpu/vk/vk_memory.cpp`. - `live_reach`, with `imported_add` and `imported_take`. They are now the table's only ways in (`pool_lookup`, `bg_drain`, `mirrors_trim`) and out (`retire_pool`, `imports_give_back`), and keep the count. - The six scans stop at `import_reach::beyond` instead of 1 GiB past an import's end: `pool_lookup`'s merge plan, `bg_drain`, `prefetch_chunks` (`imported_over`), `import_in_background`, `memory_note_gpu_write` and `memory_guest_imported`. - The two covering checks, `pool_lookup`'s `covering` and `import_in_background`'s "covered", looked only at the last import starting at or below the range. They now walk down until `beyond` too, so an import further down that holds the whole range is found. - While every live import is smaller than 1 GiB, the bound is at most 1 GiB, so the six scans stop no later than before, and sooner when the imports are small (chunk-sized imports of 4 or 16 MiB, up to 64 MiB with `import_span`, give a bound of at most 128 MiB). They go further only while an import of 1 GiB or more is live. - Unchanged: the merge escape itself (no merge above 1 GiB) and every place that binds or mirrors an import. - tests: `tests/test_import_reach.py` with `tests/import_reach_test.cpp` (new). - It cuts `struct Pool`, `retire_pool`, `bg_drain`, `prefetch_chunks`, `span_game_pools`, `pool_lookup`, `pool_of`, `import_in_background`, `memory_note_gpu_write`, `memory_guest_imported` and, where they exist, the two helpers out of `vk_memory.cpp`, and runs them against recorded imports: ranges, no payload, no Vulkan, no game. - Setup: a binding at 1 GiB imports its 16 MiB chunk C, then a 4 GiB binding from 0 goes through the merge escape and leaves the 4 GiB import A over C. Each case then works 2 GiB past C, inside A. - `lookup`, `query`, `note`, `drain`, `background` and `prefetch` each expect the scan to find A: no second import of A's bytes, `memory_guest_imported` = 1, A marked `gpu_only`, a background import inside A dropped, nothing sent to the background importer or prefetched. `retire` checks that the bound comes back down after A is retired. `header` checks the classes, the bound and the stop rule. - A source check: nothing but `imported_add` / `imported_take` changes the table, and no scan keeps the 1 GiB stop. - Every case runs in its own process. `--source <tree>` runs another tree's `vk_memory.cpp`. - tests: fixtures that cut code which now reads `live_reach` also cut the two helpers and include the header: `nx2/test_import_windows.py` (`pool_lookup`, `prefetch_chunks`) and `tests/test_parallel_rb_refresh.py` with `tests/parallel_rb_refresh_test.cpp` (`memory_note_gpu_write`). Both put their own pools in the table through `imported_add`, and `test_import_windows`'s stand-in `retire_pool` takes them out through `imported_take`. `tests/test_mirror_gpu_only.py` finds the new pool's insertion as `imported_add(p)`. Their checks are otherwise the same. - Left out: three lookups that take the last import starting at or below an address and look no further, `late_fix_build`'s page lookup, `memory_rb_refresh`'s page lookup and `trims_settle`'s start. None of them imports, so none can make a second copy. With a large import over a smaller one, they can miss the large one: a page's fix is not built or a readback page is not refreshed. That is a follow-up, not part of this fix. ## Measurements **Verdict:** fixed in the game-free test: it fails on main (the source check and 6 of 7 cases) and passes here (the source check and 8 of 8 cases). The full game-free suite changes only by the new test and the load-sensitive `test_gpu_hang_watch`. A bookkeeping fix: nothing timed (speed is under Not tested). Commits: main 6c9edffe5da1f5b5383977aeab3f544c0de25d4c (2026-11-14); this PR fix/import-lookback @ 693acd941be74ca2c259d680385ded67962130a0. Hardware: Apple M4 Max (16 CPUs), macOS 26.7.1, Apple clang 21. Game-free: no GPU, no game Scene: none (a renderer bookkeeping fix; its proof is the test, as CONTRIBUTING's `fix` row allows) Method: `tests/test_import_reach.py` against main's `gpu/vk/vk_memory.cpp` (6c9edffe) and against this PR; the full game-free suite on both How to reproduce: `python -I -B tests/test_import_reach.py --cxx clang++ --headers <Vulkan headers>` (add `--source <a tree of main>` for main's side) | | Before: main @ 6c9edffe | After: this PR @ 693acd94 | |---|---|---| | `tests/test_import_reach.py` | FAIL: the source check and 6 of 7 cases (lookup, query, note, drain, background, prefetch) | PASS: the source check and 8 of 8 cases | | full game-free suite on macOS (`tests/run_all.py --jobs 6`) | 531 passed, 32 failed, 17 skipped | 531 passed, 33 failed, 17 skipped | ``` main @ 6c9edffe: FAIL source: vk_memory.cpp has no imported_add / imported_take: nothing counts the live imports' spans FAIL: a lookup behind a small import missed the 4 GiB import covering it FAIL: memory_guest_imported missed a live import above 1 GiB FAIL: memory_note_gpu_write missed a live import above 1 GiB (its mirror could be given back) FAIL: a background import inside a live import above 1 GiB was published FAIL: import_in_background sent an import for bytes a live import above 1 GiB holds FAIL: prefetch_chunks sent a chunk a live import above 1 GiB holds PASS retire: 5 import-reach checks FAILED: source: vk_memory.cpp has no imported_add / imported_take: nothing counts the live imports' spans, lookup, query, note, drain, background, prefetch this PR @ 693acd94: PASS lookup: 7 import-reach checks PASS query: 5 import-reach checks PASS note: 6 import-reach checks PASS drain: 7 import-reach checks PASS background: 5 import-reach checks PASS prefetch: 6 import-reach checks PASS retire: 9 import-reach checks PASS header: 7 import-reach checks PASS: every scan finds the import above 1 GiB, and only imported_add / imported_take change the table ``` Screenshots: none; nothing on screen in the game-free tests. ### Flags and settings None. ## Testing **Tested:** - On an Apple M4 Max with macOS 26.7.1, game-free only. - `tests/test_import_reach.py` fails on main and passes here (above). - Before the rebase (on 5e072fc8), two deliberate breakages of this PR's `vk_memory.cpp` (`bg_drain` putting its import in with `imported.emplace`; `memory_guest_imported` with the 1 GiB stop again) each made it fail: the first in the source check, the second in the source check and `query`. `vk_memory.cpp` rebased without a change to its text. - The checks this PR touches or that cut the changed functions all pass: `nx2/test_import_windows.py`, `tests/test_parallel_rb_refresh.py`, `tests/test_mirror_gpu_only.py`, `tests/test_sparse_mirror.py`, `tests/test_retired_give_back.py`, `tests/test_import_chunk_default.py`, `tests/test_import_buffer_binding.py`, `tests/test_texel_buffer_ownership.py` and `tests/test_direct_buffer_writes.py`. - `tests/test_mirror_trim.py` fails on a Mac on both sides before it reaches anything this PR changes (its own `test_mirror_trim.cpp`: "call to 'wait' is ambiguous"). Its source check and its `trims_settle` run, called on their own, pass here. - The full suite on both sides, compared by test name. - By test name, against main 6c9edffe's run on the same Mac: 2 statuses changed: `test_import_reach`, absent on main, passes here; `test_gpu_hang_watch`: pass on main, FAIL here. - `test_gpu_hang_watch` is load-sensitive (a time bound, 3.5 s against its 3.4 s; load averages at the run's start 22.57 31.52 27.73) and is accepted without a rerun. This PR does not touch `vk_replay.cpp`. - These 32 checks fail here and on main on this Mac, by name (not triaged here: some need the macOS port's other pull requests, some are checks main added or changed since 5e072fc8): `test_appimage_package_retry`, `test_appimage_player_build`, `test_collect_wide`, `test_control`, `test_cpm_discovery`, `test_crash_dialog`, `test_crash_record`, `test_env_on`, `test_game_layout`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_longshot_ctl`, `test_low_memory_notice`, `test_make_game_cache`, `test_mem_gate`, `test_mem_gate_children`, `test_mirror_trim`, `test_oot_state_tap`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_pipeline_tour`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`, `test_readback_late`, `test_report_run`, `test_split_giants`, `test_vk_pgo`, `test_vram_budget_row`, `test_walk_key`. - The same 17 checks skipped as on main. - Crash reports during the run: recorder_sync_test.exe-2026-11-14-185200.ips (main's run: fault-arm64-2026-11-14-180704.ips, recorder_sync_test.exe-2026-11-14-180744.ips). - Object compiles of `gpu/vk/vk_memory.cpp` with the renderer's flags (`nx2/build.py`), Apple clang 21, arm64 and x86_64. - Main's renderer does not build on macOS as it is (`environ` in vk_shader.cpp, `posix_fallocate` in pipeline_cache_file.h). These compiles use a compile-only prelude that declares those two names. - With it, all 4 objects build: 2 on main and 2 here. - **After the rebase onto main aa95d35f**, at `497be2d4` (`693acd94` before the rebases onto e090cc48 and aa95d35f): Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they apply) gave the same results as before; `python -I -B tests/run_all.py` on wave B's merge `db0e2e16`, which carries this head: 569/32/18 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure (`reports/2026-11-15-rebase-sweep-3.md`). **Not tested:** - In the game, on any platform. While every live import is smaller than 1 GiB, the six scans find what they found before, only stopping sooner. The two covering checks now also find an older import that holds the range under a newer, smaller one the merge escape left over it, and use it rather than import again. With an import of 1 GiB or more live, the scans and the covering checks find it, where main imported its bytes again. - Speed. The two covering checks were one step of the map. They now walk down while an import could still reach the range: a few steps with chunk-sized imports, more while a large import is live. They run only when `pool_of`'s caches miss and, with `RX_VK_ASYNC_IMPORT=1`, for a read binding in a chunk not imported yet. Not measured. - A build on Windows (llvm-mingw) or Linux: object compiles on macOS only. The change uses `std::bit_width` (`<bit>`, as `vk_bank.cpp` does) and the map's node handles, which `vk_memory.cpp` already used. **Must be tested:** - Windows or Linux, any GPU, any driver: build this PR (head 497be2d4) and main aa95d35f the same way (llvm-mingw on Windows), run `python -I -B tests/run_all.py` in each checkout, then play a few minutes with this PR, expecting the same suite results as main plus `test_import_reach` passing here, and no change in `longshot_log.txt` or on screen; if a line `vk: pool … would take in … pools, … MB: left overlapping` appears, no later `vk: pool … imported … as lo..hi` inside that range, where main can import those bytes again (by hand: test request T-4R7H). ## Risks **What could break:** - Low. A scan now stops only where no live import can reach its address. That is further than before only while an import of 1 GiB or more is live, and no later than before otherwise. - If an import ever entered the table without `imported_add`, the bound could be too low and a scan could stop too early. `tests/test_import_reach.py`'s source check fails on that. - Landing order: - After both rebases, this PR still conflicts with #491 (macOS renderer correctness) in `vk_memory.cpp`, `tests/test_mirror_gpu_only.py` and `tests/test_parallel_rb_refresh.py` (`git merge-tree`): whichever lands second needs a rebase. The macOS port carries the same fix in its renderer branch; it rebases onto this one. - Needed when both this PR and #491 land: #658 (mac/renderer-core)'s `0030e14a` (`imported_add(Pool&&)` and the `test_import_reach` / `test_darwin_renderer` fixtures) makes #431 and #491 build together on macOS. - Rebased on `main` 6c9edffe (2026-11-14) without a conflict: main had not changed any file this PR touches since its base. **Turning it off:** revert the commit. There is no switch. **Compatibility:** Nothing changes: `vk_memory.cpp` and `import_reach.h` are not in the translator stamp (`nx2/translator_files.py`). Saves, the options file, the shader and pipeline caches and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`: on aa95d35f (#761, 2026-11-15), head `497be2d4`, with one conflict: `gpu/vk/vk_memory.cpp`'s includes met #735's reworded `mirror_budget.h` line (both kept); this PR's own commits, old → new: `07501d53` → `e968b6d5`, `693acd94` → `497be2d4`; the commit names elsewhere in this description are from before the rebase. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). On macOS: 531 passed, 17 skipped, and 32 failures that also fail on main 6c9edffe on this Mac, named under Testing, plus the load-sensitive `test_gpu_hang_watch` timing (Testing). After the rebase onto main aa95d35f (head `497be2d4`): wave B's merge `db0e2e16`, 569/32/18 against main aa95d35f's 543/33/17, no new failure. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.

mooki added 2 commits

2026-11-14 13:42:33 +00:00

fix(vk): scan the import table as far back as its largest live import 11e9ee44b4

Every scan down vk_memory.cpp's import table by start (pool_lookup's
covering import and merge plan, bg_drain, prefetch_chunks,
import_in_background, memory_note_gpu_write and memory_guest_imported)
stopped once an import ended more than 1 GiB below its address, on the
belief that no import is larger than that. pool_lookup's own merge escape
leaves an import above 1 GiB over the smaller ones inside it ("left
overlapping"), and the device allows imports up to max_import_size: a scan
stopped at a small import inside a large one, missed the large one, and
the same bytes were imported again, two memories over the same bytes.

import_reach.h counts the live imports by span class; its bound is above
every live span, within twice the largest. imported_add and imported_take,
now the table's only ways in and out, keep the count. A scan stops only
where no import further down can reach its address, so it also stops
sooner than before when the imports are small. The covering-import checks
in pool_lookup and import_in_background look past the last import at or
below the range for the same reason.

Tests: test_import_reach.py runs the actual scans with a 4 GiB import over
a 16 MiB chunk, left by the merge escape (imports are recorded ranges: no
payload, no Vulkan, no game). Its lookup, query, note, drain, background
and prefetch cases fail on main 5e072fc8 and pass here, and a source check
keeps every change to the table in the two helpers. test_import_windows
runs pool_lookup with the helpers; test_mirror_gpu_only names the new
publication point.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): the readback refresh fixture runs memory_note_gpu_write with the import table's helpers

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 5s

Details

Third-party parts / As their notes say (pull_request) Successful in 6s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

83edcf62b6

test_parallel_rb_refresh cuts memory_note_gpu_write out of vk_memory.cpp,
whose scan now stops at the live imports' reach: the fixture includes
import_reach.h, cuts live_reach, imported_add and imported_take beside it,
and puts its pool in the table through imported_add. It failed to build
at 11e9ee44 ("use of undeclared identifier 'import_reach'").

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

mooki added the

labels

2026-11-14 14:01:45 +00:00