feat(vk): log the guest pools the page table cannot reach #474
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!474
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:feat/page-table-census"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (game-free checks, no game run) · Needs: Windows or Linux, any GPU (a game run)
Why
Shaders follow pointers that no bound buffer holds through a page table of guest memory, but that table covers only two
windows of the guest's 512 GB: 8 GB from 64 GB (the heap) and 4 GB from 508 GB (where the game's GPU allocator put its storage
buffers). A pointer into a pool outside them reads 0, as the foliage culling's did before the table. Nobody has counted whether
the game makes pools out there, so whether the table has to reach the whole window, and how, is a guess. This logs them.
What changed
The renderer now counts the guest pools that come into the pool table outside the page table's two windows: it names the
first 30 as they come and logs a summary when the window closes. The page table itself is unchanged; its windows and
size only moved into a header that the table, the census and the test read.
PageTableCensus::refreshandsummaryinpage_table_census.h, then thememory_synchunk.gpu/vk/page_table_census.h(new).kPageWindows, and the table's size,kPageTableEntries, moved there fromguest_pages_setandmemory_guest_pageswith the same values.PageTableCensus: each refresh of the pool table names the pools that came outside the windows, the first 30 of them,with their bytes outside when only part of a pool is.
summary()counts every pool that came, their bytes outside, themost outside at once, how many are outside now and the lowest and highest address among them.
gpu/vk/vk_memory.cpp.guest_pages_setandmemory_guest_pagestake the windows and the size from the header.memory_syncrefreshes the census with the pool table it is about to take, under a mutex of its own (page_census_mu),before it takes
pools_mu. It runs only when the host's pool table changed, as before.page_table_census_log()(declared invk_internal.h) logs the summary.gpu/vk/vk_main.cpp:mkwvk_close_askedlogs the summary when the window closes.tests/test_page_table_census.py(new).kGuestPagesBase,kGuestPagesBytes,kGuestPagesBase2,kGuestPagesBytes2inspirv_emit_context.cpp), and checks thatguest_pages_setandmemory_guest_pagesuse theheader and that
memory_syncandmkwvk_close_askedcall the census.memory_sync's: pools at 128 GB and 504 GB and two across a window's endare named (the two by their bytes outside), pools inside are not, the same table refreshed again names nothing, past 30
the pools are counted but not named, a pool that leaves and comes back counts again, and with none outside the summary
says so.
windows reads 0 either way.
emitter's constants together; the test already holds the two to each other.
Measurements
Verdict: works, game-free: the new test fails on main and passes here, and the suite adds no failure. The census's
own cost, timed alone before the rebase (not rerun), is 2.4 µs to 29.2 µs a refresh, and a refresh happens only when
the host's pool table changed. No game run yet (Must be tested).
Commits: main
6c9edffe5d(2026-11-14); this PR feat/page-table-census @50128bf247.Hardware: Apple M4 Max (16 CPUs), macOS 26.7.1, Apple clang 21. Game-free: no GPU, no game
Scene: none. The census's proof is the test; its cost is a micro-benchmark of one refresh
Method:
tests/test_page_table_census.pyagainst main's tree (6c9edffe) and against this PR; the full game-free suite onboth; the census's
refreshtimed on tables of 2,400 and 8,192 pools (the median of 50 after a warm-up,-O2), beforethe rebase, at
3bc46da5(page_table_census.his the same after it)How to reproduce:
python -I -B tests/test_page_table_census.py --cxx clang++(add--source <a tree of main>for main'sside)
6c9edffe50128bf2tests/test_page_table_census.pytests/run_all.py --jobs 6)The census's cost, before the rebase and not rerun after it (
page_table_census.his byte for byte the same at50128bf2); main has no census:-O2)3bc46da5The lines the test expects, for a pool at 128 GB, one at 504 GB, two across a window's end, then 40 more and one back:
With no pool outside:
vk: page table census (close): no pool outside its windows, 64-72 GB and 508-512 GB (2 pools in the table now).Screenshots: none; nothing on screen.
Flags and settings
None.
Testing
Tested:
tests/test_page_table_census.pyfails on main and passes here.test_page_table_census, absenton main, passes here.
requests, some are checks main added or changed since
5e072fc8):test_appimage_package_retry,test_appimage_player_build,test_collect_wide,test_control,test_cpm_discovery,test_crash_dialog,test_crash_record,test_env_on,test_game_layout,test_lift_cleanup_error,test_linux_preflight,test_longshot_ctl,test_low_memory_notice,test_make_game_cache,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_oot_state_tap,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_pipeline_tour,test_prof_symbols,test_program_cache_cleanup,test_program_cache_one_variant,test_readback_late,test_report_run,test_split_giants,test_vk_pgo,test_vram_budget_row,test_walk_key.recorder_sync_test.exe-2026-11-14-180744.ips).
vk_memory.cpporvk_main.cpppass here (test_pool_span,test_sparse_mirror,test_fsr4_tune,test_startup_progress,test_parallel_rb_refresh,nx2/test_import_windows).gpu/vk/vk_memory.cppandgpu/vk/vk_main.cppwith the renderer's flags (nx2/build.py), Apple clang21: arm64 and x86_64, on main and here, all 8 build. Main's renderer as a whole does not build on macOS (
environinvk_shader.cpp,posix_fallocateinpipeline_cache_file.h), so these used a compile-only prelude that declares those twonames; these two files also build without it (arm64, both sides).
29.2 µs for 8,192 pools with 1,000 outside. A refresh happens only at a submit that finds the host's pool table changed.
aa95d35f, at99131946(50128bf2before the rebases ontoe090cc48andaa95d35f):Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
apply) gave the same results as before;
python -I -B tests/run_all.pyon wave S1's merge11a5b4e7, which carriesthis head: 568/22/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
(
reports/2026-11-15-rebase-sweep-3.md).Not tested:
logf, as the code around it does.Must be tested:
99131946) and mainaa95d35fthe same way, play a few minutespast the title on each and close the window, then read
longshot_log.txt, expecting this PR's log to havevk: page table census (close): ...near the end and avk: page table: pool ...line for each pool outside thewindows (the first 30), if the game makes any, where main's has neither; post both lines, since the count decides
how the table should grow (by hand: test request T-U3ND)
Risks
What could break: low. The output added is at most 30 lines in a run and one at close.
memory_syncdoes a little more work when the pool table changed (above), under a mutex of its own: inside it only thecensus runs and its lines are logged;
pools_muis taken after it is released.6c9edffe(2026-11-14) without a conflict; main had changedvk_main.cpp(#444) andvk_internal.h(#404) away from this PR's lines. No conflict with the author's other open PRs as rebased on6c9edffe(
git merge-treeof the heads, 2026-11-14).Turning it off: revert the commit.
Compatibility: nothing changes. The page table's own contents do not change: the same windows, the same values, the
same entries. The emitter's constants (
spirv_emit_context.cpp) are untouched, so the translator's stamp is the sameand no program is translated again. Saves, the options file and the vk ABI are unchanged.
Checklist
Everyone
type(scope): description) of the status line's type.main: onaa95d35f(#761, 2026-11-15), head99131946, without a conflict; this PR's owncommits, old → new:
50128bf2→99131946; the commit names elsewhere in this description are from before therebase.
python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). On macOS: 532 passed, 17skipped, and 32 failures that also fail on main
6c9edffeon this Mac, named under Testing. After the rebase onto mainaa95d35f(head99131946): wave S1's merge11a5b4e7, 568/22/17 against main aa95d35f's 543/33/17, no new failure.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.3bc46da5dd50128bf247WIP: feat(vk): log the guest pools the page table cannot reachto feat(vk): log the guest pools the page table cannot reach50128bf24744c7ebfe8144c7ebfe81991319462cView command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.