feat(vk): log the guest pools the page table cannot reach #474
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!474
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:feat/page-table-census"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (game-free checks, no game run) · Needs: Windows or Linux, any GPU (a game run)
Why
Shaders follow pointers that no bound buffer holds through a page table of guest memory, but that table covers only two
windows of the guest's 512 GB: 8 GB from 64 GB (the heap) and 4 GB from 508 GB (where the game's GPU allocator put its storage
buffers). A pointer into a pool outside them reads 0, as the foliage culling's did before the table. Nobody has counted whether
the game makes pools out there, so whether the table has to reach the whole window, and how, is a guess. This logs them.
What changed
The renderer now counts the guest pools that come into the pool table outside the page table's two windows: it names the
first 30 as they come and logs a summary when the window closes. The page table itself is unchanged; its windows and
size only moved into a header that the table, the census and the test read.
PageTableCensus::refreshandsummaryinpage_table_census.h, then thememory_synchunk.gpu/vk/page_table_census.h(new).kPageWindows, and the table's size,kPageTableEntries, moved there fromguest_pages_setandmemory_guest_pageswith the same values.PageTableCensus: each refresh of the pool table names the pools that came outside the windows, the first 30 of them,with their bytes outside when only part of a pool is.
summary()counts every pool that came, their bytes outside, themost outside at once, how many are outside now and the lowest and highest address among them.
gpu/vk/vk_memory.cpp.guest_pages_setandmemory_guest_pagestake the windows and the size from the header.memory_syncrefreshes the census with the pool table it is about to take, under a mutex of its own (page_census_mu),before it takes
pools_mu. It runs only when the host's pool table changed, as before.page_table_census_log()(declared invk_internal.h) logs the summary.gpu/vk/vk_main.cpp:mkwvk_close_askedlogs the summary when the window closes.tests/test_page_table_census.py(new).kGuestPagesBase,kGuestPagesBytes,kGuestPagesBase2,kGuestPagesBytes2inspirv_emit_context.cpp), and checks thatguest_pages_setandmemory_guest_pagesuse theheader and that
memory_syncandmkwvk_close_askedcall the census.memory_sync's: pools at 128 GB and 504 GB and two across a window's endare named (the two by their bytes outside), pools inside are not, the same table refreshed again names nothing, past 30
the pools are counted but not named, a pool that leaves and comes back counts again, and with none outside the summary
says so.
windows reads 0 either way.
emitter's constants together; the test already holds the two to each other.
Measurements
Verdict: works, game-free: the new test fails on main and passes here, and the suite adds no failure. The census's
own cost, timed alone before the rebase (not rerun), is 2.4 µs to 29.2 µs a refresh, and a refresh happens only when
the host's pool table changed. No game run yet (Must be tested).
Commits: main
6c9edffe5d(2026-11-14); this PR feat/page-table-census @50128bf247.Hardware: Apple M4 Max (16 CPUs), macOS 26.7.1, Apple clang 21. Game-free: no GPU, no game
Scene: none. The census's proof is the test; its cost is a micro-benchmark of one refresh
Method:
tests/test_page_table_census.pyagainst main's tree (6c9edffe) and against this PR; the full game-free suite onboth; the census's
refreshtimed on tables of 2,400 and 8,192 pools (the median of 50 after a warm-up,-O2), beforethe rebase, at
3bc46da5(page_table_census.his the same after it)How to reproduce:
python -I -B tests/test_page_table_census.py --cxx clang++(add--source <a tree of main>for main'sside)
6c9edffe50128bf2tests/test_page_table_census.pytests/run_all.py --jobs 6)The census's cost, before the rebase and not rerun after it (
page_table_census.his byte for byte the same at50128bf2); main has no census:-O2)3bc46da5The lines the test expects, for a pool at 128 GB, one at 504 GB, two across a window's end, then 40 more and one back:
With no pool outside:
vk: page table census (close): no pool outside its windows, 64-72 GB and 508-512 GB (2 pools in the table now).Screenshots: none; nothing on screen.
Flags and settings
None.
Testing
Tested:
tests/test_page_table_census.pyfails on main and passes here.test_page_table_census, absenton main, passes here.
requests, some are checks main added or changed since
5e072fc8):test_appimage_package_retry,test_appimage_player_build,test_collect_wide,test_control,test_cpm_discovery,test_crash_dialog,test_crash_record,test_env_on,test_game_layout,test_lift_cleanup_error,test_linux_preflight,test_longshot_ctl,test_low_memory_notice,test_make_game_cache,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_oot_state_tap,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_pipeline_tour,test_prof_symbols,test_program_cache_cleanup,test_program_cache_one_variant,test_readback_late,test_report_run,test_split_giants,test_vk_pgo,test_vram_budget_row,test_walk_key.recorder_sync_test.exe-2026-11-14-180744.ips).
vk_memory.cpporvk_main.cpppass here (test_pool_span,test_sparse_mirror,test_fsr4_tune,test_startup_progress,test_parallel_rb_refresh,nx2/test_import_windows).gpu/vk/vk_memory.cppandgpu/vk/vk_main.cppwith the renderer's flags (nx2/build.py), Apple clang21: arm64 and x86_64, on main and here, all 8 build. Main's renderer as a whole does not build on macOS (
environinvk_shader.cpp,posix_fallocateinpipeline_cache_file.h), so these used a compile-only prelude that declares those twonames; these two files also build without it (arm64, both sides).
29.2 µs for 8,192 pools with 1,000 outside. A refresh happens only at a submit that finds the host's pool table changed.
7bacf846, at1d2b1692(62c4f276onbbe6fdf2,99131946onaa95d35f): Sweep4'schecks of this head gave the same results as before:
test_page_table_census,test_pool_spanandtest_sparse_mirrorpass on arm64 and on x86_64 under Rosetta, andvk_main.cppandvk_memory.cppcompile wholefor both;
python -I -B tests/run_all.pyon wave E1's mergea1ac22b6(this sweep's 15 every-platform heads), whichcarries this head: 570/33/17 (passed/failed/skipped) against main 7bacf846's 558/34/17 on this Mac; by name no check
that passes on main fails there (
reports/Sweep4-files/EP-report.md).Not tested:
logf, as the code around it does.Must be tested:
1d2b1692) and main7bacf846the same way, play a few minutespast the title on each and close the window, then read
longshot_log.txt, expecting this PR's log to havevk: page table census (close): ...near the end and avk: page table: pool ...line for each pool outside thewindows (the first 30), if the game makes any, where main's has neither; post both lines, since the count decides
how the table should grow (by hand: test request T-U3ND)
Risks
What could break: low. The output added is at most 30 lines in a run and one at close.
memory_syncdoes a little more work when the pool table changed (above), under a mutex of its own: inside it only thecensus runs and its lines are logged;
pools_muis taken after it is released.6c9edffe(2026-11-14) without a conflict; main had changedvk_main.cpp(#444) andvk_internal.h(#404) away from this PR's lines. No conflict with the author's other open PRs as rebased on6c9edffe(
git merge-treeof the heads, 2026-11-14).Turning it off: revert the commit.
Compatibility: nothing changes. The page table's own contents do not change: the same windows, the same values, the
same entries. The emitter's constants (
spirv_emit_context.cpp) are untouched, so the translator's stamp is the sameand no program is translated again. Saves, the options file and the vk ABI are unchanged.
Checklist
Everyone
type(scope): description) of the status line's type.main: on7bacf846(#471, 2026-11-15), head1d2b1692, without a conflict here: the fork'shead
62c4f276was already onbbe6fdf2, with the one conflict (gpu/vk/vk_memory.cpp's includes, against main's#223) resolved by keeping both: main's
linux_pressure.h, then this PR'spage_table_census.h; this PR's own commits,old → new:
62c4f276→1d2b1692; the commit names elsewhere in this description are from before the rebase.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). On macOS: 532 passed, 17skipped, and 32 failures that also fail on main
6c9edffeon this Mac, named under Testing. After the rebase onto main7bacf846(head1d2b1692): wave E1's mergea1ac22b6, 570/33/17 against main 7bacf846's 558/34/17, no new failure.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.3bc46da5dd50128bf247WIP: feat(vk): log the guest pools the page table cannot reachto feat(vk): log the guest pools the page table cannot reach50128bf24744c7ebfe8144c7ebfe81991319462c991319462c62c4f276bbRebased onto
main@bbe6fdf2. One conflict, in the#includelist ofgpu/vk/vk_memory.cpp(linux_pressure.hadded on main,page_table_census.hhere): kept both. No tests were run here; the test boxes stay yours.Code review only; nothing was built or run here.
Read:
page_table_census.h, thememory_sync/guest_pages_set/memory_guest_pageshunks, the close hook and the test.{0x1000000000, 8 GB, 0},{0x7f00000000, 4 GB, 8 GB>>12}and(8 GB + 4 GB) >> 12entries match the code they replace.page_table_outsideis correct for a pool inside, across an edge, or outside a window.nextafter the sort and beforepools_mu, under its own mutex. No lock-order problem:page_census_muis never held whilepools_muis taken, andpage_table_census_logtakes only its own mutex. The refresh runs only when the maps generation changed.snprintfbuffers (288 and 512 bytes) hold the longest lines the formats can make, and a cut would only truncate the log line.MkwVkHost/MkwVkRendererchange invk_api.h, soMKWVK_API_VERSIONneeds no bump. The emitter's constants are untouched, so no program is translated again.Why this is a comment and not an approval: the description's Must be tested names a Windows or Linux game run (T-U3ND) that nobody has posted yet. The change is log-only and low risk, but the run is what the census is for.
Overlap: #475 also adds a
*_log("close")line inmkwvk_close_asked, next to this one. Whichever lands second will have a small textual conflict there; keep both lines.62c4f276bb1d2b169297Re-review after the rebase onto main
7bacf846. Code only; nothing was built or run here.jj interdiffbetween the head I reviewed (62c4f276) and the new head (1d2b1692) is empty: only the base moved. My earlier read stands (windows and table size moved unchanged, locks ordered, noMKWVK_API_VERSIONbump needed), and there are no blocking findings.Still a comment, not an approval: the description's Must be tested (a Windows or Linux game run, T-U3ND) has no result in the comments yet. The change is log-only and low risk; once the census lines from a real run are posted, this can merge.
Overlap, unchanged: #475 also adds a close-time log line in
mkwvk_close_asked. Whichever lands second has a small textual conflict there; keep both lines.