WIP: feat(vk): log the indexed descriptor arrays a program declares #475
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!475
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:feat/descriptor-array-census"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
The translator is given no device limits (
host_info()invk_shader.cppfills none), so a texture array a shader indexesis sized with the defaults, up to 1024 elements, on every GPU. Apple's M1 and M2 reach 996 samplers a stage through an
argument buffer, and other drivers have per-stage limits of their own. Nobody has counted whether the game declares arrays
that long, or image, image-buffer and texture-buffer arrays at all, so whether to give the translator real limits is a guess.
This logs them.
What changed
gpu/vk/descriptor_array_census.h(new):ArrayCensus.note()takes a program's stage and counts its arrays (descriptors with a count above 1) by stage, kind (texture,texture buffer, storage image, image buffer) and size: 2-16, 17-64, 65-256, 257-995, 996-1023, 1024 and more. A program
stage is counted once, whatever its variants.
HostTranslateInfo::DEFAULT_DESCRIPTOR_LIMIT,DESCRIPTOR_MAX_COUNT);maxPerStageDescriptorSamplers,...SampledImagesor...StorageImages.summary(): the counts, then a line for each stage with arrays, by kind and size, with its longest array.gpu/vk/vk_programs.cpp, beside the programs' census:descriptor_arrays_note()(with the device's per-stage limits)and
descriptor_arrays_log(), declared invk_programs.h.gpu/vk/vk_progcache.cpp: every program the program cache sees is noted: translated (program_cache_save, beforeit returns for a cache that is off) or read (
program_cache_read,program_cache_read_light). So a start that reads itsprograms from the cache is counted as well as one that translates them.
prepare_run(gpu/vk/vk_draw.cpp) logs the summary after the preparation, andmkwvk_close_asked(vk_main.cpp)when the window closes.
nx2/build.py:eden/,sirit/,compat/,vk_shader.cpp,vk_bank.cpp,vk_state.h,vk_api.h), so no player's programs are translated again, and the program cache's format is the same.tests/test_descriptor_array_census.py(new).of a stage already counted, a mesh fallback's stages, a 1000-sampler mesh stage, a stage it does not know, and 40 more
past the 30 named. It does so once with no device limits and once with MoltenVK's (16 samplers, 256 sampled and 256
storage images a stage).
static_assertinvk_programs.cppholds the stage bits), checks each call site, and checks that nothing of the census is in the translator's stamp.
tests/program_cache_cleanup_test.cpp,tests/program_cache_one_variant_test.cpp: a stand-in fordescriptor_arrays_notebeside theirmodule_id_learnedone, for the cache functions they cut out and compile.Flags and settings
None.
Measurements
main:
5e072fc8c2(2026-11-14)this PR: feat/descriptor-array-census @
45d2bbaa8eHardware: Apple M4 Max (16 CPUs), macOS 26.7.1, Apple clang 21. Game-free: no GPU, no game
Scene: none. The census's proof is the test; its cost is a micro-benchmark of the census alone
Method:
tests/test_descriptor_array_census.pyagainst a copy of main's tree (git archive 5e072fc8) and against thisPR; the full game-free suite on both; the census's
notetimed for 20,000 two-stage programs, new and then seen again (-O2)How to reproduce:
python -I -B tests/test_descriptor_array_census.py --cxx clang++(add--source <a tree of main>formain's side)
5e072fc845d2bbaatests/test_descriptor_array_census.pytests/run_all.py --jobs 6)Some of the lines the test expects, with MoltenVK's limits:
With no array:
vk: indexed descriptor arrays (the preparation): none in 2 program stages.Testing
Tested:
tests/test_descriptor_array_census.pyfails on main and passes here.tests/test_descriptor_array_census.py, absent on main, passes here.test_program_cache_one_variantthree lines furtherdown its cut-out functions). They fail on main on a Mac whatever this PR does: most need the macOS port's fixes, and
test_module_idslooks for acomplete_fromsignature main no longer has:test_appimage_package_retry,test_collect_wide,test_control,test_framegen_resources,test_game_layout,test_low_memory_notice,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_module_ids,test_oot_state_tap,test_pad_layout,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_prof_symbols,test_program_cache_cleanup,test_program_cache_one_variant.test_program_cache_cleanupandtest_program_cache_one_variant. On a Mac they stop atposix_fadvise, which Darwin lacks. With a compiler wrapperthat gives them a no-op
posix_fadvise, both pass on main and here. Without this PR's stand-in fordescriptor_arrays_note,test_program_cache_one_variantstops at it.test_program_cache_format,test_program_cache_index,test_load_default,test_program_life,test_program_load_wait.gpu/vk/vk_programs.cpp,vk_progcache.cpp,vk_draw.cppandvk_main.cppwith the renderer'sflags (
nx2/build.py), Apple clang 21: arm64 and x86_64, on main and here, all 16 build.RUSAGE_THREAD(vk_programs.cpp),posix_fadviseandPOSIX_FADV_WILLNEED(vk_progcache.cpp), andposix_fallocate(pipeline_cache_file.h). The compiles used a compile-only prelude that declares them, withenviron.static_assertthat holds the census's stage bits toVK_SHADER_STAGE_*is part ofvk_programs.cpp's compile.logfanddev.props.limits, as the codearound it does.
Not tested:
and no draw changes.
one (it throws "Array of texture buffers" in the SPIR-V back end), and those programs are already named by the
"not translated" lines. Image and image-buffer arrays translate today and are counted.
Needs other testers:
programs read as well), play a few minutes and close the window, then read
longshot_log.txt. Expectvk: indexed descriptor arrays (close): ...near the end, and a line for each stage with arrays. The numbers decide whether reallimits should reach the translator, on M1 and M2 especially (request to open).
Screenshots
Nothing on screen.
Risk and rollback
Reviewer notes
ArrayCensus::noteandsummaryindescriptor_array_census.h, thendescriptor_arrays_noteinvk_programs.cppand the three calls in
vk_progcache.cpp.stage their limits are checked against.
vk_shader.cpporeden/would change the translator's stamp and translateevery player's programs again, and it would miss every start that reads them from the cache.
Issues
None.
Checklist
For everyone
type(scope): description).main. Branched from5e072fc8; main has moved since, and it is rebased in one sweep before ready.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above). (Run on macOS: the 19 failures also fail on main, named under Testing.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.