forked from MagicBeanDev/Longshot
fix(nvn): a mod's own texture and sampler pools get ids of their own, found by segment #3
Loading…
Reference in a new issue
No description provided.
Delete branch "rainy/Longshot-module-gaps:fix/nvn-mod-pool-ids"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08) · Needs: a run on Windows, any GPU
TL;DR
A code mod that makes its own texture and sampler pools registered them at ids the game already uses. Longshot keeps one id table for all pools, so the game's draws showed the mod's textures even after the menu closed. This PR gives a mod's pools their own ids at the top of the tables and moves the handles to match, finding the mod's code by where the build places it. The game's pools keep their ids. A new game-free test fails without the fix and passes here. The full check suite has the same 38 failures as
mods/module-gaps, and one more pass. It ran in the game on Linux, in a full patched build on an earlier module-gaps base.Breakdown
tests/test_nvn_mod_pools.pypasses here and fails with the fix reverted.tests/test_nvn_null_pool.pystill passes.python -I -B tests/run_all.py --jobs 8has the same 38 failures asmods/module-gapsand one more pass. It also ran in the game on Linux, in a full patched build on an earlier base of the same module-gaps commits.Why
I found this while running a code mod, sloperinaGZ 0.0.8 (https://github.com/PracticeROM/sloperina/releases/tag/v0.0.8), on this branch. Its menu makes a texture pool and a sampler pool of 512 in its own module. It registers its glyph atlas at texture 256, its pictures after it, and sampler 256. Those are ids the game uses too. From the first time the menu was drawn, the game's own draws sampled the menu's atlas and panels through those ids, and the wrong textures stayed after the menu closed.
A first version found a mod's module by its DT_INIT in the build's module list, and in the game it changed nothing. This branch runs a mod's start-up at build time and builds the result in, so the module has no DT_INIT left.
write_image_cinnx2/standalone.pylists only modules with a DT_INIT, so the build'shos_image.chad main, subsdk0 and sdk, and no mod.nx2/image.pylays the mods' modules out after the game's last module, so this PR looks there. With sloperinaGZ 0.0.8 the range is 0x86020000 (mod1.subsdk9, as the build log shows) up tohos_image_end0x872ec000.On NVN each texture and sampler pool is its own table, and a command buffer picks one with
nvnCommandBufferSetTexturePoolandnvnCommandBufferSetSamplerPool. Longshot's renderer has one table of each (register_textureandregister_sampleringpu/vk/vk_bank.cpp), andNOP_SET_TEX_POOLinvk_replay.cppchanges nothing the replay resolves. That works for a game with one pool of each, and breaks for any mod that brings its own. NVN is not publicly documented, so the per-pool behaviour here comes from how the game and the mod use the API.This PR targets snaildev99's
mods/module-gapsat2756bd26, not main, because only that stack loads a mod's code module (#984) and builds its start-up in. I searched the tracker for texture pool, sampler pool and DT_INIT: no issue or PR covers it.How to reproduce
Seen on: Linux, RTX 4090, driver 615.78.08
Settings and switches: defaults, with sloperinaGZ 0.0.8 in the mods folder
Steps: build the game with the mod on
mods/module-gaps, start it, open the mod's menu, close it, and look at the game.Expected: the game's textures are unchanged after the menu closes.
Actual: the game's draws show pieces of the menu's glyph atlas and panels.
What changed
host/nvn_native.cgives a mod's own descriptor pools an id base and applies it to registrations and binds.nvn_dpoolingpu/render/rx_nvn.hgets abasefield. Newin_mod_module,mod_pool_baseandmod_handleinhost/nvn_native.c.in_mod_moduletakes the highest DT_INIT among the modules not named "mod", finds the segment that holds it, and starts the mods' range at the next code segment with permission 5 inhos_region_perm. The range ends athos_image_end, and it logs "nvn: mods' modules from ... to ..." once. A pool whose object lies in a mod's module takes a block at the top of the tables: textures count down from 2^20 and samplers from 4096, in blocks of n rounded up to 256.nvnTexturePoolRegisterTextureandnvnSamplerPoolRegisterSampleradd the base.nvnCommandBufferSetTexturePoolandnvnCommandBufferSetSamplerPoolremember each command buffer's bases in a short list, andnvnCommandBufferBindTextureandnvnCommandBufferBindTexturesmove the handle's texture and sampler fields by them.RegisterImageis unchanged, since the mod registers no images.tests/test_nvn_mod_pools.pycompiles the pool functions from the source over a made-up image shaped as the build writes it: the game's modules, then a mod's that is missing fromhos_module_init. It checks that the game's pools keep base 0, that a mod's pools get the top blocks, that binds move only on a command buffer that set a mod's pool, and that handle 0 stays 0.Measurements
Nothing to measure for the game: its pools take base 0 and its binds skip the new path after one check of an empty list.
Flags and settings
None
Testing
Tested: game-free tests on Linux with gcc, on this branch at
8aa811ec:python -I -B tests/test_nvn_mod_pools.py --cc gccpasses.host/nvn_native.candgpu/render/rx_nvn.hreverted tomods/module-gaps, it fails:ValueError: substring not foundforextern const int hos_module_count, hos_region_count;, since the pool code is not there.in_mod_module, it fails withFAIL:164 ((nvn_dpool *)(mem + 0x7100))->base == (1 << 20) - 512. The mod's pool is left at base 0.tests/test_nvn_null_pool.py,tests/test_nvn_buffer_map.py,tests/test_nvn_host_tables.py,tests/test_nvn_tex_builder_guard.pyandtests/test_nvn_texture_handle.pypass.python -I -B tests/run_all.py --jobs 8on Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08).mods/module-gaps@2756bd26: 686 passed, 38 failed, 97 skipped. this branch @8aa811ec: 687 passed, 38 failed, 97 skipped, 0 new failures compared by test name. The 38 failures also fail onmods/module-gapsat2756bd26. They are:Tests that also fail on mods/module-gaps @
2756bd26in this environmenttests/test_bug_report_defaults.pytests/test_capture_read_once.pytests/test_cpu_report.pytests/test_crash_record.pytests/test_crt_options.pytests/test_damage_debug_menu.pytests/test_dev_packs.pytests/test_display_startup_failure.pytests/test_dlss_feature_size.pytests/test_failed_compile_keeps_objects.pytests/test_guest_imported.pytests/test_high_fps_fixes.pytests/test_host_guest_writes.pytests/test_hud_layout_row.pytests/test_invalidate_quick.pytests/test_invalidate_vcore.pytests/test_libc_hooks.pytests/test_link_path_comma.pytests/test_linux_preflight.pytests/test_linux_pressure.pytests/test_linux_update.pytests/test_log_open_failure.pytests/test_missing_nup.pytests/test_mods_size_table.pytests/test_pending_guest_import.pytests/test_pipeline_play_cache.pytests/test_prepare_cache_save.pytests/test_present_thread.pytests/test_profiling_mode.pytests/test_psychov35_fsr3_fg.pytests/test_readback_timeout_source.pytests/test_run_all_changed.pytests/test_slow_frames.pytests/test_song_note_cue.pytests/test_switch_table.pytests/test_upscaler_rows.pytests/test_walk_key.pytests/test_vbytes.py(timed out at 600 s)In the game, on Linux with an RTX 4090, this fix ran as part of a full patched build: all the fixes together, on an earlier base of the same module-gaps commits (
ed1a8513). With the pool ids alone, the menu's pieces stayed on the game's draws. With the segment lookup added, they were gone after the menu closed. It was not run alone in the game on this exact branch.Not tested: Windows (no build or run there) and other GPUs.
Must be tested:
Testing with sloperinaGZ
This branch alone isn't enough to use sloperinaGZ in the game. The in-game run above also had these fixes applied:
A branch with all of them applied together: https://git.magicbean.dev/rainy/Longshot/src/branch/longslop-rc8
Risks
What could break: a mod's binds use moved ids, so a mistake in the move would show wrong textures in that mod's own draws. If a build placed something other than mods after the game's last module, its pools would be moved too, but
nx2/image.pyplaces only the mods there. More than 16 command buffers that set a mod's pools would not all be tracked. The game's own pools are not moved.Turning it off: revert the commit
Compatibility: Nothing changes for the game without mods. No save, cache or vk ABI change.
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main. This PR targets snaildev99'smods/module-gaps, not main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). The failures named below also fail onmods/module-gapsat2756bd26, which is this PR's base.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.A workaround: real NVN resolves a handle in the pool the command buffer has set, but the renderer has one table of texture ids and one of sampler ids (gpu/vk/vk_bank.cpp register_texture / register_sampler). The pool a registration names was not looked at, and setting a pool on a command buffer changes nothing the replay resolves (vk_replay.cpp NOP_SET_TEX_POOL). That matches a game with one pool of each, but sloperinaGZ 0.0.8's menu makes a texture and a sampler pool of 512 in its own module and registers its glyph atlas at texture 256, its pictures after it, and sampler 256: the game's own ids. From the first time the menu was drawn, the game's draws sampled the atlas and the menu's panels through those ids, and it stayed after the menu closed. A pool whose object lies in a mod's module now gets a base at the top of the tables (textures from 2^20, samplers from 4096, a block of n rounded up to 256 each); its registrations land at id + base, and a command buffer that has set that pool binds its handles moved by the same (texture field and sampler field). The game's pools keep base 0: nothing of the game's changes. RegisterImage is left as it was. The mods' modules are found by their segments, not by DT_INIT: a mod whose start-up the build runs ahead of time (this branch's built-in start-up) has no DT_INIT left, and nx2/standalone.py write_image_c lists only modules with one. The mods are laid out after the game's last module (nx2/image.py), so they start at the first code segment (perm 5) after the one holding the game's last DT_INIT and run to hos_image_end. With sloperinaGZ 0.0.8 that is 0x86020000 to 0x872ec000. The start is logged once ("nvn: mods' modules from ..."). tests/test_nvn_mod_pools.py compiles the pool functions over a made-up image shaped as the build writes it (the mod not in hos_module_init). Assisted-by: Grok BotThank you, and thank you for saying plainly that it is a workaround. My view on your question: take the workaround now, with two fixes, and do not build per-pool tables.
Why not per-pool tables: every lookup would have to take (pool, id):
texture_of,texture_slot_version,sampler_ofand its relatives,registrations_generation, about thirty& 0xFFFFFdecodes in the draw and shader code, and the descriptor and view caches keyed by slot. That is several hundred lines on the game's hottest renderer path and needs an in-game A/B of the game's own frame, for one mod.A smaller follow-up I'd suggest instead, a remap at replay: keep the one global table, give every pool after the first a base when it is initialised (whoever owns it, which drops the segment test and covers pools allocated on the heap), have
NOP_SET_TEX_POOL/NOP_SET_SAMP_POOLstore the set pool in the replay state, and add the pool's base where the replay resolvesNOP_BIND_TEX/SAMP/IMG. That would cover the deferred, separate and image binds and a pool set in another command buffer, without the per-buffer table. It rests on the replay carrying the set pool across one submit's command sets in order, which I have not confirmed. Not needed for this pull request.Windows run (Windows 11, the builder's clang):
tests/test_nvn_mod_pools.pypasses at8aa811ecand fails with the fix reverted;tests/test_nvn_null_pool.pystill passes.Two things to fix before merging (medium):
register_texture_copyresizes the texture table toid + 1024, so the first mod texture grows it to about a million entries of 152 bytes: about 150 MiB of host memory, against about 5 MiB for the game's 33,068. Taking blocks upward from just above the game's pool (for example from 65536) costs about 10 MiB.Small: a comment naming what is not moved (the deferred and separate binds, images, a pool object on the heap), and in
in_mod_modulecompute the range into a local and publish it once (a second thread at the first call can read 0 today).Checked and fine: the game's pools keep base 0 and nothing of the game's changes until a mod creates a pool; moved handles are stored in the recording, so replay needs nothing; two mods do not collide with each other.
Same offer: if you'd rather, I will put the two fixes on top of your commit.
Thanks for the careful review, and for the reasoning on per-pool tables. Several hundred lines on the hottest renderer path for one mod isn't worth it. Taking blocks upward from just above the game's pool is a clear win over the ~150 MiB I'd have caused. Reusing blocks keyed by pool address, plus a log line on the fallback, also fixes the silent failure, which was the part I liked least. The remap-at-replay idea is cleaner than what I had, if the set pool does carry across a submit's command sets. Please go ahead and add your follow-ups as your own commits on top of ours, your way. For credit, my commits should be authored as
rainy, with anAssisted-by: Grok Bottrailer.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.