fix(nvn): a mod's own texture and sampler pools get ids of their own, found by segment #3

Open
rainy wants to merge 1 commit from rainy/Longshot-module-gaps:fix/nvn-mod-pool-ids into mods/module-gaps
First-time contributor

Type: fix · Risk: medium · Tested on: Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08) · Needs: a run on Windows, any GPU

This is a workaround, and I'd like your view on the real fix. Real NVN looks a handle up in the pool the command buffer has set. Longshot's renderer has one global texture table and one sampler table, so a second pool's ids land on the first pool's. This PR moves a mod's pools to ids at the top of those tables. The real fix is likely per-pool tables in the renderer, with NOP_SET_TEX_POOL and NOP_SET_SAMP_POOL resolved at replay. Would you rather have that? If so, I'd close this and help there.

TL;DR

A code mod that makes its own texture and sampler pools registered them at ids the game already uses. Longshot keeps one id table for all pools, so the game's draws showed the mod's textures even after the menu closed. This PR gives a mod's pools their own ids at the top of the tables and moves the handles to match, finding the mod's code by where the build places it. The game's pools keep their ids. A new game-free test fails without the fix and passes here. The full check suite has the same 38 failures as mods/module-gaps, and one more pass. It ran in the game on Linux, in a full patched build on an earlier module-gaps base.

Breakdown

  • Who it helps: players of a code mod that draws with its own NVN texture pools, such as sloperinaGZ 0.0.8's menu.
  • What was wrong: the renderer keeps one texture id table and one sampler id table, and ignores which pool a registration or a bind names.
  • What changes: a pool whose object is inside a mod's module gets a base id near the top of the tables, and its registrations and the handles bound while it is set move by that base. The mods' modules are found by segment layout, not by DT_INIT.
  • What does not change: every pool of the game's own keeps base 0, so the game's ids and binds are as before.
  • How it was checked: the new tests/test_nvn_mod_pools.py passes here and fails with the fix reverted. tests/test_nvn_null_pool.py still passes. python -I -B tests/run_all.py --jobs 8 has the same 38 failures as mods/module-gaps and one more pass. It also ran in the game on Linux, in a full patched build on an earlier base of the same module-gaps commits.
  • What is still open: the design question above, and a run on Windows.
  • Risk: medium. It touches texture binding, but only for pools inside a mod's module.

Why

I found this while running a code mod, sloperinaGZ 0.0.8 (https://github.com/PracticeROM/sloperina/releases/tag/v0.0.8), on this branch. Its menu makes a texture pool and a sampler pool of 512 in its own module. It registers its glyph atlas at texture 256, its pictures after it, and sampler 256. Those are ids the game uses too. From the first time the menu was drawn, the game's own draws sampled the menu's atlas and panels through those ids, and the wrong textures stayed after the menu closed.

A first version found a mod's module by its DT_INIT in the build's module list, and in the game it changed nothing. This branch runs a mod's start-up at build time and builds the result in, so the module has no DT_INIT left. write_image_c in nx2/standalone.py lists only modules with a DT_INIT, so the build's hos_image.c had main, subsdk0 and sdk, and no mod. nx2/image.py lays the mods' modules out after the game's last module, so this PR looks there. With sloperinaGZ 0.0.8 the range is 0x86020000 (mod1.subsdk9, as the build log shows) up to hos_image_end 0x872ec000.

On NVN each texture and sampler pool is its own table, and a command buffer picks one with nvnCommandBufferSetTexturePool and nvnCommandBufferSetSamplerPool. Longshot's renderer has one table of each (register_texture and register_sampler in gpu/vk/vk_bank.cpp), and NOP_SET_TEX_POOL in vk_replay.cpp changes nothing the replay resolves. That works for a game with one pool of each, and breaks for any mod that brings its own. NVN is not publicly documented, so the per-pool behaviour here comes from how the game and the mod use the API.

This PR targets snaildev99's mods/module-gaps at 2756bd26, not main, because only that stack loads a mod's code module (#984) and builds its start-up in. I searched the tracker for texture pool, sampler pool and DT_INIT: no issue or PR covers it.

How to reproduce

Seen on: Linux, RTX 4090, driver 615.78.08
Settings and switches: defaults, with sloperinaGZ 0.0.8 in the mods folder
Steps: build the game with the mod on mods/module-gaps, start it, open the mod's menu, close it, and look at the game.
Expected: the game's textures are unchanged after the menu closes.
Actual: the game's draws show pieces of the menu's glyph atlas and panels.

What changed

host/nvn_native.c gives a mod's own descriptor pools an id base and applies it to registrations and binds.

  • nvn: nvn_dpool in gpu/render/rx_nvn.h gets a base field. New in_mod_module, mod_pool_base and mod_handle in host/nvn_native.c. in_mod_module takes the highest DT_INIT among the modules not named "mod", finds the segment that holds it, and starts the mods' range at the next code segment with permission 5 in hos_region_perm. The range ends at hos_image_end, and it logs "nvn: mods' modules from ... to ..." once. A pool whose object lies in a mod's module takes a block at the top of the tables: textures count down from 2^20 and samplers from 4096, in blocks of n rounded up to 256. nvnTexturePoolRegisterTexture and nvnSamplerPoolRegisterSampler add the base. nvnCommandBufferSetTexturePool and nvnCommandBufferSetSamplerPool remember each command buffer's bases in a short list, and nvnCommandBufferBindTexture and nvnCommandBufferBindTextures move the handle's texture and sampler fields by them. RegisterImage is unchanged, since the mod registers no images.
  • tests: new tests/test_nvn_mod_pools.py compiles the pool functions from the source over a made-up image shaped as the build writes it: the game's modules, then a mod's that is missing from hos_module_init. It checks that the game's pools keep base 0, that a mod's pools get the top blocks, that binds move only on a command buffer that set a mod's pool, and that handle 0 stays 0.

Measurements

Nothing to measure for the game: its pools take base 0 and its binds skip the new path after one check of an empty list.

Flags and settings

None

Testing

Tested: game-free tests on Linux with gcc, on this branch at 8aa811ec:

  • python -I -B tests/test_nvn_mod_pools.py --cc gcc passes.
  • With host/nvn_native.c and gpu/render/rx_nvn.h reverted to mods/module-gaps, it fails: ValueError: substring not found for extern const int hos_module_count, hos_region_count;, since the pool code is not there.
  • With only the DT_INIT lookup put back in in_mod_module, it fails with FAIL:164 ((nvn_dpool *)(mem + 0x7100))->base == (1 << 20) - 512. The mod's pool is left at base 0.
  • tests/test_nvn_null_pool.py, tests/test_nvn_buffer_map.py, tests/test_nvn_host_tables.py, tests/test_nvn_tex_builder_guard.py and tests/test_nvn_texture_handle.py pass.

python -I -B tests/run_all.py --jobs 8 on Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08). mods/module-gaps @ 2756bd26: 686 passed, 38 failed, 97 skipped. this branch @ 8aa811ec: 687 passed, 38 failed, 97 skipped, 0 new failures compared by test name. The 38 failures also fail on mods/module-gaps at 2756bd26. They are:

Tests that also fail on mods/module-gaps @ 2756bd26 in this environment
  • tests/test_bug_report_defaults.py
  • tests/test_capture_read_once.py
  • tests/test_cpu_report.py
  • tests/test_crash_record.py
  • tests/test_crt_options.py
  • tests/test_damage_debug_menu.py
  • tests/test_dev_packs.py
  • tests/test_display_startup_failure.py
  • tests/test_dlss_feature_size.py
  • tests/test_failed_compile_keeps_objects.py
  • tests/test_guest_imported.py
  • tests/test_high_fps_fixes.py
  • tests/test_host_guest_writes.py
  • tests/test_hud_layout_row.py
  • tests/test_invalidate_quick.py
  • tests/test_invalidate_vcore.py
  • tests/test_libc_hooks.py
  • tests/test_link_path_comma.py
  • tests/test_linux_preflight.py
  • tests/test_linux_pressure.py
  • tests/test_linux_update.py
  • tests/test_log_open_failure.py
  • tests/test_missing_nup.py
  • tests/test_mods_size_table.py
  • tests/test_pending_guest_import.py
  • tests/test_pipeline_play_cache.py
  • tests/test_prepare_cache_save.py
  • tests/test_present_thread.py
  • tests/test_profiling_mode.py
  • tests/test_psychov35_fsr3_fg.py
  • tests/test_readback_timeout_source.py
  • tests/test_run_all_changed.py
  • tests/test_slow_frames.py
  • tests/test_song_note_cue.py
  • tests/test_switch_table.py
  • tests/test_upscaler_rows.py
  • tests/test_walk_key.py
  • tests/test_vbytes.py (timed out at 600 s)

In the game, on Linux with an RTX 4090, this fix ran as part of a full patched build: all the fixes together, on an earlier base of the same module-gaps commits (ed1a8513). With the pool ids alone, the menu's pieces stayed on the game's draws. With the segment lookup added, they were gone after the menu closed. It was not run alone in the game on this exact branch.

Not tested: Windows (no build or run there) and other GPUs.

Must be tested:

  • Windows, any GPU: the same build and steps, expecting the same result (Test Farm: @Beanstalk windows, a code mod's menu opened and closed)

Testing with sloperinaGZ

This branch alone isn't enough to use sloperinaGZ in the game. The in-game run above also had these fixes applied:

  • #1683 fix(svc): svcQueryMemory free-range answer
  • #1684 fix(hid): stick direction pseudo-buttons. Without it, the mod's menu combo (R + right stick down) never fires.
  • #1685 fix(nvn): window origin and depth mode getters
  • #1686 fix(lift): SCVTF / UCVTF fixed-point
  • #1 and #2 here

A branch with all of them applied together: https://git.magicbean.dev/rainy/Longshot/src/branch/longslop-rc8

Risks

What could break: a mod's binds use moved ids, so a mistake in the move would show wrong textures in that mod's own draws. If a build placed something other than mods after the game's last module, its pools would be moved too, but nx2/image.py places only the mods there. More than 16 command buffers that set a mod's pools would not all be tracked. The game's own pools are not moved.

Turning it off: revert the commit

Compatibility: Nothing changes for the game without mods. No save, cache or vk ABI change.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. This PR targets snaildev99's mods/module-gaps, not main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). The failures named below also fail on mods/module-gaps at 2756bd26, which is this PR's base.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
**Type:** fix · **Risk:** medium · **Tested on:** Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08) · **Needs:** a run on Windows, any GPU > **This is a workaround, and I'd like your view on the real fix.** Real NVN looks a handle up in the pool the command buffer has set. Longshot's renderer has one global texture table and one sampler table, so a second pool's ids land on the first pool's. This PR moves a mod's pools to ids at the top of those tables. The real fix is likely per-pool tables in the renderer, with `NOP_SET_TEX_POOL` and `NOP_SET_SAMP_POOL` resolved at replay. Would you rather have that? If so, I'd close this and help there. ## TL;DR A code mod that makes its own texture and sampler pools registered them at ids the game already uses. Longshot keeps one id table for all pools, so the game's draws showed the mod's textures even after the menu closed. This PR gives a mod's pools their own ids at the top of the tables and moves the handles to match, finding the mod's code by where the build places it. The game's pools keep their ids. A new game-free test fails without the fix and passes here. The full check suite has the same 38 failures as `mods/module-gaps`, and one more pass. It ran in the game on Linux, in a full patched build on an earlier module-gaps base. ## Breakdown - **Who it helps:** players of a code mod that draws with its own NVN texture pools, such as sloperinaGZ 0.0.8's menu. - **What was wrong:** the renderer keeps one texture id table and one sampler id table, and ignores which pool a registration or a bind names. - **What changes:** a pool whose object is inside a mod's module gets a base id near the top of the tables, and its registrations and the handles bound while it is set move by that base. The mods' modules are found by segment layout, not by DT_INIT. - **What does not change:** every pool of the game's own keeps base 0, so the game's ids and binds are as before. - **How it was checked:** the new `tests/test_nvn_mod_pools.py` passes here and fails with the fix reverted. `tests/test_nvn_null_pool.py` still passes. `python -I -B tests/run_all.py --jobs 8` has the same 38 failures as `mods/module-gaps` and one more pass. It also ran in the game on Linux, in a full patched build on an earlier base of the same module-gaps commits. - **What is still open:** the design question above, and a run on Windows. - **Risk:** medium. It touches texture binding, but only for pools inside a mod's module. ## Why I found this while running a code mod, sloperinaGZ 0.0.8 (https://github.com/PracticeROM/sloperina/releases/tag/v0.0.8), on this branch. Its menu makes a texture pool and a sampler pool of 512 in its own module. It registers its glyph atlas at texture 256, its pictures after it, and sampler 256. Those are ids the game uses too. From the first time the menu was drawn, the game's own draws sampled the menu's atlas and panels through those ids, and the wrong textures stayed after the menu closed. A first version found a mod's module by its DT_INIT in the build's module list, and in the game it changed nothing. This branch runs a mod's start-up at build time and builds the result in, so the module has no DT_INIT left. `write_image_c` in `nx2/standalone.py` lists only modules with a DT_INIT, so the build's `hos_image.c` had main, subsdk0 and sdk, and no mod. `nx2/image.py` lays the mods' modules out after the game's last module, so this PR looks there. With sloperinaGZ 0.0.8 the range is 0x86020000 (mod1.subsdk9, as the build log shows) up to `hos_image_end` 0x872ec000. On NVN each texture and sampler pool is its own table, and a command buffer picks one with `nvnCommandBufferSetTexturePool` and `nvnCommandBufferSetSamplerPool`. Longshot's renderer has one table of each (`register_texture` and `register_sampler` in `gpu/vk/vk_bank.cpp`), and `NOP_SET_TEX_POOL` in `vk_replay.cpp` changes nothing the replay resolves. That works for a game with one pool of each, and breaks for any mod that brings its own. NVN is not publicly documented, so the per-pool behaviour here comes from how the game and the mod use the API. This PR targets snaildev99's `mods/module-gaps` at 2756bd26, not main, because only that stack loads a mod's code module (#984) and builds its start-up in. I searched the tracker for texture pool, sampler pool and DT_INIT: no issue or PR covers it. ## How to reproduce **Seen on:** Linux, RTX 4090, driver 615.78.08 **Settings and switches:** defaults, with sloperinaGZ 0.0.8 in the mods folder **Steps:** build the game with the mod on `mods/module-gaps`, start it, open the mod's menu, close it, and look at the game. **Expected:** the game's textures are unchanged after the menu closes. **Actual:** the game's draws show pieces of the menu's glyph atlas and panels. ## What changed `host/nvn_native.c` gives a mod's own descriptor pools an id base and applies it to registrations and binds. - nvn: `nvn_dpool` in `gpu/render/rx_nvn.h` gets a `base` field. New `in_mod_module`, `mod_pool_base` and `mod_handle` in `host/nvn_native.c`. `in_mod_module` takes the highest DT_INIT among the modules not named "mod", finds the segment that holds it, and starts the mods' range at the next code segment with permission 5 in `hos_region_perm`. The range ends at `hos_image_end`, and it logs "nvn: mods' modules from ... to ..." once. A pool whose object lies in a mod's module takes a block at the top of the tables: textures count down from 2^20 and samplers from 4096, in blocks of n rounded up to 256. `nvnTexturePoolRegisterTexture` and `nvnSamplerPoolRegisterSampler` add the base. `nvnCommandBufferSetTexturePool` and `nvnCommandBufferSetSamplerPool` remember each command buffer's bases in a short list, and `nvnCommandBufferBindTexture` and `nvnCommandBufferBindTextures` move the handle's texture and sampler fields by them. `RegisterImage` is unchanged, since the mod registers no images. - tests: new `tests/test_nvn_mod_pools.py` compiles the pool functions from the source over a made-up image shaped as the build writes it: the game's modules, then a mod's that is missing from `hos_module_init`. It checks that the game's pools keep base 0, that a mod's pools get the top blocks, that binds move only on a command buffer that set a mod's pool, and that handle 0 stays 0. ## Measurements Nothing to measure for the game: its pools take base 0 and its binds skip the new path after one check of an empty list. ### Flags and settings None ## Testing **Tested:** game-free tests on Linux with gcc, on this branch at 8aa811ec: - `python -I -B tests/test_nvn_mod_pools.py --cc gcc` passes. - With `host/nvn_native.c` and `gpu/render/rx_nvn.h` reverted to `mods/module-gaps`, it fails: `ValueError: substring not found` for `extern const int hos_module_count, hos_region_count;`, since the pool code is not there. - With only the DT_INIT lookup put back in `in_mod_module`, it fails with `FAIL:164 ((nvn_dpool *)(mem + 0x7100))->base == (1 << 20) - 512`. The mod's pool is left at base 0. - `tests/test_nvn_null_pool.py`, `tests/test_nvn_buffer_map.py`, `tests/test_nvn_host_tables.py`, `tests/test_nvn_tex_builder_guard.py` and `tests/test_nvn_texture_handle.py` pass. `python -I -B tests/run_all.py --jobs 8` on Linux (Omarchy, kernel 7.2.15; Intel Core Ultra 7 270K Plus; RTX 4090, driver 615.78.08). `mods/module-gaps` @ 2756bd26: 686 passed, 38 failed, 97 skipped. this branch @ 8aa811ec: 687 passed, 38 failed, 97 skipped, 0 new failures compared by test name. The 38 failures also fail on `mods/module-gaps` at 2756bd26. They are: <details> <summary>Tests that also fail on mods/module-gaps @ 2756bd26 in this environment</summary> - `tests/test_bug_report_defaults.py` - `tests/test_capture_read_once.py` - `tests/test_cpu_report.py` - `tests/test_crash_record.py` - `tests/test_crt_options.py` - `tests/test_damage_debug_menu.py` - `tests/test_dev_packs.py` - `tests/test_display_startup_failure.py` - `tests/test_dlss_feature_size.py` - `tests/test_failed_compile_keeps_objects.py` - `tests/test_guest_imported.py` - `tests/test_high_fps_fixes.py` - `tests/test_host_guest_writes.py` - `tests/test_hud_layout_row.py` - `tests/test_invalidate_quick.py` - `tests/test_invalidate_vcore.py` - `tests/test_libc_hooks.py` - `tests/test_link_path_comma.py` - `tests/test_linux_preflight.py` - `tests/test_linux_pressure.py` - `tests/test_linux_update.py` - `tests/test_log_open_failure.py` - `tests/test_missing_nup.py` - `tests/test_mods_size_table.py` - `tests/test_pending_guest_import.py` - `tests/test_pipeline_play_cache.py` - `tests/test_prepare_cache_save.py` - `tests/test_present_thread.py` - `tests/test_profiling_mode.py` - `tests/test_psychov35_fsr3_fg.py` - `tests/test_readback_timeout_source.py` - `tests/test_run_all_changed.py` - `tests/test_slow_frames.py` - `tests/test_song_note_cue.py` - `tests/test_switch_table.py` - `tests/test_upscaler_rows.py` - `tests/test_walk_key.py` - `tests/test_vbytes.py` (timed out at 600 s) </details> In the game, on Linux with an RTX 4090, this fix ran as part of a full patched build: all the fixes together, on an earlier base of the same module-gaps commits (ed1a8513). With the pool ids alone, the menu's pieces stayed on the game's draws. With the segment lookup added, they were gone after the menu closed. It was not run alone in the game on this exact branch. **Not tested:** Windows (no build or run there) and other GPUs. **Must be tested:** - Windows, any GPU: the same build and steps, expecting the same result (Test Farm: @Beanstalk windows, a code mod's menu opened and closed) **Testing with sloperinaGZ** This branch alone isn't enough to use sloperinaGZ in the game. The in-game run above also had these fixes applied: - [#1683](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1683) fix(svc): svcQueryMemory free-range answer - [#1684](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1684) fix(hid): stick direction pseudo-buttons. Without it, the mod's menu combo (R + right stick down) never fires. - [#1685](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1685) fix(nvn): window origin and depth mode getters - [#1686](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1686) fix(lift): SCVTF / UCVTF fixed-point - [#1](https://git.magicbean.dev/snaildev99/Longshot/pulls/1) and [#2](https://git.magicbean.dev/snaildev99/Longshot/pulls/2) here A branch with all of them applied together: https://git.magicbean.dev/rainy/Longshot/src/branch/longslop-rc8 ## Risks **What could break:** a mod's binds use moved ids, so a mistake in the move would show wrong textures in that mod's own draws. If a build placed something other than mods after the game's last module, its pools would be moved too, but `nx2/image.py` places only the mods there. More than 16 command buffers that set a mod's pools would not all be tracked. The game's own pools are not moved. **Turning it off:** revert the commit **Compatibility:** Nothing changes for the game without mods. No save, cache or vk ABI change. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. This PR targets snaildev99's `mods/module-gaps`, not main. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). The failures named below also fail on `mods/module-gaps` at 2756bd26, which is this PR's base. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.
A workaround: real NVN resolves a handle in the pool the command buffer
has set, but the renderer has one table of texture ids and one of sampler
ids (gpu/vk/vk_bank.cpp register_texture / register_sampler). The pool a
registration names was not looked at, and setting a pool on a command
buffer changes nothing the replay resolves (vk_replay.cpp
NOP_SET_TEX_POOL). That matches a game with one pool of each, but
sloperinaGZ 0.0.8's menu makes a texture and a sampler pool of 512 in its
own module and registers its glyph atlas at texture 256, its pictures
after it, and sampler 256: the game's own ids. From the first time the
menu was drawn, the game's draws sampled the atlas and the menu's panels
through those ids, and it stayed after the menu closed.

A pool whose object lies in a mod's module now gets a base at the top of
the tables (textures from 2^20, samplers from 4096, a block of n rounded
up to 256 each); its registrations land at id + base, and a command
buffer that has set that pool binds its handles moved by the same
(texture field and sampler field). The game's pools keep base 0: nothing
of the game's changes. RegisterImage is left as it was.

The mods' modules are found by their segments, not by DT_INIT: a mod
whose start-up the build runs ahead of time (this branch's built-in
start-up) has no DT_INIT left, and nx2/standalone.py write_image_c lists
only modules with one. The mods are laid out after the game's last module
(nx2/image.py), so they start at the first code segment (perm 5) after
the one holding the game's last DT_INIT and run to hos_image_end. With
sloperinaGZ 0.0.8 that is 0x86020000 to 0x872ec000. The start is logged
once ("nvn: mods' modules from ...").

tests/test_nvn_mod_pools.py compiles the pool functions over a made-up
image shaped as the build writes it (the mod not in hos_module_init).

Assisted-by: Grok Bot
Owner

Thank you, and thank you for saying plainly that it is a workaround. My view on your question: take the workaround now, with two fixes, and do not build per-pool tables.

Why not per-pool tables: every lookup would have to take (pool, id): texture_of, texture_slot_version, sampler_of and its relatives, registrations_generation, about thirty & 0xFFFFF decodes in the draw and shader code, and the descriptor and view caches keyed by slot. That is several hundred lines on the game's hottest renderer path and needs an in-game A/B of the game's own frame, for one mod.

A smaller follow-up I'd suggest instead, a remap at replay: keep the one global table, give every pool after the first a base when it is initialised (whoever owns it, which drops the segment test and covers pools allocated on the heap), have NOP_SET_TEX_POOL / NOP_SET_SAMP_POOL store the set pool in the replay state, and add the pool's base where the replay resolves NOP_BIND_TEX / SAMP / IMG. That would cover the deferred, separate and image binds and a pool set in another command buffer, without the per-buffer table. It rests on the replay carrying the set pool across one submit's command sets in order, which I have not confirmed. Not needed for this pull request.

Windows run (Windows 11, the builder's clang): tests/test_nvn_mod_pools.py passes at 8aa811ec and fails with the fix reverted; tests/test_nvn_null_pool.py still passes.

Two things to fix before merging (medium):

  • Memory. Blocks are taken downward from 2^20, and register_texture_copy resizes the texture table to id + 1024, so the first mod texture grows it to about a million entries of 152 bytes: about 150 MiB of host memory, against about 5 MiB for the game's 33,068. Taking blocks upward from just above the game's pool (for example from 65536) costs about 10 MiB.
  • Blocks are never reused. A pool object initialised again takes a new block each time. Sampler space (4096 down to 2048, blocks of 512) is gone after four, and then the base is 0 and the mod is back on the game's ids, silently. Keying the block by the pool object's address and reusing it fixes that; a log line when a pool falls back to base 0 would make the remaining case visible.

Small: a comment naming what is not moved (the deferred and separate binds, images, a pool object on the heap), and in in_mod_module compute the range into a local and publish it once (a second thread at the first call can read 0 today).

Checked and fine: the game's pools keep base 0 and nothing of the game's changes until a mod creates a pool; moved handles are stored in the recording, so replay needs nothing; two mods do not collide with each other.

Same offer: if you'd rather, I will put the two fixes on top of your commit.

Thank you, and thank you for saying plainly that it is a workaround. My view on your question: take the workaround now, with two fixes, and do not build per-pool tables. **Why not per-pool tables:** every lookup would have to take (pool, id): `texture_of`, `texture_slot_version`, `sampler_of` and its relatives, `registrations_generation`, about thirty `& 0xFFFFF` decodes in the draw and shader code, and the descriptor and view caches keyed by slot. That is several hundred lines on the game's hottest renderer path and needs an in-game A/B of the game's own frame, for one mod. **A smaller follow-up I'd suggest instead, a remap at replay:** keep the one global table, give every pool after the first a base when it is initialised (whoever owns it, which drops the segment test and covers pools allocated on the heap), have `NOP_SET_TEX_POOL` / `NOP_SET_SAMP_POOL` store the set pool in the replay state, and add the pool's base where the replay resolves `NOP_BIND_TEX` / `SAMP` / `IMG`. That would cover the deferred, separate and image binds and a pool set in another command buffer, without the per-buffer table. It rests on the replay carrying the set pool across one submit's command sets in order, which I have not confirmed. Not needed for this pull request. **Windows run (Windows 11, the builder's clang):** `tests/test_nvn_mod_pools.py` passes at 8aa811ec and fails with the fix reverted; `tests/test_nvn_null_pool.py` still passes. **Two things to fix before merging (medium):** - **Memory.** Blocks are taken downward from 2^20, and `register_texture_copy` resizes the texture table to `id + 1024`, so the first mod texture grows it to about a million entries of 152 bytes: about 150 MiB of host memory, against about 5 MiB for the game's 33,068. Taking blocks upward from just above the game's pool (for example from 65536) costs about 10 MiB. - **Blocks are never reused.** A pool object initialised again takes a new block each time. Sampler space (4096 down to 2048, blocks of 512) is gone after four, and then the base is 0 and the mod is back on the game's ids, silently. Keying the block by the pool object's address and reusing it fixes that; a log line when a pool falls back to base 0 would make the remaining case visible. **Small:** a comment naming what is not moved (the deferred and separate binds, images, a pool object on the heap), and in `in_mod_module` compute the range into a local and publish it once (a second thread at the first call can read 0 today). **Checked and fine:** the game's pools keep base 0 and nothing of the game's changes until a mod creates a pool; moved handles are stored in the recording, so replay needs nothing; two mods do not collide with each other. Same offer: if you'd rather, I will put the two fixes on top of your commit.
Author
First-time contributor

Thanks for the careful review, and for the reasoning on per-pool tables. Several hundred lines on the hottest renderer path for one mod isn't worth it. Taking blocks upward from just above the game's pool is a clear win over the ~150 MiB I'd have caused. Reusing blocks keyed by pool address, plus a log line on the fallback, also fixes the silent failure, which was the part I liked least. The remap-at-replay idea is cleaner than what I had, if the set pool does carry across a submit's command sets. Please go ahead and add your follow-ups as your own commits on top of ours, your way. For credit, my commits should be authored as rainy, with an Assisted-by: Grok Bot trailer.

Thanks for the careful review, and for the reasoning on per-pool tables. Several hundred lines on the hottest renderer path for one mod isn't worth it. Taking blocks upward from just above the game's pool is a clear win over the ~150 MiB I'd have caused. Reusing blocks keyed by pool address, plus a log line on the fallback, also fixes the silent failure, which was the part I liked least. The remap-at-replay idea is cleaner than what I had, if the set pool does carry across a submit's command sets. Please go ahead and add your follow-ups as your own commits on top of ours, your way. For credit, my commits should be authored as `rainy`, with an `Assisted-by: Grok Bot` trailer.
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/nvn-mod-pool-ids:rainy-fix/nvn-mod-pool-ids'
git switch 'rainy-fix/nvn-mod-pool-ids'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'mods/module-gaps'
git merge --no-ff 'rainy-fix/nvn-mod-pool-ids'
git switch 'rainy-fix/nvn-mod-pool-ids'
git rebase 'mods/module-gaps'
git switch 'mods/module-gaps'
git merge --ff-only 'rainy-fix/nvn-mod-pool-ids'
git switch 'rainy-fix/nvn-mod-pool-ids'
git rebase 'mods/module-gaps'
git switch 'mods/module-gaps'
git merge --no-ff 'rainy-fix/nvn-mod-pool-ids'
git switch 'mods/module-gaps'
git merge --squash 'rainy-fix/nvn-mod-pool-ids'
git switch 'mods/module-gaps'
git merge --ff-only 'rainy-fix/nvn-mod-pool-ids'
git switch 'mods/module-gaps'
git merge 'rainy-fix/nvn-mod-pool-ids'
git push origin 'mods/module-gaps'
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
snaildev99/Longshot!3
No description provided.