fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's #1943

Merged
HumanBean merged 3 commits from zeldaootr/Longshot:damaged-romfs-files-any-unpacker into main 2026-11-21 17:42:23 +00:00
Contributor

Summary

Fixes #1939.

A game folder whose game files Longshot Builder didn't unpack keeps the three damaged textures of an incomplete dump.
That covers a folder carried over from Hookshot, game files unpacked by another tool and chosen as folders, and a
romfs copied by hand. The game then keeps reloading a model that uses one of them, for as long as Link is in Kokiri
Forest or the market. On a Steam Deck that costs about 1.3 CPU cores and a third of the frame rate (Kokiri 30 fps
instead of 44). Nothing in the log says why.

The bug on main: take a game folder whose romfs/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs starts with
00 00 00 00 instead of 28 b5 2f fd. Stand in Kokiri with HOS_THREAD_TIMES=5. The prof: threads in 5 s: lines
show OptimizeChangeBanc at ~80% and the two ParallelDecompThread entries at ~55% together. With HOS_OCB_TRACE (a
probe build), the Hyrule Field preload never finishes: the handle for Land_FairyFountain_Base_A_01 fails about 15
times a second.

Why: the 23.3 GB dumps have three .zs textures that begin with zeros. The builder copies a similar texture over
each one (nx2/nup.py stand_in_damaged, from nx2/make_game.py stand_in_damaged_files). It does this only in a
romfs it unpacked itself, the one with the .longshot_romfs stamp. Nothing else checks those files.

This change: when the game opens one of the three textures and it doesn't start as a zstd frame, the host reads the
stand-in the builder would have used. That is the file nup.stand_in_for picks, resolved through the usual mod, overlay and game-file lookups. One warning
names both files and says the dump is incomplete:

fs: content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs is damaged in this copy of the game (zeros where its data starts): CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs, a similar texture, is read in its place. Your dump is incomplete (about 23.3 GB instead of 23.8 GB): dump the game again for the original

The host writes nothing to the player's files. A whole file (a complete dump, or a romfs the builder already fixed)
is read as it is.

Why the game and not the builder: existing installs are fixed without the player doing anything, whichever
launcher, builder or tool made the folder. The builder and nup.py say they never write to a romfs they didn't
unpack, so a one-time repair there would break that rule, and a warning alone would leave the cost in place.

Read first:

  • host/oot_damaged_files.c: the three pairs and hos_stand_in_damaged;
  • the one call in host/nn_fs.c OpenFile, after the mods and HOS_FS_OVERLAY lookups;
  • tests/test_damaged_rom_files.py.

Testing

Tested:

  • tests/test_damaged_rom_files.py compiles the real OpenFile and host/oot_damaged_files.c over a made-up romfs (no
    game bytes). It fails on main (the damaged file's zeros are served, and no line is logged) and passed on the earlier head 9d84eb80a, with
    clang++ and g++. The updated fixture has not been reported as run. It checks:

    • a damaged texture opened 15 times serves its stand-in's bytes, with one warning;
    • a damaged texture whose stand-in is damaged too is served as it is, with one warning;
    • a whole texture under the same name, and any other file, are served as they are, with no line;
    • a missing file is still not found;
    • each pair in the C table is what nup.stand_in_for picks among them.
  • Mutations: dropping the zstd check of the opened file, or the check that the stand-in is whole, each fails the test.

  • The whole suite (python3 -I -B tests/run_all.py, Linux x86-64, 16 at once), at the earlier head 9d84eb80a and on main 87c0d7b4d (its base), back to back:
    786 passed, 86 failed, 28 skipped here; 785 / 86 / 28 on main. The failing checks are the same 85 names on both (one
    failure line carries no name on either side); none is this PR's.

    The 85 checks that fail both here and on main 87c0d7b4d

    test_appimage_player_build, test_bug_report_defaults, test_capture_readback_visibility, test_capture_read_once, test_crash_dialog, test_crash_record, test_crt_options, test_damage_debug_menu, test_dev_ab, test_device_lost_exit, test_dev_packs, test_discover, test_display_startup_failure, test_dlss_feature_size, test_draw_binds, test_draw_stream_visibility, test_failed_compile_keeps_objects, test_flush_nothing_held, test_frame_overlap, test_fsr3fi_hud_layer, test_fsr_image_retirement, test_fsr_uniform_ownership, test_gpu_wait_stats, test_guest_imported, test_high_fps_fixes, test_host_guest_writes, test_image_reaper_threads, test_libc_hooks, test_lift_cache, test_lift_lookups, test_linux_preflight, test_linux_pressure, test_linux_tools, test_linux_update, test_log_open_failure, test_low_vram_texture_mips, test_make_game_cache, test_make_game_ngx, test_menu_module_slots, test_menu_modules, test_menu_save_atomic, test_menu_sections, test_mise_build, test_missing_nup, test_mods_list, test_mods_size_table, test_ngx_sdk, test_nonuniform_image_arrays, test_oot_shadow_fit, test_pads_after_menu, test_pass_barrier, test_pending_guest_import, test_pipeline_play_cache, test_prepare_cache_save, test_present_queue_default, test_present_semaphores, test_present_source_crop, test_profiling_mode, test_program_cache_cleanup, test_program_cache_one_variant, test_psychov35_fsr3_fg, test_readback_timeout_source, test_recorder_own_idle, test_recorder_sync, test_reflex_swapchain_lifetime, test_renderer_new_handler, test_renhdrum, test_replay_write_order, test_sampler_admission, test_settings_table, test_slow_frames, test_song_note_cue, test_sparse_mirror, test_spirv_pack, test_staging_burst, test_target_reuse_memory, test_texel_buffer_ownership, test_thinlto_tls, test_ub_update_batch, test_ui_record, test_upscaler_rows, test_vcore_escape, test_vcore_mask0, test_vk_occlusion_queries, test_walk

  • At the earlier head 9d84eb80a, host/oot_damaged_files.c and host/nn_fs.c compiled with -Wall -Wextra under clang and gcc for Linux and
    llvm-mingw for Windows. The new file has no warnings. nn_fs.c has the same 37 warnings as on main, all
    -Wmissing-field-initializers from the HLE macro.

  • In the game, Steam Deck OLED (SteamOS 3.9.5, Mesa RADV 26.2.0-devel, GPU at manual 1600 MHz, headless), with a
    romfs that has the three damaged textures (an unpacker's output, not the builder's), no overlay. This PR's build
    against main @ 2210931cf, three starts each, alternated (A B B A A B), on the route Kokiri, Hyrule Field, the market
    (Measurements has the settings and the table). What was seen:

    • Every start of this PR logged the warning once, for the Fairy Fountain floor texture, as Link arrived in Kokiri
      (the Hyrule Field preload opens it). No start of main logged it. It was the only warning in all six logs:
      [warn] fs: content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs is damaged in this copy of the game (zeros where its data starts): CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs, a similar texture, is read in its place. Your dump is incomplete (about 23.3 GB instead of 23.8 GB): dump the game again for the original
      
    • The Water Temple and Ganon textures logged nothing: this route doesn't open them.
    • OptimizeChangeBanc and ParallelDecompThread are gone from the prof: threads lines in Kokiri and the market
      (on main: ~80% and ~55% in Kokiri, ~75% and ~50% in the market).
    • Kokiri 44.2 fps against 32.0, the market 25.8 against 21.5; Hyrule Field, which never had the loop, is unchanged
      within the spread.
    • All six starts exited with code 0, with no error line, crash or device loss.
    • Not seen: the Great Fairy Fountain's floor itself. The runs are headless, and the control bridge has no teleport
      into a Great Fairy Fountain.

Still to test:

  • The Great Fairy Fountain's floor on screen: the sand stand-in in place of the missing floor (by hand; no teleport
    reaches it).
  • Any PC with a romfs from a complete dump: no is damaged in this copy line, fps unchanged.
    (@Beanstalk run Kokiri Forest on this PR and on main; check longshot_log.txt has no "is damaged in this copy" line)
  • Windows, a romfs from another unpacker with the damaged files: the warning appears and the stand-in is read (by hand).
  • The Water Temple gate walls and the Ganon stage texture: the stand-ins draw in place of the missing models. The
    builder already serves these same stand-ins, so this has been seen in builder-made folders. With this change it has not
    been looked at (by hand).

Measurements

Verdict: on a romfs with the damaged textures, this PR gives Kokiri +38% fps (32.0 to 44.2) and the market +20%
(21.5 to 25.8), with no overlap between the starts of the two sides in either area. Both are larger than what three
starts a side resolve (Kokiri 6-12%, the market 4-14%). The loader and decode threads go from ~1.3 cores to 0 in both
areas. Hyrule Field, which never had the loop, is unchanged within the spread (+1% to +3%). It matches the earlier
measurement with the same stand-ins served through HOS_FS_OVERLAY (Kokiri 44.3, the market 27.1).

Baseline: main @ 2210931cf. The After build is main @ 2210931cf with this PR's two commits cherry-picked
(44defbbec; the measured head was 9d84eb80a on 1d8e73acf), so the two columns differ by this PR only. The Deck's warm
shader cache fits both builds (same translator stamp).

Setup: Steam Deck OLED, SteamOS 3.9.5, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), GPU at
manual 1600 MHz before and after (never written). The Steam Deck developers' row: Minimum preset, Render 50%, 1280x800,
frame pacing on, uncapped, vsync off, frame generation off, headless. HOS_NATIVE_ZSTD=1 HOS_THREAD_TIMES=5, no
overlay. The romfs has the three damaged textures. Route tour3 from the kokiri-start save: Kokiri 45 s standing,
Hyrule Field (20 s standing, 4 camera pans, 30 s standing) and the market (arrival +20 s, then 60 s standing) by
teleport, the order of the Field and the market alternated by round. Three starts a side, A B B A A B, fps from the
game's present trace.

Before: main @ 2210931c After: this PR @ 44defbbe Δ
kokiri fps (each start) 28.6 / 32.3 / 32.0 (median 32.0) 44.1 / 44.2 / 44.5 (median 44.2) +38.1%
kokiri 1% low fps (median) 20.7 32.2 +55.7%
kokiri p99 frame time (median) 46.0 ms 28.6 ms -17.40 ms
kokiri frames ≥50 ms (each start) 47 / 0 / 4 0 / 0 / 0
kokiri OptimizeChangeBanc + decode CPU (median) 79.8% + 54.8% 0 + 0
stand1 fps (each start) 41.5 / 43.8 / 47.6 (median 43.8) 44.3 / 40.4 / 47.3 (median 44.3) +1.2%
stand1 1% low fps (median) 27.7 32.2 +16.4%
stand1 p99 frame time (median) 32.7 ms 30.1 ms -2.60 ms
turn fps (each start) 37.0 / 35.9 / 39.1 (median 37.0) 38.1 / 33.7 / 40.4 (median 38.1) +2.9%
turn 1% low fps (median) 12.8 12.7 -1.3%
turn p99 frame time (median) 62.3 ms 64.6 ms +2.30 ms
stand2 fps (each start) 45.9 / 43.8 / 44.7 (median 44.7) 45.7 / 40.0 / 47.8 (median 45.7) +2.2%
stand2 1% low fps (median) 21.8 24.6 +12.8%
stand2 p99 frame time (median) 36.1 ms 33.7 ms -2.40 ms
market fps (each start) 20.0 / 21.5 / 23.9 (median 21.5) 26.4 / 25.4 / 25.8 (median 25.8) +20.2%
market 1% low fps (median) 10.1 12.4 +22.9%
market p99 frame time (median) 84.1 ms 67.7 ms -16.40 ms
market frames ≥50 ms (each start) 467 / 401 / 205 81 / 147 / 136
market OptimizeChangeBanc + decode CPU (median) 74.8% + 50.6% 0 + 0

The spread between starts of one side was about 10 to 17% of the median in these windows; a difference smaller than
that is within the spread. The CPU rows are the
HOS_THREAD_TIMES means over each window; /proc agrees within 1 point.

A: longshot v3.0.0-81-g2210931cf (source 28f0cd0e)
A: report: gpu AMD Custom GPU 0932 (RADV VANGOGH), radv Mesa 26.2.0-devel (git-5c65d4fc31)
A: vk: switches set (10): HOS_CONTROL=<a path> (diag), HOS_PRESENT_TRACE=<a path> (diag), HOS_OPTIONS_FILE=<a path> (diag), HOS_ENV_FILE=<a path> (diag), LONGSHOT_BUILD_NAME=dmgfix A1f (diag), HOS_HEADLESS=1 (diag), HOS_OUTPUT_RES=1280x800 (player), HOS_WINDOW_SIZE=1280x800 (tune), HOS_THREAD_TIMES=5 (diag), HOS_NATIVE_ZSTD=1 (attempt)
B: longshot v3.0.0-83-g44defbbec (source 03298903)
B: report: gpu AMD Custom GPU 0932 (RADV VANGOGH), radv Mesa 26.2.0-devel (git-5c65d4fc31)
B: vk: switches set (10): HOS_CONTROL=<a path> (diag), HOS_PRESENT_TRACE=<a path> (diag), HOS_OPTIONS_FILE=<a path> (diag), HOS_ENV_FILE=<a path> (diag), LONGSHOT_BUILD_NAME=dmgfix B1f (diag), HOS_HEADLESS=1 (diag), HOS_OUTPUT_RES=1280x800 (player), HOS_WINDOW_SIZE=1280x800 (tune), HOS_THREAD_TIMES=5 (diag), HOS_NATIVE_ZSTD=1 (attempt)

The thread lines in Kokiri, main (A3f) and this PR (B3f):

A: prof: threads in 5 s: sead::MainThread 28%; Presentation Thread 43%; Resource Memory 8%; ParallelDecompThread 22%; ParallelDecompThread 32%; ... BaseProcExecutor5 14%; OptimizeChangeBanc 78%;
B: prof: threads in 5 s: sead::MainThread 31%; Presentation Thread 53%; NavMeshCAStepDepWorker1 7%; ... BaseProcExecutor5 19%;

Earlier, the same effect through an overlay (before this PR's code existed): main @ 2210931cf plus a trace-only
probe (HOS_OCB_TRACE), the same runtime on both sides, the three stand-ins served through HOS_FS_OVERLAY. Same
Deck and row; Kokiri a 60 s stand, the market from arrival +20 s a 60 s stand, two starts a side.

main, romfs as dumped main, the stand-ins in HOS_FS_OVERLAY
Kokiri fps (run 1 / run 2) 30.46 / 29.46 44.21 / 44.37
Kokiri OptimizeChangeBanc + decode CPU 79% + 55% 0 + 0
Market fps (run 1 / run 2) 20.45 / 20.32 27.59 / 26.53
Market OptimizeChangeBanc + decode CPU 74.5% + 46% / 74.5% + 44% 0 + 0
Hyrule Field fps (control) 47.84 / 47.94 47.84 / 47.28

The probe's lines, without the stand-ins (Kokiri, 55 s into the stand):

prof: threads in 5 s: sead::MainThread 28%; Presentation Thread 44%; Resource Memory 8%; ParallelDecompThread 33%; ParallelDecompThread 22%; ... OptimizeChangeBanc 80%;
ocb: in 1.00 s: loads started 0 (...), finished 0 (longest 0.0 s), cancelled 0: ...; ms in states 3 1004; changes none; thread turns 173; ...; zstd out on the host 175.4 MB; files opened 51 (not found 0, entry looks 17), the most content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs x17 (looks 17); ...
ocb: entries: 852 of 6171 from 0; handles 1561 by status: 3:1 6:1560; state 7: 1, 14: 709; ...; first at 7: entry 44 handle 1 status 3 state 7 Land_FairyFountain_Base_A_01 ...

With them:

fs: OpenFile(content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs): from the overlay
ocb: loaded: HyruleField_Child in 10.0 s, 852 entries
ocb: in 1.01 s: loads started 0 (...), finished 0 (...), cancelled 0: ...; ms in states 4 1005; changes none; thread turns 62; ...; zstd out on the host 0.0 MB; files opened 1 ...

Risks

  • What it changes: only the open of three file names, and only when the file there doesn't start as zstd. The game
    can't use such a file anyway: it fails the model and asks for it again. A complete dump, a builder-made romfs and a
    mod that supplies a whole file are read as before.
  • Cost: on every open, a prefix compare (Tex/) and up to three string compares. For the three names, two 4-byte
    reads at their open, which with this change happens about once a load instead of 15 times a second.
  • What a player sees: the Great Fairy Fountain's floor, the Water Temple's gate walls and one Ganon stage texture
    are drawn with a similar texture instead of being left out. That is what a builder-made folder from the same dump
    already shows. There is also one warning a file in longshot_log.txt.
  • No switch: it acts only on a file the game can't read, so there is no working state to go back to. A revert is
    enough to undo it.
  • Limits:
    • Only the three textures known from the 23.3 GB dumps are covered. Another damaged file, from another dump, is
      still found only by the builder's scan of a romfs it unpacked.
    • Like the builder's stand-ins, the stand-in is served without a size-table entry of its own.
  • After updating: nothing in the game folder, saves or caches changes.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. (On my Steam Deck OLED, six
    starts; see Tested.)
  • mise run test passes (any failure that also fails on main is named). (The whole tests/run_all.py here and on main 87c0d7b4d: the same 85 failing checks on both, named under Tested.)
  • Measured against main, or Measurements says why there is nothing to measure. (This PR's build against main
    @ 2210931cf, three starts a side on a Steam Deck OLED.)
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here. (The updated fixture has not been reported as run.)
  • A check under tests/ that reads changed source is updated, not weakened. (tests/test_vcore_fs_open.py: the
    new call's stand-in.)
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. (The build was reported for the earlier head; the updated code has not been reported as built.)
  • Screenshots of the same spot on main and here, for a change that shows on screen. (The Great Fairy Fountain's
    floor, with and without the change: not yet taken. The Steam Deck runs were headless, and the control bridge has
    no teleport into a Great Fairy Fountain, so it needs a run by hand.)
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary Fixes https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1939. A game folder whose game files Longshot Builder didn't unpack keeps the three damaged textures of an incomplete dump. That covers a folder carried over from Hookshot, game files unpacked by another tool and chosen as folders, and a `romfs` copied by hand. The game then keeps reloading a model that uses one of them, for as long as Link is in Kokiri Forest or the market. On a Steam Deck that costs about 1.3 CPU cores and a third of the frame rate (Kokiri 30 fps instead of 44). Nothing in the log says why. **The bug on `main`:** take a game folder whose `romfs/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs` starts with `00 00 00 00` instead of `28 b5 2f fd`. Stand in Kokiri with `HOS_THREAD_TIMES=5`. The `prof: threads in 5 s:` lines show `OptimizeChangeBanc` at ~80% and the two `ParallelDecompThread` entries at ~55% together. With `HOS_OCB_TRACE` (a probe build), the Hyrule Field preload never finishes: the handle for `Land_FairyFountain_Base_A_01` fails about 15 times a second. **Why:** the 23.3 GB dumps have three `.zs` textures that begin with zeros. The builder copies a similar texture over each one (`nx2/nup.py` `stand_in_damaged`, from `nx2/make_game.py` `stand_in_damaged_files`). It does this only in a `romfs` it unpacked itself, the one with the `.longshot_romfs` stamp. Nothing else checks those files. **This change:** when the game opens one of the three textures and it doesn't start as a zstd frame, the host reads the stand-in the builder would have used. That is the file `nup.stand_in_for` picks, resolved through the usual mod, overlay and game-file lookups. One warning names both files and says the dump is incomplete: ``` fs: content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs is damaged in this copy of the game (zeros where its data starts): CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs, a similar texture, is read in its place. Your dump is incomplete (about 23.3 GB instead of 23.8 GB): dump the game again for the original ``` The host writes nothing to the player's files. A whole file (a complete dump, or a `romfs` the builder already fixed) is read as it is. **Why the game and not the builder:** existing installs are fixed without the player doing anything, whichever launcher, builder or tool made the folder. The builder and `nup.py` say they never write to a `romfs` they didn't unpack, so a one-time repair there would break that rule, and a warning alone would leave the cost in place. **Read first:** - `host/oot_damaged_files.c`: the three pairs and `hos_stand_in_damaged`; - the one call in `host/nn_fs.c` OpenFile, after the mods and `HOS_FS_OVERLAY` lookups; - `tests/test_damaged_rom_files.py`. ## Testing **Tested:** - `tests/test_damaged_rom_files.py` compiles the real OpenFile and `host/oot_damaged_files.c` over a made-up romfs (no game bytes). It fails on `main` (the damaged file's zeros are served, and no line is logged) and passed on the earlier head 9d84eb80a, with clang++ and g++. The updated fixture has not been reported as run. It checks: - a damaged texture opened 15 times serves its stand-in's bytes, with one warning; - a damaged texture whose stand-in is damaged too is served as it is, with one warning; - a whole texture under the same name, and any other file, are served as they are, with no line; - a missing file is still not found; - each pair in the C table is what `nup.stand_in_for` picks among them. - Mutations: dropping the zstd check of the opened file, or the check that the stand-in is whole, each fails the test. - The whole suite (`python3 -I -B tests/run_all.py`, Linux x86-64, 16 at once), at the earlier head 9d84eb80a and on `main` 87c0d7b4d (its base), back to back: 786 passed, 86 failed, 28 skipped here; 785 / 86 / 28 on `main`. The failing checks are the same 85 names on both (one failure line carries no name on either side); none is this PR's. <details><summary>The 85 checks that fail both here and on main 87c0d7b4d</summary> `test_appimage_player_build`, `test_bug_report_defaults`, `test_capture_readback_visibility`, `test_capture_read_once`, `test_crash_dialog`, `test_crash_record`, `test_crt_options`, `test_damage_debug_menu`, `test_dev_ab`, `test_device_lost_exit`, `test_dev_packs`, `test_discover`, `test_display_startup_failure`, `test_dlss_feature_size`, `test_draw_binds`, `test_draw_stream_visibility`, `test_failed_compile_keeps_objects`, `test_flush_nothing_held`, `test_frame_overlap`, `test_fsr3fi_hud_layer`, `test_fsr_image_retirement`, `test_fsr_uniform_ownership`, `test_gpu_wait_stats`, `test_guest_imported`, `test_high_fps_fixes`, `test_host_guest_writes`, `test_image_reaper_threads`, `test_libc_hooks`, `test_lift_cache`, `test_lift_lookups`, `test_linux_preflight`, `test_linux_pressure`, `test_linux_tools`, `test_linux_update`, `test_log_open_failure`, `test_low_vram_texture_mips`, `test_make_game_cache`, `test_make_game_ngx`, `test_menu_module_slots`, `test_menu_modules`, `test_menu_save_atomic`, `test_menu_sections`, `test_mise_build`, `test_missing_nup`, `test_mods_list`, `test_mods_size_table`, `test_ngx_sdk`, `test_nonuniform_image_arrays`, `test_oot_shadow_fit`, `test_pads_after_menu`, `test_pass_barrier`, `test_pending_guest_import`, `test_pipeline_play_cache`, `test_prepare_cache_save`, `test_present_queue_default`, `test_present_semaphores`, `test_present_source_crop`, `test_profiling_mode`, `test_program_cache_cleanup`, `test_program_cache_one_variant`, `test_psychov35_fsr3_fg`, `test_readback_timeout_source`, `test_recorder_own_idle`, `test_recorder_sync`, `test_reflex_swapchain_lifetime`, `test_renderer_new_handler`, `test_renhdrum`, `test_replay_write_order`, `test_sampler_admission`, `test_settings_table`, `test_slow_frames`, `test_song_note_cue`, `test_sparse_mirror`, `test_spirv_pack`, `test_staging_burst`, `test_target_reuse_memory`, `test_texel_buffer_ownership`, `test_thinlto_tls`, `test_ub_update_batch`, `test_ui_record`, `test_upscaler_rows`, `test_vcore_escape`, `test_vcore_mask0`, `test_vk_occlusion_queries`, `test_walk` </details> - At the earlier head 9d84eb80a, `host/oot_damaged_files.c` and `host/nn_fs.c` compiled with `-Wall -Wextra` under clang and gcc for Linux and llvm-mingw for Windows. The new file has no warnings. `nn_fs.c` has the same 37 warnings as on `main`, all `-Wmissing-field-initializers` from the HLE macro. - **In the game, Steam Deck OLED** (SteamOS 3.9.5, Mesa RADV 26.2.0-devel, GPU at manual 1600 MHz, headless), with a `romfs` that has the three damaged textures (an unpacker's output, not the builder's), no overlay. This PR's build against `main` @ 2210931cf, three starts each, alternated (A B B A A B), on the route Kokiri, Hyrule Field, the market (Measurements has the settings and the table). What was seen: - Every start of this PR logged the warning once, for the Fairy Fountain floor texture, as Link arrived in Kokiri (the Hyrule Field preload opens it). No start of `main` logged it. It was the only warning in all six logs: ``` [warn] fs: content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs is damaged in this copy of the game (zeros where its data starts): CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs, a similar texture, is read in its place. Your dump is incomplete (about 23.3 GB instead of 23.8 GB): dump the game again for the original ``` - The Water Temple and Ganon textures logged nothing: this route doesn't open them. - `OptimizeChangeBanc` and `ParallelDecompThread` are gone from the `prof: threads` lines in Kokiri and the market (on `main`: ~80% and ~55% in Kokiri, ~75% and ~50% in the market). - Kokiri 44.2 fps against 32.0, the market 25.8 against 21.5; Hyrule Field, which never had the loop, is unchanged within the spread. - All six starts exited with code 0, with no error line, crash or device loss. - Not seen: the Great Fairy Fountain's floor itself. The runs are headless, and the control bridge has no teleport into a Great Fairy Fountain. **Still to test:** - The Great Fairy Fountain's floor on screen: the sand stand-in in place of the missing floor (by hand; no teleport reaches it). - Any PC with a `romfs` from a complete dump: no `is damaged in this copy` line, fps unchanged. (`@Beanstalk run Kokiri Forest on this PR and on main; check longshot_log.txt has no "is damaged in this copy" line`) - Windows, a `romfs` from another unpacker with the damaged files: the warning appears and the stand-in is read (by hand). - The Water Temple gate walls and the Ganon stage texture: the stand-ins draw in place of the missing models. The builder already serves these same stand-ins, so this has been seen in builder-made folders. With this change it has not been looked at (by hand). ## Measurements **Verdict:** on a `romfs` with the damaged textures, this PR gives Kokiri +38% fps (32.0 to 44.2) and the market +20% (21.5 to 25.8), with no overlap between the starts of the two sides in either area. Both are larger than what three starts a side resolve (Kokiri 6-12%, the market 4-14%). The loader and decode threads go from ~1.3 cores to 0 in both areas. Hyrule Field, which never had the loop, is unchanged within the spread (+1% to +3%). It matches the earlier measurement with the same stand-ins served through `HOS_FS_OVERLAY` (Kokiri 44.3, the market 27.1). Baseline: `main` @ 2210931cf. The After build is `main` @ 2210931cf with this PR's two commits cherry-picked (44defbbec; the measured head was 9d84eb80a on 1d8e73acf), so the two columns differ by this PR only. The Deck's warm shader cache fits both builds (same translator stamp). Setup: Steam Deck OLED, SteamOS 3.9.5, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), GPU at manual 1600 MHz before and after (never written). The Steam Deck developers' row: Minimum preset, Render 50%, 1280x800, frame pacing on, uncapped, vsync off, frame generation off, headless. `HOS_NATIVE_ZSTD=1 HOS_THREAD_TIMES=5`, no overlay. The `romfs` has the three damaged textures. Route `tour3` from the `kokiri-start` save: Kokiri 45 s standing, Hyrule Field (20 s standing, 4 camera pans, 30 s standing) and the market (arrival +20 s, then 60 s standing) by teleport, the order of the Field and the market alternated by round. Three starts a side, A B B A A B, fps from the game's present trace. | | Before: main @ 2210931c | After: this PR @ 44defbbe | Δ | |---|---|---|---| | kokiri fps (each start) | 28.6 / 32.3 / 32.0 (median 32.0) | 44.1 / 44.2 / 44.5 (median 44.2) | +38.1% | | kokiri 1% low fps (median) | 20.7 | 32.2 | +55.7% | | kokiri p99 frame time (median) | 46.0 ms | 28.6 ms | -17.40 ms | | kokiri frames ≥50 ms (each start) | 47 / 0 / 4 | 0 / 0 / 0 | | | kokiri `OptimizeChangeBanc` + decode CPU (median) | 79.8% + 54.8% | 0 + 0 | | | stand1 fps (each start) | 41.5 / 43.8 / 47.6 (median 43.8) | 44.3 / 40.4 / 47.3 (median 44.3) | +1.2% | | stand1 1% low fps (median) | 27.7 | 32.2 | +16.4% | | stand1 p99 frame time (median) | 32.7 ms | 30.1 ms | -2.60 ms | | turn fps (each start) | 37.0 / 35.9 / 39.1 (median 37.0) | 38.1 / 33.7 / 40.4 (median 38.1) | +2.9% | | turn 1% low fps (median) | 12.8 | 12.7 | -1.3% | | turn p99 frame time (median) | 62.3 ms | 64.6 ms | +2.30 ms | | stand2 fps (each start) | 45.9 / 43.8 / 44.7 (median 44.7) | 45.7 / 40.0 / 47.8 (median 45.7) | +2.2% | | stand2 1% low fps (median) | 21.8 | 24.6 | +12.8% | | stand2 p99 frame time (median) | 36.1 ms | 33.7 ms | -2.40 ms | | market fps (each start) | 20.0 / 21.5 / 23.9 (median 21.5) | 26.4 / 25.4 / 25.8 (median 25.8) | +20.2% | | market 1% low fps (median) | 10.1 | 12.4 | +22.9% | | market p99 frame time (median) | 84.1 ms | 67.7 ms | -16.40 ms | | market frames ≥50 ms (each start) | 467 / 401 / 205 | 81 / 147 / 136 | | | market `OptimizeChangeBanc` + decode CPU (median) | 74.8% + 50.6% | 0 + 0 | | The spread between starts of one side was about 10 to 17% of the median in these windows; a difference smaller than that is within the spread. The CPU rows are the `HOS_THREAD_TIMES` means over each window; `/proc` agrees within 1 point. ``` A: longshot v3.0.0-81-g2210931cf (source 28f0cd0e) A: report: gpu AMD Custom GPU 0932 (RADV VANGOGH), radv Mesa 26.2.0-devel (git-5c65d4fc31) A: vk: switches set (10): HOS_CONTROL=<a path> (diag), HOS_PRESENT_TRACE=<a path> (diag), HOS_OPTIONS_FILE=<a path> (diag), HOS_ENV_FILE=<a path> (diag), LONGSHOT_BUILD_NAME=dmgfix A1f (diag), HOS_HEADLESS=1 (diag), HOS_OUTPUT_RES=1280x800 (player), HOS_WINDOW_SIZE=1280x800 (tune), HOS_THREAD_TIMES=5 (diag), HOS_NATIVE_ZSTD=1 (attempt) B: longshot v3.0.0-83-g44defbbec (source 03298903) B: report: gpu AMD Custom GPU 0932 (RADV VANGOGH), radv Mesa 26.2.0-devel (git-5c65d4fc31) B: vk: switches set (10): HOS_CONTROL=<a path> (diag), HOS_PRESENT_TRACE=<a path> (diag), HOS_OPTIONS_FILE=<a path> (diag), HOS_ENV_FILE=<a path> (diag), LONGSHOT_BUILD_NAME=dmgfix B1f (diag), HOS_HEADLESS=1 (diag), HOS_OUTPUT_RES=1280x800 (player), HOS_WINDOW_SIZE=1280x800 (tune), HOS_THREAD_TIMES=5 (diag), HOS_NATIVE_ZSTD=1 (attempt) ``` The thread lines in Kokiri, `main` (A3f) and this PR (B3f): ``` A: prof: threads in 5 s: sead::MainThread 28%; Presentation Thread 43%; Resource Memory 8%; ParallelDecompThread 22%; ParallelDecompThread 32%; ... BaseProcExecutor5 14%; OptimizeChangeBanc 78%; B: prof: threads in 5 s: sead::MainThread 31%; Presentation Thread 53%; NavMeshCAStepDepWorker1 7%; ... BaseProcExecutor5 19%; ``` **Earlier, the same effect through an overlay** (before this PR's code existed): `main` @ 2210931cf plus a trace-only probe (`HOS_OCB_TRACE`), the same runtime on both sides, the three stand-ins served through `HOS_FS_OVERLAY`. Same Deck and row; Kokiri a 60 s stand, the market from arrival +20 s a 60 s stand, two starts a side. | | main, `romfs` as dumped | main, the stand-ins in `HOS_FS_OVERLAY` | |---|---|---| | Kokiri fps (run 1 / run 2) | 30.46 / 29.46 | 44.21 / 44.37 | | Kokiri `OptimizeChangeBanc` + decode CPU | 79% + 55% | 0 + 0 | | Market fps (run 1 / run 2) | 20.45 / 20.32 | 27.59 / 26.53 | | Market `OptimizeChangeBanc` + decode CPU | 74.5% + 46% / 74.5% + 44% | 0 + 0 | | Hyrule Field fps (control) | 47.84 / 47.94 | 47.84 / 47.28 | The probe's lines, without the stand-ins (Kokiri, 55 s into the stand): ``` prof: threads in 5 s: sead::MainThread 28%; Presentation Thread 44%; Resource Memory 8%; ParallelDecompThread 33%; ParallelDecompThread 22%; ... OptimizeChangeBanc 80%; ocb: in 1.00 s: loads started 0 (...), finished 0 (longest 0.0 s), cancelled 0: ...; ms in states 3 1004; changes none; thread turns 173; ...; zstd out on the host 175.4 MB; files opened 51 (not found 0, entry looks 17), the most content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs x17 (looks 17); ... ocb: entries: 852 of 6171 from 0; handles 1561 by status: 3:1 6:1560; state 7: 1, 14: 709; ...; first at 7: entry 44 handle 1 status 3 state 7 Land_FairyFountain_Base_A_01 ... ``` With them: ``` fs: OpenFile(content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs): from the overlay ocb: loaded: HyruleField_Child in 10.0 s, 852 entries ocb: in 1.01 s: loads started 0 (...), finished 0 (...), cancelled 0: ...; ms in states 4 1005; changes none; thread turns 62; ...; zstd out on the host 0.0 MB; files opened 1 ... ``` ## Risks - **What it changes:** only the open of three file names, and only when the file there doesn't start as zstd. The game can't use such a file anyway: it fails the model and asks for it again. A complete dump, a builder-made `romfs` and a mod that supplies a whole file are read as before. - **Cost:** on every open, a prefix compare (`Tex/`) and up to three string compares. For the three names, two 4-byte reads at their open, which with this change happens about once a load instead of 15 times a second. - **What a player sees:** the Great Fairy Fountain's floor, the Water Temple's gate walls and one Ganon stage texture are drawn with a similar texture instead of being left out. That is what a builder-made folder from the same dump already shows. There is also one warning a file in `longshot_log.txt`. - **No switch:** it acts only on a file the game can't read, so there is no working state to go back to. A revert is enough to undo it. - **Limits:** - Only the three textures known from the 23.3 GB dumps are covered. Another damaged file, from another dump, is still found only by the builder's scan of a `romfs` it unpacked. - Like the builder's stand-ins, the stand-in is served without a size-table entry of its own. - **After updating:** nothing in the game folder, saves or caches changes. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. *(On my Steam Deck OLED, six starts; see Tested.)* - [x] `mise run test` passes (any failure that also fails on `main` is named). *(The whole `tests/run_all.py` here and on `main` 87c0d7b4d: the same 85 failing checks on both, named under Tested.)* - [x] Measured against `main`, or Measurements says why there is nothing to measure. *(This PR's build against `main` @ 2210931cf, three starts a side on a Steam Deck OLED.)* - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A fix: a game-free test that fails on `main` and passes here. *(The updated fixture has not been reported as run.)* - [x] A check under `tests/` that reads changed source is updated, not weakened. *(`tests/test_vcore_fs_open.py`: the new call's stand-in.)* - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. *(The build was reported for the earlier head; the updated code has not been reported as built.)* - [ ] Screenshots of the same spot on `main` and here, for a change that shows on screen. *(The Great Fairy Fountain's floor, with and without the change: not yet taken. The Steam Deck runs were headless, and the control bridge has no teleport into a Great Fairy Fountain, so it needs a run by hand.)* - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
`tests/test_damaged_rom_files.py` compiles `host/nn_fs.c`'s OpenFile over a made-up romfs: a texture that starts with
zeros, under one of the three names the 23.3 GB dumps damage, opens its stand-in's bytes and says so once; a whole one
and any other file are read as they are; a missing stand-in is a warning. It also checks the host's pairs against
`nx2/nup.py` `stand_in_for`. Fails on main: the host serves the damaged file as it is.

Assisted-by: Claude
fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
ee110c025f
The 23.3 GB dumps hold three `.zs` textures that start with zeros. The builder copies a stand-in over each only in a
romfs it unpacked (`.longshot_romfs`); a romfs from another tool kept them, and the game asked for the Great Fairy
Fountain's model ~15 times a second in Kokiri and the Market (~1.3 cores on the Steam Deck).

`host/oot_damaged_files.c`: when OpenFile opens one of the three and it does not start as zstd, the stand-in
`nup.stand_in_for` picks is read in its place and one warning names both. Nothing is written to the player's files.
`tests/test_vcore_fs_open.py` gets the new call's stand-in.

Assisted-by: Claude
Author
Contributor

Tested in game: Tested in the game on a Steam Deck OLED (SteamOS 3.9.5, Mesa RADV, GPU fixed at 1600 MHz), at the Steam Deck developers' settings (Minimum preset, Render 50%, 1280x800, uncapped), with a copy of the game that has the three damaged textures. I ran this PR and main 2210931cf three times each, taking turns, through Kokiri Forest, Hyrule Field and the market. Every start of this PR wrote the new warning once, for the Great Fairy Fountain floor texture, and said a similar sand texture was read in its place. No start of main wrote it. The other two damaged textures (Water Temple and Ganon) were not opened on this route, so they had no warning. With the fix, the background loader and the decode threads, which used about 1.3 CPU cores on main in Kokiri and the market, dropped to zero. Kokiri went from 32 to 44 fps and the market from 21.5 to 26 fps; every start of the fix was faster than every start of main there. Hyrule Field, which never had the problem, stayed the same. All six starts ended normally, with no errors or crashes. I did not see the fountain floor itself: the runs had no screen, and the test tools cannot teleport into a Great Fairy Fountain.

Tested in game: Tested in the game on a Steam Deck OLED (SteamOS 3.9.5, Mesa RADV, GPU fixed at 1600 MHz), at the Steam Deck developers' settings (Minimum preset, Render 50%, 1280x800, uncapped), with a copy of the game that has the three damaged textures. I ran this PR and main 2210931cf three times each, taking turns, through Kokiri Forest, Hyrule Field and the market. Every start of this PR wrote the new warning once, for the Great Fairy Fountain floor texture, and said a similar sand texture was read in its place. No start of main wrote it. The other two damaged textures (Water Temple and Ganon) were not opened on this route, so they had no warning. With the fix, the background loader and the decode threads, which used about 1.3 CPU cores on main in Kokiri and the market, dropped to zero. Kokiri went from 32 to 44 fps and the market from 21.5 to 26 fps; every start of the fix was faster than every start of main there. Hyrule Field, which never had the problem, stayed the same. All six starts ended normally, with no errors or crashes. I did not see the fountain floor itself: the runs had no screen, and the test tools cannot teleport into a Great Fairy Fountain.
Member

Possibly related, for context:

  • issue #1939 Game: Kokiri Forest and the market run ~30% slower when the game files weren't unpacked by Longshot — Issue this PR explicitly fixes
  • PR #546 WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open build-time stand-in handling overlaps same damaged-texture cases
Possibly related, for context: - issue #1939 Game: Kokiri Forest and the market run ~30% slower when the game files weren't unpacked by Longshot — Issue this PR explicitly fixes - PR #546 WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open build-time stand-in handling overlaps same damaged-texture cases
BeanBot left a comment

Recommendation: Request changes. The Deck runs show that serving the stand-in stops the preload retry loop, but the implementation misses supported Windows paths and can misdiagnose mod, overlay, and save files as a bad dump. The test also fails to establish that the runtime and builder choose the same replacements.

Players with the three known zero-header textures can recover models and frame rate without rebuilding or modifying their game folder. The fix belongs in the host read path, but the current design opens matching files twice, checks a different handle from the one it serves, and applies the dump diagnosis beyond the base RomFS.

Does it help: Yes on the reported damaged Steam Deck OLED setup. Three interleaved starts per side against 2210931cf moved median Kokiri FPS from 32.0 to 44.2 and market FPS from 21.5 to 25.8. Median 1% lows improved from 20.7 to 32.2 and 10.1 to 12.4, and p99 fell from 46.0 to 28.6 ms and 84.1 to 67.7 ms. About 1.3 cores of retry and decode work disappeared. The Hyrule Field runs overlap. The reported after-build and source hashes are unavailable in the checkout, so the result for ee110c025f against its merge base and the cost for complete dumps remain unverified.
Other systems: All GPU vendors and both OSes share the host open path. The Deck run covers RADV and one of three textures. Windows compile results do not exercise its path APIs, and there are no Tier 1 desktop runs. The change can replace mod, overlay, and writable-mount reads, and the resource size table still budgets the original guest name. The fountain, Water Temple, and Ganon visuals remain unverified through the runtime path. The builder change is a docstring, write-mode opens bypass the new branch, and there are no direct changes to audio, input, shaders, caches, or the renderer ABI.
Switches: None added, changed, or removed. The existing HOS_FS_OVERLAY lookup still runs first, but the new fallback can supersede its chosen file.

Reviewed: ee110c025f against 87c0d7b4d6: Read the full seven-file diff, both commits, the complete new C file and Python test, and the changed sections of nn_fs.c, make_game.py, test_vcore_fs_open.py, and both docs. Read the relevant mount, mod, handle, Windows and POSIX file API, builder selector, source-check, and build source-discovery code. Read AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards. The review-checklist file and sibling Tooling checkout were absent. Skimmed recorded-run references and reviewed the supplied run tables and excerpts. No raw traces, CI results, or tester reviews were supplied. Unrelated parts of the large source files were outside the review.; nothing run (a code review).
Checked and fine:

  • One topic with a Conventional Commit title after the WIP marker. Both commits carry assistance trailers.
  • Game-specific logic lives in its own source file, picked up by the existing host source discovery. The diff contains no game bytes, generated output, or third-party changes.
  • The synthetic test exercises returned bytes and warnings through OpenFile. The existing virtual-core test retains its assertions, and the new I/O runs while the core is released and outside the mount and handle locks.
  • The replacement path has a capacity check, temporary probe handles are closed, and the warning counter uses atomic access. The test uses local temporary files and a local compiler with no downloads.

Blocking (to change before merge)

  1. host/oot_damaged_files.c:28: The probe uses plain CreateFileA while OpenFile uses the UTF-8 wide wrapper with extended Windows paths. A damaged texture under a path past MAX_PATH can open normally but skip the fix because its probe failed. Use the same open path for both probes and reads, and cover a long Windows path with non-ASCII components.
  2. host/nn_fs.c:429: Read mode alone does not identify a base RomFS file. A complete dump with a zero-header fault-injection overlay and a sibling stand-in will silently serve the sibling and tell the player to dump the game again. A read under a writable mount with the same relative name is also eligible. Carry the source and validated relative name from path resolution and apply dump repair only to the base RomFS. Cover overlay and mod precedence and another mount in the test.
  3. host/oot_damaged_files.c:50: The check closes each probe handle and returns a path for a separate open. If the substitute is removed or atomically replaced between those calls, the game can get a not-found result or unchecked bytes after a warning that the substitute was read. Open the original once, probe at offset zero, and keep that handle unless an already-open, validated substitute is ready to serve. This also removes the duplicate open on complete dumps.
  4. docs/user/common-issues.md:141: The guide promises a substitute for each damaged texture. The runtime handles three names with a bad magic header and an available substitute. A folder carried over from an older builder can contain damage still wrapped in the NUP's valid zstd frame, which the builder detects and this probe passes through. State these limits here and in the builder docstring, or add equivalent detection if those folders are in scope.
  5. tests/test_damaged_rom_files.py:207: The selector fixture creates its candidates from the C table it is meant to check. A wrong Water Temple stand-in with a matching suffix and strong prefix can pass because the intended candidate was never created, and the runtime fixture only opens that texture as missing. Use an independent candidate fixture with plausible competitors and assert the bytes served for all three successful substitutions.

Waiting on the author (a run, a measurement or a decision; no edit does it)

  • Tie the recorded gain to the reviewed diff and current main. The description names 9d84eb80a and 44defbbec, while the checkout is ee110c025f on 87c0d7b4 and local main has advanced. Supply evidence that the tested patches match and the numbers still hold, or rerun the interleaved comparison on the updated head.
  • Measure the healthy path too. A complete or builder-repaired RomFS pays the added checks and, as written, another open and header read per matching texture. The damaged Deck runs cannot isolate that cost. Report cold and warm relevant load times and frame pacing on a complete dump with normal player settings.
  • Finish the listed Windows and affected Tier 1 runs, and attach before-and-after screenshots for the visible substitutions. Exercise all three runtime stand-ins in their scenes to check that they render and fit the original resource budgets. Record head CI and lint results and identify the shared suite failures in the results.

Testing: Read-only review. The author reports clang++ and g++ fixture passes and Linux and llvm-mingw compilation. The reported suite still has 85 failures and 28 skips, with shared failures described in aggregate. The fixture uses a stdio stand-in and does not exercise the production Windows path layer. Green head CI, healthy-path timing, and the listed in-game runs remain pending.
Overlaps and merge order: No overlapping pull request identified in the available history. Local main has advanced beyond the merge base.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The Deck runs show that serving the stand-in stops the preload retry loop, but the implementation misses supported Windows paths and can misdiagnose mod, overlay, and save files as a bad dump. The test also fails to establish that the runtime and builder choose the same replacements. Players with the three known zero-header textures can recover models and frame rate without rebuilding or modifying their game folder. The fix belongs in the host read path, but the current design opens matching files twice, checks a different handle from the one it serves, and applies the dump diagnosis beyond the base RomFS. **Does it help:** Yes on the reported damaged Steam Deck OLED setup. Three interleaved starts per side against 2210931cf moved median Kokiri FPS from 32.0 to 44.2 and market FPS from 21.5 to 25.8. Median 1% lows improved from 20.7 to 32.2 and 10.1 to 12.4, and p99 fell from 46.0 to 28.6 ms and 84.1 to 67.7 ms. About 1.3 cores of retry and decode work disappeared. The Hyrule Field runs overlap. The reported after-build and source hashes are unavailable in the checkout, so the result for ee110c025fd3 against its merge base and the cost for complete dumps remain unverified. **Other systems:** All GPU vendors and both OSes share the host open path. The Deck run covers RADV and one of three textures. Windows compile results do not exercise its path APIs, and there are no Tier 1 desktop runs. The change can replace mod, overlay, and writable-mount reads, and the resource size table still budgets the original guest name. The fountain, Water Temple, and Ganon visuals remain unverified through the runtime path. The builder change is a docstring, write-mode opens bypass the new branch, and there are no direct changes to audio, input, shaders, caches, or the renderer ABI. **Switches:** None added, changed, or removed. The existing HOS_FS_OVERLAY lookup still runs first, but the new fallback can supersede its chosen file. **Reviewed:** ee110c025fd3 against 87c0d7b4d68d: Read the full seven-file diff, both commits, the complete new C file and Python test, and the changed sections of nn_fs.c, make_game.py, test_vcore_fs_open.py, and both docs. Read the relevant mount, mod, handle, Windows and POSIX file API, builder selector, source-check, and build source-discovery code. Read AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards. The review-checklist file and sibling Tooling checkout were absent. Skimmed recorded-run references and reviewed the supplied run tables and excerpts. No raw traces, CI results, or tester reviews were supplied. Unrelated parts of the large source files were outside the review.; nothing run (a code review). **Checked and fine:** - One topic with a Conventional Commit title after the WIP marker. Both commits carry assistance trailers. - Game-specific logic lives in its own source file, picked up by the existing host source discovery. The diff contains no game bytes, generated output, or third-party changes. - The synthetic test exercises returned bytes and warnings through OpenFile. The existing virtual-core test retains its assertions, and the new I/O runs while the core is released and outside the mount and handle locks. - The replacement path has a capacity check, temporary probe handles are closed, and the warning counter uses atomic access. The test uses local temporary files and a local compiler with no downloads. **Blocking** (to change before merge) 1. host/oot_damaged_files.c:28: The probe uses plain CreateFileA while OpenFile uses the UTF-8 wide wrapper with extended Windows paths. A damaged texture under a path past MAX_PATH can open normally but skip the fix because its probe failed. Use the same open path for both probes and reads, and cover a long Windows path with non-ASCII components. 2. host/nn_fs.c:429: Read mode alone does not identify a base RomFS file. A complete dump with a zero-header fault-injection overlay and a sibling stand-in will silently serve the sibling and tell the player to dump the game again. A read under a writable mount with the same relative name is also eligible. Carry the source and validated relative name from path resolution and apply dump repair only to the base RomFS. Cover overlay and mod precedence and another mount in the test. 3. host/oot_damaged_files.c:50: The check closes each probe handle and returns a path for a separate open. If the substitute is removed or atomically replaced between those calls, the game can get a not-found result or unchecked bytes after a warning that the substitute was read. Open the original once, probe at offset zero, and keep that handle unless an already-open, validated substitute is ready to serve. This also removes the duplicate open on complete dumps. 4. docs/user/common-issues.md:141: The guide promises a substitute for each damaged texture. The runtime handles three names with a bad magic header and an available substitute. A folder carried over from an older builder can contain damage still wrapped in the NUP's valid zstd frame, which the builder detects and this probe passes through. State these limits here and in the builder docstring, or add equivalent detection if those folders are in scope. 5. tests/test_damaged_rom_files.py:207: The selector fixture creates its candidates from the C table it is meant to check. A wrong Water Temple stand-in with a matching suffix and strong prefix can pass because the intended candidate was never created, and the runtime fixture only opens that texture as missing. Use an independent candidate fixture with plausible competitors and assert the bytes served for all three successful substitutions. **Waiting on the author** (a run, a measurement or a decision; no edit does it) - Tie the recorded gain to the reviewed diff and current main. The description names 9d84eb80a and 44defbbec, while the checkout is ee110c025fd3 on 87c0d7b4 and local main has advanced. Supply evidence that the tested patches match and the numbers still hold, or rerun the interleaved comparison on the updated head. - Measure the healthy path too. A complete or builder-repaired RomFS pays the added checks and, as written, another open and header read per matching texture. The damaged Deck runs cannot isolate that cost. Report cold and warm relevant load times and frame pacing on a complete dump with normal player settings. - Finish the listed Windows and affected Tier 1 runs, and attach before-and-after screenshots for the visible substitutions. Exercise all three runtime stand-ins in their scenes to check that they render and fit the original resource budgets. Record head CI and lint results and identify the shared suite failures in the results. **Testing:** Read-only review. The author reports clang++ and g++ fixture passes and Linux and llvm-mingw compilation. The reported suite still has 85 failures and 28 skips, with shared failures described in aggregate. The fixture uses a stdio stand-in and does not exercise the production Windows path layer. Green head CI, healthy-path timing, and the listed in-game runs remain pending. **Overlaps and merge order:** No overlapping pull request identified in the available history. Local main has advanced beyond the merge base. --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -139,1 +139,3 @@
2. **Dump the game again** and build from the new dump. The file is removed once the dump is whole.
2. **Look in `longshot_log.txt` for `is damaged in this copy of the game`.** Game files that Longshot Builder didn't
unpack (copied from another tool, or by hand) get no `longshot_damaged_files.txt`. The game then reads a similar
texture in place of each damaged one as it plays, writes nothing to your game files, and logs one line per file.
Member

Qualify this to the three known bad-header textures with available stand-ins. Damage wrapped in an older builder's valid NUP zstd frame still passes through.

Qualify this to the three known bad-header textures with available stand-ins. Damage wrapped in an older builder's valid NUP zstd frame still passes through.
host/nn_fs.c Outdated
@ -426,2 +426,4 @@
}
}
{ int hos_stand_in_damaged(const char *guest, char *path, size_t cap); /* (oot_damaged_files.c: an incomplete dump) */
if (!want_write) hos_stand_in_damaged(g, path, sizeof path); }
Member

Gate this on the resolved base RomFS source and use its validated relative name. A damaged test overlay over a complete dump can currently trigger substitution and a false redump warning. Reads from writable mounts are eligible too.

Gate this on the resolved base RomFS source and use its validated relative name. A damaged test overlay over a complete dump can currently trigger substitution and a false redump warning. Reads from writable mounts are eligible too.
@ -0,0 +25,4 @@
/* 1: the file starts as zstd data (a frame, or a skippable frame: nup.py zstd_start); 0: it does not; -1: it could not
be read, and is left to the open that follows */
static int starts_as_zstd(const char *path) {
HANDLE h = CreateFileA(path, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
Member

Use the same wide, extended-length open as OpenFile. A long Windows RomFS path can open normally while this probe fails and leaves the retry loop intact.

Use the same wide, extended-length open as OpenFile. A long Windows RomFS path can open normally while this probe fails and leaves the retry loop intact.
@ -0,0 +47,4 @@
if (strncmp(rel, "Tex/", 4)) return 0;
for (size_t i = 0; i < STAND_INS; i++) {
if (strcmp(rel, stand_ins[i].damaged)) continue;
if (starts_as_zstd(path)) return 0;
Member

Probe and serve the same opened handle. A replacement can disappear or change after its probe, and reopening also adds I/O on every matching healthy texture.

Probe and serve the same opened handle. A replacement can disappear or change after its probe, and reopening also adds I/O on every matching healthy texture.
@ -0,0 +204,4 @@
with tempfile.TemporaryDirectory(prefix="damaged-tex-") as temporary:
# (nup.stand_in_for's choice among the six: the damaged files as in the dump, zeros first; their stand-ins whole)
tex = Path(temporary) / "Tex"
for damaged, stand_in in pairs:
Member

Build the candidate set independently of the C table. With this fixture a wrong Gate mapping can supply its own winning candidate, and the runtime case only tests that Gate is missing.

Build the candidate set independently of the C table. With this fixture a wrong Gate mapping can supply its own winning candidate, and the runtime case only tests that Gate is missing.
Author
Contributor

Ran the whole check suite at this head and on main 87c0d7b4d: the same 85 checks fail on both, and they are now named in the description. One box is left before this can be marked ready: a screenshot of the Great Fairy Fountain floor with the stand-in texture.

Ran the whole check suite at this head and on main 87c0d7b4d: the same 85 checks fail on both, and they are now named in the description. One box is left before this can be marked ready: a screenshot of the Great Fairy Fountain floor with the stand-in texture.
Author
Contributor

Rebased onto main fd7dc40db (head 9d9d67ee5); the two commits applied without a conflict and the fix's checks pass at this head.

Rebased onto main fd7dc40db (head 9d9d67ee5); the two commits applied without a conflict and the fix's checks pass at this head.
zeldaootr force-pushed damaged-romfs-files-any-unpacker from ee110c025f
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 9d9d67ee51
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 10s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-21 06:54:16 +00:00
Compare
Owner
@fix
BeanBot force-pushed damaged-romfs-files-any-unpacker from 9d9d67ee51
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 10s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 4766cfa7c1
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 28s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 42s
CI / Workflow lint (actionlint) (pull_request) Successful in 47s
Third-party parts / As their notes say (pull_request) Successful in 45s
PR description / Description follows the template (pull_request_target) Failing after 40s
CI / Website build and checks (pull_request) Failing after 4m6s
2026-11-21 17:33:41 +00:00
Compare
BeanBot left a comment

Recommendation: Approve. The Steam Deck measurements show a substantial improvement for the damaged dump, but the requested visual and platform checks remain outstanding. Nothing blocking is left: what remains is for the author before merging (below). Recommend approval; not approved on the forge.

What @fix changed
I changed this pull request: the new head is 4766cfa7c1. I rebased it onto main (577bba1225). I edited its description to match the change.

Changed

  • Windows path handling in host/oot_damaged_files.c:28: The probe now uses the host’s wide, long-path-aware open helper. The fixture covers a non-ASCII path longer than the Windows path limit. Tests were not run, as requested.
  • Stand-in lookup in host/oot_damaged_files.c:50: Stand-ins now follow the mod, overlay and game-file lookup policy. Added fixture cases for damaged mod and overlay overrides with stand-ins only in the game folder.
  • Windows fixture in tests/test_damaged_rom_files.py:106: The Windows fixture uses the host’s UTF-8-to-wide conversion and Windows file APIs, with a non-ASCII temporary path.

Nothing was built or run: the checks run again on the new head.

The host substitutes known damaged textures with similar files without changing the game folder. This avoids the repeated loading observed on an incomplete dump, at the cost of showing substitute textures.

Before merging, for the author (1) (a run, a measurement or a decision; it doesn't hold the recommendation)

  • L4 (minor): The fountain floor has not been viewed, and the Water Temple and Ganon substitutions have not been viewed with this change. Complete the stated in-game visual checks and the remaining Windows and complete-dump runs; include the requested before-and-after screenshots.

Checks: they run again on this head; a failure shows on the pull request.

Also noticed (3; not asked for, not blocking)
  • The pull request description still says Windows paths over about 200 characters cannot be probed, although the probe now uses the host’s long-path wrapper.
  • The updated fixture and host builds have not been reported as run at this head.
  • The supplied base-to-HEAD diff includes many unrelated merged commits.
More: does it help, other systems, switches, what was read

Does it help: Yes on the measured Steam Deck setup: median Kokiri fps rose from 32.0 to 44.2 and market fps from 21.5 to 25.8 over three interleaved starts per side. These measurements used an earlier version of the change.

Other systems: The lookup affects Windows and Linux, including mod and overlay files. The Windows damaged-dump run, complete-dump run, and on-screen checks of all three substitutions remain outstanding.

Switches: None added, changed, or removed.

Reviewed: 4766cfa7c1 against 577bba1225: Read AGENTS.md, CONTRIBUTING.md, the pull request template, host/oot_damaged_files.c, the relevant OpenFile and path-resolution code in host/nn_fs.c, the damaged-file fixture, and the fix commit diff. Skimmed the supplied measurements and the base-to-HEAD diff; the latter includes many unrelated merged commits and was not read in full. Tooling's AGENTS.md is not present locally.; nothing run (a code review).

Checked and fine:

  • The description preserves the outstanding checks and distinguishes the earlier measurements from the current code.
  • The stand-in lookup uses the host path resolver and the same read helper as OpenFile.

Testing: No commands that build or run project code were executed. CI has not run on this head; the author reports earlier game and suite results in the description.

Overlaps and merge order: none found


Automated code review by BeanBot (gpt-6.1-terra), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.

**Recommendation: Approve.** The Steam Deck measurements show a substantial improvement for the damaged dump, but the requested visual and platform checks remain outstanding. Nothing blocking is left: what remains is for the author before merging (below). Recommend approval; not approved on the forge. **What `@fix` changed** I changed this pull request: the new head is 4766cfa7c1eb. I rebased it onto `main` (577bba122541). I edited its description to match the change. **Changed** - **Windows path handling in host/oot_damaged_files.c:28**: The probe now uses the host’s wide, long-path-aware open helper. The fixture covers a non-ASCII path longer than the Windows path limit. Tests were not run, as requested. - **Stand-in lookup in host/oot_damaged_files.c:50**: Stand-ins now follow the mod, overlay and game-file lookup policy. Added fixture cases for damaged mod and overlay overrides with stand-ins only in the game folder. - **Windows fixture in tests/test_damaged_rom_files.py:106**: The Windows fixture uses the host’s UTF-8-to-wide conversion and Windows file APIs, with a non-ASCII temporary path. Nothing was built or run: the checks run again on the new head. The host substitutes known damaged textures with similar files without changing the game folder. This avoids the repeated loading observed on an incomplete dump, at the cost of showing substitute textures. **Before merging, for the author (1)** (a run, a measurement or a decision; it doesn't hold the recommendation) - **L4** (minor): The fountain floor has not been viewed, and the Water Temple and Ganon substitutions have not been viewed with this change. Complete the stated in-game visual checks and the remaining Windows and complete-dump runs; include the requested before-and-after screenshots. **Checks:** they run again on this head; a failure shows on the pull request. <details> <summary>Also noticed (3; not asked for, not blocking)</summary> - The pull request description still says Windows paths over about 200 characters cannot be probed, although the probe now uses the host’s long-path wrapper. - The updated fixture and host builds have not been reported as run at this head. - The supplied base-to-HEAD diff includes many unrelated merged commits. </details> <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Yes on the measured Steam Deck setup: median Kokiri fps rose from 32.0 to 44.2 and market fps from 21.5 to 25.8 over three interleaved starts per side. These measurements used an earlier version of the change. **Other systems:** The lookup affects Windows and Linux, including mod and overlay files. The Windows damaged-dump run, complete-dump run, and on-screen checks of all three substitutions remain outstanding. **Switches:** None added, changed, or removed. **Reviewed:** 4766cfa7c1eb against 577bba122541: Read AGENTS.md, CONTRIBUTING.md, the pull request template, host/oot_damaged_files.c, the relevant OpenFile and path-resolution code in host/nn_fs.c, the damaged-file fixture, and the fix commit diff. Skimmed the supplied measurements and the base-to-HEAD diff; the latter includes many unrelated merged commits and was not read in full. Tooling's AGENTS.md is not present locally.; nothing run (a code review). **Checked and fine:** - The description preserves the outstanding checks and distinguishes the earlier measurements from the current code. - The stand-in lookup uses the host path resolver and the same read helper as OpenFile. **Testing:** No commands that build or run project code were executed. CI has not run on this head; the author reports earlier game and suite results in the description. **Overlaps and merge order:** none found </details> --- <sub>Automated code review by BeanBot (gpt-6.1-terra), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.</sub>
HumanBean changed title from WIP: fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's to fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's 2026-11-21 17:41:15 +00:00
HumanBean changed title from WIP: fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's to fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's 2026-11-21 17:41:16 +00:00
HumanBean deleted branch damaged-romfs-files-any-unpacker 2026-11-21 17:42:40 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1943
No description provided.