fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's #1943
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1943
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:damaged-romfs-files-any-unpacker"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Fixes #1939.
A game folder whose game files Longshot Builder didn't unpack keeps the three damaged textures of an incomplete dump.
That covers a folder carried over from Hookshot, game files unpacked by another tool and chosen as folders, and a
romfscopied by hand. The game then keeps reloading a model that uses one of them, for as long as Link is in KokiriForest or the market. On a Steam Deck that costs about 1.3 CPU cores and a third of the frame rate (Kokiri 30 fps
instead of 44). Nothing in the log says why.
The bug on
main: take a game folder whoseromfs/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zsstarts with00 00 00 00instead of28 b5 2f fd. Stand in Kokiri withHOS_THREAD_TIMES=5. Theprof: threads in 5 s:linesshow
OptimizeChangeBancat ~80% and the twoParallelDecompThreadentries at ~55% together. WithHOS_OCB_TRACE(aprobe build), the Hyrule Field preload never finishes: the handle for
Land_FairyFountain_Base_A_01fails about 15times a second.
Why: the 23.3 GB dumps have three
.zstextures that begin with zeros. The builder copies a similar texture overeach one (
nx2/nup.pystand_in_damaged, fromnx2/make_game.pystand_in_damaged_files). It does this only in aromfsit unpacked itself, the one with the.longshot_romfsstamp. Nothing else checks those files.This change: when the game opens one of the three textures and it doesn't start as a zstd frame, the host reads the
stand-in the builder would have used. That is the file
nup.stand_in_forpicks, resolved through the usual mod, overlay and game-file lookups. One warningnames both files and says the dump is incomplete:
The host writes nothing to the player's files. A whole file (a complete dump, or a
romfsthe builder already fixed)is read as it is.
Why the game and not the builder: existing installs are fixed without the player doing anything, whichever
launcher, builder or tool made the folder. The builder and
nup.pysay they never write to aromfsthey didn'tunpack, so a one-time repair there would break that rule, and a warning alone would leave the cost in place.
Read first:
host/oot_damaged_files.c: the three pairs andhos_stand_in_damaged;host/nn_fs.cOpenFile, after the mods andHOS_FS_OVERLAYlookups;tests/test_damaged_rom_files.py.Testing
Tested:
tests/test_damaged_rom_files.pycompiles the real OpenFile andhost/oot_damaged_files.cover a made-up romfs (nogame bytes). It fails on
main(the damaged file's zeros are served, and no line is logged) and passed on the earlier head 9d84eb80a, withclang++ and g++. The updated fixture has not been reported as run. It checks:
nup.stand_in_forpicks among them.Mutations: dropping the zstd check of the opened file, or the check that the stand-in is whole, each fails the test.
The whole suite (
python3 -I -B tests/run_all.py, Linux x86-64, 16 at once), at the earlier head 9d84eb80a and onmain87c0d7b4d(its base), back to back:786 passed, 86 failed, 28 skipped here; 785 / 86 / 28 on
main. The failing checks are the same 85 names on both (onefailure line carries no name on either side); none is this PR's.
The 85 checks that fail both here and on main
87c0d7b4dtest_appimage_player_build,test_bug_report_defaults,test_capture_readback_visibility,test_capture_read_once,test_crash_dialog,test_crash_record,test_crt_options,test_damage_debug_menu,test_dev_ab,test_device_lost_exit,test_dev_packs,test_discover,test_display_startup_failure,test_dlss_feature_size,test_draw_binds,test_draw_stream_visibility,test_failed_compile_keeps_objects,test_flush_nothing_held,test_frame_overlap,test_fsr3fi_hud_layer,test_fsr_image_retirement,test_fsr_uniform_ownership,test_gpu_wait_stats,test_guest_imported,test_high_fps_fixes,test_host_guest_writes,test_image_reaper_threads,test_libc_hooks,test_lift_cache,test_lift_lookups,test_linux_preflight,test_linux_pressure,test_linux_tools,test_linux_update,test_log_open_failure,test_low_vram_texture_mips,test_make_game_cache,test_make_game_ngx,test_menu_module_slots,test_menu_modules,test_menu_save_atomic,test_menu_sections,test_mise_build,test_missing_nup,test_mods_list,test_mods_size_table,test_ngx_sdk,test_nonuniform_image_arrays,test_oot_shadow_fit,test_pads_after_menu,test_pass_barrier,test_pending_guest_import,test_pipeline_play_cache,test_prepare_cache_save,test_present_queue_default,test_present_semaphores,test_present_source_crop,test_profiling_mode,test_program_cache_cleanup,test_program_cache_one_variant,test_psychov35_fsr3_fg,test_readback_timeout_source,test_recorder_own_idle,test_recorder_sync,test_reflex_swapchain_lifetime,test_renderer_new_handler,test_renhdrum,test_replay_write_order,test_sampler_admission,test_settings_table,test_slow_frames,test_song_note_cue,test_sparse_mirror,test_spirv_pack,test_staging_burst,test_target_reuse_memory,test_texel_buffer_ownership,test_thinlto_tls,test_ub_update_batch,test_ui_record,test_upscaler_rows,test_vcore_escape,test_vcore_mask0,test_vk_occlusion_queries,test_walkAt the earlier head 9d84eb80a,
host/oot_damaged_files.candhost/nn_fs.ccompiled with-Wall -Wextraunder clang and gcc for Linux andllvm-mingw for Windows. The new file has no warnings.
nn_fs.chas the same 37 warnings as onmain, all-Wmissing-field-initializersfrom the HLE macro.In the game, Steam Deck OLED (SteamOS 3.9.5, Mesa RADV 26.2.0-devel, GPU at manual 1600 MHz, headless), with a
romfsthat has the three damaged textures (an unpacker's output, not the builder's), no overlay. This PR's buildagainst
main@2210931cf, three starts each, alternated (A B B A A B), on the route Kokiri, Hyrule Field, the market(Measurements has the settings and the table). What was seen:
(the Hyrule Field preload opens it). No start of
mainlogged it. It was the only warning in all six logs:OptimizeChangeBancandParallelDecompThreadare gone from theprof: threadslines in Kokiri and the market(on
main: ~80% and ~55% in Kokiri, ~75% and ~50% in the market).within the spread.
into a Great Fairy Fountain.
Still to test:
reaches it).
romfsfrom a complete dump: nois damaged in this copyline, fps unchanged.(
@Beanstalk run Kokiri Forest on this PR and on main; check longshot_log.txt has no "is damaged in this copy" line)romfsfrom another unpacker with the damaged files: the warning appears and the stand-in is read (by hand).builder already serves these same stand-ins, so this has been seen in builder-made folders. With this change it has not
been looked at (by hand).
Measurements
Verdict: on a
romfswith the damaged textures, this PR gives Kokiri +38% fps (32.0 to 44.2) and the market +20%(21.5 to 25.8), with no overlap between the starts of the two sides in either area. Both are larger than what three
starts a side resolve (Kokiri 6-12%, the market 4-14%). The loader and decode threads go from ~1.3 cores to 0 in both
areas. Hyrule Field, which never had the loop, is unchanged within the spread (+1% to +3%). It matches the earlier
measurement with the same stand-ins served through
HOS_FS_OVERLAY(Kokiri 44.3, the market 27.1).Baseline:
main@2210931cf. The After build ismain@2210931cfwith this PR's two commits cherry-picked(44defbbec; the measured head was 9d84eb80a on
1d8e73acf), so the two columns differ by this PR only. The Deck's warmshader cache fits both builds (same translator stamp).
Setup: Steam Deck OLED, SteamOS 3.9.5, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), GPU at
manual 1600 MHz before and after (never written). The Steam Deck developers' row: Minimum preset, Render 50%, 1280x800,
frame pacing on, uncapped, vsync off, frame generation off, headless.
HOS_NATIVE_ZSTD=1 HOS_THREAD_TIMES=5, nooverlay. The
romfshas the three damaged textures. Routetour3from thekokiri-startsave: Kokiri 45 s standing,Hyrule Field (20 s standing, 4 camera pans, 30 s standing) and the market (arrival +20 s, then 60 s standing) by
teleport, the order of the Field and the market alternated by round. Three starts a side, A B B A A B, fps from the
game's present trace.
2210931cOptimizeChangeBanc+ decode CPU (median)OptimizeChangeBanc+ decode CPU (median)The spread between starts of one side was about 10 to 17% of the median in these windows; a difference smaller than
that is within the spread. The CPU rows are the
HOS_THREAD_TIMESmeans over each window;/procagrees within 1 point.The thread lines in Kokiri,
main(A3f) and this PR (B3f):Earlier, the same effect through an overlay (before this PR's code existed):
main@2210931cfplus a trace-onlyprobe (
HOS_OCB_TRACE), the same runtime on both sides, the three stand-ins served throughHOS_FS_OVERLAY. SameDeck and row; Kokiri a 60 s stand, the market from arrival +20 s a 60 s stand, two starts a side.
romfsas dumpedHOS_FS_OVERLAYOptimizeChangeBanc+ decode CPUOptimizeChangeBanc+ decode CPUThe probe's lines, without the stand-ins (Kokiri, 55 s into the stand):
With them:
Risks
can't use such a file anyway: it fails the model and asks for it again. A complete dump, a builder-made
romfsand amod that supplies a whole file are read as before.
Tex/) and up to three string compares. For the three names, two 4-bytereads at their open, which with this change happens about once a load instead of 15 times a second.
are drawn with a similar texture instead of being left out. That is what a builder-made folder from the same dump
already shows. There is also one warning a file in
longshot_log.txt.enough to undo it.
still found only by the builder's scan of a
romfsit unpacked.Checklist
Ready for review
main; the title is a Conventional Commit.starts; see Tested.)
mise run testpasses (any failure that also fails onmainis named). (The wholetests/run_all.pyhere and onmain87c0d7b4d: the same 85 failing checks on both, named under Tested.)main, or Measurements says why there is nothing to measure. (This PR's build againstmain@
2210931cf, three starts a side on a Steam Deck OLED.)third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here. (The updated fixture has not been reported as run.)tests/that reads changed source is updated, not weakened. (tests/test_vcore_fs_open.py: thenew call's stand-in.)
mainand here, for a change that shows on screen. (The Great Fairy Fountain'sfloor, with and without the change: not yet taken. The Steam Deck runs were headless, and the control bridge has
no teleport into a Great Fairy Fountain, so it needs a run by hand.)
Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Tested in game: Tested in the game on a Steam Deck OLED (SteamOS 3.9.5, Mesa RADV, GPU fixed at 1600 MHz), at the Steam Deck developers' settings (Minimum preset, Render 50%, 1280x800, uncapped), with a copy of the game that has the three damaged textures. I ran this PR and main
2210931cfthree times each, taking turns, through Kokiri Forest, Hyrule Field and the market. Every start of this PR wrote the new warning once, for the Great Fairy Fountain floor texture, and said a similar sand texture was read in its place. No start of main wrote it. The other two damaged textures (Water Temple and Ganon) were not opened on this route, so they had no warning. With the fix, the background loader and the decode threads, which used about 1.3 CPU cores on main in Kokiri and the market, dropped to zero. Kokiri went from 32 to 44 fps and the market from 21.5 to 26 fps; every start of the fix was faster than every start of main there. Hyrule Field, which never had the problem, stayed the same. All six starts ended normally, with no errors or crashes. I did not see the fountain floor itself: the runs had no screen, and the test tools cannot teleport into a Great Fairy Fountain.Possibly related, for context:
Recommendation: Request changes. The Deck runs show that serving the stand-in stops the preload retry loop, but the implementation misses supported Windows paths and can misdiagnose mod, overlay, and save files as a bad dump. The test also fails to establish that the runtime and builder choose the same replacements.
Players with the three known zero-header textures can recover models and frame rate without rebuilding or modifying their game folder. The fix belongs in the host read path, but the current design opens matching files twice, checks a different handle from the one it serves, and applies the dump diagnosis beyond the base RomFS.
Does it help: Yes on the reported damaged Steam Deck OLED setup. Three interleaved starts per side against
2210931cfmoved median Kokiri FPS from 32.0 to 44.2 and market FPS from 21.5 to 25.8. Median 1% lows improved from 20.7 to 32.2 and 10.1 to 12.4, and p99 fell from 46.0 to 28.6 ms and 84.1 to 67.7 ms. About 1.3 cores of retry and decode work disappeared. The Hyrule Field runs overlap. The reported after-build and source hashes are unavailable in the checkout, so the result foree110c025fagainst its merge base and the cost for complete dumps remain unverified.Other systems: All GPU vendors and both OSes share the host open path. The Deck run covers RADV and one of three textures. Windows compile results do not exercise its path APIs, and there are no Tier 1 desktop runs. The change can replace mod, overlay, and writable-mount reads, and the resource size table still budgets the original guest name. The fountain, Water Temple, and Ganon visuals remain unverified through the runtime path. The builder change is a docstring, write-mode opens bypass the new branch, and there are no direct changes to audio, input, shaders, caches, or the renderer ABI.
Switches: None added, changed, or removed. The existing HOS_FS_OVERLAY lookup still runs first, but the new fallback can supersede its chosen file.
Reviewed:
ee110c025fagainst87c0d7b4d6: Read the full seven-file diff, both commits, the complete new C file and Python test, and the changed sections of nn_fs.c, make_game.py, test_vcore_fs_open.py, and both docs. Read the relevant mount, mod, handle, Windows and POSIX file API, builder selector, source-check, and build source-discovery code. Read AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards. The review-checklist file and sibling Tooling checkout were absent. Skimmed recorded-run references and reviewed the supplied run tables and excerpts. No raw traces, CI results, or tester reviews were supplied. Unrelated parts of the large source files were outside the review.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Waiting on the author (a run, a measurement or a decision; no edit does it)
ee110c025fon87c0d7b4and local main has advanced. Supply evidence that the tested patches match and the numbers still hold, or rerun the interleaved comparison on the updated head.Testing: Read-only review. The author reports clang++ and g++ fixture passes and Linux and llvm-mingw compilation. The reported suite still has 85 failures and 28 skips, with shared failures described in aggregate. The fixture uses a stdio stand-in and does not exercise the production Windows path layer. Green head CI, healthy-path timing, and the listed in-game runs remain pending.
Overlaps and merge order: No overlapping pull request identified in the available history. Local main has advanced beyond the merge base.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -139,1 +139,3 @@2. **Dump the game again** and build from the new dump. The file is removed once the dump is whole.2. **Look in `longshot_log.txt` for `is damaged in this copy of the game`.** Game files that Longshot Builder didn'tunpack (copied from another tool, or by hand) get no `longshot_damaged_files.txt`. The game then reads a similartexture in place of each damaged one as it plays, writes nothing to your game files, and logs one line per file.Qualify this to the three known bad-header textures with available stand-ins. Damage wrapped in an older builder's valid NUP zstd frame still passes through.
@ -426,2 +426,4 @@}}{ int hos_stand_in_damaged(const char *guest, char *path, size_t cap); /* (oot_damaged_files.c: an incomplete dump) */if (!want_write) hos_stand_in_damaged(g, path, sizeof path); }Gate this on the resolved base RomFS source and use its validated relative name. A damaged test overlay over a complete dump can currently trigger substitution and a false redump warning. Reads from writable mounts are eligible too.
@ -0,0 +25,4 @@/* 1: the file starts as zstd data (a frame, or a skippable frame: nup.py zstd_start); 0: it does not; -1: it could notbe read, and is left to the open that follows */static int starts_as_zstd(const char *path) {HANDLE h = CreateFileA(path, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);Use the same wide, extended-length open as OpenFile. A long Windows RomFS path can open normally while this probe fails and leaves the retry loop intact.
@ -0,0 +47,4 @@if (strncmp(rel, "Tex/", 4)) return 0;for (size_t i = 0; i < STAND_INS; i++) {if (strcmp(rel, stand_ins[i].damaged)) continue;if (starts_as_zstd(path)) return 0;Probe and serve the same opened handle. A replacement can disappear or change after its probe, and reopening also adds I/O on every matching healthy texture.
@ -0,0 +204,4 @@with tempfile.TemporaryDirectory(prefix="damaged-tex-") as temporary:# (nup.stand_in_for's choice among the six: the damaged files as in the dump, zeros first; their stand-ins whole)tex = Path(temporary) / "Tex"for damaged, stand_in in pairs:Build the candidate set independently of the C table. With this fixture a wrong Gate mapping can supply its own winning candidate, and the runtime case only tests that Gate is missing.
Ran the whole check suite at this head and on main
87c0d7b4d: the same 85 checks fail on both, and they are now named in the description. One box is left before this can be marked ready: a screenshot of the Great Fairy Fountain floor with the stand-in texture.Rebased onto main
fd7dc40db(head9d9d67ee5); the two commits applied without a conflict and the fix's checks pass at this head.ee110c025f9d9d67ee51@fix
9d9d67ee514766cfa7c1Recommendation: Approve. The Steam Deck measurements show a substantial improvement for the damaged dump, but the requested visual and platform checks remain outstanding. Nothing blocking is left: what remains is for the author before merging (below). Recommend approval; not approved on the forge.
What
@fixchangedI changed this pull request: the new head is
4766cfa7c1. I rebased it ontomain(577bba1225). I edited its description to match the change.Changed
Nothing was built or run: the checks run again on the new head.
The host substitutes known damaged textures with similar files without changing the game folder. This avoids the repeated loading observed on an incomplete dump, at the cost of showing substitute textures.
Before merging, for the author (1) (a run, a measurement or a decision; it doesn't hold the recommendation)
Checks: they run again on this head; a failure shows on the pull request.
Also noticed (3; not asked for, not blocking)
More: does it help, other systems, switches, what was read
Does it help: Yes on the measured Steam Deck setup: median Kokiri fps rose from 32.0 to 44.2 and market fps from 21.5 to 25.8 over three interleaved starts per side. These measurements used an earlier version of the change.
Other systems: The lookup affects Windows and Linux, including mod and overlay files. The Windows damaged-dump run, complete-dump run, and on-screen checks of all three substitutions remain outstanding.
Switches: None added, changed, or removed.
Reviewed:
4766cfa7c1against577bba1225: Read AGENTS.md, CONTRIBUTING.md, the pull request template, host/oot_damaged_files.c, the relevant OpenFile and path-resolution code in host/nn_fs.c, the damaged-file fixture, and the fix commit diff. Skimmed the supplied measurements and the base-to-HEAD diff; the latter includes many unrelated merged commits and was not read in full. Tooling's AGENTS.md is not present locally.; nothing run (a code review).Checked and fine:
Testing: No commands that build or run project code were executed. CI has not run on this head; the author reports earlier game and suite results in the description.
Overlaps and merge order: none found
Automated code review by BeanBot (gpt-6.1-terra), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.
WIP: fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder'sto fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder'sWIP: fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder'sto fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's