WIP: feat(builder): accept .nx archives (NXFormat) as the game files #685
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!685
Loading…
Reference in a new issue
No description provided.
Delete branch "nussico/Longshot:builder/nx-format"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Windows 11, RTX 2070 · Needs: Linux, any GPU
Why
Players who keep their game as a
.nxfile (NXFormat, github.com/fab918/nxformat, made withnxconvert pack-nup) have to convert it back to a.nupor to folders before Longshot can build. With this PR the builder takes the.nxdirectly.What changed
make_game.pynow reads a.nxarchive wherever it reads a.nup: it tells the two apart by the file's first bytes and runs a new reader for the.nx. The reader reusesnup.py's extraction, so path checks, progress and the damaged-.zslist behave the same, and every file is checked against its SHA-256 in the archive.nx2/nxar.py(classNxand a CLI likenup.py's) reads NX 0.2 and 0.3 archives holding exefs/romfs trees. A.nxholding NCA files (nxconvert pack) is refused with a message saying how to convert it.nx2/test_nxar.pybuilds archives from the spec, game-free.nx2/nup.py: the step after extraction (plain NSOs, ZBIC) moved intoplain_modules(), whichnxar.pycalls too. The.nuppath is unchanged.package_script()innx2/make_game.pypicksnx2/nxar.pyornx2/nup.pyby the NXAR magic, incheck()and inmain()'s extraction;--nupand--gamehelp mention.nx. Test:tests/test_make_game_package_script.py..gitignoreignores*.nxandscripts/release.py'sNOT_A_RELEASE_FILErefuses.nx(a.nxis a whole dump); CONTRIBUTING lists it with the game files.website/src/content/docs/requirements.mdx(its copy) anddocs/development.md.The launcher's side (finding, browsing and dropping
.nxfiles) is a separate PR on LongshotLauncher, and needs a Longshot release with this change.Measurements
Nothing to measure in the game: this changes only how the builder reads the game file; the build it makes and the game are unchanged (see Testing for the extraction check and its speed).
Flags and settings
None
Testing
Tested: On Windows 11 build 26300 (llvm-mingw 20260616), with my own copy, the same code in Hookshot's tree (this PR is that change with Longshot's names). The game files never left my PC.
.nxand the.nupof the same game hold the same 186,592 files by path, size and SHA-256, with byte-identical exefs and build ide0f3ccfa..nxdecoded and SHA-256-checked: 26.27 GB in 158 s from disk (166 MB/s).make_game.py --check --nup <file.nx>: "This is the version ... supports (build e0f3ccfa).".nxextracted all 186,592 files, mademaina plain NSO and translated 100% of the code. The game it built starts and plays (NVIDIA GeForce RTX 2070, driver 32.0.16.2101).nx2/test_nxar.pyandtests/test_make_game_package_script.pypass.python -I -B tests/run_all.py: 504 passed, 19 failed, 72 skipped; the same 19 fail onmain@53f96d7on this PC: test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_cv_signal_race, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_vram_budget_row, test_walk_key.Not tested: A build from this Longshot branch in the game (only Hookshot's tree with the same change); Linux (I have no Linux PC).
Must be tested:
python -I nx2/make_game.py --check --nup <your .nx>and then a build with--nup <your .nx>(a.nxmade from your.nupwithnxconvert pack-nup), expecting the check to report build e0f3ccfa and the game to start (by hand: a tester with a.nx, since the farm's copies are.nup).Risks
What could break: Only a file starting with
NXARgoes to the new reader; every.nupstill goes tonup.py, whose extraction is unchanged except that its last step moved into a function.Turning it off: revert the commit.
Compatibility: Nothing changes for a player who builds from a
.nupor folders: no saves, options, caches or vk ABI touched.Issues
Fixes #411
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why. Not ticked yet: The in-game build ran from Hookshot's tree, and the PR says this Longshot branch was not built and run in the game.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
tests/that reads changed source updated in this PR, not weakened.012546faaec9d3ac620fRebased onto
main@7bacf846: no conflicts. No tests were run here; the test boxes stay yours.Hi @nussico, I've been helping this land. Here's what I found, and a branch you can take whole or in parts:
madebyjordan:builder/nx-format-rebased.On that branch
6d72ca93. It now conflicts with main innx2/make_game.py, but only in the docstring: main gained--romfs-from. I kept both lines and wrote ".nup or .nx" in the new one. No code changes.nx2/game_title.py, which read only a.nup. With a.nxit got "not a NUP package", and the game ended up as "Longshot" with no icon.Nxnow keeps the catalog and the media region, and reads the application's title and icon (SPEC.md §4 and §5: the icon's SHA-256 is checked; a damaged one is left out).game_title.from_nuptells a.nxby itsNXARmagic, as yourpackage_scriptdoes.nx2/test_nxar.pycovers it.nx2/prompt_assets.pyandhost/prompts.c, withtests/test_prompt_size_table_version.py(new) and a case intests/test_mods_size_table.py.The finding: a .nx's update holds the base game's files too
nxconvert pack-nupmakes an update a view in which it "shadows individual RomFS/data paths; other base files remain available" (its CLI.md, NUP migration). On my real.nxof the 1.1.1 dump, the newest title has 225 files more than the.nup's update (62 MB). Nothing is missing and no size differs. They're the base game's versions of files the update renamed:*.Product.100.*beside the update's.Product.110/.111.One of them reaches the game.
prompt_assets.size_tablemakesprompts/rsizetable.zsfrom the size table whose name sorts first, which isResourceSizeTable.Product.100(the base game's).host/prompts.cthen serves that file when the game opens its ownProduct.111. On the real build from the.nx, the served table matchedProduct.100in 219,207 of 219,208 entries.Product.111has 868 entries thatProduct.100lacks, and 47 that are larger. The game drops a file larger than its entry, so a.nxbuild would have run 1.1.1 with 1.0.0's sizes. I didn't run the game with the bad table, so the effect in play is unobserved.d2948c31 takes the name that sorts last (the newest version, the one the game opens), in both
prompt_assets.game_table_nameandhost/prompts.cgame_table, which must agree. With a single table, as from a.nup, nothing changes. After the fix, the same build's table matchedProduct.111in 219,967 of 219,968 entries (the other is the font's raised entry).Two smaller effects of the extra files, not fixed:
text_assets.pyalso rewords the base game'sMals/*.Product.100.sarc.zs. That's extra files intexts/, but the game asks by exact name.shader_census.pywalksShader/and will see the base game's four.bfshafiles. I didn't check what that adds to the shader list.Tested
Windows 11, RX 6700 XT, with a
.nxI made from my own 25.62 GB.nupusing nxconvert 1.0.4pack-nup --platform switch2(375 s).nxconvert check-nuppassed: 372,812 files, 25.58 GB verified.make_game.py --check --nup <file.nx>: "This is the version Longshot supports (build e0f3ccfa)", in 18 s. A cut-short copy: "the .nx file is incomplete ...". A file of zeros: "not a NUP package"..nxon this branch (llvm-mingw, shaders left to the first start):mainmade a plain NSO@@donenx2/test_nxar.py(with the name/icon checks),tests/test_make_game_package_script.py,tests/test_prompt_size_table_version.pyand thenx2/test_nup_*checks pass. Without each fix, its new checks fail.python -I -B tests/run_all.pyon 5e5ee079: 235 passed, 69 failed, 418 skipped. That run had no C++ compiler on its PATH. Main @6d72ca93on the same PC: 232 passed, 70 failed. No failure is new.Not tested:
.nxbuild. I set up a short start, but the folder was removed before it got its turn. It's on my list.tests/test_mods_size_table.py's folder cases (including the new one) run only there.One question
nxar.py's refusal of an NCA-holding.nxsays to runnxconvert extract <file.nx> <new folder>and pick the folder with romfs and exefs. nxconvert's help describesextractas "every snapshot as a dump: base, updates/, dlc/...". Does it really give romfs/exefs folders for such an archive? I couldn't check without a DNSP. The launcher's wording in MagicBeanDev/LongshotLauncher#159 only suggestsnxconvert pack-nupor picking the.nup.Taking it
The whole branch, with your commit as you wrote it plus the two on top:
Or only my two commits, on your branch after you rebase it yourself:
The launcher side of the error messages is MagicBeanDev/LongshotLauncher#159: a
.nxthat can't be read was shown as "the check stopped before it could tell anything", becauseGameCheckdidn't know "nxar.py stopped".Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).
**Not testable by hand:**line saying why and what stands in for it. Then tick it.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.