MagicBeanDev/Hookshot
Archived
Watch
6
0
Fork
You've already forked Hookshot
 0
Code Issues 125 Pull requests 205 Activity

bake: offline NVN translation harness + live bank writer + verify #1707

Merged
blessedplummer merged 5 commits from bake-offline into main 2026-11-10 20:00:26 +00:00
Conversation 5 Commits 5 Files changed 13 +5993
blessedplummer commented 2026-11-10 14:11:03 +00:00 (Migrated from github.com)

Developer tooling: translate archived NVN shaders to SPIR-V with no GPU, no game, no playthrough, and write/verify live-format program banks offline.

What it adds (purely additive — nothing under gpu/vk, nx2/build.py or host/; game behavior unchanged):

  • Offline translator core (seed → SPIR-V with exact runtime program keys, FNV-1a stable IDs, mesh/task coverage) — every emitted module spirv-val --target-env vulkan1.3 clean.
  • Live hookshot_programs.bin writer (kDiskMagic/disk_stamp) with read-back verify through a copy of program_cache_load.
  • Portable manifest + bake verify (seed-sha round-trip, stamp currency, per-module validation), container enumeration (BNSH/BAHS/NDSR/NDVE/vsetb/SARC incl. zstd-dictionary frames), Ampere encoding probe.
  • --target feature profiles (plain/tensor/tensor-software/rdna2/gfx1201/query): tensor-gated programs + integer-answer variants translated per profile, feature set folded into the stamp (target belongs to the stamp, never to a key), verified against live seed_compute semantics (dedup + cap-64 arithmetic identical).
  • Tests: skeleton (9 passed), mimo, ds (16 programs / 18 slots → 32/32 modules valid, 0 bad) — tests/run_all.py bake: 3 passed, 0 failed.
  • BAKE.md: design, run guide, key/stamp derivation, offline assumptions, cut list, trap index.

First real-ROM run (measured, read-only): 42,856 stages / 38,809 programs in 1m15s; 46,500 slots translated in 3m20s, 0 translate failures; 5.93 GB bank, 0 bad records; 90,132/90,135 modules valid (99.9967%) under the layout feature the build enables; 167 skips itemized (71 tensor-gate, now covered by profiles; 95 real recompiler gaps the shipping build also skips; 1 capture refusal).

Why: first-visit shader stutter is the player-facing cost of runtime translation; this moves all of it ahead of time for archived content, locally from the user's own ROM — no cache sharing, no legal gray zone, supportable upstream. Built by four independent implementations that converged (four-way stamp agreement, mutual verification).

Second commit: --target profiles + layout-aware verify gate (both MiMo-verified with independent rebuilds and measurements).

Developer tooling: translate archived NVN shaders to SPIR-V with no GPU, no game, no playthrough, and write/verify live-format program banks offline. **What it adds** (purely additive — nothing under `gpu/vk`, `nx2/build.py` or `host/`; game behavior unchanged): - Offline translator core (seed → SPIR-V with exact runtime program keys, FNV-1a stable IDs, mesh/task coverage) — every emitted module `spirv-val --target-env vulkan1.3` clean. - Live `hookshot_programs.bin` writer (kDiskMagic/disk_stamp) with read-back verify through a copy of `program_cache_load`. - Portable manifest + `bake verify` (seed-sha round-trip, stamp currency, per-module validation), container enumeration (BNSH/BAHS/NDSR/NDVE/vsetb/SARC incl. zstd-dictionary frames), Ampere encoding probe. - `--target` feature profiles (plain/tensor/tensor-software/rdna2/gfx1201/query): tensor-gated programs + integer-answer variants translated per profile, feature set folded into the stamp (target belongs to the stamp, never to a key), verified against live `seed_compute` semantics (dedup + cap-64 arithmetic identical). - Tests: skeleton (9 passed), mimo, ds (16 programs / 18 slots → 32/32 modules valid, 0 bad) — `tests/run_all.py bake`: 3 passed, 0 failed. - `BAKE.md`: design, run guide, key/stamp derivation, offline assumptions, cut list, trap index. **First real-ROM run (measured, read-only): 42,856 stages / 38,809 programs in 1m15s; 46,500 slots translated in 3m20s, 0 translate failures; 5.93 GB bank, 0 bad records; 90,132/90,135 modules valid (99.9967%) under the layout feature the build enables; 167 skips itemized (71 tensor-gate, now covered by profiles; 95 real recompiler gaps the shipping build also skips; 1 capture refusal).** **Why**: first-visit shader stutter is the player-facing cost of runtime translation; this moves all of it ahead of time for archived content, locally from the user's own ROM — no cache sharing, no legal gray zone, supportable upstream. Built by four independent implementations that converged (four-way stamp agreement, mutual verification). Second commit: --target profiles + layout-aware verify gate (both MiMo-verified with independent rebuilds and measurements).
hckrman101 (Migrated from github.com) reviewed 2026-11-10 16:40:50 +00:00
hckrman101 (Migrated from github.com) left a comment

Nice piece of tooling, and purely additive is the right shape for it. One thing will break CI though.

tests/test_bake_skeleton.py hardcodes dir="/tmp/opencode" in four tempfile.TemporaryDirectory(...) calls (lines 91, 128, 183, 245), plus tool = "/tmp/opencode/sb-tool-cache" (188, 250). run_all.py picks up every tests/test_*.py automatically, and TemporaryDirectory(dir=...) raises FileNotFoundError when that directory doesn't exist, which is the case on CI runners and on Windows. So test_bake_end_to_end and friends will fail rather than skip. Please drop the dir= argument (use the default temp dir) and put the tool cache inside the temp dir or under tempfile.gettempdir(). Also: the translate tests there need a compiler/headers, so they should skip when those are missing, the way test_bake_mimo.py does.

Smaller things (nits):

  • nx2/bake_ds/build.sh defaults OUT to /tmp/opencode/ds and looks for externals in /home/deck/azahar-build/externals. Those are your machine's paths; I'd keep only the $ROOT-relative candidates and default OUT to something under ${TMPDIR:-/tmp}. Same default appears in the test_bake_ds.py docstring.
  • gpu/shaderbatch.cpp and nx2/bake_mimo.py point to BAKE-MIMO.md for the stop-lines, but that file isn't in the PR (BAKE.md is). Point them at BAKE.md.
  • No CI checks have run on this branch yet, so the above is from reading the code, not from a failing run. Worth a merge of current main to get a CI run.

I didn't build or run the translator, so I'm taking the 90,132/90,135 module numbers from the description.

Nice piece of tooling, and purely additive is the right shape for it. One thing will break CI though. `tests/test_bake_skeleton.py` hardcodes `dir="/tmp/opencode"` in four `tempfile.TemporaryDirectory(...)` calls (lines 91, 128, 183, 245), plus `tool = "/tmp/opencode/sb-tool-cache"` (188, 250). `run_all.py` picks up every `tests/test_*.py` automatically, and `TemporaryDirectory(dir=...)` raises FileNotFoundError when that directory doesn't exist, which is the case on CI runners and on Windows. So `test_bake_end_to_end` and friends will fail rather than skip. Please drop the `dir=` argument (use the default temp dir) and put the tool cache inside the temp dir or under `tempfile.gettempdir()`. Also: the translate tests there need a compiler/headers, so they should skip when those are missing, the way test_bake_mimo.py does. Smaller things (nits): - `nx2/bake_ds/build.sh` defaults `OUT` to `/tmp/opencode/ds` and looks for externals in `/home/deck/azahar-build/externals`. Those are your machine's paths; I'd keep only the `$ROOT`-relative candidates and default OUT to something under `${TMPDIR:-/tmp}`. Same default appears in the `test_bake_ds.py` docstring. - `gpu/shaderbatch.cpp` and `nx2/bake_mimo.py` point to `BAKE-MIMO.md` for the stop-lines, but that file isn't in the PR (BAKE.md is). Point them at BAKE.md. - No CI checks have run on this branch yet, so the above is from reading the code, not from a failing run. Worth a merge of current main to get a CI run. I didn't build or run the translator, so I'm taking the 90,132/90,135 module numbers from the description.
hckrman101 (Migrated from github.com) reviewed 2026-11-10 18:12:52 +00:00
hckrman101 (Migrated from github.com) left a comment

Re-review at a2bbd4e. The new commits (verify/partial-bank work in bake.py, bake_ds, BAKE.md) don't touch the blocker from last round, and the new test added one more instance of it.

Still open: tests/test_bake_skeleton.py still hardcodes dir="/tmp/opencode" (lines 91, 128, 183, 245, and the new one at 285) plus tool = "/tmp/opencode/sb-tool-cache" (188, 250, 290). run_all.py discovers every tests/test_*.py, and TemporaryDirectory(dir=...) raises FileNotFoundError when that dir doesn't exist (CI, Windows). Dropping the dir= argument and putting the tool cache under the temp dir fixes it. These tests also need a compiler/headers, so they should skip when those are missing like test_bake_mimo.py does.

Nits, still unchanged: nx2/bake_ds/build.sh defaults (OUT=/tmp/opencode/ds, /home/deck/azahar-build/externals) and the same default in the test_bake_ds.py docstring; BAKE-MIMO.md references in gpu/shaderbatch.cpp / nx2/bake_mimo.py (check whether they're fixed, I didn't see that file added).

Also: if you add RUN_ALL_ARGS/RUN_ALL_SKIP needs for these tests, they now live in each test file (see #1822), not in run_all.py. No CI has run on this branch yet (no checks reported).

Re-review at a2bbd4e. The new commits (verify/partial-bank work in bake.py, bake_ds, BAKE.md) don't touch the blocker from last round, and the new test added one more instance of it. Still open: `tests/test_bake_skeleton.py` still hardcodes `dir="/tmp/opencode"` (lines 91, 128, 183, 245, and the new one at 285) plus `tool = "/tmp/opencode/sb-tool-cache"` (188, 250, 290). `run_all.py` discovers every `tests/test_*.py`, and `TemporaryDirectory(dir=...)` raises FileNotFoundError when that dir doesn't exist (CI, Windows). Dropping the `dir=` argument and putting the tool cache under the temp dir fixes it. These tests also need a compiler/headers, so they should skip when those are missing like test_bake_mimo.py does. Nits, still unchanged: `nx2/bake_ds/build.sh` defaults (`OUT=/tmp/opencode/ds`, `/home/deck/azahar-build/externals`) and the same default in the `test_bake_ds.py` docstring; `BAKE-MIMO.md` references in `gpu/shaderbatch.cpp` / `nx2/bake_mimo.py` (check whether they're fixed, I didn't see that file added). Also: if you add RUN_ALL_ARGS/RUN_ALL_SKIP needs for these tests, they now live in each test file (see #1822), not in run_all.py. No CI has run on this branch yet (no checks reported).
tests/test_bake_skeleton.py
@ -0,0 +1,316 @@
"""Bake skeleton end-to-end on synthetic data only (no game files, no GPU).
hckrman101 (Migrated from github.com) commented 2026-11-10 18:12:52 +00:00

Same hardcoded /tmp/opencode here (and tool at line 290), now in the new partial-bank test too. Use the default temp dir.

Same hardcoded `/tmp/opencode` here (and `tool` at line 290), now in the new partial-bank test too. Use the default temp dir.
blessedplummer commented 2026-11-10 18:48:30 +00:00 (Migrated from github.com)

Review follow-ups landed (commit 31d3b83):

  • Blocker fixed: all five dir="/tmp/opencode" dropped (default temp dirs), tool caches live inside the temp dir, and compiler-dependent tests skip when the toolchain is missing (the bake_mimo pattern — including the partial-bank test, which I initially missed and caught on the re-run: 9/9 green locally).
  • Nits: build.sh OUT is now ${TMPDIR:-/tmp}-relative, machine-specific externals path removed (docstring + BAKE.md updated); dangling doc references repointed at BAKE.md (verified no other references to non-PR files); RUN_ALL_ARGS declared in test_bake_mimo.py per #1822 (skeleton keeps in-test skips so file-level declaration doesn't nuke coverage; ds self-skips on --tool).
  • Main merged with no conflicts and no translator drift (stamp formula byte-identical, just relocated) — should get a CI run now.

Note: the new runner's header_dir doesn't probe in-tree buildext/cpm, so locally the mimo test skips at runner level while the skeleton runs its translate half via its own probe; consistent on CI (no buildext/ there). The nx2/build.py line refs in PR comments now point at moved code (comment-only).

Review follow-ups landed (commit 31d3b83): - **Blocker fixed:** all five `dir="/tmp/opencode"` dropped (default temp dirs), tool caches live inside the temp dir, and compiler-dependent tests skip when the toolchain is missing (the `bake_mimo` pattern — including the partial-bank test, which I initially missed and caught on the re-run: 9/9 green locally). - **Nits:** `build.sh` OUT is now `${TMPDIR:-/tmp}`-relative, machine-specific externals path removed (docstring + BAKE.md updated); dangling doc references repointed at `BAKE.md` (verified no other references to non-PR files); `RUN_ALL_ARGS` declared in `test_bake_mimo.py` per #1822 (skeleton keeps in-test skips so file-level declaration doesn't nuke coverage; ds self-skips on `--tool`). - **Main merged** with no conflicts and no translator drift (stamp formula byte-identical, just relocated) — should get a CI run now. Note: the new runner's `header_dir` doesn't probe in-tree `buildext/cpm`, so locally the mimo test skips at runner level while the skeleton runs its translate half via its own probe; consistent on CI (no `buildext/` there). The `nx2/build.py` line refs in PR comments now point at moved code (comment-only).
blessedplummer commented 2026-11-10 19:07:23 +00:00 (Migrated from github.com)

Conflicts resolved and pushed (merge 9b424be + one-line fix c00cfab):

  • Both conflicted files were upstream's own code (older main snapshot from the previous merge vs newer main: present-deferral in vk_replay.cpp, worklist doc) — took the newer upstream side in full. The PR diff vs main is back to exactly the 13 bake files.
  • The merge also surfaced a real bug in my own squash: the pushed commit carried self.require_toolchain() (nonexistent method) instead of the module-level call — the exact failure mode the review flagged, now fixed and green (9/9 locally; skips-not-errors where no toolchain exists).
Conflicts resolved and pushed (merge 9b424be + one-line fix c00cfab): - Both conflicted files were upstream's own code (older main snapshot from the previous merge vs newer main: present-deferral in vk_replay.cpp, worklist doc) — took the newer upstream side in full. The PR diff vs main is back to exactly the 13 bake files. - The merge also surfaced a real bug in my own squash: the pushed commit carried `self.require_toolchain()` (nonexistent method) instead of the module-level call — the exact failure mode the review flagged, now fixed and green (9/9 locally; skips-not-errors where no toolchain exists).
hckrman101 (Migrated from github.com) approved these changes 2026-11-10 19:58:57 +00:00
hckrman101 (Migrated from github.com) left a comment

Thanks, this covers what I raised before.

  • tests/test_bake_skeleton.py no longer hardcodes /tmp/opencode: all five temp dirs use TemporaryDirectory(prefix=...) in the default location, the tool caches live inside them, and the translate/verify tests skip when the compiler or SPIR-V headers aren't there. So it shouldn't fail on CI runners or Windows any more.
  • tests/test_bake_mimo.py declares RUN_ALL_ARGS with {cxx} / {spirv_headers}, which run_all.py knows.
  • build.sh defaults to ${TMPDIR:-/tmp}/hookshot-ds, and the machine-specific path is gone.
  • Nothing game-derived is committed: the ds fixture is a hand-built synthetic tree, with no dumped shaders or banks.

Nit: test_bake_skeleton.py skips inside the test rather than through RUN_ALL_SKIP, which is fine, just less visible in the run summary.

CI hasn't run on this branch yet (the fork workflows are waiting for approval), so it'd be good to confirm the new tests skip rather than fail once they do.

Thanks, this covers what I raised before. - `tests/test_bake_skeleton.py` no longer hardcodes `/tmp/opencode`: all five temp dirs use `TemporaryDirectory(prefix=...)` in the default location, the tool caches live inside them, and the translate/verify tests skip when the compiler or SPIR-V headers aren't there. So it shouldn't fail on CI runners or Windows any more. - `tests/test_bake_mimo.py` declares `RUN_ALL_ARGS` with `{cxx}` / `{spirv_headers}`, which `run_all.py` knows. - `build.sh` defaults to `${TMPDIR:-/tmp}/hookshot-ds`, and the machine-specific path is gone. - Nothing game-derived is committed: the `ds` fixture is a hand-built synthetic tree, with no dumped shaders or banks. Nit: `test_bake_skeleton.py` skips inside the test rather than through `RUN_ALL_SKIP`, which is fine, just less visible in the run summary. CI hasn't run on this branch yet (the fork workflows are waiting for approval), so it'd be good to confirm the new tests skip rather than fail once they do.
Commenting is not possible because the repository is archived.
Reviewers
No reviewers
hckrman101
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
autorelease: pending

   
autorelease: tagged

   
bug
Something isn't working

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
testing wanted

   
wontfix
This will not be worked on

No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Hookshot!1707
No description provided.