docs: a proposal for a faster development loop #2636

github-sync · 2026-11-12T03:38:55Z

jordanella commented

2026-11-12 03:38:55 +00:00

(Migrated from github.com)

Today a one-line renderer or host change costs a relink of the 426 MB exe, a cold start and a trip back to the scene, and a stray RX_* switch or a failed build makes the next build translate the whole game again. This adds a proposal, docs/dev-loop-proposal.md, for making those edits cheap and predictable and the code easier to navigate, without changing what players build or run.

Decisions needed (top of the doc, each before its work starts):

What agents may read: never work/, single excerpts on request, or local models only. scripts/cpu_categories.py already treats "a map of the game's code" as local; the same question covers a hand-written table of game-function addresses in the repo.
Where the entry point lives: nx2/dev.py (proposed) or scripts/dev.py.
Where scenario files live: scenarios/ or tests/scenarios/.
Dev warnings and clang-tidy in CI: advisory or failing, given a baseline.
Whether a dev-only split of the host out of the exe is acceptable at all, if the measurements call for it.
Whether to make players' start-up update check asynchronous: a player-facing change, decided on its own.

What it plans, as small PRs behind a week of measurement (Phase 0):

Build graph. Skip links whose inputs haven't changed. Use depfiles instead of header blobs. Make the lift key hash only the environment the lift reads. Don't relift after a compile error in a tracked source. Add compile_commands.json, and one nx2/dev.py whose status names why anything rebuilds.
Middle loop. A relaunch command on the control bridge (#926) with a staged DLL swap in dev builds, so a renderer edit reaches a scripted scene in one command. Built-in shaders loadable from disk. A live renderer swap is designed but deferred: the renderer has no teardown today.
Testing. run_all.py --changed, one shared source-cutting helper in place of about 165 copies of block(), a sanitizer lane, scenario files on hookshot_ctl.py, and run summaries that can be compared.
Diagnostics. One list of switches with a check, a header on captures, memory counters, and warnings and clang-tidy on our own code.
Docs and code maps. A docs/README.md index, subsystem maps, every generated file naming its generator, a dev.py where repo map that tells people and agents which tests cut the code they're changing, and the data boundary for agents written into AGENTS.md.

How it scores. Under workflow.md's formula, contributor tooling is S3 with no player reach, so it shouldn't take a worklist row from anyone. The case for it is throughput. The three pieces that do reach players (the full rebuild after a failed link, the blocking update check, the single-process emit on Windows) get their own worklist rows, scored normally.

Where the facts come from. Read from the source at 7297d93d, partly by AI agents, and rechecked against main where the plan leans on it. #2324's kept lift and #2518's COMPILE_ONLY_ENV were already doing parts of it, and the doc builds on them rather than re-proposing them. Nothing was measured; timings are quoted from the PRs that measured them.

Testing. Docs only: one new file, no code changed. Every relative link and anchor in it resolves; tests/test_hookshot_ctl.py (the one check that reads docs/) passes. I didn't run the full run_all.py locally, since nothing it checks changed; CI runs it. No screenshots: nothing changes on screen.

🤖 Generated with Claude Code

Today a one-line renderer or host change costs a relink of the 426 MB exe, a cold start and a trip back to the scene, and a stray `RX_*` switch or a failed build makes the next build translate the whole game again. This adds a proposal, [docs/dev-loop-proposal.md](https://github.com/jordanella/Hookshot/blob/docs/dev-loop-proposal/docs/dev-loop-proposal.md), for making those edits cheap and predictable and the code easier to navigate, without changing what players build or run. **Decisions needed** (top of the doc, each before its work starts): 1. What agents may read: never `work/`, single excerpts on request, or local models only. `scripts/cpu_categories.py` already treats "a map of the game's code" as local; the same question covers a hand-written table of game-function addresses in the repo. 2. Where the entry point lives: `nx2/dev.py` (proposed) or `scripts/dev.py`. 3. Where scenario files live: `scenarios/` or `tests/scenarios/`. 4. Dev warnings and clang-tidy in CI: advisory or failing, given a baseline. 5. Whether a dev-only split of the host out of the exe is acceptable at all, if the measurements call for it. 6. Whether to make players' start-up update check asynchronous: a player-facing change, decided on its own. **What it plans**, as small PRs behind a week of measurement (Phase 0): - **Build graph.** Skip links whose inputs haven't changed. Use depfiles instead of header blobs. Make the lift key hash only the environment the lift reads. Don't relift after a compile error in a tracked source. Add `compile_commands.json`, and one `nx2/dev.py` whose `status` names why anything rebuilds. - **Middle loop.** A `relaunch` command on the control bridge (#926) with a staged DLL swap in dev builds, so a renderer edit reaches a scripted scene in one command. Built-in shaders loadable from disk. A live renderer swap is designed but deferred: the renderer has no teardown today. - **Testing.** `run_all.py --changed`, one shared source-cutting helper in place of about 165 copies of `block()`, a sanitizer lane, scenario files on `hookshot_ctl.py`, and run summaries that can be compared. - **Diagnostics.** One list of switches with a check, a header on captures, memory counters, and warnings and clang-tidy on our own code. - **Docs and code maps.** A `docs/README.md` index, subsystem maps, every generated file naming its generator, a `dev.py where` repo map that tells people and agents which tests cut the code they're changing, and the data boundary for agents written into AGENTS.md. **How it scores.** Under workflow.md's formula, contributor tooling is S3 with no player reach, so it shouldn't take a worklist row from anyone. The case for it is throughput. The three pieces that do reach players (the full rebuild after a failed link, the blocking update check, the single-process emit on Windows) get their own worklist rows, scored normally. **Where the facts come from.** Read from the source at `7297d93d`, partly by AI agents, and rechecked against `main` where the plan leans on it. #2324's kept lift and #2518's `COMPILE_ONLY_ENV` were already doing parts of it, and the doc builds on them rather than re-proposing them. Nothing was measured; timings are quoted from the PRs that measured them. **Testing.** Docs only: one new file, no code changed. Every relative link and anchor in it resolves; `tests/test_hookshot_ctl.py` (the one check that reads `docs/`) passes. I didn't run the full `run_all.py` locally, since nothing it checks changed; CI runs it. No screenshots: nothing changes on screen. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hckrman101 (Migrated from github.com) reviewed

2026-11-12 03:49:21 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks for writing this up: §2 is the clearest description of the build and the boot path we have, and most of the plan is right. I've checked the parts of §2 the plan leans on against main (the lift key, forget_failed_build, object_key/flag_text, load_vulkan) and they hold. CI's failures are source checks unrelated to a docs-only change.

I run a local dev setup that already does a few of these things, so here are numbers and lessons for Phase 0, plus what I'd change.

What it does, with numbers (Linux, main, 22 jobs):

One content-addressed object store shared by every checkout/worktree. Objects are named by source + flags + headers (as object_key already does), so every workspace produces the same names. Each build gets a private object dir, hard-linked from the store before the build and linked back after (atomic link, every few seconds), so two builds at once never race on the store.
- Fresh worktree, cold store: ~720 s (renderer 81 s, lift 107 s, compile 515 s).
- The same main in a second fresh worktree, warm store: 18 s, 0 of 258 files compiled, lift reused, both links included.
- One gpu/vk header + one .cpp changed: 26 s (16 renderer objects, because of the header blob §2.2 describes, plus 1 other).
- Outputs byte-identical either way.
A lift snapshot cache. The finished lift is keyed by lift_key minus the work/game paths (the lifted C is byte-identical across folders), snapshotted with reflinks, and reflinked into a new work folder with the same key, so make_game skips the lift.
A renderer-only path (RX_VK_ONLY, ~30 s cold, seconds warm) dropped into a game folder, with an ABI check first (below).
A build/GPU queue: builds and game runs take tickets (exclusive, shared, timing = exclusive with no builds), so fps measurements never run beside a compile.
In-session A/B runs: one game session with a file-based switch toggled every N seconds, so both halves see the same scenes and load, plus scripted live input and a fresh copy of a clean save per run.

What that means for the plan

A2 misses the paths. lift_key hashes env, and make_game's env carries OOT_WORK, OOT_EXEFS, OOT_ROMFS and OOT_DUMP. Two checkouts of the same tree, or a moved work folder, therefore never share a lift, although the exefs contents are already hashed. LIFT_ENV should leave out path-valued variables (or hash what they point to), not list OOT_EXEFS. Dropping the paths is what makes our snapshot cache work. (inline)
The biggest lever isn't in the doc: an object store shared across work folders. OBJ_CACHE is <work>/objcache, so every new worktree pays the full ~720 s even though flag_text already makes the keys folder-independent. An opt-in shared store (a switch pointing objcache elsewhere, private per-build dir, atomic link-back) took that to 18 s here. That's the worktree-per-PR workflow agents use, and it needs no change for players. Two catches we hit:
- the git-ignored inputs in work/ (hot_funcs.txt, hot_order.txt, chunks.txt) must be seeded into a new checkout, or its chunk boundaries and so every lifted object name differ (make_game copies them from ROOT/work, which is empty in a fresh worktree);
- the PGO profile's path ends up in the hot pieces' keys, so it has to sit at a content-addressed path to be shared.
A4 and C2: data point for Phase 0. With nothing to compile, the whole build including both links was 18 s on Linux with lld, so the exe link there is seconds, not minutes. A4 is still worth doing (it's S), but on this evidence I'd expect C2 to be unnecessary; it's worth measuring the link on Windows before designing anything.
A6 is validated: the one-header edit above rebuilt 16 renderer objects where a depfile would rebuild its includers.
B1's staged swap needs an ABI check, not just MKWVK_API_VERSION. The renderer .so also takes symbols from the exe. An exe and a .so built from different trees fail at load, before mkwvk_init's version check runs, and to the user it looks like the game silently never opens a window. We check nm -D --undefined-only hookshot_vk.so | c++filt | grep mkwvk is empty before dropping it in. dev.py should do the equivalent before renaming .new over the DLL, and load_vulkan's failure line on Linux should carry dlerror()'s text. (inline)
D4/D5: compare inside a session, not between sessions. For a flicker-type metric, counts varied 5–20x from one session to the next on the same build, so --compare against the last run would mostly report noise. What worked: an A/B mode in the scenario itself (a file-based switch flipped every N seconds, per-half results), always with a sanity metric next to the target one. We record mean luma: one "fix" showed 0 flashes because it had blown the exposure out. I'd make in-session A/B a first-class scenario option and give D5's summaries a variance/sanity column. A fresh copy of a clean save per run (which D4's per-scenario user folder covers) mattered too.
Missing from D5/§12: runs must not share the machine with a build. fps and frame-time numbers taken beside a compile are worthless, and with dev.py run --watch rebuilding on save that will happen by default. A simple lock (a build waits for a timing run, a timing run waits for builds) in dev.py covers it.

I'm happy to upstream the shared-store wrapper (as an opt-in on build.py/make_game.py) and the in-session A/B pieces if they'd help. Say if you want them; they'd come as their own PRs under A and D.

On the decisions

What agents may read: strict. Agents never read work/ or anything made from the game's files: lifted C, functions.txt, image.bin, linker maps of a build, dumps, captures with game data. That follows directly from CONTRIBUTING.md and is how scripts/cpu_categories.py already treats a map of the game's code. The related question gets the same answer: a hand-written table of game-function addresses with names and meanings is a map of the game's code and stays local, not in the repo. F4's in-repo part goes with it; a local-only lookup is fine. (inline)
Warnings and clang-tidy: advisory to start. Everything builds with -Wno-everything today, so the first baseline will be large; let it settle before anything fails CI.
Async update check: agreed it's its own player-facing PR with a worklist row, measured on a slow network, not part of this work.

Across all of it: anything shared or dev-only stays opt-in, and players' paths and the work-folder layout the Builders use don't change. 2, 3 and 5 I'll come back to; for 5 I'd wait for the Windows link numbers.

Nit: the doc is long for a proposal. Once the decisions are in, §2 could move into the subsystem maps (F1) it is meant to seed, and the proposal could keep just the plan.

Thanks for writing this up: §2 is the clearest description of the build and the boot path we have, and most of the plan is right. I've checked the parts of §2 the plan leans on against `main` (the lift key, `forget_failed_build`, `object_key`/`flag_text`, `load_vulkan`) and they hold. CI's failures are source checks unrelated to a docs-only change. I run a local dev setup that already does a few of these things, so here are numbers and lessons for Phase 0, plus what I'd change. **What it does, with numbers** (Linux, `main`, 22 jobs): - **One content-addressed object store shared by every checkout/worktree.** Objects are named by source + flags + headers (as `object_key` already does), so every workspace produces the same names. Each build gets a private object dir, hard-linked from the store before the build and linked back after (atomic link, every few seconds), so two builds at once never race on the store. - Fresh worktree, cold store: ~720 s (renderer 81 s, lift 107 s, compile 515 s). - The same `main` in a second fresh worktree, warm store: **18 s**, 0 of 258 files compiled, lift reused, both links included. - One `gpu/vk` header + one `.cpp` changed: 26 s (16 renderer objects, because of the header blob §2.2 describes, plus 1 other). - Outputs byte-identical either way. - **A lift snapshot cache.** The finished lift is keyed by `lift_key` minus the work/game *paths* (the lifted C is byte-identical across folders), snapshotted with reflinks, and reflinked into a new work folder with the same key, so `make_game` skips the lift. - **A renderer-only path** (`RX_VK_ONLY`, ~30 s cold, seconds warm) dropped into a game folder, with an ABI check first (below). - **A build/GPU queue:** builds and game runs take tickets (exclusive, shared, timing = exclusive with no builds), so fps measurements never run beside a compile. - **In-session A/B runs:** one game session with a file-based switch toggled every N seconds, so both halves see the same scenes and load, plus scripted live input and a fresh copy of a clean save per run. **What that means for the plan** - **A2 misses the paths.** `lift_key` hashes `env`, and make_game's `env` carries `OOT_WORK`, `OOT_EXEFS`, `OOT_ROMFS` and `OOT_DUMP`. Two checkouts of the same tree, or a moved work folder, therefore never share a lift, although the exefs *contents* are already hashed. `LIFT_ENV` should leave out path-valued variables (or hash what they point to), not list `OOT_EXEFS`. Dropping the paths is what makes our snapshot cache work. (inline) - **The biggest lever isn't in the doc: an object store shared across work folders.** `OBJ_CACHE` is `<work>/objcache`, so every new worktree pays the full ~720 s even though `flag_text` already makes the keys folder-independent. An opt-in shared store (a switch pointing `objcache` elsewhere, private per-build dir, atomic link-back) took that to 18 s here. That's the worktree-per-PR workflow agents use, and it needs no change for players. Two catches we hit: - the git-ignored inputs in `work/` (`hot_funcs.txt`, `hot_order.txt`, `chunks.txt`) must be seeded into a new checkout, or its chunk boundaries and so every lifted object name differ (make_game copies them from `ROOT/work`, which is empty in a fresh worktree); - the PGO profile's path ends up in the hot pieces' keys, so it has to sit at a content-addressed path to be shared. - **A4 and C2: data point for Phase 0.** With nothing to compile, the whole build including both links was 18 s on Linux with lld, so the exe link there is seconds, not minutes. A4 is still worth doing (it's S), but on this evidence I'd expect C2 to be unnecessary; it's worth measuring the link on Windows before designing anything. - **A6 is validated:** the one-header edit above rebuilt 16 renderer objects where a depfile would rebuild its includers. - **B1's staged swap needs an ABI check, not just `MKWVK_API_VERSION`.** The renderer `.so` also takes symbols from the exe. An exe and a `.so` built from different trees fail at load, before `mkwvk_init`'s version check runs, and to the user it looks like the game silently never opens a window. We check `nm -D --undefined-only hookshot_vk.so | c++filt | grep mkwvk` is empty before dropping it in. `dev.py` should do the equivalent before renaming `.new` over the DLL, and `load_vulkan`'s failure line on Linux should carry `dlerror()`'s text. (inline) - **D4/D5: compare inside a session, not between sessions.** For a flicker-type metric, counts varied 5–20x from one session to the next on the same build, so `--compare` against the last run would mostly report noise. What worked: an A/B mode in the scenario itself (a file-based switch flipped every N seconds, per-half results), always with a sanity metric next to the target one. We record mean luma: one "fix" showed 0 flashes because it had blown the exposure out. I'd make in-session A/B a first-class scenario option and give D5's summaries a variance/sanity column. A fresh copy of a clean save per run (which D4's per-scenario user folder covers) mattered too. - **Missing from D5/§12: runs must not share the machine with a build.** fps and frame-time numbers taken beside a compile are worthless, and with `dev.py run --watch` rebuilding on save that will happen by default. A simple lock (a build waits for a timing run, a timing run waits for builds) in `dev.py` covers it. I'm happy to upstream the shared-store wrapper (as an opt-in on `build.py`/`make_game.py`) and the in-session A/B pieces if they'd help. Say if you want them; they'd come as their own PRs under A and D. **On the decisions** 1. **What agents may read: strict.** Agents never read `work/` or anything made from the game's files: lifted C, `functions.txt`, `image.bin`, linker maps of a build, dumps, captures with game data. That follows directly from CONTRIBUTING.md and is how `scripts/cpu_categories.py` already treats a map of the game's code. The related question gets the same answer: a hand-written table of game-function addresses with names and meanings is a map of the game's code and stays local, not in the repo. F4's in-repo part goes with it; a local-only lookup is fine. (inline) 4. **Warnings and clang-tidy: advisory to start.** Everything builds with `-Wno-everything` today, so the first baseline will be large; let it settle before anything fails CI. 6. **Async update check:** agreed it's its own player-facing PR with a worklist row, measured on a slow network, not part of this work. Across all of it: anything shared or dev-only stays opt-in, and players' paths and the work-folder layout the Builders use don't change. 2, 3 and 5 I'll come back to; for 5 I'd wait for the Windows link numbers. Nit: the doc is long for a proposal. Once the decisions are in, §2 could move into the subsystem maps (F1) it is meant to seed, and the proposal could keep just the plan.

docs/dev-loop-proposal.md

					
				@ -0,0 +1,910 @@

				# Proposal: a faster development loop

My position: no. A hand-written table of game-function addresses with names and meanings is a map of the game's code, the same thing cpu_categories.py already keeps local. It can live in a local file outside the repo that a local tool reads with --local, but not in docs/. That also settles F4: the in-repo table goes, and a local-only lookup is fine.

My position: no. A hand-written table of game-function addresses with names and meanings is a map of the game's code, the same thing `cpu_categories.py` already keeps local. It can live in a local file outside the repo that a local tool reads with `--local`, but not in `docs/`. That also settles F4: the in-repo table goes, and a local-only lookup is fine.

OOT_EXEFS is a path. lift_key already hashes the exefs files' contents, and today it also hashes env's OOT_WORK / OOT_EXEFS / OOT_ROMFS / OOT_DUMP values, so two checkouts of the same tree (or a moved work folder) never share a lift. I'd make LIFT_ENV cover only the variables whose values change the lifted C, and hash path-valued inputs by content. In our setup, keying the lift without the paths gave byte-identical lifted C across folders.

`OOT_EXEFS` is a path. `lift_key` already hashes the exefs files' *contents*, and today it also hashes `env`'s `OOT_WORK` / `OOT_EXEFS` / `OOT_ROMFS` / `OOT_DUMP` values, so two checkouts of the same tree (or a moved work folder) never share a lift. I'd make `LIFT_ENV` cover only the variables whose values change the lifted C, and hash path-valued inputs by content. In our setup, keying the lift without the paths gave byte-identical lifted C across folders.

Comparing against the last run is weaker than it looks: for a flicker-type metric, counts varied 5–20x between sessions of the same build here. An in-session A/B mode (a switch flipped every N seconds through a file, results per half) was the only comparison that held up. Also record a sanity metric beside the target one; mean luma caught a "fix" whose 0 flashes came from a blown-out exposure.

docs/dev-loop-proposal.md

					
				@ -0,0 +104,4 @@

				**Compiling** ([`nx2/build.py`](../nx2/build.py)):

				- **Objects are keyed by content.** `object_key` is a sha1 of the source, the normalised flags and a blob of headers. Objects are hard-linked from `<work>/objcache/<key>.o` when present (a home-grown ccache).

The cache is per work folder (OBJ_CACHE = <work>/objcache), but flag_text already makes keys folder-independent. An opt-in store shared across work folders, each build using a private dir linked from it and atomically linked back, took a fresh worktree of main from ~720 s to 18 s here, byte-identical. I think it deserves an item under A. It also needs the git-ignored work/ inputs (hot list, hot order, chunk boundaries) seeded into new checkouts, or their object names differ.

The cache is per work folder (`OBJ_CACHE = <work>/objcache`), but `flag_text` already makes keys folder-independent. An opt-in store shared across work folders, each build using a private dir linked from it and atomically linked back, took a fresh worktree of `main` from ~720 s to 18 s here, byte-identical. I think it deserves an item under A. It also needs the git-ignored `work/` inputs (hot list, hot order, chunk boundaries) seeded into new checkouts, or their object names differ.

docs/dev-loop-proposal.md

					
				@ -0,0 +458,4 @@

				The bridge already does the hard part: launch, wait for `in_play` or a map, teleport, screenshots, saves. B1 adds:

				1. **A `relaunch` command** in the bridge (`host/control.c`) and in `hookshot_ctl.py`. It takes the Restart path the Insert menu uses, and the client reconnects to the new process.

				2. **A staged DLL swap, dev only** (`HOS_DEV_DLL_SWAP`). `dev.py` builds `hookshot_vk.dll.new`. At start-up, before `load_vulkan`, the host renames it over the old DLL. The existing `HOS_RESTART_OF` wait already orders that after the old process has exited. Release never looks for `.new`.

Worth adding an ABI check before the rename. The .so/DLL also resolves symbols from the exe, so an exe and a renderer built from different trees fail at load, before mkwvk_init's version check. On Linux it looks like the game just never opens a window. We check nm -D --undefined-only hookshot_vk.so | c++filt | grep mkwvk is empty first. On Linux load_vulkan's failure log should also carry dlerror() so the missing symbol is named.

Worth adding an ABI check before the rename. The `.so`/DLL also resolves symbols from the exe, so an exe and a renderer built from different trees fail at load, before `mkwvk_init`'s version check. On Linux it looks like the game just never opens a window. We check `nm -D --undefined-only hookshot_vk.so | c++filt | grep mkwvk` is empty first. On Linux `load_vulkan`'s failure log should also carry `dlerror()` so the missing symbol is named.

hckrman101 commented

2026-11-12 03:54:58 +00:00

(Migrated from github.com)

On the decisions still open after my review:

Entry point: nx2/dev.py. It imports make_game's profiles (the proposed nx2/profiles.py), and scripts/ is for standalone tools.
Scenarios: tests/scenarios/, no new top-level folder. Mark them as needing the game, so run_all.py skips them in CI like the other game-only checks.
Warnings / clang-tidy: advisory to start, as I said. Once the baseline has settled, new findings in changed renderer files can fail CI. That gets its own PR.
Dev-only host split: not now. Linux links in seconds here, so decide it only if Phase 0's Windows link times say the exe link is the bottleneck.
Async update check: yes, as its own player-facing PR with a worklist row, measured on a slow network.

With those settled, the A items can go ahead as small PRs once Phase 0 has numbers.

On the decisions still open after my review: 2. **Entry point:** `nx2/dev.py`. It imports make_game's profiles (the proposed `nx2/profiles.py`), and `scripts/` is for standalone tools. 3. **Scenarios:** `tests/scenarios/`, no new top-level folder. Mark them as needing the game, so `run_all.py` skips them in CI like the other game-only checks. 4. **Warnings / clang-tidy:** advisory to start, as I said. Once the baseline has settled, new findings in changed renderer files can fail CI. That gets its own PR. 5. **Dev-only host split:** not now. Linux links in seconds here, so decide it only if Phase 0's Windows link times say the exe link is the bottleneck. 6. **Async update check:** yes, as its own player-facing PR with a worklist row, measured on a slow network. With those settled, the A items can go ahead as small PRs once Phase 0 has numbers.