feat(build): a terminal dashboard for the build's steps and files #425

Open

suporte_de_cadeira wants to merge 4 commits from suporte_de_cadeira/Longshot:feat/build-dashboard into main

pull from: suporte_de_cadeira/Longshot:feat/build-dashboard

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 5 Commits 4 Files changed 9 +993 -15

suporte_de_cadeira commented 2026-11-14 13:12:10 +00:00

First-time contributor

• Copy link
• Copy Markdown

Why

A build started from the command line prints one line a file and says nothing about which file is compiling now, so a long first build is opaque: it is hard to tell whether it is stuck, how far the compile is, or which worker is on what. This adds a terminal dashboard that reads the @@ lines the builder already prints.

What changed

• nx2/build_ui.py: runs build.py --standalone (or make_game.py with --make); full screen on a terminal, plain lines elsewhere; --attach follows a log another build writes and ends when the log says the build ended; --demo shows the screen without a game; Ctrl+C stops the build and its compilers; no new dependencies.
• nx2/build.py: prints @@work <name> <MB> <worker> when a module's compiler starts, inside the try whose finally releases the memory-gate lease, and writes every @@ line under one lock so two compilers cannot interleave one; @@file still says when it ends. This is the only new protocol line.
• nx2/make_game.py: the protocol documents @@work.
• tests/test_build_ui.py: the parser, the counts, the screen text, the full-screen redraw, more than six workers, two sources with one base name, the attached log's end and LONGSHOT_TIMES.
• tests/test_compiler_failure.py and tests/test_compile_retry_jobs.py: their fixtures cut compile_one, which now uses threading and prints inside its try; the failure test also checks that a closed stdout releases the memory-gate lease, and that 200 threads get whole @@ lines through announce.
• docs/development.md: the dashboard's command beside the standalone build.

Flags and settings

The dashboard is a command-line tool; it adds no new environment variable, menu row or options-file key. It sets the existing LONGSHOT_TIMES for the build.py it starts, as make_game.py does, so the build reports its plan and per-file times; a value already in the environment is kept.

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
LONGSHOT_TIMES	the record of the earlier builds' per-file seconds; build_ui.py points the build.py it starts at <work>/build_times.json	set by build_ui.py when it runs build.py alone; make_game.py sets it itself	on: the file is only read, and a missing one is the first build	the dashboard's counts, ETA and per-file times	a build.py run by hand (unchanged)	Linux or macOS full game builds

Measurements

This is a build-time tool; the game does not run it. As CONTRIBUTING and AGENTS say for a game-free change, the game-free checks and the build they drive are the measurement. The full game build below is the end-to-end test.

main: 7ff5256e (2026-11-14)
this PR: feat/build-dashboard @ 5a2cbd6b
Hardware: Windows 11, RTX 3080 10 GB, Python 3.14.4, the pinned llvm-mingw 20261104; mooki's macOS 26.7.1 run on the Mac integration branch
Scene: the full game build from the installed dump (romfs + exefs), 643 files at 6 jobs; the built game's first start at 1280x720
Method: tests/run_all.py, the dashboard's tests, the full build through build_ui.py --make --jobs 6, and the built game's first start
How to reproduce: python -I -B tests/test_build_ui.py; python -I -B tests/run_all.py; python -I nx2/build_ui.py --make --jobs 6 -- --romfs <dump>/romfs --exefs <dump>/exefs --out <folder> --work <work> --language en-US --compile-shaders most; then <folder>/longshot.exe

	main @ 7ff5256e	this PR @ 5a2cbd6b
tests/run_all.py	the 8 failures below fail here too, each run alone	492 passed, 8 failed, 72 skipped
tests/test_build_ui.py	not present	PASS
tests/test_compiler_failure.py	1 test, PASS	3 tests, PASS
the renderer through build_ui.py	n/a (no dashboard)	205 files, 0 failed, linked; 01:42 (compile 01:41)
the full game through build_ui.py --make	n/a (no dashboard)	643 files, 0 failed; 33:25 (compile 16:28); 24,453 pipelines prepared
the built game's first start	n/a	60.0 fps; 600 presents; 1280x720

PASS: the build dashboard reads the stages, the plan, the files in flight, the cache and the error path
@@plan 643 643
@@done <folder>
build done in 33:25 (compile 16:28); log: <work>/build_ui.log

Testing

Tested: Windows 11, Python 3.14.4. tests/run_all.py gives 492 passed, 8 failed, 72 skipped in 148 s. The 8 also fail on unmodified main 7ff5256e, each run alone: test_fpx_sticky and test_spawn_emit (the a64emit build), test_bug_report (NoneType has no splitlines), test_oot_state_tap (no game listens), test_tested_configurations, test_release (openssl missing), test_cpm_discovery, and test_walk_key. The dashboard's tests pass, including the broken-pipe lease test, which fails on the old print placement and passes with it inside the try.

A real renderer build through build_ui.py --plain --jobs 6 compiled 205 files at 6 jobs, 0 failed, and linked; LONGSHOT_TIMES gave @@plan 205 205, 205 @@work, 205 @@file and 206 @@eta lines, and --plain printed no @@eta chatter. The first such run gave 204 @@work for 205 @@file: the workers' prints could interleave, so announce() now writes every @@ line under one lock, and a test starts 200 threads through it. --attach on that log ended at the linked line and reported the compile's 01:41 from compiled in 101s.

A full game build through build_ui.py --make --jobs 6, from the installed dump's romfs and exefs, ran read, lift, compile, finish and shaders: 643 files, 0 failed, 33:25 in all (compile 16:28), and the shaders' preparation built 24,453 pipelines in 430.7 s. It wrote longshot.exe (454,863,360 bytes) and longshot_vk.dll (13,630,464 bytes) into a new folder. The built game started and ran at 60.0 fps (600 presents, 1280x720); its screenshot is below. T-RVZR passed with this run.

mooki's macOS run on the Mac integration branch built 212 renderer files and exercised the full screen, the six workers, a live resize, Ctrl+C and the plain output; its five defects are the fix commit's subject. The 16-row screenshot below repeats the full-screen case after the scroll fix.

Not tested: an extended gameplay session beyond the first start (the field rendered at 60 fps; no save was played); Linux and macOS full game builds; Terminal.app and iTerm2 by eye; the SIGHUP/SIGTERM path on a real terminal (the handler is in place, and the broken-pipe lease test covers the leak it prevents).

Needs other testers: None.

Screenshots

The demo (no game files) at 110x34: the compile with six workers in flight, and the finished state. The third is the same demo in a 16-row terminal after the scroll fix: the title stays visible and the workers beyond the rows are counted.

The real build through build_ui.py --make --jobs 6: the compiler with six workers, and [Compile ok] with 643/643 while the shaders prepare. The command line and log are cropped, so no local paths are shown.

The built game's first start at 1280x720:

Risk and rollback

The dashboard is terminal-only and runs the existing builder; the game, the renderer and the build results are unchanged. The one new protocol line is additive, and a reader that does not know it ignores it. Reverting the commit is enough; there is no flag to turn off.

Reviewer notes

The fix commit 8bf1493c addresses the review's points in order: the @@work print inside compile_one's try and SIGHUP/SIGTERM (1); LONGSHOT_TIMES from build_ui.py when it runs build.py alone (2); no newline after the last screen line (3); --attach ends on @@done/@@error, a build.py log's linked line, or a make_game.py @@times, reads the compile time from compiled in Ns, says what Ctrl+C does, and names a missing path (4); all workers shown or counted, one row cleared per @@file when two sources share a base name, --plain without @@eta and with build_ui: for its own lines, and the log under OOT_WORK (5). The last commit 5a2cbd6b makes each @@ line one write after a clean build showed 204 @@work for 205 files. The broken-pipe test fails on the old placement and passes now. End-to-end, a full game build from the dump through build_ui.py --make --jobs 6 (643 files, 0 failed, 33:25, compile 16:28) produced the game that started at 60.0 fps; T-RVZR passed.

Issues

None.

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• Screenshots for a change that shows on screen.
• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened.
• Docs updated.

## Why A build started from the command line prints one line a file and says nothing about which file is compiling now, so a long first build is opaque: it is hard to tell whether it is stuck, how far the compile is, or which worker is on what. This adds a terminal dashboard that reads the `@@` lines the builder already prints. ## What changed - `nx2/build_ui.py`: runs `build.py --standalone` (or `make_game.py` with `--make`); full screen on a terminal, plain lines elsewhere; `--attach` follows a log another build writes and ends when the log says the build ended; `--demo` shows the screen without a game; `Ctrl+C` stops the build and its compilers; no new dependencies. - `nx2/build.py`: prints `@@work <name> <MB> <worker>` when a module's compiler starts, inside the `try` whose `finally` releases the memory-gate lease, and writes every `@@` line under one lock so two compilers cannot interleave one; `@@file` still says when it ends. This is the only new protocol line. - `nx2/make_game.py`: the protocol documents `@@work`. - `tests/test_build_ui.py`: the parser, the counts, the screen text, the full-screen redraw, more than six workers, two sources with one base name, the attached log's end and `LONGSHOT_TIMES`. - `tests/test_compiler_failure.py` and `tests/test_compile_retry_jobs.py`: their fixtures cut `compile_one`, which now uses `threading` and prints inside its `try`; the failure test also checks that a closed stdout releases the memory-gate lease, and that 200 threads get whole `@@` lines through `announce`. - `docs/development.md`: the dashboard's command beside the standalone build. ## Flags and settings The dashboard is a command-line tool; it adds no new environment variable, menu row or options-file key. It sets the existing `LONGSHOT_TIMES` for the `build.py` it starts, as `make_game.py` does, so the build reports its plan and per-file times; a value already in the environment is kept. | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_TIMES` | the record of the earlier builds' per-file seconds; `build_ui.py` points the `build.py` it starts at `<work>/build_times.json` | set by `build_ui.py` when it runs `build.py` alone; `make_game.py` sets it itself | on: the file is only read, and a missing one is the first build | the dashboard's counts, ETA and per-file times | a `build.py` run by hand (unchanged) | Linux or macOS full game builds | ## Measurements This is a build-time tool; the game does not run it. As CONTRIBUTING and AGENTS say for a game-free change, the game-free checks and the build they drive are the measurement. The full game build below is the end-to-end test. main: 7ff5256e (2026-11-14) this PR: feat/build-dashboard @ 5a2cbd6b Hardware: Windows 11, RTX 3080 10 GB, Python 3.14.4, the pinned llvm-mingw 20261104; mooki's macOS 26.7.1 run on the Mac integration branch Scene: the full game build from the installed dump (romfs + exefs), 643 files at 6 jobs; the built game's first start at 1280x720 Method: `tests/run_all.py`, the dashboard's tests, the full build through `build_ui.py --make --jobs 6`, and the built game's first start How to reproduce: `python -I -B tests/test_build_ui.py`; `python -I -B tests/run_all.py`; `python -I nx2/build_ui.py --make --jobs 6 -- --romfs <dump>/romfs --exefs <dump>/exefs --out <folder> --work <work> --language en-US --compile-shaders most`; then `<folder>/longshot.exe` | | main @ 7ff5256e | this PR @ 5a2cbd6b | |---|---|---| | `tests/run_all.py` | the 8 failures below fail here too, each run alone | 492 passed, 8 failed, 72 skipped | | `tests/test_build_ui.py` | not present | PASS | | `tests/test_compiler_failure.py` | 1 test, PASS | 3 tests, PASS | | the renderer through `build_ui.py` | n/a (no dashboard) | 205 files, 0 failed, linked; 01:42 (compile 01:41) | | the full game through `build_ui.py --make` | n/a (no dashboard) | 643 files, 0 failed; 33:25 (compile 16:28); 24,453 pipelines prepared | | the built game's first start | n/a | 60.0 fps; 600 presents; 1280x720 | ``` PASS: the build dashboard reads the stages, the plan, the files in flight, the cache and the error path @@plan 643 643 @@done <folder> build done in 33:25 (compile 16:28); log: <work>/build_ui.log ``` ## Testing **Tested:** Windows 11, Python 3.14.4. `tests/run_all.py` gives 492 passed, 8 failed, 72 skipped in 148 s. The 8 also fail on unmodified main `7ff5256e`, each run alone: `test_fpx_sticky` and `test_spawn_emit` (the a64emit build), `test_bug_report` (`NoneType` has no `splitlines`), `test_oot_state_tap` (no game listens), `test_tested_configurations`, `test_release` (openssl missing), `test_cpm_discovery`, and `test_walk_key`. The dashboard's tests pass, including the broken-pipe lease test, which fails on the old print placement and passes with it inside the `try`. A real renderer build through `build_ui.py --plain --jobs 6` compiled 205 files at 6 jobs, 0 failed, and linked; `LONGSHOT_TIMES` gave `@@plan 205 205`, 205 `@@work`, 205 `@@file` and 206 `@@eta` lines, and `--plain` printed no `@@eta` chatter. The first such run gave 204 `@@work` for 205 `@@file`: the workers' prints could interleave, so `announce()` now writes every `@@` line under one lock, and a test starts 200 threads through it. `--attach` on that log ended at the `linked` line and reported the compile's 01:41 from `compiled in 101s`. A full game build through `build_ui.py --make --jobs 6`, from the installed dump's romfs and exefs, ran read, lift, compile, finish and shaders: 643 files, 0 failed, 33:25 in all (compile 16:28), and the shaders' preparation built 24,453 pipelines in 430.7 s. It wrote `longshot.exe` (454,863,360 bytes) and `longshot_vk.dll` (13,630,464 bytes) into a new folder. The built game started and ran at 60.0 fps (600 presents, 1280x720); its screenshot is below. T-RVZR passed with this run. mooki's macOS run on the Mac integration branch built 212 renderer files and exercised the full screen, the six workers, a live resize, `Ctrl+C` and the plain output; its five defects are the fix commit's subject. The 16-row screenshot below repeats the full-screen case after the scroll fix. **Not tested:** an extended gameplay session beyond the first start (the field rendered at 60 fps; no save was played); Linux and macOS full game builds; Terminal.app and iTerm2 by eye; the SIGHUP/SIGTERM path on a real terminal (the handler is in place, and the broken-pipe lease test covers the leak it prevents). **Needs other testers:** None. ## Screenshots The demo (no game files) at 110x34: the compile with six workers in flight, and the finished state. The third is the same demo in a 16-row terminal after the scroll fix: the title stays visible and the workers beyond the rows are counted.    The real build through `build_ui.py --make --jobs 6`: the compiler with six workers, and `[Compile ok]` with 643/643 while the shaders prepare. The command line and log are cropped, so no local paths are shown.   The built game's first start at 1280x720:  ## Risk and rollback The dashboard is terminal-only and runs the existing builder; the game, the renderer and the build results are unchanged. The one new protocol line is additive, and a reader that does not know it ignores it. Reverting the commit is enough; there is no flag to turn off. ## Reviewer notes The fix commit `8bf1493c` addresses the review's points in order: the `@@work` print inside `compile_one`'s `try` and SIGHUP/SIGTERM (1); `LONGSHOT_TIMES` from `build_ui.py` when it runs `build.py` alone (2); no newline after the last screen line (3); `--attach` ends on `@@done`/`@@error`, a `build.py` log's `linked` line, or a `make_game.py` `@@times`, reads the compile time from `compiled in Ns`, says what `Ctrl+C` does, and names a missing path (4); all workers shown or counted, one row cleared per `@@file` when two sources share a base name, `--plain` without `@@eta` and with `build_ui:` for its own lines, and the log under `OOT_WORK` (5). The last commit `5a2cbd6b` makes each `@@` line one write after a clean build showed 204 `@@work` for 205 files. The broken-pipe test fails on the old placement and passes now. End-to-end, a full game build from the dump through `build_ui.py --make --jobs 6` (643 files, 0 failed, 33:25, compile 16:28) produced the game that started at 60.0 fps; T-RVZR passed. ## Issues None. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] Screenshots for a change that shows on screen. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.

fullbuild-compile-safe.png

11 KiB

fullbuild-shaders-safe.png

6.1 KiB

game-first-start.png

998 KiB

longshot-demo-compile.png

19 KiB

longshot-demo-done.png

21 KiB

longshot-demo-16rows.png

16 KiB

suporte_de_cadeira added 1 commit 2026-11-14 13:12:10 +00:00

feat(build): a terminal dashboard for the build's steps and files ... d36c5e75de

A build from the command line prints one line a file and does not say which file is compiling now. build_ui.py runs build.py (or make_game.py with --make) and draws the steps, the compile plan and cache, the files in flight per worker and the ETA from the @@ lines; --plain keeps the build's log for CI, --attach follows a log another build writes, and --demo shows the screen without a game.

build.py now prints @@work when a module's compiler starts (make_game.py documents it); @@file still says when it ends. No new dependencies: the full screen uses ANSI on a terminal and falls back to plain lines elsewhere.

The two checks that cut compile_one from build.py now stub threading, which the new line uses.

suporte_de_cadeira added 1 commit 2026-11-14 13:19:39 +00:00

docs(build): the dashboard's command in the development table ... 0771bec524

The standalone build row now names build_ui.py and its --make and --plain forms, so the tool is found beside the build it reads.

suporte_de_cadeira commented 2026-11-14 13:30:51 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Added the dashboard's row to the development command table in docs/development.md (0771bec5); no behavior change, and the screenshots and test results above still apply.

Added the dashboard's row to the development command table in docs/development.md (0771bec5); no behavior change, and the screenshots and test results above still apply.

mooki added the

kind

feature

priority

low

labels 2026-11-14 13:34:07 +00:00

mooki added the

area/install

label 2026-11-14 16:27:55 +00:00

mooki commented 2026-11-14 16:49:36 +00:00

Member

• Copy link
• Copy Markdown

I tested T-RVZR on a Mac, the only machine here that builds the game. In short: the six workers and their files
move, and Ctrl+C stops the build and all of its compilers, both full screen and piped. I found one real bug: closing
the terminal leaves build.py hung for ever, and the root cause is in the new @@work line. There are also a few
display problems. Repros are below.

Platform

• macOS 26.7.1 (25G241) on an Apple M4 Max (16 cores, 64 GB); Python 3.14.10.
• Toolchain: Apple clang 21.0.0 (clang-2100.3.34.2) from the Command Line Tools, MacOSX27.0.sdk, deployment
  target 14.4, arm64.
• Terminal: a real pty (Python's pty.fork, 110×34, TERM=xterm-256color), driven by a script. Ctrl+C was the 0x03
  byte written to the pty, so the pty's line discipline delivered SIGINT exactly as a terminal does. I did not watch
  it by eye in Terminal.app or iTerm2.

Base, and why

• main cannot build on macOS yet, so I tested on the Mac integration branch (macos/integrated 1c53cff1) plus this
  PR's two commits, which cherry-picked cleanly.
• Merging the PR head instead conflicts in 17 files, none of them yours. The merge would also bring in the 39 main
  commits between the Mac branch's base and 5e072fc8, and those are what conflict.
• The build: build_ui.py --jobs 6, which runs build.py --standalone --jobs 6, with RX_VK_ONLY=1 and a fresh
  OOT_WORK. Every renderer source really compiles (212 files, 50–60 s at 6 jobs) and longshot_vk.dylib links.
• I did not compile the lifted game: a fresh game work folder needs about 27 GB, and the disk did not have that free.
  The dashboard's code path is the same; only the files differ.

What worked

• tests/test_build_ui.py, tests/test_compiler_failure.py and tests/test_compile_retry_jobs.py pass on macOS.
• Full screen: six worker rows, each naming a file, redrawn about every 0.1 s (the median time between redraws was
  0.07–0.10 s).
  • In a full run, six rows were on screen in 741 of 763 redraws.
  • Each worker (#0 to #5) showed 27–39 different files in a run.
  • ps showed six compilers in the build's process group most of the time.
  • No line was wider than the terminal. After a live resize to 80×24 and to 60×12 (with --demo), the next
    redraws fit within half a second.
• With LONGSHOT_TIMES set, as make_game sets it:
  • the compile line went from 0/212 to 212/212 (new 212, cached 0);
  • the step went 3% → 99% → ok, and the ETA counted down from 00:44 to 00:00;
  • the rows cleared at the end, and it exited 0 with build done in 01:02 (compile 00:58).
• Ctrl+C mid-compile, full screen (two runs):
  • exit 130 within 0.07–0.21 s, with build cancelled in 00:21 (compile --:--), and the screen and cursor restored;
  • every compiler of the build was gone within 1 s (in one run one compiler was still exiting at the moment build_ui
    returned), and nothing of the build was left at 1, 3 and 6 s;
  • the object folder had as many .sha1 stamps as objects (77/77, 29/29), so no half-written object is kept.
• Ctrl+C with the output piped (SIGINT to the job's process group, as a shell sends it): the same result. It
  exited 130 in 0.07 s, left nothing running, and kept 114/114 objects and stamps. The plain output is the
  build_ui: command line, build.py's own lines and the summary.

Defects

1. Closing the terminal window, or kill <build_ui pid>, leaves build.py running for ever.

   • build.py is started in a session of its own (start_new_session=True), so the terminal's hangup never reaches
     it, and stop_tree runs only on KeyboardInterrupt.

   • The compilers in flight finish their files (with lifted files that would be minutes and GBs each), then
     build.py hangs. It was still there (parent 1, sleeping, its six executor threads waiting) 5 minutes after a
     SIGHUP and 60 s after a SIGTERM, until I killed it.

   • Why it hangs: in compile_one, the new print("@@work …") comes after GATE.enter(src) and before the try:
     whose finally calls GATE.leave(lease). Once the reader of build.py's stdout is gone, the print raises
     BrokenPipeError, and the memory-gate lease is never given back. After enough leaked leases,
     MemoryGate.enter() waits for ever, and the ThreadPoolExecutor shutdown waits on it.

   • Repro in a terminal: run python -I nx2/build_ui.py --jobs 6; while it compiles, close the terminal window (or
     kill -TERM build_ui); then run pgrep -fl 'nx2/build.py --standalone'.

   • Game-free repro, no compiler:

     1. Take compile_one, MemoryGate, mem_limit_mb, mem_need and rss_mb from build.py, as
        tests/test_compiler_failure.py takes compile_one.
     2. Set LONGSHOT_MEM_LIMIT_GB=2 and make sys.stdout.write raise BrokenPipeError.
     3. Call compile_one twice on a .cpp job.

     Result: both leases stay held, and a third GATE.enter() never returns. On the Mac branch without this PR,
     0 leases are held. With the print moved to the first line inside the try:, 0 are held too. That makes a test
     that fails on this branch and passes with the fix.

   • Suggested fix:

     • move the print inside the try:;
     • have build_ui treat SIGHUP and SIGTERM like Ctrl+C, stopping the build's process group before it exits.

   • Not run: I expect any other reader of build.py's output that goes away first (make_game ended alone, or a cancel
     that ends make_game but not its children) to leave build.py hung the same way.

2. The test request's own command shows no counts and leaves stale rows.

   • build.py prints @@plan, @@file and @@eta only when LONGSHOT_TIMES is set. make_game sets it;
     build_ui.py does not.
   • So with python -I nx2/build_ui.py --jobs 6:
     • the compile line is empty (no bar, no counts);
     • [Compile] never shows as current or done, while [Link] and [Finish] turn ok;
     • the ETA stays --:--, and the summary says compile --:--;
     • at done, the six worker rows are still listed with the last file each worker compiled (aged 4–8 s), because
       only @@file clears a row.
   • With LONGSHOT_TIMES pointing at a file that does not exist, all of this works (the LONGSHOT_TIMES run above),
     and build.py does not create that file. Passing it from build_ui when it runs build.py itself would fix this.

3. The title row scrolls off once the screen is full.

   • Screen.draw writes ESC[K\n after every line, including the last one.
   • Once the log tail fills the screen, render_lines returns exactly height lines: 34 lines at 34 rows in a real
     build.
   • The newline on the bottom row then scrolls the screen up by one line at every redraw. The top line
     (Longshot build … elapsed … ETA) is never visible, and the bottom row is blank.
   • Measured with a VT model (a line feed on the bottom row scrolls, as in any VT100-style terminal): 107 of 175
     redraws of a real build at 110×34 scrolled and lost the title, every redraw after the log filled the screen.
   • The demo at 110×34 never fills the screen (22 lines at most), which is why your screenshots look right.
   • Repro: python -I -B nx2/build_ui.py --demo in a terminal 16 rows high.
   • Fix: no newline after the last line, or draw height - 1 lines.

4. --attach never ends.

   • On a finished log it keeps the screen until Ctrl+C, then exits 0.
   • On a build.py log, which has no @@done, the status stays running.
   • The elapsed and compile timers count from the attach: compile 00:12 for a 58 s compile.
   • The footer says Ctrl+C stops the build, but in attach mode Ctrl+C leaves the build alone.
   • Attached to a path that does not exist yet, the screen stays blank with no message.

5. Smaller:

   • At most 6 worker rows are drawn (min(6, …)), and nothing says more are running. A game-free check on
     render_lines with 8 files in flight drew 6 rows and no mention of the other 2. On a 16-core Mac, --jobs
     above 6 is likely.
   • Rows are matched by base name, and two pairs of renderer sources share one (opcodes.cpp twice in gpu/vk/eden;
     type.cpp in gpu/vk/eden and gpu/vk/sirit). So the @@file of one clears both rows: 210 names were seen
     for 212 files.
   • Piped with LONGSHOT_TIMES, a build: eta N line follows every file: 213 of the 436 lines. build.py's own
     build: macOS arm64 (baseline ISA) line carries the same build: prefix as the dashboard's lines.
   • The default log is <repo>/work/build_ui.log even when OOT_WORK points elsewhere.

@@work under make_game's Mac path: make_game prints every line build.py prints. So @@work should reach
build_ui.py --make on the Mac too, and it should also appear raw in make_game's own console, beside the @@file
lines that console already shows. This is from reading the source; I did not run --make.

Not tested

• A game build (lifted files: their names, MB above 0, compiles of minutes), and --make / make_game.
• Terminal.app and iTerm2 by eye. Point 3 comes from the VT model and the line counts.
• Ctrl+C during the link or the retry pass; a failing compile (failed / again); real compiles at other --jobs;
  --no-color / NO_COLOR.
• tests/run_all.py (only the three tests above); Windows and Linux.

I tested T-RVZR on a Mac, the only machine here that builds the game. In short: the six workers and their files move, and Ctrl+C stops the build and all of its compilers, both full screen and piped. I found one real bug: closing the terminal leaves `build.py` hung for ever, and the root cause is in the new `@@work` line. There are also a few display problems. Repros are below. **Platform** - macOS 26.7.1 (25G241) on an Apple M4 Max (16 cores, 64 GB); Python 3.14.10. - Toolchain: Apple clang 21.0.0 (clang-2100.3.34.2) from the Command Line Tools, MacOSX27.0.sdk, deployment target 14.4, arm64. - Terminal: a real pty (Python's `pty.fork`, 110×34, `TERM=xterm-256color`), driven by a script. Ctrl+C was the 0x03 byte written to the pty, so the pty's line discipline delivered SIGINT exactly as a terminal does. I did not watch it by eye in Terminal.app or iTerm2. **Base, and why** - main cannot build on macOS yet, so I tested on the Mac integration branch (`macos/integrated` 1c53cff1) plus this PR's two commits, which cherry-picked cleanly. - Merging the PR head instead conflicts in 17 files, none of them yours. The merge would also bring in the 39 main commits between the Mac branch's base and 5e072fc8, and those are what conflict. - The build: `build_ui.py --jobs 6`, which runs `build.py --standalone --jobs 6`, with `RX_VK_ONLY=1` and a fresh `OOT_WORK`. Every renderer source really compiles (212 files, 50–60 s at 6 jobs) and `longshot_vk.dylib` links. - I did not compile the lifted game: a fresh game work folder needs about 27 GB, and the disk did not have that free. The dashboard's code path is the same; only the files differ. **What worked** - `tests/test_build_ui.py`, `tests/test_compiler_failure.py` and `tests/test_compile_retry_jobs.py` pass on macOS. - **Full screen:** six worker rows, each naming a file, redrawn about every 0.1 s (the median time between redraws was 0.07–0.10 s). - In a full run, six rows were on screen in 741 of 763 redraws. - Each worker (#0 to #5) showed 27–39 different files in a run. - `ps` showed six compilers in the build's process group most of the time. - No line was wider than the terminal. After a live resize to 80×24 and to 60×12 (with `--demo`), the next redraws fit within half a second. - **With `LONGSHOT_TIMES` set, as make_game sets it:** - the compile line went from 0/212 to 212/212 (new 212, cached 0); - the step went 3% → 99% → ok, and the ETA counted down from 00:44 to 00:00; - the rows cleared at the end, and it exited 0 with `build done in 01:02 (compile 00:58)`. - **Ctrl+C mid-compile, full screen (two runs):** - exit 130 within 0.07–0.21 s, with `build cancelled in 00:21 (compile --:--)`, and the screen and cursor restored; - every compiler of the build was gone within 1 s (in one run one compiler was still exiting at the moment build_ui returned), and nothing of the build was left at 1, 3 and 6 s; - the object folder had as many `.sha1` stamps as objects (77/77, 29/29), so no half-written object is kept. - **Ctrl+C with the output piped** (SIGINT to the job's process group, as a shell sends it): the same result. It exited 130 in 0.07 s, left nothing running, and kept 114/114 objects and stamps. The plain output is the `build_ui:` command line, build.py's own lines and the summary. **Defects** 1. **Closing the terminal window, or `kill <build_ui pid>`, leaves `build.py` running for ever.** - build.py is started in a session of its own (`start_new_session=True`), so the terminal's hangup never reaches it, and `stop_tree` runs only on `KeyboardInterrupt`. - The compilers in flight finish their files (with lifted files that would be minutes and GBs each), then build.py hangs. It was still there (parent 1, sleeping, its six executor threads waiting) 5 minutes after a SIGHUP and 60 s after a SIGTERM, until I killed it. - Why it hangs: in `compile_one`, the new `print("@@work …")` comes after `GATE.enter(src)` and before the `try:` whose `finally` calls `GATE.leave(lease)`. Once the reader of build.py's stdout is gone, the print raises `BrokenPipeError`, and the memory-gate lease is never given back. After enough leaked leases, `MemoryGate.enter()` waits for ever, and the `ThreadPoolExecutor` shutdown waits on it. - Repro in a terminal: run `python -I nx2/build_ui.py --jobs 6`; while it compiles, close the terminal window (or `kill -TERM` build_ui); then run `pgrep -fl 'nx2/build.py --standalone'`. - Game-free repro, no compiler: 1. Take `compile_one`, `MemoryGate`, `mem_limit_mb`, `mem_need` and `rss_mb` from build.py, as `tests/test_compiler_failure.py` takes `compile_one`. 2. Set `LONGSHOT_MEM_LIMIT_GB=2` and make `sys.stdout.write` raise `BrokenPipeError`. 3. Call `compile_one` twice on a `.cpp` job. Result: both leases stay held, and a third `GATE.enter()` never returns. On the Mac branch without this PR, 0 leases are held. With the print moved to the first line inside the `try:`, 0 are held too. That makes a test that fails on this branch and passes with the fix. - Suggested fix: - move the print inside the `try:`; - have build_ui treat SIGHUP and SIGTERM like Ctrl+C, stopping the build's process group before it exits. - Not run: I expect any other reader of build.py's output that goes away first (make_game ended alone, or a cancel that ends make_game but not its children) to leave build.py hung the same way. 2. **The test request's own command shows no counts and leaves stale rows.** - build.py prints `@@plan`, `@@file` and `@@eta` only when `LONGSHOT_TIMES` is set. make_game sets it; `build_ui.py` does not. - So with `python -I nx2/build_ui.py --jobs 6`: - the compile line is empty (no bar, no counts); - `[Compile]` never shows as current or done, while `[Link]` and `[Finish]` turn ok; - the ETA stays `--:--`, and the summary says `compile --:--`; - at `done`, the six worker rows are still listed with the last file each worker compiled (aged 4–8 s), because only `@@file` clears a row. - With `LONGSHOT_TIMES` pointing at a file that does not exist, all of this works (the `LONGSHOT_TIMES` run above), and build.py does not create that file. Passing it from build_ui when it runs build.py itself would fix this. 3. **The title row scrolls off once the screen is full.** - `Screen.draw` writes `ESC[K\n` after every line, including the last one. - Once the log tail fills the screen, `render_lines` returns exactly `height` lines: 34 lines at 34 rows in a real build. - The newline on the bottom row then scrolls the screen up by one line at every redraw. The top line (`Longshot build … elapsed … ETA`) is never visible, and the bottom row is blank. - Measured with a VT model (a line feed on the bottom row scrolls, as in any VT100-style terminal): 107 of 175 redraws of a real build at 110×34 scrolled and lost the title, every redraw after the log filled the screen. - The demo at 110×34 never fills the screen (22 lines at most), which is why your screenshots look right. - Repro: `python -I -B nx2/build_ui.py --demo` in a terminal 16 rows high. - Fix: no newline after the last line, or draw `height - 1` lines. 4. **`--attach` never ends.** - On a finished log it keeps the screen until Ctrl+C, then exits 0. - On a `build.py` log, which has no `@@done`, the status stays `running`. - The elapsed and compile timers count from the attach: `compile 00:12` for a 58 s compile. - The footer says `Ctrl+C stops the build`, but in attach mode Ctrl+C leaves the build alone. - Attached to a path that does not exist yet, the screen stays blank with no message. 5. **Smaller:** - At most 6 worker rows are drawn (`min(6, …)`), and nothing says more are running. A game-free check on `render_lines` with 8 files in flight drew 6 rows and no mention of the other 2. On a 16-core Mac, `--jobs` above 6 is likely. - Rows are matched by base name, and two pairs of renderer sources share one (`opcodes.cpp` twice in `gpu/vk/eden`; `type.cpp` in `gpu/vk/eden` and `gpu/vk/sirit`). So the `@@file` of one clears both rows: 210 names were seen for 212 files. - Piped with `LONGSHOT_TIMES`, a `build: eta N` line follows every file: 213 of the 436 lines. build.py's own `build: macOS arm64 (baseline ISA)` line carries the same `build: ` prefix as the dashboard's lines. - The default log is `<repo>/work/build_ui.log` even when `OOT_WORK` points elsewhere. **`@@work` under make_game's Mac path:** make_game prints every line build.py prints. So `@@work` should reach `build_ui.py --make` on the Mac too, and it should also appear raw in make_game's own console, beside the `@@file` lines that console already shows. This is from reading the source; I did not run `--make`. **Not tested** - A game build (lifted files: their names, MB above 0, compiles of minutes), and `--make` / make_game. - Terminal.app and iTerm2 by eye. Point 3 comes from the VT model and the line counts. - Ctrl+C during the link or the retry pass; a failing compile (`failed` / `again`); real compiles at other `--jobs`; `--no-color` / `NO_COLOR`. - `tests/run_all.py` (only the three tests above); Windows and Linux.

mooki added the

platform/linux

platform/windows

labels 2026-11-14 18:23:50 +00:00

suporte_de_cadeira added 1 commit 2026-11-14 19:31:41 +00:00

fix(build): the dashboard keeps no gate lease, ends --attach and its title ... 80c5cb53e3

The review's Mac run found the new @@work print outside compile_one's try: when the reader of build.py's stdout goes away, the print raised BrokenPipeError and the memory gate lease was never released. The print is now the first line inside the try, and build_ui stops the build on SIGHUP and SIGTERM as on Ctrl+C.

build_ui now sets LONGSHOT_TIMES when it runs build.py alone (make_game sets it itself), so the compile counts and ETA appear; it no longer writes a newline after the last screen line, so a full screen keeps its title; --attach ends on @@done/@@error, on a build.py log's linked line, and reads its compile time from 'compiled in Ns'; the footer says what Ctrl+C does there.

Also: all workers are shown (or counted), two files with one base name clear one row each, --plain drops @@eta and uses build_ui: for its lines, and the default log follows OOT_WORK. Tests cover each.

suporte_de_cadeira force-pushed feat/build-dashboard from 80c5cb53e3 to 8bf1493cff 2026-11-14 19:50:14 +00:00 Compare

suporte_de_cadeira commented 2026-11-14 19:50:14 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Reworked in 8bf1493c after mooki's Mac run: @@work is inside compile_one's try, so a closed stdout releases the memory-gate lease, and build_ui stops the build on SIGHUP/SIGTERM as on Ctrl+C (the new test fails on the old placement); build_ui sets LONGSHOT_TIMES when it runs build.py alone, so the counts, ETA and cleared rows appear (a real 205-file renderer build gave plan 205 205, 0 failed); the last screen line has no newline, so a full screen keeps its title (the 16-row screenshot is in the description); --attach ends on @@done/@@error, a build.py linked line or make_game's @@times, reads the compile time from 'compiled in Ns', says Ctrl+C stops following, and names a missing path; all workers are shown or counted, duplicate base names clear one row, --plain drops @@eta and uses build_ui:, and the log follows OOT_WORK. Rebased onto 7ff5256e; run_all is 492 passed, 8 failed (the same 8 on main), 72 skipped.

Reworked in 8bf1493c after mooki's Mac run: @@work is inside compile_one's try, so a closed stdout releases the memory-gate lease, and build_ui stops the build on SIGHUP/SIGTERM as on Ctrl+C (the new test fails on the old placement); build_ui sets LONGSHOT_TIMES when it runs build.py alone, so the counts, ETA and cleared rows appear (a real 205-file renderer build gave plan 205 205, 0 failed); the last screen line has no newline, so a full screen keeps its title (the 16-row screenshot is in the description); --attach ends on @@done/@@error, a build.py linked line or make_game's @@times, reads the compile time from 'compiled in Ns', says Ctrl+C stops following, and names a missing path; all workers are shown or counted, duplicate base names clear one row, --plain drops @@eta and uses build_ui:, and the log follows OOT_WORK. Rebased onto 7ff5256e; run_all is 492 passed, 8 failed (the same 8 on main), 72 skipped.

suporte_de_cadeira changed title from WIP: feat(build): a terminal dashboard for the build's steps and files to feat(build): a terminal dashboard for the build's steps and files 2026-11-14 19:51:12 +00:00

suporte_de_cadeira added 1 commit 2026-11-14 20:10:42 +00:00

fix(build): each @@ line is one write, so two compilers cannot interleave one ... 5a2cbd6b09

A clean 205-file renderer build had 205 @@file but 204 @@work: the worker threads' prints could interleave and one line was lost. announce() writes every @@ line (and the compile loop's human lines) under one lock; a clean build now has 205/205. A test starts 200 threads through it, and the checks that cut compile_one or read the FAILED line provide the new name.

suporte_de_cadeira commented 2026-11-14 20:11:29 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

A clean 205-file build after the review fixes showed 205 @@file but 204 @@work: the workers' prints could interleave, and one line was lost. announce() now writes every @@ line under one lock; a clean build gives 205/205 and a test starts 200 threads through it. Head 5a2cbd6b; run_all is 492 passed, 8 failed (the same 8 on main 7ff5256e), 72 skipped.

A clean 205-file build after the review fixes showed 205 @@file but 204 @@work: the workers' prints could interleave, and one line was lost. announce() now writes every @@ line under one lock; a clean build gives 205/205 and a test starts 200 threads through it. Head 5a2cbd6b; run_all is 492 passed, 8 failed (the same 8 on main 7ff5256e), 72 skipped.

suporte_de_cadeira commented 2026-11-14 21:13:04 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Full end-to-end run passed on Windows: build_ui.py --make --jobs 6 from the installed dump built 643 files in 33:25 (compile 16:28), 0 failed, and prepared 24,453 shader pipelines; the built game started at 60.0 fps (600 presents, 1280x720). The description now has the real-build and game screenshots; T-RVZR passes.

Full end-to-end run passed on Windows: build_ui.py --make --jobs 6 from the installed dump built 643 files in 33:25 (compile 16:28), 0 failed, and prepared 24,453 shader pipelines; the built game started at 60.0 fps (600 presents, 1280x720). The description now has the real-build and game screenshots; T-RVZR passes.

Some checks failed

Hide checks

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Failing after 18s

Details

beanstalk/approval Test farm not requested

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

Some workflows are waiting to be reviewed.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/build-dashboard:suporte_de_cadeira-feat/build-dashboard'

git switch 'suporte_de_cadeira-feat/build-dashboard'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

beanstalk: approved @ 1697603

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 2c5a62e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 4fd82b7

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 64d6d09

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 6f91fae

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 70a78f9

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 8d361dc

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 9d8356e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ abf2227

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ d82f202

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ e5f1e40

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ ee8ab5f

Set by Beanstalk for show; the farm's database is what counts

  

ci: approved

A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

regression

Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

beanstalk: approved @ 1697603

beanstalk: approved @ 2c5a62e

beanstalk: approved @ 4fd82b7

beanstalk: approved @ 64d6d09

beanstalk: approved @ 6f91fae

beanstalk: approved @ 70a78f9

beanstalk: approved @ 8d361dc

beanstalk: approved @ 9d8356e

beanstalk: approved @ abf2227

beanstalk: approved @ d82f202

beanstalk: approved @ e5f1e40

beanstalk: approved @ ee8ab5f

ci: approved

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

regression

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!425

No description provided.