MagicBeanDev/Longshot

Watch

Fork 71

chore(scripts): cover.py records which lifted functions a play session runs, and when #419

Open

madebyjordan wants to merge 1 commit from madebyjordan/Longshot:feat/coverage into main

madebyjordan commented

2026-11-14 12:08:09 +00:00

Contributor

Why

The clang-free cold tier proposed in #362 needs a hot list that covers what play actually runs. Nobody could measure that, because the profiler (HOS_SAMPLE_ALL) sees only where time goes, not every function that runs once. This adds a tool that records which lifted functions a real play session runs, and when each first ran.

What changed

scripts: scripts/coverage/cover.py (new).
- run <game folder> starts bin\longshot.exe the way the launcher's Play does, with cover_dll.c loaded before its entry point, and waits for the game to end (or --seconds).
  - Nothing is rebuilt: the lifted functions come from the exe's own symbol table (f_<guest address>, via llvm-nm). The DLL is compiled from source with the launcher's llvm-mingw.
  - Ctrl+Alt+M marks moments from inside the game: the title screen, being in play, then "mark 3" and so on. Windows' default sound confirms each one. A note typed into the terminal works too.
- report <run folder> gives the functions that ran, their share of the host code, and, with --gen (the lifted C) or --csize, their share of the lifted C and of its compile cost.
  - The cost uses nx2/chunks.py's model (size + size²/350000), scaled by --compiler-s to the whole lifted C's -O1 compiler-seconds. The default is 6,478, measured in nightly-34's first build on a Ryzen 5 5600.
  - It also shows how many functions had run by each mark, the arrival of new functions per 100 ms, 1 s and 10 s, and new functions per minute. --hot adds how much of a session a given hot list covers. covered.txt lists the functions in first-run order.
- compare <listed run> <played run>: one session's functions treated as a shipped list, and another session played against it. It counts what the second session runs that the list lacks, and when.
- report and compare work on any OS. run is Windows only.
scripts: scripts/coverage/cover_dll.c (new).
- It writes an int3 over the first byte of each listed function.
- A vectored handler puts the byte back the first time the function is entered, notes the time, and lets the thread go on at the same address. Breakpoints that aren't its own are passed on.
- Every 5 s, and at a normal exit, it writes cover.bin (temp file, then rename).
- Cost: one exception per function, once; ~400 MB more committed memory, because the code pages become private copies.
tests: tests/test_coverage_report.py (new, any OS) checks report and compare on a made-up run folder: counts, the compile-cost model, the marks, covered.txt's order, a hot list's cover and compare's misses. It also checks that cover_dll.c writes the header read_cover reads.
tests: tests/test_coverage_run.py (new, Windows, --cc clang from llvm-mingw) builds a stand-in game with four f_ functions, two of them called, and runs it under the real DLL. Exactly the two called functions must be marked, in the order they first ran, and the game must end normally.
docs: docs/development.md, a row in the commands table.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
`cover.py --compiler-s <n>` (new, a script option)	The -O1 compiler-seconds the cost model is scaled to	6,478 (nightly-34's first build, Ryzen 5 5600)	Set it to your own PC's figure for absolute numbers; the shares don't depend on it	n/a	n/a	n/a

Measurements

A development tool that the game doesn't run, so it's measured by its game-free checks and by one real session.

main: 5e072fc8 (2026-11-14)
this PR: feat/coverage @ 12472833
Hardware: Ryzen 5 5600, 16 GB, AMD Radeon RX 6700 XT, Windows 11
Scene: nightly-130 (0caf0618) as the launcher built it, played by a person for 100 minutes from boot, reaching play at 1:48.
Method: cover.py run <game folder> --gen <a recent main lift's sa/gen> --hot <the shipped hot_order.txt>. The session results are posted on #362 (#362 (comment)).
How to reproduce: the command above on any built game folder on Windows. Close the game normally to finish.

	main @ `5e072fc8`	this PR @ `12472833`
the session	no tool	77,725 of 295,270 functions ran (41.5% of the lifted C; ~2,500 of 6,478 est. compiler-s); 34,520 by the title screen, 39,326 by the start of play
marking at the start	n/a	295,270 functions in 1,256 ms
the game under the tool	n/a	played normally for 100 minutes, no crash
`tests/test_coverage_report.py`, `tests/test_coverage_run.py`	absent	pass
`tests/run_all.py --jobs 4`	485 passed, 6 failed, 70 skipped (at `0caf0618`; the 6 below)	487 passed (the 2 new), the same 6 failed, 70 skipped. The 6 fail on main on this PC and touch nothing here: test_module_ids, test_oot_state_tap, test_pad_layout, test_portable_builder_home, test_portable_builder_startup, test_tested_configurations

Testing

Tested:

The 100-minute session above (Windows 11, AMD), and shorter scripted runs with --seconds during development. The session ran the tool as it was before this PR's last edits: run and report are the same apart from --compiler-s, which took the place of the fixed 6,478, and the Windows-only parts, which are now kept out of the way on other OSes. test_coverage_run.py runs the final version end to end.
Both new tests on Windows with llvm-mingw clang 23.
python -I -B tests/run_all.py --jobs 4 on this branch.

Not tested:

report and compare on Linux: they use only the standard library, and test_coverage_report.py runs there. run is Windows only, and says so: cover_dll.c is a Windows development DLL, not part of the game, and takes its paths as wide characters (the W calls), so UTF-8 folder names work.
A game started some other way than run (for example under the launcher with its own arguments): --args and --env pass those on, but that wasn't exercised.

Needs other testers:
None: a development tool, which players don't run.

Screenshots

Nothing on screen.

Risk and rollback

Nothing in the game, the build or the launcher changes. The tool runs only when someone starts it.
What it writes (%LOCALAPPDATA%\LongshotCover\<date-time>) is offsets, names (guest addresses) and times: no game code or data. The --gen sizes are read locally and only counts are written.
Reverting the commit removes the tool.

Issues

For #362 (sizing the cold tier's hot list).

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A check under tests/ that reads changed source updated in this PR, not weakened.
Docs updated.

## Why The clang-free cold tier proposed in #362 needs a hot list that covers what play actually runs. Nobody could measure that, because the profiler (`HOS_SAMPLE_ALL`) sees only where time goes, not every function that runs once. This adds a tool that records which lifted functions a real play session runs, and when each first ran. ## What changed - scripts: `scripts/coverage/cover.py` (new). - `run <game folder>` starts `bin\longshot.exe` the way the launcher's Play does, with `cover_dll.c` loaded before its entry point, and waits for the game to end (or `--seconds`). - Nothing is rebuilt: the lifted functions come from the exe's own symbol table (`f_<guest address>`, via `llvm-nm`). The DLL is compiled from source with the launcher's llvm-mingw. - Ctrl+Alt+M marks moments from inside the game: the title screen, being in play, then "mark 3" and so on. Windows' default sound confirms each one. A note typed into the terminal works too. - `report <run folder>` gives the functions that ran, their share of the host code, and, with `--gen` (the lifted C) or `--csize`, their share of the lifted C and of its compile cost. - The cost uses `nx2/chunks.py`'s model (size + size²/350000), scaled by `--compiler-s` to the whole lifted C's -O1 compiler-seconds. The default is 6,478, measured in nightly-34's first build on a Ryzen 5 5600. - It also shows how many functions had run by each mark, the arrival of new functions per 100 ms, 1 s and 10 s, and new functions per minute. `--hot` adds how much of a session a given hot list covers. `covered.txt` lists the functions in first-run order. - `compare <listed run> <played run>`: one session's functions treated as a shipped list, and another session played against it. It counts what the second session runs that the list lacks, and when. - `report` and `compare` work on any OS. `run` is Windows only. - scripts: `scripts/coverage/cover_dll.c` (new). - It writes an `int3` over the first byte of each listed function. - A vectored handler puts the byte back the first time the function is entered, notes the time, and lets the thread go on at the same address. Breakpoints that aren't its own are passed on. - Every 5 s, and at a normal exit, it writes `cover.bin` (temp file, then rename). - Cost: one exception per function, once; ~400 MB more committed memory, because the code pages become private copies. - tests: `tests/test_coverage_report.py` (new, any OS) checks `report` and `compare` on a made-up run folder: counts, the compile-cost model, the marks, `covered.txt`'s order, a hot list's cover and compare's misses. It also checks that `cover_dll.c` writes the header `read_cover` reads. - tests: `tests/test_coverage_run.py` (new, Windows, `--cc clang` from llvm-mingw) builds a stand-in game with four `f_` functions, two of them called, and runs it under the real DLL. Exactly the two called functions must be marked, in the order they first ran, and the game must end normally. - docs: `docs/development.md`, a row in the commands table. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `cover.py --compiler-s <n>` (new, a script option) | The -O1 compiler-seconds the cost model is scaled to | 6,478 (nightly-34's first build, Ryzen 5 5600) | Set it to your own PC's figure for absolute numbers; the shares don't depend on it | n/a | n/a | n/a | ## Measurements A development tool that the game doesn't run, so it's measured by its game-free checks and by one real session. main: 5e072fc8 (2026-11-14) this PR: feat/coverage @ 12472833 Hardware: Ryzen 5 5600, 16 GB, AMD Radeon RX 6700 XT, Windows 11 Scene: nightly-130 (0caf0618) as the launcher built it, played by a person for 100 minutes from boot, reaching play at 1:48. Method: `cover.py run <game folder> --gen <a recent main lift's sa/gen> --hot <the shipped hot_order.txt>`. The session results are posted on #362 (https://git.magicbean.dev/MagicBeanDev/Longshot/issues/362#issuecomment-17894). How to reproduce: the command above on any built game folder on Windows. Close the game normally to finish. | | main @ 5e072fc8 | this PR @ 12472833 | |---|---|---| | the session | no tool | 77,725 of 295,270 functions ran (41.5% of the lifted C; ~2,500 of 6,478 est. compiler-s); 34,520 by the title screen, 39,326 by the start of play | | marking at the start | n/a | 295,270 functions in 1,256 ms | | the game under the tool | n/a | played normally for 100 minutes, no crash | | `tests/test_coverage_report.py`, `tests/test_coverage_run.py` | absent | pass | | `tests/run_all.py --jobs 4` | 485 passed, 6 failed, 70 skipped (at 0caf0618; the 6 below) | 487 passed (the 2 new), the same 6 failed, 70 skipped. The 6 fail on main on this PC and touch nothing here: test_module_ids, test_oot_state_tap, test_pad_layout, test_portable_builder_home, test_portable_builder_startup, test_tested_configurations | ## Testing **Tested:** - The 100-minute session above (Windows 11, AMD), and shorter scripted runs with `--seconds` during development. The session ran the tool as it was before this PR's last edits: `run` and `report` are the same apart from `--compiler-s`, which took the place of the fixed 6,478, and the Windows-only parts, which are now kept out of the way on other OSes. `test_coverage_run.py` runs the final version end to end. - Both new tests on Windows with llvm-mingw clang 23. - `python -I -B tests/run_all.py --jobs 4` on this branch. **Not tested:** - `report` and `compare` on Linux: they use only the standard library, and `test_coverage_report.py` runs there. `run` is Windows only, and says so: `cover_dll.c` is a Windows development DLL, not part of the game, and takes its paths as wide characters (the `W` calls), so UTF-8 folder names work. - A game started some other way than `run` (for example under the launcher with its own arguments): `--args` and `--env` pass those on, but that wasn't exercised. **Needs other testers:** None: a development tool, which players don't run. ## Screenshots Nothing on screen. ## Risk and rollback - Nothing in the game, the build or the launcher changes. The tool runs only when someone starts it. - What it writes (`%LOCALAPPDATA%\LongshotCover\<date-time>`) is offsets, names (guest addresses) and times: no game code or data. The `--gen` sizes are read locally and only counts are written. - Reverting the commit removes the tool. ## Issues For #362 (sizing the cold tier's hot list). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.

madebyjordan added 1 commit

2026-11-14 12:08:09 +00:00

chore(scripts): cover.py records which lifted functions a play session runs, and when

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

12472833de

scripts/coverage/cover.py starts the built game with cover_dll.c loaded before its entry point. The DLL puts a
breakpoint on the first byte of every f_ function (found in the exe's own symbol table) and notes the time each is
first entered. report gives their share of the lifted C and of its compile cost (nx2/chunks.py's model, scaled to
--compiler-s), how many had run by each mark (Ctrl+Alt+M), and a hot list's cover; compare plays one session against
another's list. The data for sizing a hot list (#362).

tests: test_coverage_report.py (report and compare on a made-up run folder, any OS) and test_coverage_run.py (a
stand-in game under the real DLL: exactly the called functions are marked; Windows, llvm-mingw).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

madebyjordan changed title from ~~WIP: chore(scripts): cover.py records which lifted functions a play session runs, and when~~ to chore(scripts): cover.py records which lifted functions a play session runs, and when

2026-11-14 12:08:40 +00:00