fix(host): HOS_CHAIN_RECORDS=<n> sizes the HOS_CHAIN buffer, so a capture can cover a whole measured window #1282

Open
klimovich008 wants to merge 2 commits from klimovich008/Longshot:perf/chain-capacity into main
Contributor

Type: fix · Risk: low · Tested on: Windows 11, RX 9070 XT · Needs: Linux, any GPU

Why

HOS_CHAIN, the capture that scripts/chain.py reads for the frame's critical path, stops when its fixed buffer of 2,097,152 records is full: in a 20 s Hyrule Field window that is the first 5.8 s (28 % of the window's frames), so the path it reports is only the window's first third. With HOS_CHAIN_RECORDS=8388608 the same capture covers the whole window (99.9 % of its frames). No issue was filed for it (tracker searched for chain, critical path and profiling).

How to reproduce

Seen on: Windows 11, AMD Radeon RX 9070 XT, AMD's Windows driver
Settings and switches: HOS_CHAIN=chain.bin in the environment; threaded rendering on, HOS_FPS_MAX=240
Steps: load into Hyrule Field, make chain.bin.go beside the game, turn the camera for 20 s, delete chain.bin.go, then python -I scripts/chain.py chain.bin
Expected: the capture holds the whole 20 s, about 1,400 frames
Actual: the buffer fills after about 6 s: 2,097,152 records, and chain.py walks about 400 frames; nothing in main can make the buffer larger

What changed

A new switch, HOS_CHAIN_RECORDS=<n>, sets how many records the HOS_CHAIN buffer holds, from 1,048,576 to 33,554,432 (32 MB to 1 GB). Without it, or with any value that isn't a whole number in that range, the size stays today's 2,097,152, and the game's log says which size it records with. It is read only when HOS_CHAIN is set, so nothing changes for anyone else.

  • host: host/hos_chain.c
    • CHAIN_CAP became CHAIN_CAP_DEFAULT (2,097,152, unchanged), with CHAIN_CAP_MIN, CHAIN_CAP_MAX and static LONG64 chain_cap; the recorder's full test in hos_chain_note() and the writer's clamp in chain_write() use chain_cap.
    • hos_chain_start() reads HOS_CHAIN_RECORDS once, before the recorder thread starts, through chain_records() (strtoll, base 10, with end and errno checks); a refused value is logged: chain: HOS_CHAIN_RECORDS=<v> is not a number of records from 1048576 to 33554432; 2097152 instead.
    • chain_thread(): if the larger buffer can't be allocated, it logs chain: no memory for N records (HOS_CHAIN_RECORDS); 2097152 instead and tries the default; if that fails too, no recording, as before. The start line now names the size: chain: recording (<file>, at most N records).
    • chain_cap is set only before recording first starts and never changes while it records (the comment beside it says why every thread then reads the size its buffer was made with).
  • docs: the HOS_CHAIN row in docs/development.md gives the default size and the new switch; the file's header comment too.
  • tests: tests/test_chain_records.py (new, found by tests/run_all.py): compiles the real recorder, writer and start code cut from hos_chain.c with stubs and fills the buffer past the default and past a larger size.
  • scripts/chain.py and the file format are unchanged: chain.py reads the record count from the file's header and has no size of its own.

Measurements

Verdict: fixed: with HOS_CHAIN_RECORDS=8388608 the capture covers the whole 20 s window (1,466 of 1,467 frames walked, not capacity-limited), where today's size covers 5.8 s (397 of 1,400 frames, 28.4 %). The new test fails on main and passes here. Normal play is unchanged: the switch is only read with HOS_CHAIN set, and one run each of main and this PR without HOS_CHAIN are in the same band.

Baseline: the game runs were made on this branch's base, main @ 0cab3548, and this PR @ a9d942d6, before main (8e9cfc3a) was merged in. They still hold: none of main's 34 newer commits touch host/hos_chain.c, and scripts/chain.py's 3-line change there doesn't touch how it reads the file. The game-free checks were run again after the merge (Testing).
Hardware: Windows 11 (build 26300), AMD Radeon RX 9070 XT (AMD's Windows driver), Ryzen 7 5700X3D (8 cores, 16 threads), 64 GB
Scene: Hyrule Field, the camera turning steadily; 16:9 window, threaded rendering on, HOS_FPS_MAX=240
Method: one 20 s window per run, taken after the load; the capture switched on and off with the window (chain.bin.go made at its start, deleted at its end). Coverage: the records' times against the frames presented in the window (both on the game's hos_now_ns clock), and python -I scripts/chain.py chain.bin on each file. Normal play: frames a second, CPU time a frame and frame times over the same window, with no HOS_CHAIN, one run each.
How to reproduce: start the game with HOS_CHAIN=chain.bin and HOS_CHAIN_RECORDS=8388608 in the environment, go to Hyrule Field, make chain.bin.go beside the game, turn the camera for 20 s, delete chain.bin.go, wait for the log's chain: ... records ... written line, then python -I scripts/chain.py chain.bin.

The capture, 20 s of Hyrule Field:

Before: main @ 0cab3548 After, HOS_CHAIN_RECORDS unset: this PR @ a9d942d6 After, HOS_CHAIN_RECORDS=8388608: this PR @ a9d942d6 After, HOS_CHAIN_RECORDS=33554432: this PR @ a9d942d6
buffer size (the log's start line) 2,097,152 (not logged) 2,097,152 8,388,608 33,554,432
records written 2,097,152 (full) 2,097,152 (full) 7,949,300 (254 MB, 94.8 % of the buffer) 7,531,818 (241 MB)
part of the window recorded 3.5 s (see note) 0.0 to 5.8 s 0.0 to 20.0 s 0.1 to 20.1 s
the window's frames walked by chain.py 215 of 1,300 (16.5 %) 397 of 1,400 (28.4 %) 1,466 of 1,467 (99.9 %) 1,260 of 1,265 (99.6 %)
tests/test_chain_records.py FAIL PASS PASS PASS

Note: on main's run the window opened late (the area's load came later than usual), so the capture began 3.7 s before the window and only 3.5 s of the window fell inside it. The "unset" column is the comparison at today's size: it runs the same code path as main with the same buffer.

Normal play, no HOS_CHAIN (one run each, a sanity check rather than a measurement: none of this PR's code runs without HOS_CHAIN):

Before: main @ 0cab3548 After: this PR @ a9d942d6
fps 73.99 74.04
CPU ms a frame (all threads) 69.25 70.61 (inside one build's run-to-run spread here, 68.5 to 70.7)
p99 / worst frame 19.54 / 33.6 ms 18.89 / 34.6 ms
frames over 50 / 100 / 250 ms 0 / 0 / 0 0 / 0 / 0
game clock against real time 1.001 1.000
main @ 0cab3548:              chain: recording (chain.bin)
this PR, unset:               chain: recording (chain.bin, at most 2097152 records)
this PR, =8388608:            chain: recording (chain.bin, at most 8388608 records)
this PR, =33554432:           chain: recording (chain.bin, at most 33554432 records)

python -I scripts/chain.py chain.bin, this PR, unset:
2097152 records, 51 threads; 397 frames presented by Presentation_Thread, 14.51 ms a frame (68.9 fps)
python -I scripts/chain.py chain.bin, this PR, =8388608:
7949300 records, 52 threads; 1468 frames presented by Presentation_Thread, 13.64 ms a frame (73.3 fps)

tests/test_chain_records.py with main's host/hos_chain.c:
FAIL: HOS_CHAIN_RECORDS=2500000: recording did not stop at exactly note 2500000
tests/test_chain_records.py on this PR:
ok: the broken variant (the writer's clamp left at the default size) fails: HOS_CHAIN_RECORDS=2500000: the file does not hold the first 2500000 notes in order
ok: the broken variant (the recorder's full test left at the default size) fails: HOS_CHAIN_RECORDS=2500000: recording did not stop at exactly note 2500000
PASS: HOS_CHAIN keeps HOS_CHAIN_RECORDS records (2,097,152 unset), stops at exactly that many, and falls back to the default for 0, words, out-of-range values and no memory; both broken variants fail

Flags and settings

One new switch for profiling, off unless set; the default is unchanged.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
HOS_CHAIN_RECORDS=<n> how many records the HOS_CHAIN buffer holds (32 bytes each) not there: 2,097,152, fixed unset: 2,097,152; read only with HOS_CHAIN; 1,048,576 to 33,554,432, any other value the default (logged) a capture can run longer before the buffer fills; the buffer's memory (up to 1 GB) is taken when recording starts helps any capture longer than about 6 s of a busy scene; tried on Windows at 8,388,608 and 33,554,432; untested in the game on Linux unset for everyone; for a 20 s window of a busy scene, 16,777,216 (8,388,608 already fills 94.8 % in Hyrule Field)

Testing

Tested:

  • Windows 11, RX 9070 XT, in the game (the runs above): the log's size line at the default, 8,388,608 and the ceiling; the whole 20 s window captured at 8,388,608; the buffer filling at the default and recording stopping there, written whole; each file complete (its header's count times 32 bytes is its size) and read by scripts/chain.py. The memory at the ceiling: +1.07 GiB of commit and +0.26 GiB of working set against a run without HOS_CHAIN (Windows commits the buffer at once; only the pages that fill become resident).
  • tests/test_chain_records.py: HOS_CHAIN_RECORDS=2500000 stops at exactly record 2,500,000 with all of them in the file in order; unset, exactly 2,097,152; the floor; a window that ends before the buffer is full; 0, lots, 8M, 0x800000, one below the floor, one over the ceiling, a negative and an overflowing number all give the default with the log line (empty gives it silently); the ceiling with the allocation failing gives the default, logged; no memory at all gives no recording, as before; without HOS_CHAIN, HOS_CHAIN_RECORDS isn't read. It fails with main's hos_chain.c, and two broken variants (the writer's clamp and the recorder's full test left at the default) fail inside it. It passes on Windows with llvm-mingw clang and MinGW gcc, and on Linux (WSL Ubuntu) with clang and gcc.
  • After merging main (8e9cfc3a): python -I -B tests/run_all.py --strict (what mise run test runs) on this branch and on main @ 8e9cfc3a, Windows, llvm-mingw 20261104: this branch 623 passed, 46 failed, 45 skipped (the Linux-only checks); main 622 passed, 46 failed, 45 skipped. The 46 failures are the same list on both (they fail on this Windows checkout on main too; none reads hos_chain.c or the docs row); the extra pass is tests/test_chain_records.py. tests/test_chain.py passes on both. Before the merge, on main @ 0cab3548, the same comparison gave the same picture (25 shared failures, the new test the one extra pass).
  • host/hos_chain.c compiled for Windows (llvm-mingw clang, -Wall -Wextra: no warnings) and for Linux (WSL clang 21 and gcc with the Linux build's flags, host/posix first: no warnings; objects only).

Not tested:

  • Linux in the game (only the compile and the test there); the Steam Deck.
  • A PC with 16 GB of RAM: at the ceiling the buffer adds 1 GB of commit at once; if Windows can't give it, the game logs it and records with the default size (the test's no-memory case), not a crash.
  • Several guest threads filling the buffer at once in the test: it proves the counts on one thread. The size is fixed before recording starts (before the recorder thread is made, or before hos_chain_on is first set) and never changes while it records, the same ordering the buffer's pointer already relies on.
  • mise run lint (pre-commit): mise isn't installed on this PC, so the pre-commit hooks were not run.

Must be tested:

  • linux, any GPU: start the game with HOS_CHAIN=chain.bin HOS_CHAIN_RECORDS=8388608, make chain.bin.go in Hyrule Field, turn the camera for 20 s, delete chain.bin.go, expecting the log's chain: recording (chain.bin, at most 8388608 records), then a chain: <N> records ... written line, and python -I scripts/chain.py chain.bin walking about 20 s of frames; then once with HOS_CHAIN_RECORDS=lots, expecting the is not a number of records ... 2097152 instead line and a 2,097,152-record buffer (Test Farm: @Beanstalk linux, as in the comment below)

Risks

What could break: only runs with HOS_CHAIN set. A large size takes its memory when recording starts (8,388,608: 256 MB; the ceiling: 1 GB of commit) and writes a file that size when the capture ends or the buffer fills; if the buffer fills inside a measured window, that write lands in it, as it does at today's size. A failed allocation falls back to the default, then to no recording, each logged.

Turning it off: leave HOS_CHAIN_RECORDS unset (or revert the commit).

Compatibility: Nothing changes: no save, options file, cache, chain file format or vk ABI change; HOS_CHAIN without the new switch records as before.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. (main @ 8e9cfc3a merged in; the one conflict, the HOS_CHAIN row next to #1279's new rows, kept both)
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened. (tests/test_chain_records.py is new and compiles the whole of host/hos_chain.c; no existing check of that file was changed)
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows: the game built and run; Linux: compiled to objects with the Linux build's flags, not linked)
  • Docs updated.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, RX 9070 XT · **Needs:** Linux, any GPU ## Why `HOS_CHAIN`, the capture that `scripts/chain.py` reads for the frame's critical path, stops when its fixed buffer of 2,097,152 records is full: in a 20 s Hyrule Field window that is the first 5.8 s (28 % of the window's frames), so the path it reports is only the window's first third. With `HOS_CHAIN_RECORDS=8388608` the same capture covers the whole window (99.9 % of its frames). No issue was filed for it (tracker searched for chain, critical path and profiling). ## How to reproduce **Seen on:** Windows 11, AMD Radeon RX 9070 XT, AMD's Windows driver **Settings and switches:** `HOS_CHAIN=chain.bin` in the environment; threaded rendering on, `HOS_FPS_MAX=240` **Steps:** load into Hyrule Field, make `chain.bin.go` beside the game, turn the camera for 20 s, delete `chain.bin.go`, then `python -I scripts/chain.py chain.bin` **Expected:** the capture holds the whole 20 s, about 1,400 frames **Actual:** the buffer fills after about 6 s: 2,097,152 records, and `chain.py` walks about 400 frames; nothing in `main` can make the buffer larger ## What changed A new switch, `HOS_CHAIN_RECORDS=<n>`, sets how many records the `HOS_CHAIN` buffer holds, from 1,048,576 to 33,554,432 (32 MB to 1 GB). Without it, or with any value that isn't a whole number in that range, the size stays today's 2,097,152, and the game's log says which size it records with. It is read only when `HOS_CHAIN` is set, so nothing changes for anyone else. - host: `host/hos_chain.c` - `CHAIN_CAP` became `CHAIN_CAP_DEFAULT` (2,097,152, unchanged), with `CHAIN_CAP_MIN`, `CHAIN_CAP_MAX` and `static LONG64 chain_cap`; the recorder's full test in `hos_chain_note()` and the writer's clamp in `chain_write()` use `chain_cap`. - `hos_chain_start()` reads `HOS_CHAIN_RECORDS` once, before the recorder thread starts, through `chain_records()` (`strtoll`, base 10, with end and `errno` checks); a refused value is logged: `chain: HOS_CHAIN_RECORDS=<v> is not a number of records from 1048576 to 33554432; 2097152 instead`. - `chain_thread()`: if the larger buffer can't be allocated, it logs `chain: no memory for N records (HOS_CHAIN_RECORDS); 2097152 instead` and tries the default; if that fails too, no recording, as before. The start line now names the size: `chain: recording (<file>, at most N records)`. - `chain_cap` is set only before recording first starts and never changes while it records (the comment beside it says why every thread then reads the size its buffer was made with). - docs: the `HOS_CHAIN` row in `docs/development.md` gives the default size and the new switch; the file's header comment too. - tests: `tests/test_chain_records.py` (new, found by `tests/run_all.py`): compiles the real recorder, writer and start code cut from `hos_chain.c` with stubs and fills the buffer past the default and past a larger size. - `scripts/chain.py` and the file format are unchanged: `chain.py` reads the record count from the file's header and has no size of its own. ## Measurements **Verdict:** fixed: with `HOS_CHAIN_RECORDS=8388608` the capture covers the whole 20 s window (1,466 of 1,467 frames walked, not capacity-limited), where today's size covers 5.8 s (397 of 1,400 frames, 28.4 %). The new test fails on `main` and passes here. Normal play is unchanged: the switch is only read with `HOS_CHAIN` set, and one run each of `main` and this PR without `HOS_CHAIN` are in the same band. Baseline: the game runs were made on this branch's base, main @ 0cab3548, and this PR @ a9d942d6, before main (8e9cfc3a) was merged in. They still hold: none of main's 34 newer commits touch `host/hos_chain.c`, and `scripts/chain.py`'s 3-line change there doesn't touch how it reads the file. The game-free checks were run again after the merge (Testing). Hardware: Windows 11 (build 26300), AMD Radeon RX 9070 XT (AMD's Windows driver), Ryzen 7 5700X3D (8 cores, 16 threads), 64 GB Scene: Hyrule Field, the camera turning steadily; 16:9 window, threaded rendering on, `HOS_FPS_MAX=240` Method: one 20 s window per run, taken after the load; the capture switched on and off with the window (`chain.bin.go` made at its start, deleted at its end). Coverage: the records' times against the frames presented in the window (both on the game's `hos_now_ns` clock), and `python -I scripts/chain.py chain.bin` on each file. Normal play: frames a second, CPU time a frame and frame times over the same window, with no `HOS_CHAIN`, one run each. How to reproduce: start the game with `HOS_CHAIN=chain.bin` and `HOS_CHAIN_RECORDS=8388608` in the environment, go to Hyrule Field, make `chain.bin.go` beside the game, turn the camera for 20 s, delete `chain.bin.go`, wait for the log's `chain: ... records ... written` line, then `python -I scripts/chain.py chain.bin`. The capture, 20 s of Hyrule Field: | | Before: main @ 0cab3548 | After, `HOS_CHAIN_RECORDS` unset: this PR @ a9d942d6 | After, `HOS_CHAIN_RECORDS=8388608`: this PR @ a9d942d6 | After, `HOS_CHAIN_RECORDS=33554432`: this PR @ a9d942d6 | |---|---|---|---|---| | buffer size (the log's start line) | 2,097,152 (not logged) | 2,097,152 | 8,388,608 | 33,554,432 | | records written | 2,097,152 (full) | 2,097,152 (full) | 7,949,300 (254 MB, 94.8 % of the buffer) | 7,531,818 (241 MB) | | part of the window recorded | 3.5 s (see note) | 0.0 to 5.8 s | 0.0 to 20.0 s | 0.1 to 20.1 s | | the window's frames walked by `chain.py` | 215 of 1,300 (16.5 %) | 397 of 1,400 (28.4 %) | 1,466 of 1,467 (99.9 %) | 1,260 of 1,265 (99.6 %) | | `tests/test_chain_records.py` | FAIL | PASS | PASS | PASS | Note: on `main`'s run the window opened late (the area's load came later than usual), so the capture began 3.7 s before the window and only 3.5 s of the window fell inside it. The "unset" column is the comparison at today's size: it runs the same code path as `main` with the same buffer. Normal play, no `HOS_CHAIN` (one run each, a sanity check rather than a measurement: none of this PR's code runs without `HOS_CHAIN`): | | Before: main @ 0cab3548 | After: this PR @ a9d942d6 | |---|---|---| | fps | 73.99 | 74.04 | | CPU ms a frame (all threads) | 69.25 | 70.61 (inside one build's run-to-run spread here, 68.5 to 70.7) | | p99 / worst frame | 19.54 / 33.6 ms | 18.89 / 34.6 ms | | frames over 50 / 100 / 250 ms | 0 / 0 / 0 | 0 / 0 / 0 | | game clock against real time | 1.001 | 1.000 | ``` main @ 0cab3548: chain: recording (chain.bin) this PR, unset: chain: recording (chain.bin, at most 2097152 records) this PR, =8388608: chain: recording (chain.bin, at most 8388608 records) this PR, =33554432: chain: recording (chain.bin, at most 33554432 records) python -I scripts/chain.py chain.bin, this PR, unset: 2097152 records, 51 threads; 397 frames presented by Presentation_Thread, 14.51 ms a frame (68.9 fps) python -I scripts/chain.py chain.bin, this PR, =8388608: 7949300 records, 52 threads; 1468 frames presented by Presentation_Thread, 13.64 ms a frame (73.3 fps) tests/test_chain_records.py with main's host/hos_chain.c: FAIL: HOS_CHAIN_RECORDS=2500000: recording did not stop at exactly note 2500000 tests/test_chain_records.py on this PR: ok: the broken variant (the writer's clamp left at the default size) fails: HOS_CHAIN_RECORDS=2500000: the file does not hold the first 2500000 notes in order ok: the broken variant (the recorder's full test left at the default size) fails: HOS_CHAIN_RECORDS=2500000: recording did not stop at exactly note 2500000 PASS: HOS_CHAIN keeps HOS_CHAIN_RECORDS records (2,097,152 unset), stops at exactly that many, and falls back to the default for 0, words, out-of-range values and no memory; both broken variants fail ``` ### Flags and settings One new switch for profiling, off unless set; the default is unchanged. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `HOS_CHAIN_RECORDS=<n>` | how many records the `HOS_CHAIN` buffer holds (32 bytes each) | not there: 2,097,152, fixed | unset: 2,097,152; read only with `HOS_CHAIN`; 1,048,576 to 33,554,432, any other value the default (logged) | a capture can run longer before the buffer fills; the buffer's memory (up to 1 GB) is taken when recording starts | helps any capture longer than about 6 s of a busy scene; tried on Windows at 8,388,608 and 33,554,432; untested in the game on Linux | unset for everyone; for a 20 s window of a busy scene, 16,777,216 (8,388,608 already fills 94.8 % in Hyrule Field) | ## Testing **Tested:** - Windows 11, RX 9070 XT, in the game (the runs above): the log's size line at the default, 8,388,608 and the ceiling; the whole 20 s window captured at 8,388,608; the buffer filling at the default and recording stopping there, written whole; each file complete (its header's count times 32 bytes is its size) and read by `scripts/chain.py`. The memory at the ceiling: +1.07 GiB of commit and +0.26 GiB of working set against a run without `HOS_CHAIN` (Windows commits the buffer at once; only the pages that fill become resident). - `tests/test_chain_records.py`: `HOS_CHAIN_RECORDS=2500000` stops at exactly record 2,500,000 with all of them in the file in order; unset, exactly 2,097,152; the floor; a window that ends before the buffer is full; `0`, `lots`, `8M`, `0x800000`, one below the floor, one over the ceiling, a negative and an overflowing number all give the default with the log line (empty gives it silently); the ceiling with the allocation failing gives the default, logged; no memory at all gives no recording, as before; without `HOS_CHAIN`, `HOS_CHAIN_RECORDS` isn't read. It fails with `main`'s `hos_chain.c`, and two broken variants (the writer's clamp and the recorder's full test left at the default) fail inside it. It passes on Windows with llvm-mingw clang and MinGW gcc, and on Linux (WSL Ubuntu) with clang and gcc. - After merging main (8e9cfc3a): `python -I -B tests/run_all.py --strict` (what `mise run test` runs) on this branch and on main @ 8e9cfc3a, Windows, llvm-mingw 20261104: this branch 623 passed, 46 failed, 45 skipped (the Linux-only checks); main 622 passed, 46 failed, 45 skipped. The 46 failures are the same list on both (they fail on this Windows checkout on `main` too; none reads `hos_chain.c` or the docs row); the extra pass is `tests/test_chain_records.py`. `tests/test_chain.py` passes on both. Before the merge, on main @ 0cab3548, the same comparison gave the same picture (25 shared failures, the new test the one extra pass). - `host/hos_chain.c` compiled for Windows (llvm-mingw clang, `-Wall -Wextra`: no warnings) and for Linux (WSL clang 21 and gcc with the Linux build's flags, `host/posix` first: no warnings; objects only). **Not tested:** - Linux in the game (only the compile and the test there); the Steam Deck. - A PC with 16 GB of RAM: at the ceiling the buffer adds 1 GB of commit at once; if Windows can't give it, the game logs it and records with the default size (the test's no-memory case), not a crash. - Several guest threads filling the buffer at once in the test: it proves the counts on one thread. The size is fixed before recording starts (before the recorder thread is made, or before `hos_chain_on` is first set) and never changes while it records, the same ordering the buffer's pointer already relies on. - `mise run lint` (pre-commit): mise isn't installed on this PC, so the pre-commit hooks were not run. **Must be tested:** - linux, any GPU: start the game with `HOS_CHAIN=chain.bin HOS_CHAIN_RECORDS=8388608`, make `chain.bin.go` in Hyrule Field, turn the camera for 20 s, delete `chain.bin.go`, expecting the log's `chain: recording (chain.bin, at most 8388608 records)`, then a `chain: <N> records ... written` line, and `python -I scripts/chain.py chain.bin` walking about 20 s of frames; then once with `HOS_CHAIN_RECORDS=lots`, expecting the `is not a number of records ... 2097152 instead` line and a 2,097,152-record buffer (Test Farm: @Beanstalk linux, as in the comment below) ## Risks **What could break:** only runs with `HOS_CHAIN` set. A large size takes its memory when recording starts (8,388,608: 256 MB; the ceiling: 1 GB of commit) and writes a file that size when the capture ends or the buffer fills; if the buffer fills inside a measured window, that write lands in it, as it does at today's size. A failed allocation falls back to the default, then to no recording, each logged. **Turning it off:** leave `HOS_CHAIN_RECORDS` unset (or revert the commit). **Compatibility:** Nothing changes: no save, options file, cache, chain file format or vk ABI change; `HOS_CHAIN` without the new switch records as before. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. (main @ 8e9cfc3a merged in; the one conflict, the `HOS_CHAIN` row next to #1279's new rows, kept both) - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. (`tests/test_chain_records.py` is new and compiles the whole of `host/hos_chain.c`; no existing check of that file was changed) - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows: the game built and run; Linux: compiled to objects with the Linux build's flags, not linked) - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The critical-path recorder kept at most 2,097,152 records (64 MB), about 6 s of Hyrule Field with the
camera turning: a 20 s window's capture was its first third. HOS_CHAIN_RECORDS=<n>, read only when
HOS_CHAIN is set and before its thread starts, sizes the buffer from 1,048,576 to 33,554,432 records
(32 MB to 1 GB). Unset, 0, not a whole number or out of range: the default, logged. With no memory
for the size asked for when recording starts, the default size instead, logged. The recording's log
line names the size it records with. The size is fixed before recording turns on and never changed
after. No change to the file's format: scripts/chain.py reads the count from the header.

tests/test_chain_records.py compiles the whole recorder (after its includes) with stubs and runs it,
its thread driven through Sleep(): past the default, unset, the floor, a window that ends first (read
back by scripts/chain.py), the fallbacks and no memory. Two broken variants (the writer's clamp or the
recorder's full test left at the default) must fail.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Merge Longshot main (8e9cfc3a) into perf/chain-capacity
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 4s
e8df1bf8f2
The HOS_CHAIN row of docs/development.md conflicted with #1279's new
profiling rows: main's rows kept, the HOS_CHAIN row this branch's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Contributor

Tested in game: Windows 11, AMD Radeon RX 9070 XT (AMD's Windows driver), Ryzen 7 5700X3D, this PR @ a9d942d6 on main @ 0cab3548. Hyrule Field, camera turning, 20 s window: with HOS_CHAIN=chain.bin HOS_CHAIN_RECORDS=8388608 the log said chain: recording (chain.bin, at most 8388608 records), 7,949,300 records were written and scripts/chain.py walked 1,466 of the window's 1,467 frames; unset, 2,097,152 records as before (5.8 s); at the ceiling (33,554,432) the size line and a whole capture too. Without HOS_CHAIN, normal play in the same band as main (74.0 fps both, no frame over 50 ms).

Tested in game: Windows 11, AMD Radeon RX 9070 XT (AMD's Windows driver), Ryzen 7 5700X3D, this PR @ a9d942d6 on main @ 0cab3548. Hyrule Field, camera turning, 20 s window: with `HOS_CHAIN=chain.bin HOS_CHAIN_RECORDS=8388608` the log said `chain: recording (chain.bin, at most 8388608 records)`, 7,949,300 records were written and `scripts/chain.py` walked 1,466 of the window's 1,467 frames; unset, 2,097,152 records as before (5.8 s); at the ceiling (33,554,432) the size line and a whole capture too. Without `HOS_CHAIN`, normal play in the same band as main (74.0 fps both, no frame over 50 ms).
Author
Contributor

@Beanstalk linux, any GPU: please build this PR and start the game with HOS_CHAIN=chain.bin and HOS_CHAIN_RECORDS=8388608 in the environment. In Hyrule Field, create an empty file chain.bin.go beside the game, turn the camera for about 20 s, then delete chain.bin.go. Expected in the log: chain: recording (chain.bin, at most 8388608 records) at the start, then a chain: <N> records of <M> threads written to chain.bin line; python -I scripts/chain.py chain.bin should report about 20 s worth of frames (not about 6 s). Then once more with HOS_CHAIN_RECORDS=lots: expected chain: HOS_CHAIN_RECORDS=lots is not a number of records from 1048576 to 33554432; 2097152 instead and a capture of at most 2,097,152 records. This PC is Windows only; the switch is Linux-built from the same file (host/posix), compiled and unit-tested there, but not run in the game.

@Beanstalk linux, any GPU: please build this PR and start the game with `HOS_CHAIN=chain.bin` and `HOS_CHAIN_RECORDS=8388608` in the environment. In Hyrule Field, create an empty file `chain.bin.go` beside the game, turn the camera for about 20 s, then delete `chain.bin.go`. Expected in the log: `chain: recording (chain.bin, at most 8388608 records)` at the start, then a `chain: <N> records of <M> threads written to chain.bin` line; `python -I scripts/chain.py chain.bin` should report about 20 s worth of frames (not about 6 s). Then once more with `HOS_CHAIN_RECORDS=lots`: expected `chain: HOS_CHAIN_RECORDS=lots is not a number of records from 1048576 to 33554432; 2097152 instead` and a capture of at most 2,097,152 records. This PC is Windows only; the switch is Linux-built from the same file (`host/posix`), compiled and unit-tested there, but not run in the game.
Member

@klimovich008 only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@klimovich008 only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Collaborator

Possibly related, for context:

  • PR #1044 WIP: fix(profiling): the Windows all-thread sampler stamps each sample with its own time under HOS_CHAIN — Related profiling work documents HOS_CHAIN filling its fixed buffer during capture.
Possibly related, for context: - PR #1044 WIP: fix(profiling): the Windows all-thread sampler stamps each sample with its own time under HOS_CHAIN — Related profiling work documents HOS_CHAIN filling its fixed buffer during capture.
klimovich008 changed title from WIP: fix(host): HOS_CHAIN_RECORDS=<n> sizes the HOS_CHAIN buffer, so a capture can cover a whole measured window to fix(host): HOS_CHAIN_RECORDS=<n> sizes the HOS_CHAIN buffer, so a capture can cover a whole measured window 2026-11-17 00:30:21 +00:00
Author
Contributor

Ready for review. Description: added the checklist box for the new source-reading test (tests/test_chain_records.py compiles the whole of host/hos_chain.c). The measurements were made on main 0cab3548; the Baseline line says why they hold on 8e9cfc3a (none of main's newer commits touch host/hos_chain.c).

Ready for review. Description: added the checklist box for the new source-reading test (tests/test_chain_records.py compiles the whole of host/hos_chain.c). The measurements were made on main 0cab3548; the Baseline line says why they hold on 8e9cfc3a (none of main's newer commits touch host/hos_chain.c).
TrashPanda left a comment

HOS_CHAIN_RECORDS sets how many records a chain capture holds, from 1,048,576 to 33,554,432. Unset, or not a whole number in that range, it stays 2,097,152. I read head e8df1bf8. It applies onto current main, and this stays a comment.

The size is read once, before the recorder thread starts. A buffer that cannot be allocated falls back to the default before recording begins, and both the recorder and the writer stop at that size. The check fills past the default and past a larger size, and refuses 0, words and values outside the range. On Windows, an RX 9070 XT, 8,388,608 records covered a 20 second Hyrule Field window (1,466 of 1,467 frames). The default still fills at about 6 seconds. A run without HOS_CHAIN matches main.

The Linux run this asks for is still open: HOS_CHAIN_RECORDS=8388608 for about 20 seconds, then HOS_CHAIN_RECORDS=lots, expecting the default and the log line. The farm request is waiting on a BigBean.

`HOS_CHAIN_RECORDS` sets how many records a chain capture holds, from 1,048,576 to 33,554,432. Unset, or not a whole number in that range, it stays 2,097,152. I read head `e8df1bf8`. It applies onto current main, and this stays a comment. The size is read once, before the recorder thread starts. A buffer that cannot be allocated falls back to the default before recording begins, and both the recorder and the writer stop at that size. The check fills past the default and past a larger size, and refuses 0, words and values outside the range. On Windows, an RX 9070 XT, 8,388,608 records covered a 20 second Hyrule Field window (1,466 of 1,467 frames). The default still fills at about 6 seconds. A run without `HOS_CHAIN` matches main. The Linux run this asks for is still open: `HOS_CHAIN_RECORDS=8388608` for about 20 seconds, then `HOS_CHAIN_RECORDS=lots`, expecting the default and the log line. The farm request is waiting on a BigBean.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 4s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/chain-capacity:klimovich008-perf/chain-capacity'
git switch 'klimovich008-perf/chain-capacity'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1282
No description provided.