fix(host): HOS_CHAIN_RECORDS=<n> sizes the HOS_CHAIN buffer, so a capture can cover a whole measured window #1282
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1282
Loading…
Reference in a new issue
No description provided.
Delete branch "klimovich008/Longshot:perf/chain-capacity"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11, RX 9070 XT · Needs: Linux, any GPU
Why
HOS_CHAIN, the capture thatscripts/chain.pyreads for the frame's critical path, stops when its fixed buffer of 2,097,152 records is full: in a 20 s Hyrule Field window that is the first 5.8 s (28 % of the window's frames), so the path it reports is only the window's first third. WithHOS_CHAIN_RECORDS=8388608the same capture covers the whole window (99.9 % of its frames). No issue was filed for it (tracker searched for chain, critical path and profiling).How to reproduce
Seen on: Windows 11, AMD Radeon RX 9070 XT, AMD's Windows driver
Settings and switches:
HOS_CHAIN=chain.binin the environment; threaded rendering on,HOS_FPS_MAX=240Steps: load into Hyrule Field, make
chain.bin.gobeside the game, turn the camera for 20 s, deletechain.bin.go, thenpython -I scripts/chain.py chain.binExpected: the capture holds the whole 20 s, about 1,400 frames
Actual: the buffer fills after about 6 s: 2,097,152 records, and
chain.pywalks about 400 frames; nothing inmaincan make the buffer largerWhat changed
A new switch,
HOS_CHAIN_RECORDS=<n>, sets how many records theHOS_CHAINbuffer holds, from 1,048,576 to 33,554,432 (32 MB to 1 GB). Without it, or with any value that isn't a whole number in that range, the size stays today's 2,097,152, and the game's log says which size it records with. It is read only whenHOS_CHAINis set, so nothing changes for anyone else.host/hos_chain.cCHAIN_CAPbecameCHAIN_CAP_DEFAULT(2,097,152, unchanged), withCHAIN_CAP_MIN,CHAIN_CAP_MAXandstatic LONG64 chain_cap; the recorder's full test inhos_chain_note()and the writer's clamp inchain_write()usechain_cap.hos_chain_start()readsHOS_CHAIN_RECORDSonce, before the recorder thread starts, throughchain_records()(strtoll, base 10, with end anderrnochecks); a refused value is logged:chain: HOS_CHAIN_RECORDS=<v> is not a number of records from 1048576 to 33554432; 2097152 instead.chain_thread(): if the larger buffer can't be allocated, it logschain: no memory for N records (HOS_CHAIN_RECORDS); 2097152 insteadand tries the default; if that fails too, no recording, as before. The start line now names the size:chain: recording (<file>, at most N records).chain_capis set only before recording first starts and never changes while it records (the comment beside it says why every thread then reads the size its buffer was made with).HOS_CHAINrow indocs/development.mdgives the default size and the new switch; the file's header comment too.tests/test_chain_records.py(new, found bytests/run_all.py): compiles the real recorder, writer and start code cut fromhos_chain.cwith stubs and fills the buffer past the default and past a larger size.scripts/chain.pyand the file format are unchanged:chain.pyreads the record count from the file's header and has no size of its own.Measurements
Verdict: fixed: with
HOS_CHAIN_RECORDS=8388608the capture covers the whole 20 s window (1,466 of 1,467 frames walked, not capacity-limited), where today's size covers 5.8 s (397 of 1,400 frames, 28.4 %). The new test fails onmainand passes here. Normal play is unchanged: the switch is only read withHOS_CHAINset, and one run each ofmainand this PR withoutHOS_CHAINare in the same band.Baseline: the game runs were made on this branch's base, main @
0cab3548, and this PR @a9d942d6, before main (8e9cfc3a) was merged in. They still hold: none of main's 34 newer commits touchhost/hos_chain.c, andscripts/chain.py's 3-line change there doesn't touch how it reads the file. The game-free checks were run again after the merge (Testing).Hardware: Windows 11 (build 26300), AMD Radeon RX 9070 XT (AMD's Windows driver), Ryzen 7 5700X3D (8 cores, 16 threads), 64 GB
Scene: Hyrule Field, the camera turning steadily; 16:9 window, threaded rendering on,
HOS_FPS_MAX=240Method: one 20 s window per run, taken after the load; the capture switched on and off with the window (
chain.bin.gomade at its start, deleted at its end). Coverage: the records' times against the frames presented in the window (both on the game'shos_now_nsclock), andpython -I scripts/chain.py chain.binon each file. Normal play: frames a second, CPU time a frame and frame times over the same window, with noHOS_CHAIN, one run each.How to reproduce: start the game with
HOS_CHAIN=chain.binandHOS_CHAIN_RECORDS=8388608in the environment, go to Hyrule Field, makechain.bin.gobeside the game, turn the camera for 20 s, deletechain.bin.go, wait for the log'schain: ... records ... writtenline, thenpython -I scripts/chain.py chain.bin.The capture, 20 s of Hyrule Field:
0cab3548HOS_CHAIN_RECORDSunset: this PR @a9d942d6HOS_CHAIN_RECORDS=8388608: this PR @a9d942d6HOS_CHAIN_RECORDS=33554432: this PR @a9d942d6chain.pytests/test_chain_records.pyNote: on
main's run the window opened late (the area's load came later than usual), so the capture began 3.7 s before the window and only 3.5 s of the window fell inside it. The "unset" column is the comparison at today's size: it runs the same code path asmainwith the same buffer.Normal play, no
HOS_CHAIN(one run each, a sanity check rather than a measurement: none of this PR's code runs withoutHOS_CHAIN):0cab3548a9d942d6Flags and settings
One new switch for profiling, off unless set; the default is unchanged.
HOS_CHAIN_RECORDS=<n>HOS_CHAINbuffer holds (32 bytes each)HOS_CHAIN; 1,048,576 to 33,554,432, any other value the default (logged)Testing
Tested:
scripts/chain.py. The memory at the ceiling: +1.07 GiB of commit and +0.26 GiB of working set against a run withoutHOS_CHAIN(Windows commits the buffer at once; only the pages that fill become resident).tests/test_chain_records.py:HOS_CHAIN_RECORDS=2500000stops at exactly record 2,500,000 with all of them in the file in order; unset, exactly 2,097,152; the floor; a window that ends before the buffer is full;0,lots,8M,0x800000, one below the floor, one over the ceiling, a negative and an overflowing number all give the default with the log line (empty gives it silently); the ceiling with the allocation failing gives the default, logged; no memory at all gives no recording, as before; withoutHOS_CHAIN,HOS_CHAIN_RECORDSisn't read. It fails withmain'shos_chain.c, and two broken variants (the writer's clamp and the recorder's full test left at the default) fail inside it. It passes on Windows with llvm-mingw clang and MinGW gcc, and on Linux (WSL Ubuntu) with clang and gcc.8e9cfc3a):python -I -B tests/run_all.py --strict(whatmise run testruns) on this branch and on main @8e9cfc3a, Windows, llvm-mingw 20261104: this branch 623 passed, 46 failed, 45 skipped (the Linux-only checks); main 622 passed, 46 failed, 45 skipped. The 46 failures are the same list on both (they fail on this Windows checkout onmaintoo; none readshos_chain.cor the docs row); the extra pass istests/test_chain_records.py.tests/test_chain.pypasses on both. Before the merge, on main @0cab3548, the same comparison gave the same picture (25 shared failures, the new test the one extra pass).host/hos_chain.ccompiled for Windows (llvm-mingw clang,-Wall -Wextra: no warnings) and for Linux (WSL clang 21 and gcc with the Linux build's flags,host/posixfirst: no warnings; objects only).Not tested:
hos_chain_onis first set) and never changes while it records, the same ordering the buffer's pointer already relies on.mise run lint(pre-commit): mise isn't installed on this PC, so the pre-commit hooks were not run.Must be tested:
HOS_CHAIN=chain.bin HOS_CHAIN_RECORDS=8388608, makechain.bin.goin Hyrule Field, turn the camera for 20 s, deletechain.bin.go, expecting the log'schain: recording (chain.bin, at most 8388608 records), then achain: <N> records ... writtenline, andpython -I scripts/chain.py chain.binwalking about 20 s of frames; then once withHOS_CHAIN_RECORDS=lots, expecting theis not a number of records ... 2097152 insteadline and a 2,097,152-record buffer (Test Farm: @Beanstalk linux, as in the comment below)Risks
What could break: only runs with
HOS_CHAINset. A large size takes its memory when recording starts (8,388,608: 256 MB; the ceiling: 1 GB of commit) and writes a file that size when the capture ends or the buffer fills; if the buffer fills inside a measured window, that write lands in it, as it does at today's size. A failed allocation falls back to the default, then to no recording, each logged.Turning it off: leave
HOS_CHAIN_RECORDSunset (or revert the commit).Compatibility: Nothing changes: no save, options file, cache, chain file format or vk ABI change;
HOS_CHAINwithout the new switch records as before.Checklist
Everyone
type(scope): description) of the status line's type.main. (main @8e9cfc3amerged in; the one conflict, theHOS_CHAINrow next to #1279's new rows, kept both)python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened. (tests/test_chain_records.pyis new and compiles the whole ofhost/hos_chain.c; no existing check of that file was changed)🤖 Generated with Claude Code
8e9cfc3a) into perf/chain-capacityTested in game: Windows 11, AMD Radeon RX 9070 XT (AMD's Windows driver), Ryzen 7 5700X3D, this PR @
a9d942d6on main @0cab3548. Hyrule Field, camera turning, 20 s window: withHOS_CHAIN=chain.bin HOS_CHAIN_RECORDS=8388608the log saidchain: recording (chain.bin, at most 8388608 records), 7,949,300 records were written andscripts/chain.pywalked 1,466 of the window's 1,467 frames; unset, 2,097,152 records as before (5.8 s); at the ceiling (33,554,432) the size line and a whole capture too. WithoutHOS_CHAIN, normal play in the same band as main (74.0 fps both, no frame over 50 ms).@Beanstalk linux, any GPU: please build this PR and start the game with
HOS_CHAIN=chain.binandHOS_CHAIN_RECORDS=8388608in the environment. In Hyrule Field, create an empty filechain.bin.gobeside the game, turn the camera for about 20 s, then deletechain.bin.go. Expected in the log:chain: recording (chain.bin, at most 8388608 records)at the start, then achain: <N> records of <M> threads written to chain.binline;python -I scripts/chain.py chain.binshould report about 20 s worth of frames (not about 6 s). Then once more withHOS_CHAIN_RECORDS=lots: expectedchain: HOS_CHAIN_RECORDS=lots is not a number of records from 1048576 to 33554432; 2097152 insteadand a capture of at most 2,097,152 records. This PC is Windows only; the switch is Linux-built from the same file (host/posix), compiled and unit-tested there, but not run in the game.@klimovich008 only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.Possibly related, for context:
WIP: fix(host): HOS_CHAIN_RECORDS=<n> sizes the HOS_CHAIN buffer, so a capture can cover a whole measured windowto fix(host): HOS_CHAIN_RECORDS=<n> sizes the HOS_CHAIN buffer, so a capture can cover a whole measured windowReady for review. Description: added the checklist box for the new source-reading test (tests/test_chain_records.py compiles the whole of host/hos_chain.c). The measurements were made on main
0cab3548; the Baseline line says why they hold on8e9cfc3a(none of main's newer commits touch host/hos_chain.c).HOS_CHAIN_RECORDSsets how many records a chain capture holds, from 1,048,576 to 33,554,432. Unset, or not a whole number in that range, it stays 2,097,152. I read heade8df1bf8. It applies onto current main, and this stays a comment.The size is read once, before the recorder thread starts. A buffer that cannot be allocated falls back to the default before recording begins, and both the recorder and the writer stop at that size. The check fills past the default and past a larger size, and refuses 0, words and values outside the range. On Windows, an RX 9070 XT, 8,388,608 records covered a 20 second Hyrule Field window (1,466 of 1,467 frames). The default still fills at about 6 seconds. A run without
HOS_CHAINmatches main.The Linux run this asks for is still open:
HOS_CHAIN_RECORDS=8388608for about 20 seconds, thenHOS_CHAIN_RECORDS=lots, expecting the default and the log line. The farm request is waiting on a BigBean.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.