WIP: fix(vk): a rewritten finalized target goes at three quarters only where the rest is in memory, with a retire reason of its own #1489
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1489
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:fix/finalized-target-review"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux, RTX 4090 (a 6 GB card's allowance by switch); game-free checks · Needs: Windows, any GPU; an 8 GB card
TL;DR
Follows #1462, which lets go of a finalized terrain tile once the game has rewritten its memory. HumanBean's review of it (1687, 1690) made three suggestions that were not blocking: apply the three-quarters rule only where the untouched quarter is known to be in memory, name these textures in the "textures retired" line instead of calling them "drawn over", and run the check once per texture per pass. This PR does all three, each with a test that failed first. A 16-minute walk at a simulated 6 GB allowance on this head kept the render-target images in a band; the cost of the texture pass is not measured yet (the night's timings were invalid; queued again).
Breakdown
written_back_for == last_written), all of the storage otherwise; a fourth retire reason, "rewritten by the game N"; the answer kept in the texture for the pass.tex_frame_ms, not measured yet: the night's timings were invalid and are queued again), a real 8 GB card, Windows execution.Why
#1462 (merged) stops the render-target images of #1443 from climbing, by letting go of finalized terrain tiles the game has rewritten. Its review found three things to improve and left them to the author. The first is a correctness one on a narrow path: a target that the write-back never covers has the part the game did not rewrite only in its image, which the first form of the rule did not take into account. The others are a log reason and a repeated computation.
Follows #1462. Refs #1443.
How to reproduce
Seen on: Linux, Arch, RTX 4090, NVIDIA 615.78.08, main @
93bc8a7fb(the merge of #1462)Settings and switches: defaults
Steps: the three are shown by the new game-free test,
python -I -B tests/test_target_retention.py, run on main @93bc8a7fbwith this PR's test copied in; in the game, read the "textures retired" line of a walk through Hyrule FieldExpected: a one-level or over-1 MB target rewritten by three quarters stays; the retired line has a column for the textures the game rewrote; a tile's written runs are read once a pass
Actual: on main the first two cases fail ("one level, three quarters written over: stays" and "over 1 MB, three quarters written over: stays"), the line counts the textures as drawn over, and a tile's runs are read up to four times a pass
What changed
Three review points of #1462, each as a test commit that fails on the commit before it and its fix.
finalized_obsolete()ingpu/vk/vk_texture.cpp, HumanBean's first suggestion): a finalized texture the GPU drew, of 32 KB or more, goes at three quarters of its storage rewritten only ift.written_back_for == t.last_written(a mip chain up to 1 MB, whichdrawn_writeback.hwrites back: the terrain tiles); any other needs all of its storage rewritten. The fixture models the write-back withdrawn_writeback.h's ownnotedandkFrames.gpu/vk/evict_reason.h, the second suggestion):OBSOLETE;classifytakes whether the texture is obsolete; the line readsbudget N, drawn over N, dead targets N, rewritten by the game N(the first three unchanged); theEvictrecord carries the flag instead of the or; the line's buffer grows from 160 to 192 bytes.Texture::obsolete_passandTexture::obsoleteingpu/vk/vk_texture.h, the third suggestion): the answer is kept in the texture with theframe_nothe pass runs under, so the page generations are walked once. HumanBean put the cost at tens of microseconds a pass; the timing, queued again, is the measurement.fd954e25d337f9cdf11bcb62b92a1022ad7bOBSOLETE, "rewritten by the game N"750dffe9a761394e7dfinalized_obsoleteanswers once per texture per passThe branch is on main @
310e7c9dfand merges with current main @a6c8ed570without a conflict. It was first made on main @93bc8a7fb, the merge of #1462 (the six commits had been prepared on #1462's own branch before it merged), and rebased;git range-diffof the two ranges shows all six patches identical. Main's changes in between add 225 lines togpu/vk/vk_texture.cpp; intexture_framethere are two single lines (ahead_budget_bytes = size_budget;before the budget rules andrendered_forget(gone);after the eviction loop), neither in a line this PR changes. The 16-minute walk below ran on the first form (5f83ff7c7, on93bc8a7fb); a 9-minute walk on this head follows it.Measurements
Verdict: the walk below is the in-game run of this change: over 16 minutes at a simulated 6 GB allowance the render-target images stay in a band (no steady rise from minute to minute) and the new retire reason fires. The cost of the texture pass is not measured yet. The two paired exclusive timings of the night (17 Nov, 22:37 to 22:41) each ended after their first run with no measuring window (the walk driver teleported at the first play state and the arrival never came; details in the comment of 18 Nov), so there is no number and no quiet verdict, not a result of the machine being loud. A repaired driver is being rehearsed; the timing is queued again and this table is filled in a dated update. A first look from the diagnostic runs is in the comment of 18 Nov (the texture pass's own time per frame, 0.19 ms before #1462 and 0.16 ms with it, ordinary runs, no speed claim).
Hardware: Linux, RTX 4090, NVIDIA 615.78.08, i9-13900K, 96 GB; the walk is an ordinary run, the timing runs are exclusive holds under the quiet rule
Scene: Hyrule Field (child) for the timing, Kokiri Forest and Hyrule Field with area changes for the walk; DLAA 2048x1152
Method: the frame trace (
RX_VK_FRAME_TRACE=1):tex_frame_ms(the texture pass's own time) and the frame times of the window, main, head, main, head in one hold, two holds in all; the walk samples the game's own log linesHow to reproduce: the workbench's paired timing hold with the field-walk scenario and its lap script; the commands are under How it was tested
93bc8a7fb5f83ff7c7tex_frame_ms, mean and p99 of the window415a8944c(before) against main @93bc8a7fb(with it), same two rowsA walk of 16 minutes at a simulated 6 GB allowance (a 24 GB card, not an 8 GB card)
The game is the build of this change's first form (
5f83ff7c7: main @93bc8a7fband the six commits),RX_VK_TEST_ALLOWED_MB=6144(the renderer sees a 6144 MB allowance on the 24 GB RTX 4090), windowless, a child-Link save in Kokiri Forest. The lap script's laps: teleport to Hyrule Field, walk a planned route for 6000 game presents, teleport back, with a lap's end sampled; it ran 989.7 s of wall time and 44,708 presents (6 laps; the machine was busy at the start, load 69 falling to 12, so the game ran at about 20 to 45 frames a second: an ordinary run, no speed claim). Exit 0, fatal 0, crash 0, device lost 0, kernel Xid/NVRM lines 0 (journal coverage complete), no "ran without a core" lines.At the lap ends (0 to 6): render-target images 93, 126, 140, 165, 138, 142, 208; kept finalized targets 3, 51, 63, 77, 70, 70, 54. So after the first laps the images move between 126 and 208 and the kept targets between 51 and 77, with no steady rise; the renderer's VRAM stays between 4.6 and 5.5 GB (one lap-end sample reads 6,226 MB, 82 MB over the 6,144 MB allowed) and the process's RSS falls from 13.4 GB to 6.1 GB. The last sample (208 images, minute 17) is the highest of the series: I do not know whether that is the start of a rise or a one-off, and there is no main beside this walk, so this is "no growth beyond a band in 16 minutes", not a proof of a plateau and not a comparison with main (that comparison is #1462's, on its laps). The retired lines of the walk sum to 32,093 textures: budget 25,318, drawn over 1,962, dead targets 4,403, rewritten by the game 410, so the new reason shows and the terrain tiles are still let go under the narrowed rule.
Pictures: 10 taken while walking and 11 at the ends of laps and legs, looked at as contact sheets (320 x 180 tiles): the ground, grass, rock faces, a path and a cliff draw normally; no streaks along edges, black patches or missing tiles at that size. The last lap ended with the teleport back refused ("an event is playing": a character's text box was on screen in the last walking picture), so lap 6 ends in Hyrule Field. A subtle streak would not show at this size; a closer look by a person is under Must be tested. In the dark passage out of Kokiri Forest a row of small white flecks shows on the ground; a walk on main before #1462 (
415a8944c, with counters only) shows the same flecks at the same spot, so they do not come from this change.What an earlier pair on the pre-rebase twin of #1462's review head (
4cd44a7a3: #1462 and these six patches over main @645d58e68) showed for the retired line: a 375 s walk summed to budget 12,187, drawn over 434, dead targets 990, rewritten by the game 144, against main's 9,898, 1,147 and 538 with no such column; that pair was not route-matched (229 m against 92 m in lap 1) and is not used for the leak, which #1462's own runs showed.A walk of 9 minutes on this head, at the same allowance
The same route and setting on
761394e7ditself (the game logsv3.0.0-rc.7-67-g761394e7d), a full build over main @310e7c9df: 555 s, 22,578 presents, 3 laps; exit 0, no fatal, crash, device-lost, kernel or NVRM line. Render-target images moved between 98 and 170 by minute and read 107, 143 and 156 at the lap ends; kept finalized targets rose from 22 to 87 by minute 10, the same shape as the first 10 minutes of the 16-minute walk (3 to 77); the new retire reason fired 180 times in 13,339 retirements. The walk and lap pictures, looked at as a contact sheet, draw normally (ground, grass, rock faces, caves); the white flecks in the dark passage are the ones main shows. Ten minutes say nothing about a plateau either way: this is the game run of this head, not a leak measurement.The Test Farm's three reports on #1462 and the review's reading
The Test Farm ran field-retention on #1462's head (
1c09f9d12) against its base: one game start a side, a 20 s field window, the 60 fps cap. RTX 4070: field p99 18.27 to 23.19 ms (+26.9 %, over the 20 % rule, failed). RX 9070 XT: 19.93 to 20.63 (+3.5 %, passed). RTX 3070 8 GB: 19.88 to 19.89 (+0 %, passed). The sticky's median over the two Windows reports it counted was +15 %, and the farm marked the 4070's report as disagreeing with its peers.HumanBean's reading (review 1690) is that this is noise, not the change: main's side of the 4070 run was unusually smooth (field max 20.7 ms against 47 to 88 ms on main for the other PCs), p50 did not move (16.67 to 16.62), there were no stutters or hitches on either side, the other two PCs do not show it, field VRAM is flat on all three (+0 % to +1.7 %), and the work added per pass is tens of microseconds. I find that consistent with what the three reports show, and I am not using it as a measurement: the direct checks are the paired
tex_frame_msabove and a diagnostic run on my PC (not part of this PR; running now) that counts rendered textures let go and made again within 30, 120 and 600 frames on the same field walk, main before and with #1462, next to the frame trace. Their numbers go into this description as a dated update.Flags and settings
None.
How it was tested
Everything here was run by an AI agent (Claude Sonnet 5.5) working for me; commands are given so anyone can rerun them. The tools are the repository's own game-free tests and, for the game runs, the workbench's (https://github.com/mooki000/ls-workbench: the run wrapper with its GPU gate, the scripted-play drivers over the control bridge described in https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/docs/control.md, the regression worker's paired timing hold and the idle-class test gate).
python -I -B tests/test_target_retention.pycuts, by their text,texture_frame's choice of which textures go andfinalized_obsolete()out ofgpu/vk/vk_texture.cpp(tests/srccut.py), compiles them with stand-ins for the Texture fields they read, and runs 30 laps of 60 render targets the size of the terrain tiles (260 x 260 RG8, nine levels, 176 KB of texels, 325 KB as the allocator gives it), each drawn once, finalized, never used again, with the game writing over a lap's storage two laps on (all of it, a quarter, half, three quarters, or a 16 KB texture); the write-back isdrawn_writeback.h's ownnotedandkFrames. Each test commit was run on the commit before it:fd954e25d337f9cdf11bcb62b92test_evict_reason.pyno longer finds the newclassifycall; the fixture does not compile (noobsoletemember ofEvict)a1022ad7b750dffe9a761394e7dEach fix was also broken on purpose in a copy of the tree and the test failed every time: the first with the old rule (always three quarters), with always all of the storage ("three quarters is" fails) and with the condition inverted; the second with the pass handing the flag back as drawn over ("each went as obsolete" fails); the third without the memo (the read count is four) and with a memo that never expires (five cases fail).
2. The neighbours.
test_evict_reason.py,test_readback_timeout_reuse.py,test_vram_compact.pyandtest_drawn_writeback.py, which also cuttexture_frame, pass on the rebased head.3. Whole suite and Windows syntax, on the build of the first form (
5f83ff7c7, over main @93bc8a7fb).python -I -B tests/run_all.py --jobs 6 --strictthrough the workbench's idle-class gate (the idle wrapper, three slots, core dumps off, a fresh HOME): 675 passed, 38 failed, 17 skipped. Compared by name with main: 37 of the 38 are in the list of the earlier whole-suite run of main @645d58e68, and the 38th,test_log_open_failure.py, is a test main added since; run alone on main @93bc8a7fbit fails the same way (its C fixture does not compile with this PC's gcc and-Werror).test_crash_record.py, which failed on main @645d58e68, passed in this run. I did not run the whole suite on main @93bc8a7fb. The workbench's Windows fixture check (llvm-mingwucrt-x86_64) compiles the changed C++ fixtures for the Windows target: PASS, 2 translation units, 0 failures, 0 unjudged. It is a syntax check, not a Windows run.4. The walk. The game was driven over the control bridge by the lap script of the workbench's scripted-play drivers: it starts the game through the run wrapper with a save and the allowance switch, teleports to Hyrule Field, follows a planned route with the left stick (the position and camera from the bridge's status give the stick's angle), teleports back, and samples every 4 s the process's memory from
/proc, the card's memory fromnvidia-smiand the newest values of the renderer's own log lines (image blocks of the render targets, finalized textures kept, VRAM, textures, the retired line), with a picture of the game's frame every 90 s of walking. The table takes the last value of each line before a minute's mark; the numbers come from the run'ssamples.csvthrough a small script, nothing is averaged.5. Still to run on my PC: (a) done: the diagnostic counting of let-go-then-remade targets, main before and with #1462 (ordinary runs; the result is in the comment of 18 Nov); (b) the paired exclusive timing, now split over two holds (main then head, head then main, so each side runs twice; 225 s windows on the field walk; the quiet rule: mean other work under 3 cores, highest 5 s under 6, same-arm repeats within 5 %), reading the frame trace's
tex_frame_msand the p99 of each window. A run that is not quiet is kept and said, not averaged away.Testing
Tested: on my own PC. On this head
761394e7d(on main @310e7c9df): the five focused tests and a 9-minute walk in the game at a simulated 6 GB allowance (Measurements). On its first form5f83ff7c7(on main @93bc8a7fb;git range-diff: the same six patches): the three red-first tests and their fixes, the four neighbouring tests, the whole suite by name against main, the Windows-target syntax check, the Linux build, and the 16-minute walk.Not tested: the cost of the texture pass (the night's timings were invalid; queued again); whether the Windows field p99 of the Test Farm's 4070 report is real (the review's reading is noise; the Linux diagnostic of the comment of 18 Nov fits that reading, and the paired timing is queued again); the whole suite on this head itself (it ran on the first form); Windows execution of the changed C++ fixtures; a real 8 GB card and a 20-minute walk on it; AMD and Intel GPUs.
Must be tested:
runs: 3), expecting field p99 within 20 % of main (Test Farm: field-retention on windows).Risks
What could break: the rule now lets go of less than #1462's did (a one-level or over-1 MB target needs all of its storage rewritten), so it can only keep a texture longer, not drop one earlier; the terrain tiles still go at three quarters. The log line has one more field at its end, so a script that matched the whole line, not its first three columns, would need the new column.
Cost: the check reads a texture's written ranges once per pass now (it was up to four times), and returns before reading for anything that is not a released, GPU-drawn pool texture of 32 KB or more; the memo adds two fields to
Texture. Measured: not yet. The night's paired timings were invalid (no measuring window), so the cost of the pass is still unmeasured; the diagnostic runs show 0.16 to 0.19 ms a frame for the whole pass on both sides (ordinary runs, no speed claim) and the same upload and destroy mix in the slowest 1 % of frames, which is why I expect the difference to be small, but that is an expectation, not a measurement.Turning it off: revert the commits.
Compatibility: the "textures retired" log line has one more field; nothing else changes.
Issues
Follows #1462. Refs #1443.
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks. Not ticked yet: the texture pass's own time (
tex_frame_ms) is not measured; the night's paired timings were invalid (no measuring window) and are queued again.main(main @310e7c9df; merges with main @a6c8ed570without a conflict).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named): 38 tests fail, all of them failures of main: test_bug_report_defaults.py, test_cpu_report.py, test_crt_options.py, test_damage_debug_menu.py, test_display_startup_failure.py, test_dlss_feature_size.py, test_failed_compile_keeps_objects.py, test_framegen_preset.py, test_fsr_uniform_ownership.py, test_guest_imported.py, test_hud_layout_row.py, test_import_reach.py, test_invalidate_vcore.py, test_linux_preflight.py, test_linux_pressure.py, test_linux_update.py, test_log_open_failure.py, test_mirror_budget.py, test_missing_nup.py, test_mods_size_table.py, test_nonuniform_image_arrays.py, test_parallel_init_order.py, test_pending_guest_import.py, test_pipeline_play_cache.py, test_present_thread.py, test_profiling_mode.py, test_readback_timeout_source.py, test_replay_write_order.py, test_rumble_menu.py, test_run_all_changed.py, test_slow_frames.py, test_staging_burst.py, test_target_image_pool.py, test_upscaler_round_trip.py, test_upscaler_rows.py, test_vcore_escape.py, test_vk_occlusion_queries.py, test_walk_key.py.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened (tests/test_evict_reason.pyfollows the newclassifycall; nothing removed).mainand passing here.New on this forge; not opened on GitHub.
Pushed the rebased head,
761394e7don main @310e7c9df(it merges with today's main @a6c8ed570without a conflict).git range-diffagainst the earlier head5f83ff7c7(on93bc8a7fb) shows all six patches identical; only the base moved.This head had its own game run before the push: a 9-minute walk on the same route and setting as the 16-minute one (a simulated 6 GB allowance on a 24 GB card), a full build over
310e7c9df. Exit 0, no fatal, crash, device-lost, kernel or NVRM line; render-target images between 98 and 170 by minute; the new retire reason fired 180 times; the pictures draw normally. The details are in the description's new "A walk of 9 minutes on this head" section, and the Rebased box is now ticked.Still open, as before: the cost of the texture pass (paired timing tonight, after 22:00, under the quiet rule) and the "Nothing gets worse" box that waits on it. The guide-screenshot check is red on main itself; #1501 fixes main, and once it merges I add a docs-only commit here with the review line for
gpu/vk/vk_texture.h(two data members the Graphics view never draws). A docs-only commit leaves the built game identical, so this walk stays the run of the code.New on this forge; not opened on GitHub.
5f83ff7c72761394e7d9Cost of the texture pass and the Test Farm's 4070 report: what I have, and what I do not
No timing number yet, and the box "Nothing gets worse" stays unticked. The two paired exclusive timings of the night (17 Nov, 22:37 to 22:41: main with #1462 against this PR's head, and main before #1462 against main with it) each ended after their first run with no measuring window. That is a fault of my walk driver, not a loud machine: no quiet verdict was computed for either, and no frame time exists. The driver teleported to Hyrule Field at the first play state; the game reported the teleport as sent, kept rendering Kokiri Forest at about 97 fps, and the arrival never came in the 48 s the run had left. All three runs that failed this way (a rehearsal and the two holds) were the first start of a game copy that had never been run, with a 62 s boot where a warm copy boots in 20 s; I also skipped the usual unmeasured warm-up. A rehearsal after a repair (12 s of walking in the save's own place before the teleport, none of it counted toward the window) on a warm copy arrived in 6 s and opened a window of 144 s with 10,014 frames. I cannot tell from these runs whether the settle walk or the warm copy did it; the next runs have both.
What the rehearsal shows (ordinary run, not a timing: the quiet rule flagged it, other work peaked at 6.1 cores). The texture pass's own time (the frame trace's
tex_frame_ms) over the window: mean 0.162 ms, median 0.130 ms, p99 0.79 ms, longest 4.5 ms, against a mean frame of 14.4 ms: about 1 % of the frame. That says how big the thing I am measuring is, not what this PR changed.The diagnostic runs (main before #1462 against main with it, two laps each, ordinary runs at load 10 to 30, a diagnostic build that is not part of this PR). The class the new rule adds to ("drawn over or obsolete") was let go 1,325 times before and 1,361 times with #1462 in runs of about 16,700 frames, and 795 against 841 of them came back as a new texture within 600 frames: the same 60 to 62 %. The slowest 1 % of frames spent the same mix (frames with an upload over 1 ms: 51 of 170 and 60 of 167; frames with textures destroyed: 51 and 52). The pass's mean time per frame was 0.190 ms and 0.159 ms. I find no sign that targets let go as obsolete are made again soon in numbers that could move a field p99. That fits the reading of the review that the 4070 report is noise; it is Linux and one run a side, so it does not measure that Windows PC.
What would let me tick the box. Two exclusive holds under the quiet rule, 225 s runs, main with #1462 (
93bc8a7fb) and this PR's head (the timed build is5f83ff7c7, six patches identical to the pushed761394e7dbygit range-diff, on the same base93bc8a7fb): one hold main then head, the other head then main, so each side runs twice and the order is balanced over the two holds (a 480 s hold cannot hold four runs of this length; the quiet rule's usual one-hold A B A B does not fit). I will tick the box only if both holds are valid and quiet and the difference intex_frame_msand in p99 is within the spread between the two runs of the same side; if a hold is not quiet I will say so here and give no number. A warm-up of the main copy is queued before them; the holds wait for the machine's night window (22:00). The numbers, valid or not, go here as a dated update.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.