fix(shaders): read a blob's words from the file, not stdio's stale buffer #2153
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2153
Loading…
Reference in a new issue
No description provided.
Delete branch "dazemc/Longshot:fix/spirv-pack-stale-read"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Store::readis meant to refuse a SPIR-V blob whose words changed on disk: it verifies a content hash after reading. On Linux that check is defeated, because the store reads through a per-threadFILE*opened once: glibc'sfseekohas a fast path that moves the pointer inside the stdio buffer instead of callinglseek/read, so a blob overwritten behind the handle is served from the old buffered bytes — which are exactly the bytes whose hash is in the head. A minimal reproduction understraceshows the second read issuing noread()syscall at all (only seeks) and returning the old word.Windows passes because its
fseekpath re-reads.What changed (
gpu/vk/spirv_pack.h): on POSIX the pack handles used byread()and byhead_is()are unbuffered (setvbuf(..., _IONBF, 0)), so every read goes to the file; Windows keeps its existing buffered path. The hash check stays as it was. Rejected alternatives: re-hashing after the read (already present — stale bytes hash to the head),posix_fadvise(kernel page cache, not stdio's buffer),O_DIRECT(alignment, slower), reopening per read (hot path).Testing
Tested:
tests/spirv_pack_test.cppextended. The test writes a blob, reads it (filling the handle's buffer), overwrites a word in the file through a separatefopen(..., "r+b"), then requires the read to be refused and returns the changed word; a second block overwrites a head afterhead_isfirst read it and requiresputto write the words again. After the rebase ontoa2dc5385, #2171's version of theflipblock read the blob back on a thread of its own, so the calling thread's stdio handle held nothing when the disk changed, and the newCHECK(got[10] == 0xDEADBEEF)passed onmainwithout exercising the bug. The block now reads back on the same thread that reads after the change — the handle whose buffer glibc's stalefseekoserves — which is the condition the bug needs and the way the test read it before #2171; the assertion is not weakened. Run againsta2dc5385's owngpu/vk/spirv_pack.h(onlytests/spirv_pack_test.cpptaken from this head), it fails for the bug at lines 138, 142 and 144 (theflipcase) and 169 (the head case):With the fix:
In the game: a pack over
1885234b, from this branch at59ce70bb, Kokiri Forest (child), 20 s after warm-up: 1833 frames, 91.7 fps, median 10.93 ms, 0 cold frames (base: 1821 frames, 91.1 fps, 11.08 ms).Game run at
59ce70bb; the head1a8a053bdiffers only intests/spirv_pack_test.cpp(the rework after #2171's change to it) and the rebase ontoa2dc5385(upstreammaincommits, not part of this change), andgpu/vk/spirv_pack.his byte-identical between the two, so the run covers this change's game code.Still to test:
Measurements
Verdict: no win in this run: 4 KB sequential is 3.0 → 3.4 µs/read (+0.4 µs, ~13 %), 64 KB sequential is 30.3 → 32.2 µs (+1.9 µs, ~6 %), 4 KB scattered is equal, and 64 KB scattered is 33.2 → 32.3 µs. The description records medians over several runs and no spread, so none of the differences is called noise; the 4 KB sequential cost is the worst case and is stated in Risks.
Setup: Arch Linux, Ryzen 9 5950X, page-cached synthetic blobs, hashes precomputed, 2000 blobs, several runs, median of runs; a game-free microbenchmark of
Store::read.1885234b1a8a053bBaseline: the numbers were measured on
bc44f697; the changed lines ingpu/vk/spirv_pack.hare byte-identical on1a8a053b, and onlytests/spirv_pack_test.cppchanged in the rebase. No commit in344deb6c..a2dc5385touches the header, so the numbers still hold.The largest relative cost is the tiny 4 KB sequential read (3.0 → 3.4 µs, ~13 %); at 64 KB — the size real SPIR-V stages are — sequential is 30.3 → 32.2 µs, while scattered is equal at 4 KB and slightly lower at 64 KB. The setup records medians only, so these are stated as measured rather than as noise.
read()always reads the whole blob at a fresh offset, so stdio's buffer mostly missed anyway and only added a copy; unbufferedfreadreads straight intoout.Risks
A store written concurrently by another process while a blob is read now returns the changed words (and the hash refuses them) instead of stale ones. No switch; a revert is enough. Nothing changes for saves,
longshot_options.inior the game folder's caches; the shader store is read-only from the game's side.The measured cost is the 4 KB sequential read: 3.0 → 3.4 µs/read (+0.4 µs, ~13 %), the largest relative regression in the table; 64 KB sequential is 30.3 → 32.2 µs (+1.9 µs, ~6 %), 4 KB scattered is unchanged, and 64 KB scattered is slightly lower. The description records medians over several runs and no spread, so this is not dismissed as noise. The change is a correctness fix, and this is its measured price.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). Rebased ontoa2dc5385;tests/run_all.py --strict --jobs 3insidesystemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0with the pinned headers.mainat9e10f6f7was 911/22/18; ata2dc5385it is 911 passed / 23 failed / 18 skipped in 635 s — the 23rd istests/test_sim_turf_pools.py; this head1a8a053b: 911 / 23 / 18 in 619 s. Failing names identical (23, compared name by name):nx2/test_import_windows.py,tests/test_close_waits_for_save.py,tests/test_compile_zero_byte.py,tests/test_cpm_discovery.py,tests/test_damaged_rom_files.py,tests/test_dev_packs.py,tests/test_draw_stream_visibility.py,tests/test_fs_save_commit.py,tests/test_fsr_uniform_ownership.py,tests/test_guest_copies.py,tests/test_hold_toggle.py,tests/test_lower_atomg_cas.py,tests/test_menu_i18n.py,tests/test_menu_keys.py,tests/test_menu_slider_ends.py,tests/test_mise_build.py,tests/test_pads_after_menu.py,tests/test_recording_admission.py,tests/test_sim_turf_pools.py,tests/test_sim_turf_settings.py,tests/test_switch_table.py,tests/test_tls_objects.py,tests/test_vibration_menu.py. The 23rd,tests/test_sim_turf_pools.py, is new ona2dc5385since9e10f6f7and is #2174's stale check (it asserts the old zero turf floor, while #2174 made the floor the game's size); it fails identically here.tests/test_spirv_pack.pypasses (2.1 s), both reworked cases included.main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened (tests/spirv_pack_test.cppgains the two cases; no existing assertion removed).Ready to merge
@Beanstalkrequest was posted on 2026-11-22 00:02, BeanBot answered that only a member ofBigBeanscan start it, and no approval has come yet. The by-handfliprework has run (in Tested). Ticks when that result is in Tested.pr status: Reviews: none); the Beanstalk approval is waiting for a BigBean.main, measured again or a comment says why the numbers still hold. On head1a8a053bafter the rebase: the description check passed, 11 checks still running, every build and test jobBlocked by required conditions. The measurements namemain @ 1885234b; the production lines ingpu/vk/spirv_pack.hare identical on this head, and onlytests/spirv_pack_test.cppchanged in the rebase, so they still hold. Re-confirm CI when it reports.Tested in game: Kokiri Forest (child), pack over
1885234b, 20 s: 91.7 fps median 10.93 ms, 0 cold frames; the shader store loads and plays. Arch Linux/Hyprland, RTX 4070 Ti, NVIDIA 620.70.06.Possibly related, for context:
b531da874abc44f69726Rebased onto
344deb6c(headbc44f697); gpu/vk/spirv_pack.h is untouched by the commits since, so the benchmark and the run hold.Measurements were taken on
1885234b. The head has since moved (bc44f697) and is based on344deb6c, which is now 3 commits behind current maindd5ffe6a, so here is why the numbers still hold rather than a fresh run.gpu/vk/spirv_pack.his untouched by everything that landed in between — #2161, #2140 and #2163 — and nothing in the change depends on those commits;merge-treeagainstdd5ffe6ais clean. A local check agrees:git log 344deb6c..dd5ffe6a -- gpu/vk/spirv_pack.his empty.@Beanstalk Windows: start the game on this branch on a pack and confirm the shader store loads and programs serve — no shader errors in the log, the scene renders. Windows keeps the buffered path, which is unchanged here, so this is a no-regression check.
Ready otherwise.
@dazemc only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was forbc44f69, and the PR has changed since (1a8a053), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.fix(shaders): read a blob's words from the file, not stdio's stale bufferto WIP: fix(shaders): read a blob's words from the file, not stdio's stale bufferBack to WIP for two things:
344deb6c, andmainhas since moved toa2dc5385with #2171 (9e10f6f7) fixing the game-free checks (~199 test files). The suite numbers here were measured against the oldmain(813/116/18 there, now 911/22/18), so 'mise run test' is unticked until a rebase ontoa2dc5385and a re-run.flipcase intests/spirv_pack_test.cppneeds reworking: #2171 moved that block's read-back onto a thread of its own, so the calling thread's handle holds no old buffered words and the newCHECK(got[10] == 0xDEADBEEF)passes onmainalready. The new head case (a head overwritten underput) still fails there, so the test still catches the bug — but thefliphalf should fail onmaintoo. Listed in Still to test.The in-game smoke stands: its record (
59ce70bb, 1833 frames, 91.7 fps, 0 cold frames) matches, andgpu/vk/spirv_pack.h's changed lines are identical to the head's. Measurements still namemain @ 1885234b; the comment saying why they hold was written againstdd5ffe6a, so re-confirm after the rebase.The Ready to merge boxes stay unticked, each with its reason beside it: Windows is with the Test Farm, waiting for a BigBean; no review yet; CI's build/test jobs are still pending.
bc44f697261a8a053bc0Rebased onto
a2dc5385: head is now1a8a053b. The rebase was clean, and theflipcase was reworked on top (test-only).Full game-free suite, run in the same shape as a clean
a2dc5385baseline (tests/run_all.py --strict --jobs 3,systemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0, the pinned headers):a2dc5385: 911 passed / 23 failed / 18 skipped in 635 s1a8a053b: 911 / 23 / 18 in 619 sFailing names identical, compared name by name (23). The one name new on
a2dc5385since9e10f6f7istests/test_sim_turf_pools.py: #2174 changed the simulated-turf floor while that old check still asserts a zero floor, so it is stale onmainand red identically here.test_spirv_packpasses (2.1 s), both new cases included.The rework is test-only and honest. #2171 moved the
flipblock's read-back onto a thread of its own, so the calling thread's stdio handle held nothing andCHECK(got[10] == 0xDEADBEEF)passed onmainwithout exercising the bug. The block now reads back on the same thread that reads after the disk changes — the handle whose buffer glibc's stalefseekoserves — which is the condition the bug needs, as the test read it before #2171. The assertion is not weakened. Run againsta2dc5385's owngpu/vk/spirv_pack.h(only this head'stests/spirv_pack_test.cpp), it fails for the bug:The production hunk in
gpu/vk/spirv_pack.his byte-identical to the pre-rebase head; the rework changed no production code. Jev on the test change: 0.88, likely honest. Diff stat fora2dc5385..1a8a053bisgpu/vk/spirv_pack.h | 11 +andtests/spirv_pack_test.cpp | 33 +-; outsidetests/the only file is the original runtime fix, unchanged frombc44f697.In-game evidence (Jev: 0.68, likely valid): the smoked
59ce70bbrecord matches, the changed lines ingpu/vk/spirv_pack.hare identical to this head, and no commit in344deb6c..a2dc5385touches it. No new in-game run was made.Measurements column corrected to name the rebased head, with (where the numbers were taken) in a Baseline line: the production hunk in gpu/vk/spirv_pack.h is byte-identical, only the test changed in the rebase.
WIP: fix(shaders): read a blob's words from the file, not stdio's stale bufferto fix(shaders): read a blob's words from the file, not stdio's stale bufferCorrection to the comment above: the measurement commit is
bc44f697; the rebased head is1a8a053b.@Beanstalk Windows: start the game on this branch on a pack and confirm the shader store loads and programs serve — no shader errors in the log, the scene renders. Windows keeps the buffered path, which is unchanged here, so this is a no-regression check.
Head:
1a8a053b(rebased ontoa2dc5385).@dazemc only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.Testing now names the run commit and what separates it from the head: on the branch side only
tests/spirv_pack_test.cppdiffers between59ce70bband1a8a053b(the rework after #2171's change), plus the rebase ontoa2dc5385, andgpu/vk/spirv_pack.his byte-identical, so the in-game run covers this change's game code. No code change; stays ready.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.