fix(shaders): read a blob's words from the file, not stdio's stale buffer #2153

Open
dazemc wants to merge 1 commit from dazemc/Longshot:fix/spirv-pack-stale-read into main
Contributor

Summary

Store::read is meant to refuse a SPIR-V blob whose words changed on disk: it verifies a content hash after reading. On Linux that check is defeated, because the store reads through a per-thread FILE* opened once: glibc's fseeko has a fast path that moves the pointer inside the stdio buffer instead of calling lseek/read, so a blob overwritten behind the handle is served from the old buffered bytes — which are exactly the bytes whose hash is in the head. A minimal reproduction under strace shows the second read issuing no read() syscall at all (only seeks) and returning the old word.

Windows passes because its fseek path re-reads.

What changed (gpu/vk/spirv_pack.h): on POSIX the pack handles used by read() and by head_is() are unbuffered (setvbuf(..., _IONBF, 0)), so every read goes to the file; Windows keeps its existing buffered path. The hash check stays as it was. Rejected alternatives: re-hashing after the read (already present — stale bytes hash to the head), posix_fadvise (kernel page cache, not stdio's buffer), O_DIRECT (alignment, slower), reopening per read (hot path).

Testing

Tested: tests/spirv_pack_test.cpp extended. The test writes a blob, reads it (filling the handle's buffer), overwrites a word in the file through a separate fopen(..., "r+b"), then requires the read to be refused and returns the changed word; a second block overwrites a head after head_is first read it and requires put to write the words again. After the rebase onto a2dc5385, #2171's version of the flip block read the blob back on a thread of its own, so the calling thread's stdio handle held nothing when the disk changed, and the new CHECK(got[10] == 0xDEADBEEF) passed on main without exercising the bug. The block now reads back on the same thread that reads after the change — the handle whose buffer glibc's stale fseeko serves — which is the condition the bug needs and the way the test read it before #2171; the assertion is not weakened. Run against a2dc5385's own gpu/vk/spirv_pack.h (only tests/spirv_pack_test.cpp taken from this head), it fails for the bug at lines 138, 142 and 144 (the flip case) and 169 (the head case):

FAIL line 138: !s.read(at, a.size(), hash(a), got.data())
FAIL line 142: got[10] == 0xDEADBEEF
FAIL line 144: again_at > at && reads(s, again_at, a)
FAIL line 169: s.put(q.data(), q.size()) > at

With the fix:

pass 2.1s tests/test_spirv_pack.py   1 passed, 0 failed, 0 skipped

In the game: a pack over 1885234b, from this branch at 59ce70bb, Kokiri Forest (child), 20 s after warm-up: 1833 frames, 91.7 fps, median 10.93 ms, 0 cold frames (base: 1821 frames, 91.1 fps, 11.08 ms).

Game run at 59ce70bb; the head 1a8a053b differs only in tests/spirv_pack_test.cpp (the rework after #2171's change to it) and the rebase onto a2dc5385 (upstream main commits, not part of this change), and gpu/vk/spirv_pack.h is byte-identical between the two, so the run covers this change's game code.

Still to test:

  • Windows (llvm-mingw): the unchanged buffered path, a start (by hand).

Measurements

Verdict: no win in this run: 4 KB sequential is 3.0 → 3.4 µs/read (+0.4 µs, ~13 %), 64 KB sequential is 30.3 → 32.2 µs (+1.9 µs, ~6 %), 4 KB scattered is equal, and 64 KB scattered is 33.2 → 32.3 µs. The description records medians over several runs and no spread, so none of the differences is called noise; the 4 KB sequential cost is the worst case and is stated in Risks.

Setup: Arch Linux, Ryzen 9 5950X, page-cached synthetic blobs, hashes precomputed, 2000 blobs, several runs, median of runs; a game-free microbenchmark of Store::read.

Before: main @ 1885234b After: this PR @ 1a8a053b
4 KB, sequential (µs/read) 3.0 3.4
4 KB, scattered (µs/read) 3.4 3.4
64 KB, sequential (µs/read) 30.3 32.2
64 KB, scattered (µs/read) 33.2 32.3

Baseline: the numbers were measured on bc44f697; the changed lines in gpu/vk/spirv_pack.h are byte-identical on 1a8a053b, and only tests/spirv_pack_test.cpp changed in the rebase. No commit in 344deb6c..a2dc5385 touches the header, so the numbers still hold.

The largest relative cost is the tiny 4 KB sequential read (3.0 → 3.4 µs, ~13 %); at 64 KB — the size real SPIR-V stages are — sequential is 30.3 → 32.2 µs, while scattered is equal at 4 KB and slightly lower at 64 KB. The setup records medians only, so these are stated as measured rather than as noise. read() always reads the whole blob at a fresh offset, so stdio's buffer mostly missed anyway and only added a copy; unbuffered fread reads straight into out.

Risks

A store written concurrently by another process while a blob is read now returns the changed words (and the hash refuses them) instead of stale ones. No switch; a revert is enough. Nothing changes for saves, longshot_options.ini or the game folder's caches; the shader store is read-only from the game's side.

The measured cost is the 4 KB sequential read: 3.0 → 3.4 µs/read (+0.4 µs, ~13 %), the largest relative regression in the table; 64 KB sequential is 30.3 → 32.2 µs (+1.9 µs, ~6 %), 4 KB scattered is unchanged, and 64 KB scattered is slightly lower. The description records medians over several runs and no spread, so this is not dismissed as noise. The change is a correctness fix, and this is its measured price.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named). Rebased onto a2dc5385; tests/run_all.py --strict --jobs 3 inside systemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0 with the pinned headers. main at 9e10f6f7 was 911/22/18; at a2dc5385 it is 911 passed / 23 failed / 18 skipped in 635 s — the 23rd is tests/test_sim_turf_pools.py; this head 1a8a053b: 911 / 23 / 18 in 619 s. Failing names identical (23, compared name by name): nx2/test_import_windows.py, tests/test_close_waits_for_save.py, tests/test_compile_zero_byte.py, tests/test_cpm_discovery.py, tests/test_damaged_rom_files.py, tests/test_dev_packs.py, tests/test_draw_stream_visibility.py, tests/test_fs_save_commit.py, tests/test_fsr_uniform_ownership.py, tests/test_guest_copies.py, tests/test_hold_toggle.py, tests/test_lower_atomg_cas.py, tests/test_menu_i18n.py, tests/test_menu_keys.py, tests/test_menu_slider_ends.py, tests/test_mise_build.py, tests/test_pads_after_menu.py, tests/test_recording_admission.py, tests/test_sim_turf_pools.py, tests/test_sim_turf_settings.py, tests/test_switch_table.py, tests/test_tls_objects.py, tests/test_vibration_menu.py. The 23rd, tests/test_sim_turf_pools.py, is new on a2dc5385 since 9e10f6f7 and is #2174's stale check (it asserts the old zero turf floor, while #2174 made the floor the game's size); it fails identically here. tests/test_spirv_pack.py passes (2.1 s), both reworked cases included.
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A check under tests/ that reads changed source is updated, not weakened (tests/spirv_pack_test.cpp gains the two cases; no existing assertion removed).

Ready to merge

  • Everything in Still to test has run, its results in Tested. Windows has not: the @Beanstalk request was posted on 2026-11-22 00:02, BeanBot answered that only a member of BigBeans can start it, and no approval has come yet. The by-hand flip rework has run (in Tested). Ticks when that result is in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). One Tier 1 machine has run: Arch Linux, RTX 4070 Ti, NVIDIA 620.70.06 (the in-game smoke above). Tier 1's Windows half has not, which is the same pending Windows item.
  • Approved in review, every comment answered. No review yet (pr status: Reviews: none); the Beanstalk approval is waiting for a BigBean.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold. On head 1a8a053b after the rebase: the description check passed, 11 checks still running, every build and test job Blocked by required conditions. The measurements name main @ 1885234b; the production lines in gpu/vk/spirv_pack.h are identical on this head, and only tests/spirv_pack_test.cpp changed in the rebase, so they still hold. Re-confirm CI when it reports.
## Summary `Store::read` is meant to refuse a SPIR-V blob whose words changed on disk: it verifies a content hash after reading. On Linux that check is defeated, because the store reads through a per-thread `FILE*` opened once: glibc's `fseeko` has a fast path that moves the pointer *inside the stdio buffer* instead of calling `lseek`/`read`, so a blob overwritten behind the handle is served from the old buffered bytes — which are exactly the bytes whose hash is in the head. A minimal reproduction under `strace` shows the second read issuing **no `read()` syscall at all** (only seeks) and returning the old word. Windows passes because its `fseek` path re-reads. What changed (`gpu/vk/spirv_pack.h`): on POSIX the pack handles used by `read()` and by `head_is()` are unbuffered (`setvbuf(..., _IONBF, 0)`), so every read goes to the file; Windows keeps its existing buffered path. The hash check stays as it was. Rejected alternatives: re-hashing after the read (already present — stale bytes hash to the head), `posix_fadvise` (kernel page cache, not stdio's buffer), `O_DIRECT` (alignment, slower), reopening per read (hot path). ## Testing **Tested:** `tests/spirv_pack_test.cpp` extended. The test writes a blob, reads it (filling the handle's buffer), overwrites a word in the file through a separate `fopen(..., "r+b")`, then requires the read to be refused *and* returns the changed word; a second block overwrites a head after `head_is` first read it and requires `put` to write the words again. After the rebase onto `a2dc5385`, #2171's version of the `flip` block read the blob back on a thread of its own, so the calling thread's stdio handle held nothing when the disk changed, and the new `CHECK(got[10] == 0xDEADBEEF)` passed on `main` without exercising the bug. The block now reads back on the same thread that reads after the change — the handle whose buffer glibc's stale `fseeko` serves — which is the condition the bug needs and the way the test read it before #2171; the assertion is not weakened. Run against `a2dc5385`'s own `gpu/vk/spirv_pack.h` (only `tests/spirv_pack_test.cpp` taken from this head), it fails for the bug at lines 138, 142 and 144 (the `flip` case) and 169 (the head case): ``` FAIL line 138: !s.read(at, a.size(), hash(a), got.data()) FAIL line 142: got[10] == 0xDEADBEEF FAIL line 144: again_at > at && reads(s, again_at, a) FAIL line 169: s.put(q.data(), q.size()) > at ``` With the fix: ``` pass 2.1s tests/test_spirv_pack.py 1 passed, 0 failed, 0 skipped ``` In the game: a pack over `1885234b`, from this branch at `59ce70bb`, Kokiri Forest (child), 20 s after warm-up: 1833 frames, 91.7 fps, median 10.93 ms, 0 cold frames (base: 1821 frames, 91.1 fps, 11.08 ms). Game run at `59ce70bb`; the head `1a8a053b` differs only in `tests/spirv_pack_test.cpp` (the rework after #2171's change to it) and the rebase onto `a2dc5385` (upstream `main` commits, not part of this change), and `gpu/vk/spirv_pack.h` is byte-identical between the two, so the run covers this change's game code. **Still to test:** - Windows (llvm-mingw): the unchanged buffered path, a start (by hand). ## Measurements **Verdict:** no win in this run: 4 KB sequential is 3.0 → 3.4 µs/read (**+0.4 µs, ~13 %**), 64 KB sequential is 30.3 → 32.2 µs (+1.9 µs, ~6 %), 4 KB scattered is equal, and 64 KB scattered is 33.2 → 32.3 µs. The description records medians over several runs and no spread, so none of the differences is called noise; the 4 KB sequential cost is the worst case and is stated in Risks. Setup: Arch Linux, Ryzen 9 5950X, page-cached synthetic blobs, hashes precomputed, 2000 blobs, several runs, median of runs; a game-free microbenchmark of `Store::read`. | | Before: main @ 1885234b | After: this PR @ 1a8a053b | |---|---|---| | 4 KB, sequential (µs/read) | 3.0 | 3.4 | | 4 KB, scattered (µs/read) | 3.4 | 3.4 | | 64 KB, sequential (µs/read) | 30.3 | 32.2 | | 64 KB, scattered (µs/read) | 33.2 | 32.3 | **Baseline:** the numbers were measured on `bc44f697`; the changed lines in `gpu/vk/spirv_pack.h` are byte-identical on `1a8a053b`, and only `tests/spirv_pack_test.cpp` changed in the rebase. No commit in `344deb6c..a2dc5385` touches the header, so the numbers still hold. The largest relative cost is the tiny 4 KB sequential read (3.0 → 3.4 µs, ~13 %); at 64 KB — the size real SPIR-V stages are — sequential is 30.3 → 32.2 µs, while scattered is equal at 4 KB and slightly lower at 64 KB. The setup records medians only, so these are stated as measured rather than as noise. `read()` always reads the whole blob at a fresh offset, so stdio's buffer mostly missed anyway and only added a copy; unbuffered `fread` reads straight into `out`. ## Risks A store written concurrently by another process while a blob is read now returns the changed words (and the hash refuses them) instead of stale ones. No switch; a revert is enough. Nothing changes for saves, `longshot_options.ini` or the game folder's caches; the shader store is read-only from the game's side. The measured cost is the 4 KB sequential read: 3.0 → 3.4 µs/read (**+0.4 µs, ~13 %**), the largest relative regression in the table; 64 KB sequential is 30.3 → 32.2 µs (+1.9 µs, ~6 %), 4 KB scattered is unchanged, and 64 KB scattered is slightly lower. The description records medians over several runs and no spread, so this is not dismissed as noise. The change is a correctness fix, and this is its measured price. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). Rebased onto `a2dc5385`; `tests/run_all.py --strict --jobs 3` inside `systemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0` with the pinned headers. `main` at `9e10f6f7` was 911/22/18; at `a2dc5385` it is **911 passed / 23 failed / 18 skipped in 635 s** — the 23rd is `tests/test_sim_turf_pools.py`; this head `1a8a053b`: **911 / 23 / 18 in 619 s**. Failing names identical (23, compared name by name): `nx2/test_import_windows.py`, `tests/test_close_waits_for_save.py`, `tests/test_compile_zero_byte.py`, `tests/test_cpm_discovery.py`, `tests/test_damaged_rom_files.py`, `tests/test_dev_packs.py`, `tests/test_draw_stream_visibility.py`, `tests/test_fs_save_commit.py`, `tests/test_fsr_uniform_ownership.py`, `tests/test_guest_copies.py`, `tests/test_hold_toggle.py`, `tests/test_lower_atomg_cas.py`, `tests/test_menu_i18n.py`, `tests/test_menu_keys.py`, `tests/test_menu_slider_ends.py`, `tests/test_mise_build.py`, `tests/test_pads_after_menu.py`, `tests/test_recording_admission.py`, `tests/test_sim_turf_pools.py`, `tests/test_sim_turf_settings.py`, `tests/test_switch_table.py`, `tests/test_tls_objects.py`, `tests/test_vibration_menu.py`. The 23rd, `tests/test_sim_turf_pools.py`, is new on `a2dc5385` since `9e10f6f7` and is #2174's stale check (it asserts the old zero turf floor, while #2174 made the floor the game's size); it fails identically here. `tests/test_spirv_pack.py` passes (2.1 s), both reworked cases included. - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A check under `tests/` that reads changed source is updated, not weakened (`tests/spirv_pack_test.cpp` gains the two cases; no existing assertion removed). ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. Windows has not: the `@Beanstalk` request was posted on 2026-11-22 00:02, BeanBot answered that only a member of `BigBeans` can start it, and no approval has come yet. The by-hand `flip` rework has run (in Tested). Ticks when that result is in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). One Tier 1 machine has run: Arch Linux, RTX 4070 Ti, NVIDIA 620.70.06 (the in-game smoke above). Tier 1's Windows half has not, which is the same pending Windows item. - [ ] Approved in review, every comment answered. No review yet (`pr status`: Reviews: none); the Beanstalk approval is waiting for a BigBean. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. On head `1a8a053b` after the rebase: the description check passed, 11 checks still running, every build and test job `Blocked by required conditions`. The measurements name `main @ 1885234b`; the production lines in `gpu/vk/spirv_pack.h` are identical on this head, and only `tests/spirv_pack_test.cpp` changed in the rebase, so they still hold. Re-confirm CI when it reports.
fix(shaders): read a blob's words from the file, not stdio's stale buffer
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
b531da874a
On Linux, read() verified a blob's hash against words stdio had buffered
before the file changed: glibc's fseeko seeks within its buffer instead of
reading the file again when the target offset falls in it, so an edit made
behind the handle was never seen and the old words, which match the head's
hash, were given out. The store's pack handles are now unbuffered on POSIX
(the words go straight into out; the buffer only ever saved a copy), so
read() hashes what the file holds now. Windows' fseek already re-reads; its
path is unchanged.

The test asserts the bytes a refused read returned are the changed ones,
and that put's head check sees a head changed after it first read it.
Author
Contributor

Tested in game: Kokiri Forest (child), pack over 1885234b, 20 s: 91.7 fps median 10.93 ms, 0 cold frames; the shader store loads and plays. Arch Linux/Hyprland, RTX 4070 Ti, NVIDIA 620.70.06.

Tested in game: Kokiri Forest (child), pack over 1885234b, 20 s: 91.7 fps median 10.93 ms, 0 cold frames; the shader store loads and plays. Arch Linux/Hyprland, RTX 4070 Ti, NVIDIA 620.70.06.
Member

Possibly related, for context:

  • PR #1809 perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its content — Merged SPIR-V pack implementation in the same file; this fixes its stale-buffer read path.
Possibly related, for context: - PR #1809 perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its content — Merged SPIR-V pack implementation in the same file; this fixes its stale-buffer read path.
dazemc force-pushed fix/spirv-pack-stale-read from b531da874a
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to bc44f69726
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 27s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-21 23:33:47 +00:00
Compare
Author
Contributor

Rebased onto 344deb6c (head bc44f697); gpu/vk/spirv_pack.h is untouched by the commits since, so the benchmark and the run hold.

Rebased onto 344deb6c (head bc44f697); gpu/vk/spirv_pack.h is untouched by the commits since, so the benchmark and the run hold.
Author
Contributor

Measurements were taken on 1885234b. The head has since moved (bc44f697) and is based on 344deb6c, which is now 3 commits behind current main dd5ffe6a, so here is why the numbers still hold rather than a fresh run.

gpu/vk/spirv_pack.h is untouched by everything that landed in between — #2161, #2140 and #2163 — and nothing in the change depends on those commits; merge-tree against dd5ffe6a is clean. A local check agrees: git log 344deb6c..dd5ffe6a -- gpu/vk/spirv_pack.h is empty.

Measurements were taken on `1885234b`. The head has since moved (`bc44f697`) and is based on `344deb6c`, which is now 3 commits behind current main `dd5ffe6a`, so here is why the numbers still hold rather than a fresh run. `gpu/vk/spirv_pack.h` is untouched by everything that landed in between — #2161, #2140 and #2163 — and nothing in the change depends on those commits; `merge-tree` against `dd5ffe6a` is clean. A local check agrees: `git log 344deb6c..dd5ffe6a -- gpu/vk/spirv_pack.h` is empty.
Author
Contributor

@Beanstalk Windows: start the game on this branch on a pack and confirm the shader store loads and programs serve — no shader errors in the log, the scene renders. Windows keeps the buffered path, which is unchanged here, so this is a no-regression check.

Ready otherwise.

@Beanstalk Windows: start the game on this branch on a pack and confirm the shader store loads and programs serve — no shader errors in the log, the scene renders. Windows keeps the buffered path, which is unchanged here, so this is a no-regression check. Ready otherwise.
Member

@dazemc only members of BigBeans can start the farm.

One of them can approve this request by reacting 👍 to it or to this comment. This request was for bc44f69, and the PR has changed since (1a8a053), so a 👍 no longer approves it. Comment @Beanstalk again to ask for the new commit.

@dazemc only members of `BigBeans` can start the farm. ~~One of them can approve this request by reacting 👍 to it or to this comment.~~ This request was for `bc44f69`, and the PR has changed since (`1a8a053`), so a 👍 no longer approves it. Comment `@Beanstalk` again to ask for the new commit.
dazemc changed title from fix(shaders): read a blob's words from the file, not stdio's stale buffer to WIP: fix(shaders): read a blob's words from the file, not stdio's stale buffer 2026-11-22 01:41:58 +00:00
Author
Contributor

Back to WIP for two things:

  1. This head's merge base is 344deb6c, and main has since moved to a2dc5385 with #2171 (9e10f6f7) fixing the game-free checks (~199 test files). The suite numbers here were measured against the old main (813/116/18 there, now 911/22/18), so 'mise run test' is unticked until a rebase onto a2dc5385 and a re-run.
  2. After that rebase, the new flip case in tests/spirv_pack_test.cpp needs reworking: #2171 moved that block's read-back onto a thread of its own, so the calling thread's handle holds no old buffered words and the new CHECK(got[10] == 0xDEADBEEF) passes on main already. The new head case (a head overwritten under put) still fails there, so the test still catches the bug — but the flip half should fail on main too. Listed in Still to test.

The in-game smoke stands: its record (59ce70bb, 1833 frames, 91.7 fps, 0 cold frames) matches, and gpu/vk/spirv_pack.h's changed lines are identical to the head's. Measurements still name main @ 1885234b; the comment saying why they hold was written against dd5ffe6a, so re-confirm after the rebase.

The Ready to merge boxes stay unticked, each with its reason beside it: Windows is with the Test Farm, waiting for a BigBean; no review yet; CI's build/test jobs are still pending.

Back to WIP for two things: 1. This head's merge base is `344deb6c`, and `main` has since moved to `a2dc5385` with **#2171** (`9e10f6f7`) fixing the game-free checks (~199 test files). The suite numbers here were measured against the old `main` (813/116/18 there, now 911/22/18), so 'mise run test' is unticked until a rebase onto `a2dc5385` and a re-run. 2. After that rebase, the new `flip` case in `tests/spirv_pack_test.cpp` needs reworking: #2171 moved that block's read-back onto a thread of its own, so the calling thread's handle holds no old buffered words and the new `CHECK(got[10] == 0xDEADBEEF)` passes on `main` already. The new head case (a head overwritten under `put`) still fails there, so the test still catches the bug — but the `flip` half should fail on `main` too. Listed in Still to test. The in-game smoke stands: its record (`59ce70bb`, 1833 frames, 91.7 fps, 0 cold frames) matches, and `gpu/vk/spirv_pack.h`'s changed lines are identical to the head's. Measurements still name `main @ 1885234b`; the comment saying why they hold was written against `dd5ffe6a`, so re-confirm after the rebase. The Ready to merge boxes stay unticked, each with its reason beside it: Windows is with the Test Farm, waiting for a BigBean; no review yet; CI's build/test jobs are still pending.
dazemc force-pushed fix/spirv-pack-stale-read from bc44f69726
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 27s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 1a8a053bc0
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 5s
2026-11-22 01:48:33 +00:00
Compare
Author
Contributor

Rebased onto a2dc5385: head is now 1a8a053b. The rebase was clean, and the flip case was reworked on top (test-only).

Full game-free suite, run in the same shape as a clean a2dc5385 baseline (tests/run_all.py --strict --jobs 3, systemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0, the pinned headers):

  • clean a2dc5385: 911 passed / 23 failed / 18 skipped in 635 s
  • this head 1a8a053b: 911 / 23 / 18 in 619 s

Failing names identical, compared name by name (23). The one name new on a2dc5385 since 9e10f6f7 is tests/test_sim_turf_pools.py: #2174 changed the simulated-turf floor while that old check still asserts a zero floor, so it is stale on main and red identically here. test_spirv_pack passes (2.1 s), both new cases included.

The rework is test-only and honest. #2171 moved the flip block's read-back onto a thread of its own, so the calling thread's stdio handle held nothing and CHECK(got[10] == 0xDEADBEEF) passed on main without exercising the bug. The block now reads back on the same thread that reads after the disk changes — the handle whose buffer glibc's stale fseeko serves — which is the condition the bug needs, as the test read it before #2171. The assertion is not weakened. Run against a2dc5385's own gpu/vk/spirv_pack.h (only this head's tests/spirv_pack_test.cpp), it fails for the bug:

FAIL line 138: !s.read(at, a.size(), hash(a), got.data())
FAIL line 142: got[10] == 0xDEADBEEF
FAIL line 144: again_at > at && reads(s, again_at, a)
FAIL line 169: s.put(q.data(), q.size()) > at

The production hunk in gpu/vk/spirv_pack.h is byte-identical to the pre-rebase head; the rework changed no production code. Jev on the test change: 0.88, likely honest. Diff stat for a2dc5385..1a8a053b is gpu/vk/spirv_pack.h | 11 + and tests/spirv_pack_test.cpp | 33 +-; outside tests/ the only file is the original runtime fix, unchanged from bc44f697.

In-game evidence (Jev: 0.68, likely valid): the smoked 59ce70bb record matches, the changed lines in gpu/vk/spirv_pack.h are identical to this head, and no commit in 344deb6c..a2dc5385 touches it. No new in-game run was made.

Rebased onto `a2dc5385`: head is now `1a8a053b`. The rebase was clean, and the `flip` case was reworked on top (test-only). Full game-free suite, run in the same shape as a clean `a2dc5385` baseline (`tests/run_all.py --strict --jobs 3`, `systemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0`, the pinned headers): - clean `a2dc5385`: **911 passed / 23 failed / 18 skipped in 635 s** - this head `1a8a053b`: **911 / 23 / 18 in 619 s** Failing names identical, compared name by name (23). The one name new on `a2dc5385` since `9e10f6f7` is `tests/test_sim_turf_pools.py`: #2174 changed the simulated-turf floor while that old check still asserts a zero floor, so it is stale on `main` and red identically here. `test_spirv_pack` passes (2.1 s), both new cases included. **The rework is test-only and honest.** #2171 moved the `flip` block's read-back onto a thread of its own, so the calling thread's stdio handle held nothing and `CHECK(got[10] == 0xDEADBEEF)` passed on `main` without exercising the bug. The block now reads back on the same thread that reads after the disk changes — the handle whose buffer glibc's stale `fseeko` serves — which is the condition the bug needs, as the test read it before #2171. The assertion is not weakened. Run against `a2dc5385`'s own `gpu/vk/spirv_pack.h` (only this head's `tests/spirv_pack_test.cpp`), it fails for the bug: ``` FAIL line 138: !s.read(at, a.size(), hash(a), got.data()) FAIL line 142: got[10] == 0xDEADBEEF FAIL line 144: again_at > at && reads(s, again_at, a) FAIL line 169: s.put(q.data(), q.size()) > at ``` The production hunk in `gpu/vk/spirv_pack.h` is byte-identical to the pre-rebase head; the rework changed no production code. Jev on the test change: **0.88, likely honest**. Diff stat for `a2dc5385..1a8a053b` is `gpu/vk/spirv_pack.h | 11 +` and `tests/spirv_pack_test.cpp | 33 +-`; outside `tests/` the only file is the original runtime fix, unchanged from `bc44f697`. In-game evidence (Jev: **0.68, likely valid**): the smoked `59ce70bb` record matches, the changed lines in `gpu/vk/spirv_pack.h` are identical to this head, and no commit in `344deb6c..a2dc5385` touches it. No new in-game run was made.
Author
Contributor

Measurements column corrected to name the rebased head, with (where the numbers were taken) in a Baseline line: the production hunk in gpu/vk/spirv_pack.h is byte-identical, only the test changed in the rebase.

Measurements column corrected to name the rebased head, with (where the numbers were taken) in a Baseline line: the production hunk in gpu/vk/spirv_pack.h is byte-identical, only the test changed in the rebase.
dazemc changed title from WIP: fix(shaders): read a blob's words from the file, not stdio's stale buffer to fix(shaders): read a blob's words from the file, not stdio's stale buffer 2026-11-22 02:59:06 +00:00
Author
Contributor

Correction to the comment above: the measurement commit is bc44f697; the rebased head is 1a8a053b.

Correction to the comment above: the measurement commit is `bc44f697`; the rebased head is `1a8a053b`.
Author
Contributor

@Beanstalk Windows: start the game on this branch on a pack and confirm the shader store loads and programs serve — no shader errors in the log, the scene renders. Windows keeps the buffered path, which is unchanged here, so this is a no-regression check.

Head: 1a8a053b (rebased onto a2dc5385).

@Beanstalk Windows: start the game on this branch on a pack and confirm the shader store loads and programs serve — no shader errors in the log, the scene renders. Windows keeps the buffered path, which is unchanged here, so this is a no-regression check. Head: `1a8a053b` (rebased onto `a2dc5385`).
Member

@dazemc only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@dazemc only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Author
Contributor

Testing now names the run commit and what separates it from the head: on the branch side only tests/spirv_pack_test.cpp differs between 59ce70bb and 1a8a053b (the rework after #2171's change), plus the rebase onto a2dc5385, and gpu/vk/spirv_pack.h is byte-identical, so the in-game run covers this change's game code. No code change; stays ready.

Testing now names the run commit and what separates it from the head: on the branch side only `tests/spirv_pack_test.cpp` differs between `59ce70bb` and `1a8a053b` (the rework after #2171's change), plus the rebase onto `a2dc5385`, and `gpu/vk/spirv_pack.h` is byte-identical, so the in-game run covers this change's game code. No code change; stays ready.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 5s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/spirv-pack-stale-read:dazemc-fix/spirv-pack-stale-read'
git switch 'dazemc-fix/spirv-pack-stale-read'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2153
No description provided.