fix(vk): widen RGB rows with each format's own alpha one #923

Open
jcabarca wants to merge 2 commits from jcabarca/Longshot:fix/vk-widen-alpha into main
Contributor

Type: fix · Risk: low · Tested on: Windows, AMD Radeon RX 9070 XT · Needs: none

Why

RGB textures that are widened to RGBA for upload got an alpha chosen by component size alone, so several formats read a wrong alpha: R8G8B8_SNORM -0.008, R16G16B16_UNORM 0.23, R16G16B16_SNORM 0.47, and the integer formats 255, 15360 or 0x3F800000 instead of 1. Anything the game blends or tests with that alpha comes out wrong.

How to reproduce

Seen on: not seen in the game: found by reading widen_rows() in gpu/vk/vk_texture.cpp; it happens for every texture in one of those RGB formats on a device that cannot sample the three-component format (most devices)
Settings and switches: defaults
Steps: game-free: python -I -B tests/test_widen_alpha.py --cxx clang++ --headers <Vulkan-Headers include> with --source pointing at a main checkout (the test from this PR); it compiles widen_rows() out of the source and checks the alpha bytes of the twelve widened formats
Expected: each widened texel gets one in its own format as alpha
Actual: on main there is no per-format alpha (widen_alpha(VkFormat) does not exist); the test fails

What changed

A new widen_alpha(VkFormat) returns the bit pattern of one for each format widened() widens, and widen_rows() writes that pattern instead of picking one by component size.

  • vk: gpu/vk/vk_texture.cpp:
    • widen_alpha(VkFormat) (new): 0x7F for R8G8B8_SNORM, 1 for the UINT and SINT formats, 0x3C00 for R16G16B16_SFLOAT, 0xFFFF for R16G16B16_UNORM, 0x7FFF for R16G16B16_SNORM, 0x3F800000 for R32G32B32_SFLOAT, 0xFF otherwise (R8G8B8_UNORM and SRGB).
    • widen_rows() takes the alpha as an argument and copies its low to - from bytes after each texel (little endian).
    • upload_inner() keeps the source format in src_vf and passes widen_alpha(src_vf).
  • tests: tests/test_widen_alpha.py (new): compiles widen_alpha() and widen_rows() out of the source and checks all twelve formats.

The bug, step by step:

  1. Most Vulkan devices cannot sample three-component formats, so upload_inner() asks widened() for an RGBA format and widens each row with widen_rows().
  2. widen_rows() chose the alpha by component size alone: 0xFF for 1-byte components, half-float 1.0 (0x3C00) for 2-byte ones, float 1.0 for 4-byte ones.
  3. That is one only for UNORM8, SFLOAT16 and SFLOAT32. For the others the same bits mean something else:
Format Alpha written Reads as One is
R8G8B8_SNORM 0xFF -0.008 0x7F
R16G16B16_UNORM 0x3C00 0.23 0xFFFF
R16G16B16_SNORM 0x3C00 0.47 0x7FFF
R8G8B8_UINT / SINT 0xFF 255 1
R16G16B16_UINT / SINT 0x3C00 15360 1
R32G32B32_UINT / SINT 0x3F800000 1065353216 1

This breaks no Vulkan rule: the bytes are valid, only the alpha they hold is wrong.

Before:

void widen_rows(uint8_t* dst, const uint8_t* src, size_t texels, int from, int to) {
    const int comp = from / 3;
    for (size_t i = 0; i < texels; i++) {
        std::memcpy(dst + i * to, src + i * from, from);
        uint8_t* a = dst + i * to + from;
        if (comp == 1) a[0] = 0xFF;
        else if (comp == 2) a[0] = 0x00, a[1] = 0x3C;   // (half 1.0; UNORM16 would want 0xFFFF: close enough for alpha)
        else { const float one = 1.0f; std::memcpy(a, &one, 4); }
    }
}

After:

uint32_t widen_alpha(VkFormat f) {
    switch (f) {
    case VK_FORMAT_R8G8B8_SNORM: return 0x7F;
    case VK_FORMAT_R8G8B8_UINT: case VK_FORMAT_R8G8B8_SINT: case VK_FORMAT_R16G16B16_UINT:
    case VK_FORMAT_R16G16B16_SINT: case VK_FORMAT_R32G32B32_UINT: case VK_FORMAT_R32G32B32_SINT: return 1;
    case VK_FORMAT_R16G16B16_SFLOAT: return 0x3C00;
    case VK_FORMAT_R16G16B16_UNORM: return 0xFFFF;
    case VK_FORMAT_R16G16B16_SNORM: return 0x7FFF;
    case VK_FORMAT_R32G32B32_SFLOAT: return 0x3F800000;
    default: return 0xFF;
    }
}
void widen_rows(uint8_t* dst, const uint8_t* src, size_t texels, int from, int to, uint32_t alpha) {
    const int n = to - from;
    for (size_t i = 0; i < texels; i++) {
        std::memcpy(dst + i * to, src + i * from, from);
        std::memcpy(dst + i * to + from, &alpha, n);   // (little endian: the low n bytes)
    }
}

Measurements

Verdict: it fixes the bug: tests/test_widen_alpha.py fails on main (no per-format alpha) and passes with this PR, in both interleaved runs. No speed change is expected: one memcpy of the alpha per texel instead of a branch and a store.

Hardware: AMD Radeon RX 9070 XT, Windows; the test itself is game-free (it compiles a cut-out of the source with clang++ from llvm-mingw and runs it on the CPU); Python 3.12
Scene: none, game-free
Method: tests/test_widen_alpha.py from this PR with --source set to each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gave
How to reproduce: python -I -B tests/test_widen_alpha.py --cxx clang++ --headers <Vulkan-Headers include> --source <checkout> for each side

Before: main @ b8d2b04e After: this PR @ f045f945
tests/test_widen_alpha.py, run 1 fails passes
tests/test_widen_alpha.py, run 2 fails passes
Median fails passes
run 1, main @ b8d2b04e:
widen_rows picks the alpha by component size alone: no widen_alpha(VkFormat)
exit 1
run 1, this PR @ f045f945:
RGB formats widened to RGBA get their own alpha one: ok
exit 0
run 2, main @ b8d2b04e:
widen_rows picks the alpha by component size alone: no widen_alpha(VkFormat)
exit 1
run 2, this PR @ f045f945:
RGB formats widened to RGBA get their own alpha one: ok
exit 0

What the test checks:

  • vk_texture.cpp has uint32_t widen_alpha(VkFormat f) and upload_inner() calls widen_alpha(src_vf).
  • A fixture built from the source's widen_alpha() and widen_rows() widens two texels of each of the twelve formats into a buffer pre-filled with 0xEE, and checks that the RGB bytes are copied unchanged and the alpha is exactly that format's one.

Flags and settings

None.

Testing

Tested: on this branch, Windows, Python 3.12, clang++ from llvm-mingw: test_widen_alpha passes (fails on main), and the other checks that read widened() pass: test_image_view_retry, test_pipeline_state_portable.

Built and played: this commit builds on Windows (llvm-mingw back end) and on Linux (longshot_vk.so, without and with AMD FSR 4). Played on Windows, AMD Radeon RX 9070 XT: the game runs normally.

Not tested: a texture in one of the affected formats looked at in the game: not checked in the game (which textures use them was not traced). Linux was built, not run.

Must be tested: None: the change is the same on every GPU and OS, and the game-free test covers it.

Risks

What could break: textures in the affected formats now read alpha one where they read the wrong value before; if a scene happened to look right with the wrong alpha it now changes, but one is what the format's missing component means. R8G8B8_UNORM/SRGB, R16G16B16_SFLOAT and R32G32B32_SFLOAT write the same bytes as before.

Turning it off: revert the commit.

Compatibility: Nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched). (no issue fits)
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (the checks that read the changed code, on this PR, Windows: test_widen_alpha, test_image_view_retry, test_pipeline_state_portable, all pass)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here. (tests/test_widen_alpha.py)
  • A check under tests/ that reads changed source updated in this PR, not weakened. (tests/test_widen_alpha.py added; no check removed)
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (built on Windows with llvm-mingw and on Linux, longshot_vk.so without and with AMD FSR 4; no OS-specific code or paths)
**Type:** fix · **Risk:** low · **Tested on:** Windows, AMD Radeon RX 9070 XT · **Needs:** none ## Why RGB textures that are widened to RGBA for upload got an alpha chosen by component size alone, so several formats read a wrong alpha: R8G8B8_SNORM -0.008, R16G16B16_UNORM 0.23, R16G16B16_SNORM 0.47, and the integer formats 255, 15360 or 0x3F800000 instead of 1. Anything the game blends or tests with that alpha comes out wrong. ## How to reproduce **Seen on:** not seen in the game: found by reading `widen_rows()` in `gpu/vk/vk_texture.cpp`; it happens for every texture in one of those RGB formats on a device that cannot sample the three-component format (most devices) **Settings and switches:** defaults **Steps:** game-free: `python -I -B tests/test_widen_alpha.py --cxx clang++ --headers <Vulkan-Headers include>` with `--source` pointing at a main checkout (the test from this PR); it compiles `widen_rows()` out of the source and checks the alpha bytes of the twelve widened formats **Expected:** each widened texel gets one in its own format as alpha **Actual:** on main there is no per-format alpha (`widen_alpha(VkFormat)` does not exist); the test fails ## What changed A new `widen_alpha(VkFormat)` returns the bit pattern of one for each format `widened()` widens, and `widen_rows()` writes that pattern instead of picking one by component size. - vk: `gpu/vk/vk_texture.cpp`: - `widen_alpha(VkFormat)` (new): 0x7F for R8G8B8_SNORM, 1 for the UINT and SINT formats, 0x3C00 for R16G16B16_SFLOAT, 0xFFFF for R16G16B16_UNORM, 0x7FFF for R16G16B16_SNORM, 0x3F800000 for R32G32B32_SFLOAT, 0xFF otherwise (R8G8B8_UNORM and SRGB). - `widen_rows()` takes the alpha as an argument and copies its low `to - from` bytes after each texel (little endian). - `upload_inner()` keeps the source format in `src_vf` and passes `widen_alpha(src_vf)`. - tests: `tests/test_widen_alpha.py` (new): compiles `widen_alpha()` and `widen_rows()` out of the source and checks all twelve formats. The bug, step by step: 1. Most Vulkan devices cannot sample three-component formats, so `upload_inner()` asks `widened()` for an RGBA format and widens each row with `widen_rows()`. 2. `widen_rows()` chose the alpha by component size alone: `0xFF` for 1-byte components, half-float 1.0 (`0x3C00`) for 2-byte ones, float 1.0 for 4-byte ones. 3. That is one only for UNORM8, SFLOAT16 and SFLOAT32. For the others the same bits mean something else: | Format | Alpha written | Reads as | One is | |---|---|---|---| | R8G8B8_SNORM | 0xFF | -0.008 | 0x7F | | R16G16B16_UNORM | 0x3C00 | 0.23 | 0xFFFF | | R16G16B16_SNORM | 0x3C00 | 0.47 | 0x7FFF | | R8G8B8_UINT / SINT | 0xFF | 255 | 1 | | R16G16B16_UINT / SINT | 0x3C00 | 15360 | 1 | | R32G32B32_UINT / SINT | 0x3F800000 | 1065353216 | 1 | This breaks no Vulkan rule: the bytes are valid, only the alpha they hold is wrong. Before: ```cpp void widen_rows(uint8_t* dst, const uint8_t* src, size_t texels, int from, int to) { const int comp = from / 3; for (size_t i = 0; i < texels; i++) { std::memcpy(dst + i * to, src + i * from, from); uint8_t* a = dst + i * to + from; if (comp == 1) a[0] = 0xFF; else if (comp == 2) a[0] = 0x00, a[1] = 0x3C; // (half 1.0; UNORM16 would want 0xFFFF: close enough for alpha) else { const float one = 1.0f; std::memcpy(a, &one, 4); } } } ``` After: ```cpp uint32_t widen_alpha(VkFormat f) { switch (f) { case VK_FORMAT_R8G8B8_SNORM: return 0x7F; case VK_FORMAT_R8G8B8_UINT: case VK_FORMAT_R8G8B8_SINT: case VK_FORMAT_R16G16B16_UINT: case VK_FORMAT_R16G16B16_SINT: case VK_FORMAT_R32G32B32_UINT: case VK_FORMAT_R32G32B32_SINT: return 1; case VK_FORMAT_R16G16B16_SFLOAT: return 0x3C00; case VK_FORMAT_R16G16B16_UNORM: return 0xFFFF; case VK_FORMAT_R16G16B16_SNORM: return 0x7FFF; case VK_FORMAT_R32G32B32_SFLOAT: return 0x3F800000; default: return 0xFF; } } void widen_rows(uint8_t* dst, const uint8_t* src, size_t texels, int from, int to, uint32_t alpha) { const int n = to - from; for (size_t i = 0; i < texels; i++) { std::memcpy(dst + i * to, src + i * from, from); std::memcpy(dst + i * to + from, &alpha, n); // (little endian: the low n bytes) } } ``` ## Measurements **Verdict:** it fixes the bug: `tests/test_widen_alpha.py` fails on main (no per-format alpha) and passes with this PR, in both interleaved runs. No speed change is expected: one `memcpy` of the alpha per texel instead of a branch and a store. Hardware: AMD Radeon RX 9070 XT, Windows; the test itself is game-free (it compiles a cut-out of the source with clang++ from llvm-mingw and runs it on the CPU); Python 3.12 Scene: none, game-free Method: `tests/test_widen_alpha.py` from this PR with `--source` set to each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gave How to reproduce: `python -I -B tests/test_widen_alpha.py --cxx clang++ --headers <Vulkan-Headers include> --source <checkout>` for each side | | Before: main @ b8d2b04e | After: this PR @ f045f945 | |---|---|---| | `tests/test_widen_alpha.py`, run 1 | fails | passes | | `tests/test_widen_alpha.py`, run 2 | fails | passes | | Median | fails | passes | ``` run 1, main @ b8d2b04e: widen_rows picks the alpha by component size alone: no widen_alpha(VkFormat) exit 1 run 1, this PR @ f045f945: RGB formats widened to RGBA get their own alpha one: ok exit 0 run 2, main @ b8d2b04e: widen_rows picks the alpha by component size alone: no widen_alpha(VkFormat) exit 1 run 2, this PR @ f045f945: RGB formats widened to RGBA get their own alpha one: ok exit 0 ``` What the test checks: - `vk_texture.cpp` has `uint32_t widen_alpha(VkFormat f)` and `upload_inner()` calls `widen_alpha(src_vf)`. - A fixture built from the source's `widen_alpha()` and `widen_rows()` widens two texels of each of the twelve formats into a buffer pre-filled with `0xEE`, and checks that the RGB bytes are copied unchanged and the alpha is exactly that format's one. ### Flags and settings None. ## Testing **Tested:** on this branch, Windows, Python 3.12, clang++ from llvm-mingw: `test_widen_alpha` passes (fails on main), and the other checks that read `widened()` pass: `test_image_view_retry`, `test_pipeline_state_portable`. **Built and played:** this commit builds on Windows (llvm-mingw back end) and on Linux (`longshot_vk.so`, without and with AMD FSR 4). Played on Windows, AMD Radeon RX 9070 XT: the game runs normally. **Not tested:** a texture in one of the affected formats looked at in the game: not checked in the game (which textures use them was not traced). Linux was built, not run. **Must be tested:** None: the change is the same on every GPU and OS, and the game-free test covers it. ## Risks **What could break:** textures in the affected formats now read alpha one where they read the wrong value before; if a scene happened to look right with the wrong alpha it now changes, but one is what the format's missing component means. R8G8B8_UNORM/SRGB, R16G16B16_SFLOAT and R32G32B32_SFLOAT write the same bytes as before. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). (no issue fits) - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (the checks that read the changed code, on this PR, Windows: `test_widen_alpha`, `test_image_view_retry`, `test_pipeline_state_portable`, all pass) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. (`tests/test_widen_alpha.py`) - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. (`tests/test_widen_alpha.py` added; no check removed) - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (built on Windows with llvm-mingw and on Linux, `longshot_vk.so` without and with AMD FSR 4; no OS-specific code or paths)
fix(vk): widen RGB rows with each format's own alpha one
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
f045f945db
widen_rows picked the added alpha by component size alone (0xFF, half 1.0, 1.0f), so R8G8B8_SNORM read -0.008, R16G16B16_UNORM 0.23, R16G16B16_SNORM 0.47 and the integer formats 255, 15360 or 0x3F800000. widen_alpha gives each widened format its own one; tests/test_widen_alpha.py checks all twelve.

Assisted-by: GitHub Copilot
jcabarca changed title from WIP: fix(vk): widen RGB rows with each format's own alpha one to fix(vk): widen RGB rows with each format's own alpha one 2026-11-15 20:49:36 +00:00
Member

Code-only review of f045f945db; informational findings, with final approval left to another reviewer.

No new correctness defect found in the diff. The alpha encodings cover all twelve formats currently handled by widened(), and the 1/2/4-byte copies match their component widths on the supported x86-64 targets. The source format is preserved before widening; GPU unswizzle and the all-zero clear shortcut already exclude widened formats, so those routes do not bypass this correction.

The added fixture checks real output bytes for two texels of every supported widened format, including preservation of RGB. Reverting a corrected alpha value would fail those cases. Its failure on main is a missing-helper check rather than an execution of the old faulty algorithm, so that baseline result alone is not a before/after rendering measurement. The reported Windows RX 9070 XT playthrough and Windows/Linux builds are useful smoke coverage; an affected texture was not traced in-game, as the description states.

Read the full diff, upload/creation paths, test and related texture-fix diffs. The ASTC, sparse-unswizzle and packed-format fixes in 920, 925 and 924 address distinct paths; no dependency on them was found. No tests, builds or game runs performed for this review.

Code-only review of f045f945db045bb0a6ce914da3c38beb23c27510; informational findings, with final approval left to another reviewer. No new correctness defect found in the diff. The alpha encodings cover all twelve formats currently handled by `widened()`, and the 1/2/4-byte copies match their component widths on the supported x86-64 targets. The source format is preserved before widening; GPU unswizzle and the all-zero clear shortcut already exclude widened formats, so those routes do not bypass this correction. The added fixture checks real output bytes for two texels of every supported widened format, including preservation of RGB. Reverting a corrected alpha value would fail those cases. Its failure on main is a missing-helper check rather than an execution of the old faulty algorithm, so that baseline result alone is not a before/after rendering measurement. The reported Windows RX 9070 XT playthrough and Windows/Linux builds are useful smoke coverage; an affected texture was not traced in-game, as the description states. Read the full diff, upload/creation paths, test and related texture-fix diffs. The ASTC, sparse-unswizzle and packed-format fixes in [920](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/920), [925](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/925) and [924](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/924) address distinct paths; no dependency on them was found. No tests, builds or game runs performed for this review.
Merge branch 'main' into fix/vk-widen-alpha
Some checks are pending
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 5s
c415a10738
Some checks are pending
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 5s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/vk-widen-alpha:jcabarca-fix/vk-widen-alpha'
git switch 'jcabarca-fix/vk-widen-alpha'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!923
No description provided.