perf(vk): copy captured pages with non-temporal stores #1619

Merged

itaskaev2 merged 1 commit from perf/capture-nt-copy into main

2026-11-10 17:00:09 +00:00

itaskaev2 commented

2026-11-10 11:21:10 +00:00

(Migrated from github.com)

What changed

gpu/vk/vk_memory.cpp, the capture's guarded copy:

On x86-64, copy_nt copies whole 64-byte lines of the destination with streaming stores (_mm256_stream_si256 with AVX2, _mm_stream_si128 without), prefetching the source a few lines ahead. The head up to the destination's next line and the tail are plain copies.
- No read for ownership of the buffer (memory last written several frames ago, in no cache by now), and the copy does not push the game's own data out of the L3. The GPU takes the buffer from memory either way.
- It ends with an sfence; copy_part now also fences after its page-by-page retry, whose copies can leave at a fault before reaching their own.
- A fault on the source leaves through guarded_copy's handler as before (copy_nt is a call inside the __try, not inlined).
RX_VK_CAPTURE_NT=0: memcpy, as before. Other CPUs: memcpy.
tests/test_capture_copy_nt.py compiles copy_nt out of the source with SSE2 and with AVX2 and checks it against memcpy for sizes 0-9000 at every alignment of source and destination, with guard bytes around the destination. It also checks that the guarded copy uses it and that copy_part fences after its retry.

Testing

Ryzen 7 5800X3D, RTX 4080 SUPER (driver 616.92), Windows 11, 4K output, DLSS, Auto resolution. Kokiri Forest, 16 camera swings from the spawn, 130 s runs (the summary is of the camera part). The same hookshot_vk.dll with the copy off and on, alternating, two runs each:

	median frame	fps	p99	capture a submit
`RX_VK_CAPTURE_NT=0`	19.9 / 20.1 ms	49.5 / 49	32.3 / 29.6 ms	0.73 / 0.72 ms
default (streaming)	19.2 / 19.6 ms	51 / 50	29.1 / 29.0 ms	0.57 / 0.59 ms

The capture column is the "vk: capture: a submit" log line. No flash lines in the runs' logs.

python -I -B tests/run_all.py on Windows: everything passes but nx2/test_nup_paths.py, which needs the right to make symbolic links on this PC (WinError 1314) and does not touch this code.

Notes

#1375 changes copy_pages and take_capture's texture shadow pass around this code; the two should merge without trouble. One interaction to keep in mind: the shadow pass reads the texture storage's pages back from the capture buffer, which after a streaming copy come from memory rather than the cache. Kokiri Forest copies no texture storage pages ("0 texture pages copied" in the take's log line), so the runs above do not measure that; a scene that writes texture storage every frame would be the one to check.
Only x86-64 builds change.

The game thread copies the pages the CPU wrote into a capture buffer at every submit (~29 MB a frame in Kokiri Forest), and a plain copy of that size costs more than it needs: it reads every line of the buffer before overwriting it, then leaves the copy in the caches, where nothing reads it again. ## What changed `gpu/vk/vk_memory.cpp`, the capture's guarded copy: - On x86-64, `copy_nt` copies whole 64-byte lines of the destination with streaming stores (`_mm256_stream_si256` with AVX2, `_mm_stream_si128` without), prefetching the source a few lines ahead. The head up to the destination's next line and the tail are plain copies. - No read for ownership of the buffer (memory last written several frames ago, in no cache by now), and the copy does not push the game's own data out of the L3. The GPU takes the buffer from memory either way. - It ends with an `sfence`; `copy_part` now also fences after its page-by-page retry, whose copies can leave at a fault before reaching their own. - A fault on the source leaves through `guarded_copy`'s handler as before (`copy_nt` is a call inside the `__try`, not inlined). - `RX_VK_CAPTURE_NT=0`: `memcpy`, as before. Other CPUs: `memcpy`. - `tests/test_capture_copy_nt.py` compiles `copy_nt` out of the source with SSE2 and with AVX2 and checks it against `memcpy` for sizes 0-9000 at every alignment of source and destination, with guard bytes around the destination. It also checks that the guarded copy uses it and that `copy_part` fences after its retry. ## Testing Ryzen 7 5800X3D, RTX 4080 SUPER (driver 616.92), Windows 11, 4K output, DLSS, Auto resolution. Kokiri Forest, 16 camera swings from the spawn, 130 s runs (the summary is of the camera part). The same `hookshot_vk.dll` with the copy off and on, alternating, two runs each: | | median frame | fps | p99 | capture a submit | |---|---|---|---|---| | `RX_VK_CAPTURE_NT=0` | 19.9 / 20.1 ms | 49.5 / 49 | 32.3 / 29.6 ms | 0.73 / 0.72 ms | | default (streaming) | 19.2 / 19.6 ms | 51 / 50 | 29.1 / 29.0 ms | 0.57 / 0.59 ms | The capture column is the "vk: capture: a submit" log line. No flash lines in the runs' logs. `python -I -B tests/run_all.py` on Windows: everything passes but `nx2/test_nup_paths.py`, which needs the right to make symbolic links on this PC (WinError 1314) and does not touch this code. ## Notes - #1375 changes `copy_pages` and `take_capture`'s texture shadow pass around this code; the two should merge without trouble. One interaction to keep in mind: the shadow pass reads the texture storage's pages back from the capture buffer, which after a streaming copy come from memory rather than the cache. Kokiri Forest copies no texture storage pages ("0 texture pages copied" in the take's log line), so the runs above do not measure that; a scene that writes texture storage every frame would be the one to check. - Only x86-64 builds change.

hckrman101 (Migrated from github.com) approved these changes

2026-11-10 16:39:51 +00:00

hckrman101 (Migrated from github.com) left a comment

Nits, none blocking:

vk_memory.cpp doesn't include <immintrin.h> itself; the intrinsics come in through windows.h / other headers today. An explicit #include <immintrin.h> under the x86-64 guard would keep it from breaking if those headers change.
getenv(...) is called twice in capture_nt(); a single local would read a bit cleaner. Also for getenv/atoi is not included explicitly.
Perf numbers are a small win (~0.15 ms/submit, ~3% median frame) from two runs each, so the gain is modest. As the PR notes, the texture-storage shadow pass reading back from memory after streaming stores (#1375) is the one to re-check once a scene exercises it.

Looks good. I read copy_nt through: head/line/tail split is right (stream stores only on whole destination-aligned lines, the unaligned source uses loadu), the sfence before the tail is fine, and it stays a real call inside the __try so a source fault still goes through the handler. The extra fence after the page-by-page retry covers the case where a copy leaves at a fault before reaching its own sfence. The test is registered in run_all.py and CI is green. Nits, none blocking: - vk_memory.cpp doesn't include <immintrin.h> itself; the intrinsics come in through windows.h / other headers today. An explicit `#include <immintrin.h>` under the x86-64 guard would keep it from breaking if those headers change. - `getenv(...)` is called twice in capture_nt(); a single local would read a bit cleaner. Also <cstdlib> for getenv/atoi is not included explicitly. - Perf numbers are a small win (~0.15 ms/submit, ~3% median frame) from two runs each, so the gain is modest. As the PR notes, the texture-storage shadow pass reading back from memory after streaming stores (#1375) is the one to re-check once a scene exercises it.