AMD (RX 6600) fixes for Ocarina of Time: GPU hang, fog flicker, exposure

YunaraDebugs commented

2026-11-08 01:00:39 +00:00

(Migrated from github.com)

Fixes for Ocarina of Time on AMD, tested on an RX 6600 (Windows, driver 0x800161) in Kokiri Forest. Each change has an env switch that turns it off.

GPU hang one to four minutes into Kokiri (device lost)
On the RX 6600, the task and mesh programs of the foliage culling hung the GPU when the driver's optimizer built them. AMD buffer markers showed a different one stuck first each time (918adef1afaa39d4, 6f9142def3b6a978...). Runs with them built without the optimizer never hung, ours and upstream's with RX_VK_NOOPT=1. On AMD, task and mesh pipelines are now built with DISABLE_OPTIMIZATION; everything else keeps the optimizer. RX_VK_MESH_OPT=1: as before.

Fog flickering in Kokiri's light: the exposure meter
#296 made readbacks wait for their write-back, but the meter still read 2x or 0 about every two seconds. The CPU's reset to 0 landed in guest memory after the next submission had synced its mirror, so the GPU added onto the old sum. Small readback ranges (≤ 16 KB) whose write-backs stay on the queue are now copied from guest memory into the mirror at the start of each submission, last of the uploads. Logging the meter at the same spot for 20 s gave 9 bad reads before and 0 after. RX_VK_NO_RB_REFRESH=1: off.

One-frame ground-fog dips
In sync mode, a texture or program release made while queues are stopped at a wait forced the stopped work on at once (drain 1, about 1,000 times a session). That work then ran before what it waits for, and the ground fog came out thinner for a frame, about every 13th frame. Releases made while queues are stopped are now kept until the stopped work goes on, at its wait, a present or hold_ms. The title and menu text stayed correct with this. RX_VK_RELEASE_DRAIN=1: as before.

Driver optimizer per pipeline
AMD's compiler fails some pipelines with the optimizer (-13 on 5fa56485a6b07619) and takes minutes over others. Each pipeline now builds with the optimizer on a below-normal thread, with a budget (RX_VK_OPT_BUDGET_MS, 1500). If it fails or runs out of time, a build without the optimizer is used, and a late optimized one replaces it. hookshot_noopt.txt beside the exe remembers the slow and failed ones, so later starts skip the optimizer for those. The prepare_run compute warm-up uses the same path, so a slow optimizer build no longer holds the start.

Write-backs by DMA
Mirror → guest write-backs go through a staging ring on a transfer-only queue, which is AMD's SDMA engine. On the RX 6600 this cut GPU time from 18.2 to 14.0 ms per frame in Kokiri. Every reader of guest memory waits for copies over its range: mirror syncs, mirror make/free, guest bindings, texel views, texture flushes and the page table's first fill. RX_VK_WB_DMA=0: off.

Diagnostics

RX_VK_CHECKPOINTS=1: AMD buffer markers around draws and dispatches. A device lost lists what was done, under way and not reached, with each program's stages.
RX_VK_LOG_READBACK_FILE=<path>: logs every readback while that file exists.

Not tested: NVIDIA, Linux, and AMD cards other than the RX 6600. The DMA, readback-refresh and release changes apply to all GPUs. The optimizer race and the per-stage warp profile (RX_VK_WARP_WIDE=1, off by default) are also in this change.

🤖 Generated with Claude Code

Fixes for Ocarina of Time on AMD, tested on an RX 6600 (Windows, driver 0x800161) in Kokiri Forest. Each change has an env switch that turns it off. **GPU hang one to four minutes into Kokiri (device lost)** On the RX 6600, the task and mesh programs of the foliage culling hung the GPU when the driver's optimizer built them. AMD buffer markers showed a different one stuck first each time (918adef1afaa39d4, 6f9142def3b6a978...). Runs with them built without the optimizer never hung, ours and upstream's with `RX_VK_NOOPT=1`. On AMD, task and mesh pipelines are now built with `DISABLE_OPTIMIZATION`; everything else keeps the optimizer. `RX_VK_MESH_OPT=1`: as before. **Fog flickering in Kokiri's light: the exposure meter** #296 made readbacks wait for their write-back, but the meter still read 2x or 0 about every two seconds. The CPU's reset to 0 landed in guest memory after the next submission had synced its mirror, so the GPU added onto the old sum. Small readback ranges (≤ 16 KB) whose write-backs stay on the queue are now copied from guest memory into the mirror at the start of each submission, last of the uploads. Logging the meter at the same spot for 20 s gave 9 bad reads before and 0 after. `RX_VK_NO_RB_REFRESH=1`: off. **One-frame ground-fog dips** In sync mode, a texture or program release made while queues are stopped at a wait forced the stopped work on at once (drain 1, about 1,000 times a session). That work then ran before what it waits for, and the ground fog came out thinner for a frame, about every 13th frame. Releases made while queues are stopped are now kept until the stopped work goes on, at its wait, a present or `hold_ms`. The title and menu text stayed correct with this. `RX_VK_RELEASE_DRAIN=1`: as before. **Driver optimizer per pipeline** AMD's compiler fails some pipelines with the optimizer (-13 on 5fa56485a6b07619) and takes minutes over others. Each pipeline now builds with the optimizer on a below-normal thread, with a budget (`RX_VK_OPT_BUDGET_MS`, 1500). If it fails or runs out of time, a build without the optimizer is used, and a late optimized one replaces it. `hookshot_noopt.txt` beside the exe remembers the slow and failed ones, so later starts skip the optimizer for those. The prepare_run compute warm-up uses the same path, so a slow optimizer build no longer holds the start. **Write-backs by DMA** Mirror → guest write-backs go through a staging ring on a transfer-only queue, which is AMD's SDMA engine. On the RX 6600 this cut GPU time from 18.2 to 14.0 ms per frame in Kokiri. Every reader of guest memory waits for copies over its range: mirror syncs, mirror make/free, guest bindings, texel views, texture flushes and the page table's first fill. `RX_VK_WB_DMA=0`: off. **Diagnostics** - `RX_VK_CHECKPOINTS=1`: AMD buffer markers around draws and dispatches. A device lost lists what was done, under way and not reached, with each program's stages. - `RX_VK_LOG_READBACK_FILE=<path>`: logs every readback while that file exists. **Not tested:** NVIDIA, Linux, and AMD cards other than the RX 6600. The DMA, readback-refresh and release changes apply to all GPUs. The optimizer race and the per-stage warp profile (`RX_VK_WARP_WIDE=1`, off by default) are also in this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Tempus3214 commented

2026-11-08 01:18:34 +00:00

(Migrated from github.com)

hopefully this gets merged
borderline unplayable for people that have rx 6600

hopefully this gets merged borderline unplayable for people that have rx 6600

pr0vId3nce commented

2026-11-08 05:41:54 +00:00

(Migrated from github.com)

Tested on an RTX 4080 / Windows 11 / Ryzen 9 7900X: main 4cfa08f with this PR merged, warm caches, Kokiri Forest standing still, 60 s RX_VK_FRAMELOG per run, windowed 1280x720, frame generation off. One run per build, plus two runs of plain main.

Build	fps	White frames	Small flashes/min	Device lost
`main` (2 runs)	37.2 / 40.5	0	3.0 / 4.0	0
`main` + #430	36.9	0	2.0	0

No regression on NVIDIA: same fps range, no white frames, no device lost. Plain main barely flickers on this card, so this only shows the PR does no harm here; the AMD-specific parts can't be judged on this GPU.

Tested on an RTX 4080 / Windows 11 / Ryzen 9 7900X: `main` 4cfa08f with this PR merged, warm caches, Kokiri Forest standing still, 60 s `RX_VK_FRAMELOG` per run, windowed 1280x720, frame generation off. One run per build, plus two runs of plain `main`. | Build | fps | White frames | Small flashes/min | Device lost | |---|---|---|---|---| | `main` (2 runs) | 37.2 / 40.5 | 0 | 3.0 / 4.0 | 0 | | `main` + #430 | 36.9 | 0 | 2.0 | 0 | No regression on NVIDIA: same fps range, no white frames, no device lost. Plain `main` barely flickers on this card, so this only shows the PR does no harm here; the AMD-specific parts can't be judged on this GPU.

NotRexed commented

2026-11-08 08:21:30 +00:00

(Migrated from github.com)

Thank you, this was very useful. Three parts are ported into main by hand and shipped in https://github.com/NotRexed/Hookshot/releases/tag/ultrawide, each with your env switch and credit: the readback refresh for the exposure meter (it reduced one-frame lighting flicker here too, on an RTX 3060), task/mesh pipelines without the optimizer on AMD, and holding releases while queues are stopped. The DMA write-backs, the per-pipeline optimizer race and the warp profile are not in yet (they conflict with main now and are untested on NVIDIA); if you'd like them in, a rebased PR with just those would be great.

Rows
Columns

AMD (RX 6600) fixes for Ocarina of Time: GPU hang, fog flicker, exposure #430