fix(vk): count and log only the guest-memory imports that reached the driver #2165

Merged
HumanBean merged 2 commits from HumanBean/Longshot:fix/import-log-reached-driver into main 2026-11-22 00:16:23 +00:00
Owner

Summary

When the driver refuses to import a range of guest memory, the log claims huge numbers of imports that never happened,
and the refusal itself drops out of the log. #859's AMD Windows log shows this: after about 1.15 GB was imported, the
driver answered VK_ERROR_INVALID_EXTERNAL_HANDLE, and from then on the log showed about 1.25 million "imports" at
0.0 ms each.

The cause is in pool_lookup in gpu/vk/vk_memory.cpp. It calls import_pool up to twice per lookup, counts every
call as an import (n++), and logs "imported in" on every 256th call, whether the import worked or not. After a
failure, ImportHolds holds the range for kGapMs (2 s, gpu/vk/import_hold.h). While it is held, import_pool
returns false at once without calling the driver, and import_cooling hides the "not imported" line. So every lookup of
a held range counted as an import that took 0.0 ms. To see it on main, run tests/test_import_log.py: its driver stub
refuses one range, and it then counts 1000 lookups of that range, logged as "(256 imports, 0 ms in all)" and so on.

This PR changes only the counters and the log lines. Nothing about when an import is tried changes. Each attempt is now
counted by how it ended:

  • made: the driver accepted it. Only these count in "imported in ... (N imports, M ms in all)". When nothing was
    refused or held, the line reads exactly as on main. Otherwise it adds ; R refused, H lookups held.
  • refused: import_pool asked the driver and failed. The line is "not imported as lo..hi in X ms (VkResult V;
    R refused)", capped at 16 lines as before. Every refusal is counted, including those past the cap. import_pool now
    also keeps the VkResult of a refused host-pointer properties query or buffer create; before, it kept only a refused
    allocate or bind.
  • held: the range failed less than kGapMs ago, so the driver was not asked. These are counted apart. A line is
    logged for the 1st and every 65536th: "... not tried again yet (H lookups waited on a held range)".

At close (mkwvk_close_asked), a new line gives all three counts: "vk: imports at close: N made, R refused by the
driver, H lookups waited on a held range".

Read first: pool_lookup's import loop and import_stats_log in gpu/vk/vk_memory.cpp, and tests/test_import_log.py.

No script under scripts/ or tests/ reads these lines (searched for "imported in", "imports,", "not imported", "ms in
all"). tests/import_reach_test.cpp compiles pool_lookup whole, so it now declares the three counters and includes
import_hold.h.

See #859, #1443

Testing

Tested: Linux, RTX 3090, NVIDIA 620.70.06, this PR @ d9a1e62f as a pack, through the job queue: a memory tour from kokiri-start through Zora's Domain, Hyrule Field, Kakariko, the Market and Kokiri Forest, 30 s each, in play, clean quit, no crash. The "imported in ... (N imports, M ms in all)" lines look as on main, with no "refused"/"held" suffix and no "not imported" line, and the close line reads vk: imports at close: 331 made, 0 refused by the driver, 0 lookups waited on a held range. The game-free test tests/test_import_log.py is new. It cuts pool_lookup's import passes,
import_on_hold, import_remember_failure and import_stats_log out of the source and runs them over a driver stub
that refuses. It fails on main for the reason: the refusal is not counted, and the held lookups are logged as
imports. It was written without being run on the author's PC (no test suites run there), so CI on the forge runs it,
along with tests/test_import_reach.py and tests/test_import_buffer_binding.py, which compile the changed functions.

Still to test:

  • Windows, AMD (#859's setup, where the driver refuses): play past about 1.15 GB of imports, expecting "not imported ...
    (VkResult -1000072003 ...)" lines and no run of 0.0 ms "imported in" lines; the close line gives the refused and held
    counts (@Beanstalk Windows AMD: a long memtour, then report the "not imported" lines and the "vk: imports at close" line).

Measurements

Verdict: A logging fix: the counts are now correct. Nothing about speed changes, because no driver call or
decision about when to import changed.

Before: main @ 8aa74983 After: this PR @ d9a1e62f
tests/test_import_log.py fails: refusal not counted, held lookups logged as imports (CI) passes (CI)
memtour on NVIDIA, Linux: import lines as before the same shape, no suffix; close: 331 made, 0 refused, 0 held

Risks

The log format changes only after a refusal or a hold: the "imported in" line gains a suffix, and the "not imported"
line gains the attempted range, its time and the refused count. No script parses either line. Behaviour is unchanged
and so is the per-lookup work: one atomic increment per attempt, under the lock pool_lookup already holds. A revert is
enough to undo it. Players notice nothing.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named). (not run on this PC; CI runs it)
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here. (written as test commit then fix commit; the result on main and here comes from CI, not run locally)
  • A check under tests/ that reads changed source is updated, not weakened. (tests/import_reach_test.cpp declares the new counters)
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary When the driver refuses to import a range of guest memory, the log claims huge numbers of imports that never happened, and the refusal itself drops out of the log. #859's AMD Windows log shows this: after about 1.15 GB was imported, the driver answered `VK_ERROR_INVALID_EXTERNAL_HANDLE`, and from then on the log showed about 1.25 million "imports" at 0.0 ms each. The cause is in `pool_lookup` in `gpu/vk/vk_memory.cpp`. It calls `import_pool` up to twice per lookup, counts every call as an import (`n++`), and logs "imported in" on every 256th call, whether the import worked or not. After a failure, `ImportHolds` holds the range for `kGapMs` (2 s, `gpu/vk/import_hold.h`). While it is held, `import_pool` returns false at once without calling the driver, and `import_cooling` hides the "not imported" line. So every lookup of a held range counted as an import that took 0.0 ms. To see it on `main`, run `tests/test_import_log.py`: its driver stub refuses one range, and it then counts 1000 lookups of that range, logged as "(256 imports, 0 ms in all)" and so on. This PR changes only the counters and the log lines. Nothing about when an import is tried changes. Each attempt is now counted by how it ended: - **made**: the driver accepted it. Only these count in "imported in ... (N imports, M ms in all)". When nothing was refused or held, the line reads exactly as on `main`. Otherwise it adds `; R refused, H lookups held`. - **refused**: `import_pool` asked the driver and failed. The line is "not imported as lo..hi in X ms (VkResult V; R refused)", capped at 16 lines as before. Every refusal is counted, including those past the cap. `import_pool` now also keeps the `VkResult` of a refused host-pointer properties query or buffer create; before, it kept only a refused allocate or bind. - **held**: the range failed less than `kGapMs` ago, so the driver was not asked. These are counted apart. A line is logged for the 1st and every 65536th: "... not tried again yet (H lookups waited on a held range)". At close (`mkwvk_close_asked`), a new line gives all three counts: "vk: imports at close: N made, R refused by the driver, H lookups waited on a held range". Read first: `pool_lookup`'s import loop and `import_stats_log` in `gpu/vk/vk_memory.cpp`, and `tests/test_import_log.py`. No script under `scripts/` or `tests/` reads these lines (searched for "imported in", "imports,", "not imported", "ms in all"). `tests/import_reach_test.cpp` compiles `pool_lookup` whole, so it now declares the three counters and includes `import_hold.h`. See #859, #1443 ## Testing **Tested:** Linux, RTX 3090, NVIDIA 620.70.06, this PR @ d9a1e62f as a pack, through the job queue: a memory tour from `kokiri-start` through Zora's Domain, Hyrule Field, Kakariko, the Market and Kokiri Forest, 30 s each, in play, clean quit, no crash. The "imported in ... (N imports, M ms in all)" lines look as on main, with no "refused"/"held" suffix and no "not imported" line, and the close line reads `vk: imports at close: 331 made, 0 refused by the driver, 0 lookups waited on a held range`. The game-free test `tests/test_import_log.py` is new. It cuts `pool_lookup`'s import passes, `import_on_hold`, `import_remember_failure` and `import_stats_log` out of the source and runs them over a driver stub that refuses. It fails on `main` for the reason: the refusal is not counted, and the held lookups are logged as imports. It was written without being run on the author's PC (no test suites run there), so CI on the forge runs it, along with `tests/test_import_reach.py` and `tests/test_import_buffer_binding.py`, which compile the changed functions. **Still to test:** - Windows, AMD (#859's setup, where the driver refuses): play past about 1.15 GB of imports, expecting "not imported ... (VkResult -1000072003 ...)" lines and no run of 0.0 ms "imported in" lines; the close line gives the refused and held counts (`@Beanstalk Windows AMD: a long memtour, then report the "not imported" lines and the "vk: imports at close" line`). ## Measurements **Verdict:** A logging fix: the counts are now correct. Nothing about speed changes, because no driver call or decision about when to import changed. | | Before: main @ 8aa74983 | After: this PR @ d9a1e62f | |---|---|---| | `tests/test_import_log.py` | fails: refusal not counted, held lookups logged as imports (CI) | passes (CI) | | memtour on NVIDIA, Linux: import lines | as before | the same shape, no suffix; close: 331 made, 0 refused, 0 held | ## Risks The log format changes only after a refusal or a hold: the "imported in" line gains a suffix, and the "not imported" line gains the attempted range, its time and the refused count. No script parses either line. Behaviour is unchanged and so is the per-lookup work: one atomic increment per attempt, under the lock `pool_lookup` already holds. A revert is enough to undo it. Players notice nothing. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). (not run on this PC; CI runs it) - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A fix: a game-free test that fails on `main` and passes here. (written as test commit then fix commit; the result on `main` and here comes from CI, not run locally) - [x] A check under `tests/` that reads changed source is updated, not weakened. (`tests/import_reach_test.cpp` declares the new counters) - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
A game-free check that cuts pool_lookup's import passes out of vk_memory.cpp and runs them over a driver
stub that refuses: on main a held range's lookups count and log as imports in 0.0 ms.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(vk): count and log only the imports that reached the driver; held retries and refusals apart
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 21s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 31s
CI / Workflow lint (actionlint) (pull_request) Successful in 33s
CI / Website build and checks (pull_request) Failing after 34s
Third-party parts / As their notes say (pull_request) Successful in 37s
PR description / Description follows the template (pull_request_target) Failing after 11s
beanstalk/approval approved by HumanBean (reviewed)
beanstalk/windows+amd+ram16gb 0/1 pass, 0 running, 0 queued
beanstalk/windows+nvidia 0/1 pass, 0 running, 0 queued
d9a1e62f19
pool_lookup counted every import_pool call as an import, but a range that failed within ImportHolds::kGapMs
returns at once with no driver call: #859's refused range read as 1.25M imports in 0.0 ms, its not-imported
line hidden. Made, refused (with VkResult, capped at 16 lines) and held attempts are counted apart, the
imports line counts made ones (the others beside it when nonzero), and the close prints all three.
import_pool now keeps the VkResult of a refused properties query or buffer create too.

See #859, #1443

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Owner

Tested in game: Linux, RTX 3090, NVIDIA 620.70.06, through the job queue: memory tour of 5 areas from kokiri-start, in play, clean quit; import lines as on main with no refused/held suffix; close line: 331 made, 0 refused, 0 held. The refusal path needs AMD Windows (Test Farm).

Tested in game: Linux, RTX 3090, NVIDIA 620.70.06, through the job queue: memory tour of 5 areas from kokiri-start, in play, clean quit; import lines as on main with no refused/held suffix; close line: 331 made, 0 refused, 0 held. The refusal path needs AMD Windows (Test Farm).
Author
Owner

@Beanstalk Windows AMD (where the driver refuses host-pointer imports, #859's setup), 16 GB RAM if available: a long memory tour from kokiri-start through several areas, then report the 'not imported' lines and the 'vk: imports at close' line; expect no run of 0.0 ms 'imported in' lines. Also Windows NVIDIA: a normal tour, expecting the close line with 0 refused, 0 held.

@Beanstalk Windows AMD (where the driver refuses host-pointer imports, #859's setup), 16 GB RAM if available: a long memory tour from kokiri-start through several areas, then report the 'not imported' lines and the 'vk: imports at close' line; expect no run of 0.0 ms 'imported in' lines. Also Windows NVIDIA: a normal tour, expecting the close line with 0 refused, 0 held.
Member

Beanstalk test farm · 🔄 testing (0 of 2 reports so far)

Commit d9a1e62 · approved by HumanBean (reviewed) · dashboard

⚠️ Note: The 16 GB AMD target waits if no matching Windows AMD PC is available; online RAM and GPU counts do not establish that combination.
⚠️ Note: The shared plan cannot enforce zero refused and held only on NVIDIA, or distinguish a consecutive run of 0.0 ms import lines from isolated legitimate lines; it reports those counts for interpretation.
⚠️ Note: A refusal may not occur during this tour, and the new close summary is absent on main; refusal-path correctness then remains unjudged.

Done (started 2026-11-21 23:48 UTC). Comment @Beanstalk to run it again.

Target Result Reports Frame time p99 (ms), main → PR VRAM peak (MB) Crashes
windows+amd+ram16gb (16 GB RAM) ⏳ queued 0 of 1 — — —
windows+nvidia ⏳ queued 0 of 1 — — —
What was tested

What will be tested · AI plan (gpt-6.1-terra)

On Windows AMD with 16 GB RAM and Windows NVIDIA, tour five areas from the Kokiri checkpoint and compare main with the pull request. The roughly 25-minute run records refusal and close-line counts, and fails for a crash or a missing close summary. Whether the AMD driver refuses imports is not guaranteed; the reported counts and logs show which path was reached. No tester online now matches windows+amd+ram16gb: that target waits for one.

  • Where: windows+amd+ram16gb (16 GB RAM, 1-2 reports), windows+nvidia (1-2 reports).
  • memory-tour: from the repository's save kokiri-start, then teleport to Zora's Domain, teleport to Hyrule Field, teleport to Kakariko Village, teleport to Market, teleport to Kokiri Forest; 2 runs on each side.
  • Measured: crashes, the game's log, numbers from the game's log.
  • The PR fails if: it crashes; or expected log lines missing is above 0.
  • Time: about 30 min of game runs per tester (4 game starts), plus the two builds.
<!-- beanstalk:sticky --> ### Beanstalk test farm · 🔄 testing (0 of 2 reports so far) Commit `d9a1e62` · approved by HumanBean (reviewed) · [dashboard](https://beanstalk.magicbean.dev/prs/2165) ⚠️ **Note:** The 16 GB AMD target waits if no matching Windows AMD PC is available; online RAM and GPU counts do not establish that combination. ⚠️ **Note:** The shared plan cannot enforce zero refused and held only on NVIDIA, or distinguish a consecutive run of 0.0 ms import lines from isolated legitimate lines; it reports those counts for interpretation. ⚠️ **Note:** A refusal may not occur during this tour, and the new close summary is absent on main; refusal-path correctness then remains unjudged. Done (started 2026-11-21 23:48 UTC). Comment `@Beanstalk` to run it again. | Target | Result | Reports | Frame time p99 (ms), main → PR | VRAM peak (MB) | Crashes | | --- | --- | --- | --- | --- | --- | | `windows+amd+ram16gb` (16 GB RAM) | ⏳ queued | 0 of 1 | — | — | — | | `windows+nvidia` | ⏳ queued | 0 of 1 | — | — | — | <details><summary>What was tested</summary> **What will be tested** · AI plan (`gpt-6.1-terra`) On Windows AMD with 16 GB RAM and Windows NVIDIA, tour five areas from the Kokiri checkpoint and compare main with the pull request. The roughly 25-minute run records refusal and close-line counts, and fails for a crash or a missing close summary. Whether the AMD driver refuses imports is not guaranteed; the reported counts and logs show which path was reached. No tester online now matches windows+amd+ram16gb: that target waits for one. - **Where:** windows+amd+ram16gb (16 GB RAM, 1-2 reports), windows+nvidia (1-2 reports). - **memory-tour:** from the repository's save `kokiri-start`, then teleport to Zora's Domain, teleport to Hyrule Field, teleport to Kakariko Village, teleport to Market, teleport to Kokiri Forest; 2 runs on each side. - **Measured:** crashes, the game's log, numbers from the game's log. - **The PR fails if:** it crashes; or expected log lines missing is above 0. - **Time:** about 30 min of game runs per tester (4 game starts), plus the two builds. </details>
HumanBean changed title from WIP: fix(vk): count and log only the guest-memory imports that reached the driver to fix(vk): count and log only the guest-memory imports that reached the driver 2026-11-22 00:15:49 +00:00
HumanBean deleted branch fix/import-log-reached-driver 2026-11-22 00:16:25 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2165
No description provided.