[Tracking] Floating trees, missing mountains and vanishing near vegetation #994
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#994
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This issue collects every open report of floating trees and missing or vanishing vegetation, so there is one place to follow it. The individual reports were closed pointing here; their logs, screenshots and discussion stay in each linked issue.
Summary
All vendors are affected. The v2.0.6 release notes list floating trees and missing near vegetation as a known issue; it is not fixed yet. The bright red/green plant stems are a different bug, fixed in v2.0.6 (PR #759).
Affected GPUs and versions
Evidence from the logs
Workarounds
None known in the released builds.
Reports merged here (18 issues, about 42 people)
4cfa08f(RTX 4080, reporter's save)cabb432, dark Link regression); s…Adding a report
Please update to v2.0.6 first (https://github.com/NotRexed/Hookshot/releases/tag/v2.0.6; Update next to Play in the launcher). Please add: the place (a screenshot helps), GPU, version, and
hookshot_log.txt.All tracking issues
A character, not vegetation: King Zora draws flat white at a distance and fixes itself one body part at a time as Link approaches
6b1dd5d6), Linux AppImage, built with--clifrom a complete.nup(23.86 GiB).hookshot_env.txtonly hasHOS_PIN=Presentation=300,*=FCFF(CPU thread pinning).sampled as the dummy0 times, 0 draws skipped or left out, no unmapped reads, 0 programs failing to lower. VRAM 8.2 GB used of 21.5 GB allowed, textures 5.5 GB of the 6.1 GB budget, so not the over-budget case of #993. Fullhookshot_log.txt(3.5 MB) attached. No screenshot, sorry; I can go back and take one if it helps.v2.0.6, side by side with a real Switch 2: Hyrule Field's hills, plateau and the Kakariko windmill are missing, the trees on them stay
What's different:
Log:
hookshot_log.txt
Screenshots attached.
Switch 2 (Left) ------------ Hookshot (Right)
Far terrain missing behind Hyrule Castle (floating trees): what the storage buffer pass changes, v2.0.8-rc.3 / rc.5, RTX 3070 (Windows)
The scene can be reproduced exactly: the Hyrule Castle autosave, then the camera turned left toward the cliff. A wall of conifers stands in the sky, and the hills under them are missing. Every run below uses
RX_VK_DRAW_STATSon the same frame.The chain, as in my #422 trace:
b612216258832b9c(94 groups) writes the dispatch arguments of the terrain culling.@...960000:44511d69,6344c53d,cb2e06dd) is filled.@...99e000:4247f1b4,abccf81b,f0fed2bb) is 0 groups.68642bb8/cb71f79f,e28434e1/6b4dea60,9d8403e2/c36a4f3f) get no work. This includese28434e1/6b4dea60, whose arguments come from44511d69, which does run (63 groups).hookshot_env.txtb6122162only (local test switch)EDEN_NX2_GM_SPLIT=13cb76827/4f0c0b40get 247 groups and emit 0 meshes: hides the trees, does not bring the hills backEDEN_NX2_GM_LEGACY=0(all stages)68642bb8So on this card no setting of the storage buffer pass brings the far terrain back. The split only removes the trees standing on it.
b6122162's second texture (c0+0xa8) is slot0x102here: an RGBA8 4x4 that the game creates once at start-up (frame 1234), uploaded once and never written again. It looks like an engine default texture. In my earlier capture from Hyrule Field, that sampler had a 1024² BC1 forest mask instead.No game data attached. Logs on request.
Adding to @Nitnek's analysis from another scene, so nobody repeats these: Kakariko's entrance with the camera turned
toward Hyrule Field, RTX 3070 8 GB, Windows 11, v2.0.8-rc.1 (
8b2e1d3b) plus local changes, frames captured every 30presents through a scripted camera turn. Same conclusion: no setting of the storage buffer pass brings the hills back.
(The fps are from single runs, so they are only indicative.)
EDEN_NX2_GM_SPLIT=1, warm program cacheRX_VK_SSBO_STRETCH=1RX_VK_ROBUST=0cb71f79f,9d8403e2,68642bb8,6b4dea60,739acd67,d905f52b,c36a4f3f) read mesh storage buffer slots 0-3 (c0+0x770..0x7a0, the mesh table invk_bank.cpp'skSsboTable), and the default tracking binds exactly those.program, the port leaves all 19 accesses of
cb71f79fto the run-time search when the program is prepared at start-up(nothing mapped yet), and agrees with the default tracking (0 differ) when the same program is translated in play. That
is why its prepared programs are translated again in play, and why the first pass after a new build crawls.
changes nothing at that spot. That fits the far-terrain mesh programs getting no work at all, as in Nitnek's trace.
No game data attached.
hookshot_log.1.txt
hookshot_log.2.txt
hookshot_log.3.txt
hookshot_log.4.txt
hookshot_log.txt
hookshot_version.txt
v2.0.8
It keeps crashing; it opened once, but then closed the second time I tried. My resolution—1920x1080p—isn't showing up.
RTX 5070 9800x3d 32gb
@NotRexed
The
RX_VK_PARALLELpart of this has a cause and a fix: #1184With
RX_VK_PARALLELset, houses, big rocks and bushes go missing near Link: in Kokiri Forest, for example, the tree over the shop. That is a different bug from the far-terrain cases above. Serial replay (RX_VK_PARALLEL=0) draws them.Cause: a pool's VRAM mirror is made at its first binding. Its fill went into the uploads of whichever segment's worker bound the pool first, but the page table pointed at the mirror at once. An earlier segment, submitted first, read the mirror empty. That is also why such an object may come back after a few minutes of play.
RX 7900 XTX, RADV, Linux, v2.0.7 plus #1069 and #1108, scripted camera turns in front of the Kokiri shop:
RX_VK_PARALLEL=4was missing the shop tree; with the fix, 8 of 8 screenshots were complete.RX_VK_PARALLEL_SEG=40gave torn geometry; with the fix, 8 of 8 were complete.If you see missing objects and have
RX_VK_PARALLELset, tryRX_VK_PARALLEL=0(orRX_VK_MIRROR_DEFER=1) to tell which bug it is.The far terrain goes missing on the CPU side: the culling chain is given nothing to draw (RTX 3070, Windows, v2.0.8
plus local diagnostics; Kakariko's entrance with the camera held toward Hyrule Field)
Following @Nitnek's chain (
b6122162→ lists A/B → the far-terrain mesh pairs), the GPU part seems to do what it is told.What it is told is empty:
b6122162splits its warps over categories by cumulative counts it reads from its first uniform buffer (c3,+0x660..+0x6e0in steps of 16, the total at+0x6f0). As the dispatch reads them (theRX_VK_DRAW_STATSdump ofthat buffer, with the stream's inline updates):
0, 0, 43, 62, 131, 131, 168, 235, 274, total281. Categories 0, 1and 5 get no warps at all. List A's group counts match two of the others (43 warps → 42 groups, 19 → 19); list B gets
1, 1 and 0 groups. These counts are written by the game's CPU code, so the culling, the lists and the mesh pairs only
follow them.
HOS_FS_TRACE=1, in that scene the game opensTerrainArcforKakarikoVillage_01(its root tile and 19level-2 tiles) and
KakarikoVillage_02, but neverHyruleField_02: no.tscb, no tile. It does load Hyrule Field'stree textures (the spruces) and its environment parameters. The floating trees look like Hyrule Field's trees standing
on terrain that is never loaded.
RX_VK_SSBO_STRETCH=1,RX_VK_ROBUST=0), copying what the culling writes through its bare table entries back to guest memory (a local testswitch), the game's occlusion queries (
HOOKSHOT_OCCLUSION_OFF=1), and #1127 (it does apply there: the culling's poolsare remade from retired mirrors, but the counts stay the same).
So the question seems to be why the game never loads
HyruleField_02's terrain from Kakariko, or more generally whatfills those categories on the console. No game data attached.
Follow-up: in Hyrule Field the hills are drawn; one tree program pair draws far too many trees over them
Re-checked on v2.0.10 (built from source, plus #850, RTX 5080): unchanged, same tree wall at the same spot. Headless runs from one save, looking from the field towards Death Mountain.
Switching programs off one at a time (
RX_VK_SKIP_FILE): skipping only the task+mesh pair3cb7682790143836/4f0c0b40ab3a32cd(indirect args written byb612216258832b9c) removes the wall of trees, and what is left matches the Switch 2 picture: hills, the plateau, the Kakariko windmill. So in this view the terrain is not missing; that one pair draws a dense forest where the console shows a few small trees. (The other vegetation programs, including grass and bushes, don't affect it.)EDEN_NX2_GM_SPLIT=1andEDEN_NX2_GM_LEGACY=0also uncover the hills, but they remove every tree and all the grass, so they aren't a fix.No change with: DLSS off, Resolution Auto, Ultrawide off, Handheld profile;
RX_VK_FOLLOW_MESH=1;RX_VK_FOLLOW_STORES=1;RX_VK_MESH_ARGS=1+RX_VK_MESH_FIRST=1(the tree draw's own first is 0).What the pair reads looks right in game memory: the tree data the task stage reads through its c0+0x670 pointer (a count of 2968 cells, packed cell codes, prefix-sum starts, then triplets), the camera in c3+0x9d0 (near 0.1, far 25000, 16:9, 50° fov) and the frustum planes in c4+0x4c0.
One anomaly: with
RX_VK_MISS_LOG=1the task stage of the pair makes about 100k global reads per frame that no buffer holds, about 33 to 36 MB past the tree data. Turned back into indices they are packed cell codes (0x808090,0x8984B2) where list indices belong. So some task workgroups take stale or garbage entries from the visible listb6122162writes (whose entries are an index, or an index with bit 31 set).Lichii (Eden-based,
9e9ee6a215oot/nx2-tip) at the same spot draws it like the Switch 2: sparse small trees, hills, windmill. So the game's data is fine and the difference is in how Hookshot runsb6122162or the tree task stage. That fits @Nitnek's chain above (b6122162also fills List B).Ruling out the Ampere -> Maxwell lowering: I dumped the native code of
b6122162(= Lichii's52612da1509ec28d, byte for byte), the shared task staged91e4f46becaba90and the two mesh stages (9d026703ad5da78c,b86297c24c8d8c7a), then lowered each with Hookshot's and Lichii'slower_to_maxwell.hin a small standalone tool. The task stage lowers identically. In the others the differences are equivalent: aUIADD3.Xwith!PTas its second carry, a wrappedSHF.R.U32.HIasSHR.WvsSHF.R.W, branch offsets, and REGMAP markers vs shadow moves. So the difference is after lowering (the IR passes or what the shaders reach at run time), not in the instruction translation.The two textures
b6122162samples at this spot are 1024² masks, one R8 (five dark blobs) and one grey map shaped like Hyrule Field. Dumped from the GPU, they look correctly decoded, so this spot isn't the 4x4 default-texture case from the castle.Following @blushOFF's CPU-side lead at my spot, which is inside the field:
HOS_FS_TRACE=1showsHyruleField_02.tscband its tiles (levels 10 and 31-36) open without errors, so here the terrain files do load.b6122162's category starts atc3+0x660..0x6e0are0, 0, 134, 292, 485, 505, 782, 944, 1074(total 1167): category 0 is empty here too, the others are filled.Together with the lowering result and Lichii drawing this spot correctly, that points at the counts the game's CPU code hands the culling (recompiled code, or something the game asks the system, such as a memory budget), not at the GPU translation.
Happy to run more traces if anyone wants a specific switch or dump from this spot.
A third spot, same as @blushOFF's Kakariko one: from Hyrule Castle the field's terrain is never opened (v2.0.10 built from source, RTX 3070, Windows 11)
HOS_FS_TRACE=1shows: from loading the save to the cliff view, the only terrain the game opens isTerrainArc/HrlCastle_01.tscband 94 files ofTerrainArc/HrlCastle_01/.HyruleField_02is never opened, yet the spruces standing on it are drawn: that is the wall of floating trees.b612216258832b9cruns 94 groups;4247f1b4c3ce6cf3gets 0 groups;68642bb8gets no work;3cb7682790143836draws ~2.0M fragments.fs: OpenFile(content:/Tex/....bntx.zs): not foundlines (DummyBlack_Gn2,CmnTex_BakeLightDefault,Rock_Pot_A_Rgh...). None of those files exist in the romfs, with or without.zs. The game only probes for loose textures, so they are not the cause.So three spots now agree: Kakariko, Hyrule Castle, and @KrysGG's field view (where
HyruleField_02does load and category 0 is still empty). The terrain the far view needs is never requested by the game's own code. A next step could be the same trace in Lichii at one of these spots, to see whether it opensHyruleField_02there.@Nitnek I tried that Lichii trace at my field spot. With
Service.FS:Debug, Lichii logs ~15k reads but no game file names: it serves the RomFS as one storage and logs only offsets (fs_i_storage.cpp Read: offset=0x257f6a730, length=131072). Only the save files show by name. So comparingTerrainArcopens needs those offsets mapped back to RomFS paths. Lichii already parses the RomFS, so a small local log there (the path for each read's offset) would do it. I can try that unless someone has a quicker way.Correction of my 16:55Z comment, and what the floating trees are at Kakariko's entrance (RTX 3070, Windows 11, v2.0.8 plus local changes)
Please disregard the "CPU side" conclusion of my 16:55Z comment. I read
b6122162's category counts as terrain patches, and I now have no evidence for that: itsc3+0x220..0x22care bit-identical to the four floats in the header of the scene'sForestplacement file (.flnt), and it is the shader that writes the buffer holding the visible list of @KrysGG's tree pair. As far as I can now show, an empty category tells nothing about terrain, so please don't do a Lichii file trace because of that comment.At this spot, the floating trees are that tree pair drawing too many records:
3cb7682790143836and4f0c0b40ab3a32cdinRX_VK_SKIP_FILE, no floating tree is left in a scripted camera sweep.eof the visible list per workgroup and computes the number of trees of that cell asmin(offs[e+1] - offs[e], 256),offsbeing the prefix-sum table of the.flnt(read in its IR).RX_VK_GPU_DUMP), the GPU held 11 entries in that list, all below the file's 223 cells, and the file holds 82 trees for those 11 cells. But the mesh stage ran 1184 threads and made 2082 primitives, and each mesh thread that reaches the output writes one quad.offs[e]tooffs[e] + 255, placed in celle, land on the trees drawn in the sky. In the upper-right part of the frame, where the floating trees are, 98 to 100 % of the sample points along those 256 records are on non-sky pixels, and 62 to 70 % for the records after them, when 62 % of that area is not sky anyway. The four cells seen in that area hold 3, 5, 14 and 31 trees in the file.So in that frame those cells are drawn as if they held 256 trees, and the extra ones are the records of the following cells, placed in the wrong cell.
In a run of the same scene, the GPU's copy of the file matched it on the four 16-byte samples I watched (
RX_VK_WATCH_ADDRS: header, cell table, offset table), so nothing points to stale data so far.I don't know yet why the count comes out at the clamp: I'm tracing those loads. I have not checked other spots, and this says nothing about the hills @KrysGG and @Nitnek describe.
Following @blushOFF's correction: I withdraw the conclusion of my last comment, and add a data point that fits the tree-count finding.
Withdrawn. That
HyruleField_02is never opened from the castle is still what the trace shows. But I read it as "the far terrain is never requested by the game's code", leaning on the category reading that is now retracted, and that does not follow: the far view may come from elsewhere. Please ignore the Lichii file-trace suggestion too.What fits the 256 clamp. In the castle cliff view (my 15:10 comment), the same tree pair
3cb7682790143836/4f0c0b40ab3a32cdbehaves very differently with the storage buffer pass's port on the geometry stages:b6122162's list)EDEN_NX2_GM_SPLIT=1So under the legacy tracking the per-cell count comes out at the clamp (256), and under the port it seems to come out as 0, from the same visible list. That would fit the task stage's
offs[e]/offs[e+1]loads being bound to the wrong buffer, a different wrong one in each mode, rather than the data being wrong. It also fits @KrysGG's miss log: ~100k task-stage global reads a frame that no buffer holds, 33-36 MB past the tree data.A next check, if someone gets to it before me: the pointer the storage buffer pass picks for those two loads in the shared task stage (
d91e4f46becaba90per @KrysGG), e.g. withRX_VK_PTR_DIAG=1orRX_VK_MISS_LOG_PROG, under both settings.Why the count is 256 at Kakariko: the task stage's
offs[e+1]load goes to a wrong address, and neither the IR nor the SPIR-V says so (RTX 3070, Windows 11, NVIDIA driver 610.88 as Vulkan reports it, v2.0.8 plus local changes)Follow-up to my 19:14Z comment, same spot and scripted camera.
What the task stage of
3cb7682790143836reads. A local probe logged the address and the value of each 32-bit global load, for lane 0 of each workgroup (10 workgroups, each seen twice in the frame):n = load(file + 4)returns 223, the file's cell count. Thecells[e]andoffs[e]loads return the file's values, and so do the 520 later loads in the log (nagain,offs[e], the records).offs[e+1]load is the only wrong one, in all 20 cases. Its address should befile + (n + 9 + e) * 4. The address the shader used isfile + (cells[e] + 9 + e) * 4: the cell's packed code wherenbelongs. Fore = 122(code0x808178) that is 33.7 MB past the start of the file. No buffer holds it, and the load returns 0.min(0 - offs[e], 256)in unsigned arithmetic: 256 for every cell. These are the misses of the miss log (3200 a frame here). It would also be why @KrysGG's missed addresses turn back into packed cell codes: the code is in the address.Where it goes wrong.
spirv-valhere): the third call's address is built from the result id of the first call of the load helper, not the second's.DontInline. With it markedInlinefor the task and mesh stages only (a local change), in the same scene: no miss is left; the pair's mesh stage runs 160 threads and makes 78 primitives, against 1184 and 2082; the file holds 81 trees for the 10 cells of the traced frame; and no tree is left in the sky in the scripted camera runs, with a cold program cache too.So at this spot the compiled task stage does not compute what its SPIR-V says while the helper is not inlined, and does once it is. I can't tell from here whether that is the driver's compiler or something in the module that I missed.
Not checked yet: other spots (a tour of the areas is running), other GPUs and drivers, frame rate and compile times over a session. This does not explain @Nitnek's
EDEN_NX2_GM_SPLIT=1case (no trees at all), and it says nothing about the hills. If the tour holds, I'll open a PR with the change.@blushOFF great find, and it corrects my 20:06Z guess. I read the 256 as the
offsloads being bound to the wrong buffer; your trace shows the buffer is right and the address is wrong (the cell code wherenbelongs), in a compiled helper whose SPIR-V is correct. So please disregard the "bound to the wrong buffer" part of that comment.When your branch is up I can run it at the Hyrule Castle cliff view from my 15:10Z comment (RTX 3070, Windows 11). The tree wall there is large:
3cb7682790143836drew ~2.0M fragments, with 45504 mesh threads. That gives a second spot, on another driver version, with before/after draw stats.The PR is up: #1334. It is the change from my last comment (the global-load helper inlined in task and mesh stages,
EDEN_NX2_GM_INLINE=0turns it off). @Nitnek, thanks for offering to run it at the castle cliff.The "vanishing vegetation at mid distance" part has its own cause and a fix: #1338 (separate from the floating trees @blushOFF traced to the load helper).
Trees full from far, bare at mid distance and full again near (#1151) come from the foliage LODs' crossfade. It is dithered against a 64³ noise volume that the game renders while an area loads (
53509044401d78d2). Hookshot left that pass out by default after it lost the device on first starts, so the noise stayed zero and the crossfade became a hard cut. The PR fixes the two reasons the pass failed: BC images had no colour attachment usage for their raw views, and the pass was sized in texels instead of blocks. It also turns the pass back on. RTX 3070: needles kept at mid distance, 12 of 12 starts with the pass running and no device lost.Floating trees / missing background hills in Kakariko Village (v2.0.11, RTX 4070 Ti)
d6369f1e), built via HookshotBuilder on Windows.@Oliverito that is expected on v2.0.11: the fix for the floating trees (#1334) was merged after that release, so it will be in the next one. Until then, only a build from current
mainhas it.Data point on a build without #1338: the foliage/mountain LOD problem on an RX 7800 XT / Windows 11 (main
0828863+ #1130).On the path from Zora's River toward Zora's Domain (child Link), the top of a mountain shows dark angular polygon patches, like parts of the rock drawn in a much darker tone with hard edges. The log around it is clean (60 fps, no error, no
not made, no slow submit). This build has #1334 (merged) but not #1338 (still open). I have not run #1338 yet; I will build main with it on top and compare the same spot, and post the result here. Until then this is only a possible match, not a confirmation.Fixed in the v2.0.12 branch, closing when it ships. The floating trees fix (#1334: the global-load helper inlined in task and mesh stages) is merged; testers confirm the Castle and Kakariko trees are gone with it.
Also in the v2.0.12 branch: the mid-distance foliage fix (#1338, OoT's noise volume pass no longer left out), on top of the floating trees fix. Closing at release.
The recomp curse, it's even harder to track down bugs like this because it's a recomp. The game could be doing everything correctly but somewhere down the line it ends up getting scrambled. On a more positive note you guys are doing an amazing job finding and fixing them. The work on this is nothing short of incredible
Linux data point for #1334, since Linux was listed as not tested. CachyOS, kernel 7.3.3, RTX 5070 Ti (driver 620.70.06), built from main
5609af4.Same build, same spot (looking up at the sky beside the Hyrule Castle wall):
EDEN_NX2_GM_INLINE=0: a wall of floating trees across the sky.I only checked this one spot, and I have not done the before/after on AMD.
Still happening on v2.0.11.
System: Ryzen 7 5700X3D · RTX 5070 Ti (NVIDIA driver 32.0.16.2114 / 621.14) · 32 GB RAM · Windows 11 Pro build 26300 · Vulkan. 100-180 FPS, performance fine.
Version: v2.0.11 (some screenshots taken on v2.0.10; resumed the same save in the same area on v2.0.11 hours later and the bugs are unchanged).
Progress: ~2+ hours in, young Link, Spanish language. Now on Death Mountain Trail.
Huge numbers of pine trees float in the sky, detached from terrain. Seen on the Hyrule Castle walls/grounds (sneaking in to meet Zelda), the castle road gate, Kakariko Village at night, the Kakariko guard gate to Death Mountain and in the rain at night. Consistent every time, not a one-off.
Fixed in v2.0.12 (https://github.com/NotRexed/Hookshot/releases/tag/v2.0.12): the floating trees and missing mountains (#1334: the global-load helper inlined in the task and mesh stages; #1338: OoT's noise volume pass no longer left out). Update with the Hookshot Builder (or the game's update offer) and rebuild. If it still happens on v2.0.12, reopen this with your
hookshot_log.txt.Also seeing vegetation vanish when approaching it (bushes disappear as the player gets close).
GPU: NVIDIA GeForce GTX 1080 (no mesh shaders, "Mesh shaders: Compute" row)
CPU / RAM: Intel Core i7-12700, 32 GB
OS: Windows 11
Hookshot: v2.1.0 (source 633eb435)
Where: Hyrule Castle
Settings: Fullscreen On, GPU device GTX 1080, Frame generation 2x, Frame pacing Off, Reflex Off, Frame rate cap 60,
1920x1080 output, FSR 3.1 Quality 67% (1280x720), Sharpness 50%, Anisotropic 4x, Profile Docked, Ultrawide 16:9,
Robust buffers Off, Mesh shaders Compute, Descriptor reuse On, Sampler cache On.
hookshot_log.txt