feat(vk): draw the menu, FPS counter and start-up screen with Dear ImGui #1459
No reviewers
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!1459
Loading…
Reference in a new issue
No description provided.
Delete branch "imgui-menu"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The Insert menu, the FPS counter and the start-up screen were text drawn by GDI (and a small GDI on Linux, on stb_truetype) into bitmaps, then copied onto the window's image. This PR draws them with Dear ImGui instead, in the style of the game's own options screen.
What changes for players
How it works
gpu/vk/vk_imgui.cpp: the ImGui context and its Vulkan backend (dynamic rendering, our own function loader). Font and logo textures are uploaded with copies on the present's own command buffer. The backend's own upload path (which callsvkQueueWaitIdle) is not used. Fonts are loaded once. The menu's text sizes are baked ahead while the start-up screen shows, so the first open has no hitch.gpu/vk/vk_ui.h: the look (palette, cells, banner, help box, prompts).gpu/vk/vk_menu.cpp: every page is drawn anew from the menu's state at each present while something shows.menu_recordrecords nothing at all while nothing shows, so a closed menu (and no FPS counter) costs nothing.gpu/vk/vk_menu_hit.h: the drawing records each control's rectangle, and the mouse is tested against that. The picture and the hit test cannot drift apart at any size.COLOR_ATTACHMENTuse and image views (for the render pass). On an sRGB-only surface, the swapchain is made mutable (VK_KHR_swapchain_mutable_format) and the overlays draw through UNORM views. Without that extension, the colours are made linear. A new swapchain format makes the pipeline again.MkwVkHost.romfs_dir, API version 11).gdi_draw, the bitmap uploads and copies),host/posix/gdi_posix.cand its part ofwindows.h,third_party/stb,vk_startup_paint.h. The Windows DLL no longer imports GDI32.The game's font
The game's menu font is a commercial font licensed to Nintendo. Hookshot never ships it. It is read from the player's own romfs (the folder the game runs from), decompressed (zstd, with the game's own dictionary) and decoded in memory only. It is never written to disk, a cache or a release. If it is not found, Libertinus is used.
RX_VK_UI_GAME_FONT=0forces Libertinus.Third-party parts added
Each folder has its licence and a
PROVENANCE.mdwith the source and the archive's sha256. The Linux AppImage ships these licences in itslicensesfolder. Removed: stb_truetype (only the GDI path used it).Testing
tests/run_all.py: 206 passed, 0 failed (5 skipped), Linux. New tests:test_ui_record.py: the realui::recordwith stubs (barriers, load and store, order);test_game_font.py: the font reader on synthetic files (good, truncated, bad name offset, inexact name, wrong size, wrong tag);test_appimage_licenses.py: the AppImage's licence copies.RX_VK_MENU_SCRIPT: the options tabs, Controls, Cheats, Teleport, Inventory, an "After restart" change and the restart question.menu_recordper present, 1080p:tests/test_menu_hit.cpp. The headless runs have no pointer, so clicks were not tried in a game run.Screenshots
Taken in the game (headless, a fresh save). The menu is drawn over the game, dimmed.
1080p, keyboard prompts
4K TV size (Menu size Auto picks Large in fullscreen with a controller), controller prompts
Narrow window (1024x1024): the help sits below the rows
Start-up screen (the percentage is centred on the bar and stays readable across the fill edge)
Frame generation 4x: the menu and the FPS counter over generated frames
The images are on the fork's
pr-assetsbranch, which is not part of this PR.Self-review notes.
Checked and fine:
ui::recordtakes the window image from TRANSFER_DST (after ALL_TRANSFER / TRANSFER_WRITE, which covers the blit, the clear and frame generation's copy) to COLOR_ATTACHMENT, runs one LOAD/STORE pass, and goes back to TRANSFER_DST forimage_barrier's move to PRESENT_SRC. Texture copies are recorded before the pass, with UNDEFINED or SHADER_READ_ONLY to TRANSFER_DST and back. Nothing is recorded for an index with no view.test_ui_record.pyruns the realrecord()andlayout()and checks all of these masks.menu_tickormenu_record, and both run insidepresent_imageunderpresent_image_mu.startup_screen(on the start-up thread) only swaps theStartupViewunderstartup_mu, anddraw_startupcopies it under the same lock. Context creation and font loading happen in the firstui::begin, so they are under the lock too.present_imagekeeps 3 frames in flight, and the backend getsImageCount = 4for its vertex buffers.ui::idlekeeps reaping while nothing shows. When the format changes, the pipeline is rebuilt aftercreate_swapchainhas waited for the device, so the old pipeline is no longer in use. The views are dropped before the old swapchain is destroyed.VK_KHR_swapchain_mutable_formatis present, the swapchain gets UNORM views (with a format list and the mutable flag only in that case). Without it, the vertex colours are made linear and the logo is sampled as sRGB. Glyphs are white with coverage in alpha, so they look the same either way.unzstdcaps the output at 256 MB and needs a known content size.sarc_filechecks the header size, the BOM, each node against the buffer, the name offset and length (with its NUL), ands0 <= s1,data + s1 <= size.bfotf_fontchecks the size word against the real size and the sfnt tag. The font goes to the atlas viaIM_ALLOCand is never written to disk. WithRX_VK_UI_GAME_FONT=0, or when any of the three faces is missing, all three fall back to Libertinus, so the faces are never mixed.cp -aand has a test), Auto decided when the menu opens (auto_pad), whole UTF-8 characters (utf8_whole,char_startsinfit), the format change and sRGB,--romfsthroughMkwVkHost.romfs_dir(API 11), texture lifetime, the barrier and font tests, the glyph pre-bake (the dry draw restoreshits,marquee_max,list_lines,ease_stepandscroll_at), and the Windows test fixes.menu_tickhunks only change the hit map (hits.at/hits.drag),list_linesin place of the fixed 20 (with a re-reveal when it changes),pad_last/auto_pad, and the deletedredraw_menucalls. Keys, pages, presets, ini and the restart flow are untouched. The Menu size row is saved by the generic row loop.menu_ticklines from main (Esc back to Controls, tabs from Motion, the Motion row openingTAB_MOTION) are intact.scroll_athas 5 slots: tabs 0-2, Motion at 3, Controls page atkControlsSlot= 4.tab_selectedhas 4. The deadzone row is drawn and measured like any other row.kMax = 96is enough. Boxes are clipped to the visible list area.menu_recorddoesui::idle()(a counter and an empty reap) and returns, with no ImGui frame and no pass. Themenu_tickadditions are a few branches.fopenfrom the same ANSI path the game itself opens romfs with. zstd is built withZSTD_DISABLE_ASM, no threading and no legacy code. ImGui is built withIMGUI_DISABLE_WIN32_FUNCTIONSandIMGUI_DISABLE_DEFAULT_SHELL_FUNCTIONS, so there is no imm32/shell32.third_party/stbhas no remaining users.run_all.pyon the PR head, Linux, with the CPM headers: 206 passed, 1 failed, 5 skipped. The failure istest_program_cache_cleanup.py, which fails the same way on main. The old GDI barrier test now lives intest_ui_record.py, with stricter masks.test_menu_hit.cppnow tests the recorded map instead of fixed geometry, andtest_menu_hit.pychecks that each Part is recorded where it is drawn. Only one check got looser:test_device_lost_exit.pyno longer needshos_log_flushto be last in the table, which is needed now thathost_romfs_dirfollows it.RX_VK_MENU_DUMPis gone along with the bitmaps.RX_VK_UI_GAME_FONTandRX_VK_UI_PROMPTSare both in the readme. The readme's menu text, the Menu size row, the font note and the licence list match the code.Worth a look before merge:
scroll_rowshere as the other pages do, or shrink the pitch to fit. Related: on a narrow window the Cheats help box sits at 62% of the height, on top of the last rows.create_buffer/create_imageout of memory), it stays WantCreate withTexIDinvalid, but that frame's draw list still uses it. In this build (NDEBUG)ImGui_ImplVulkan_RenderDrawDatathen binds a null descriptor set. This only happens under memory pressure, but skippingRenderDrawDatawhile any texture is not OK would be safer.nx2/build.pystill passes-lgdi32to the DLL link. lld drops the unused import, so the DLL is fine as described, but the flag could go.@ -0,0 +70,4 @@const size_t name = names + (size_t)(u32(node + 4) & 0xFFFFFF) * 4;if (name > a.size() || a.size() - name < len + 1 || memcmp(a.data() + name, want, len + 1)) continue; // (with its 0)const size_t s0 = u32(node + 8), s1 = u32(node + 12);if (data + s1 > a.size() || s1 < s0) return false;Checked the bounds on a 64-bit size_t: every node is checked before it is read, the name is checked for offset and length including its NUL, and the data range is
s0 <= s1anddata + s1 <= size. The u32 values can't overflow the sum.unzstdcaps output at 256 MB, andbfotf_fontchecks the size word and the sfnt tag.test_game_font.cppcovers the truncated and bad-offset cases.@ -0,0 +222,4 @@VkBuffer buf = VK_NULL_HANDLE;VmaAllocation buf_alloc = nullptr;void* map = nullptr;if (!size || !create_buffer(size, VK_BUFFER_USAGE_TRANSFER_SRC_BIT, true, buf, buf_alloc, &map)) return;If this fails (or
create_image/AddTexturebelow), the texture stays WantCreate withTexIDinvalid, but the draw list built this frame still refers to it. The build has NDEBUG, so ImGui's assert is gone and the backend binds a null descriptor set. It only happens out of memory. Still, it would be safer forrecordto skipRenderDrawDatafor the frame when any texture indd->Texturesis still not OK after the update loop.@ -0,0 +355,4 @@pi.PipelineRenderingCreateInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_RENDERING_CREATE_INFO;pi.PipelineRenderingCreateInfo.colorAttachmentCount = 1;pi.PipelineRenderingCreateInfo.pColorAttachmentFormats = &pipeline_format;ImGui_ImplVulkan_CreateMainPipeline(&pi);Checked:
CreateMainPipelinedestroys the old pipeline right away. That is safe because a format change only comes fromcreate_swapchain, which has already waited for the device, and thisbeginis the first ImGui recording since then.@ -0,0 +412,4 @@if (!told) told = true, logf("vk: ui: a texture could not be made (out of memory?): the overlays wait for it");return;}layout(cmd, image, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL, VK_PIPELINE_STAGE_2_ALL_TRANSFER_BIT,Checked: srcStage ALL_TRANSFER / TRANSFER_WRITE covers whatever wrote the image last in
present_image(the clear,blit_crop_to_rect, frame generation's copy). The barrier at the end goes back to TRANSFER_DST forimage_barrier's ALL_COMMANDS move to PRESENT_SRC.test_ui_record.pyasserts both.Checked: the mutable flag and the format list are only used when the extension is on, the format really has a different UNORM twin, and COLOR_ATTACHMENT was granted. In every other case the swapchain is created exactly as before, and the views use the swapchain's own format.
This is the bug from the review body. The 11 Cheats rows use a fixed 48-point pitch from
hh + 24 * swith no scroll. At Large the window is 554 points tall, the prompt strip starts at about 487, and the last row would end at about 612. Teleport overlaps the prompts, Switch age is cut off, and Inventory is off screen (seen in the 4K run). The options and Controls pages already usescroll_rows, and this page should too, or the pitch should shrink to fit betweentopand the prompt bar.Merge check:
scroll_athas TAB_MOTION + 2 slots. The options page scrolls bycur_tab(0-3, Motion at 3) and the Controls page by this slot (4), so the Motion page added on main gets its own slot.prebakesaves and restores an array of the same size.@ -2377,3 +2502,3 @@}}} else if (press[7] && &r == &rows[kCheatsRow]) page = 2, changed = true;} else if (r.cheat >= 0) cheat_press(r.cheat, press, changed); // (a debug tool: on the Debug tab itself)Pre-bake runs only from
menu_record, so only while something shows. Once the start-up screen is gone and the FPS counter is off, a resize or F11 means the first open at the new unit bakes on the spot (the old 2.6 ms first-open hitch, once). Minor. One option is to also callprebakewhen the size changes, even with nothing drawn.Self-review items addressed:
28af753. The Cheats rows now scroll and keep the selected row in view, the hit map is clipped to the visible rows, and on narrow windows the help sits below the rows. Checked at 4K Large, 1080p, 1280x1024 and 1024x1024 (4K and narrow screenshots added to the description).d5b14d9. While any texture failed to create, no barriers or pass are recorded and the texture is retried next frame (test_ui_recordcovers the failure and the recovery).e01dbe0. After a window or Menu size change while nothing shows, one empty frame bakes the glyphs and measures the page, so the first open has no hitch.a394ccf. It is gone from the DLL link; the DLL imports only KERNEL32 and USER32.run_all: 207 passed, 0 failed. Linux build and Windows DLL cross-build OK.Changes since the last update, from playing the build and from main moving:
Fixes
02b0903): the menu confirmed with the bottom face button and went back with the right one, whatever the layout. With the Nintendo layout (the default) the game's A is the right button, so A went back instead of opening Debug tools, Motion or Controls. Main's menu has the same bug. The menu now confirms with whatever button A is bound to and goes back with B's. The prompt bar says what A does on each row (Open, Open list, Apply, Toggle, Switch).test_menu_keys.pycovers every layout and every page.bd1616f): the help box could run down over the prompt strip and cut it off. The strip is now laid out first and drawn on top; the help box and the rows end above it, the help moves up or shrinks if it must, and prompts shrink to fit a narrow window.bd1616f): the tag no longer sits in the name cell. The help box says "Takes effect after a restart" for the selected row, and other rows get a small gold diamond. While there:menu_restart.hhad room for 8 restart rows but the menu has 10, so Audio output and Button prompts readstart[-1]and Button prompts showed "After restart" with nothing changed. Main has the same bug.kMaxis now 16, and a test checks the count.bd1616f): the two cells of a binding row now draw like other value cells (same padding, labels centred, a gap between them).Changes to the menu
a4c3a8d): it was a picker (Left/Right changed the map) plus a list behind Enter. It now only opens the map list, like Inventory.e256616): no separate page. The tools show right below the Debug tools switch while it is on, in the same list. Teleport and Inventory still open their pages, and back returns to the Debug tab on the same row.e256616): at the bottom of the Debug tab, always shown. Descriptor reuse and Sampler cache (#1330) move there from Graphics, with the same ini keys and restart behaviour.70e2209): on windows wider than 16:9 the pages draw in a centred 16:9 column, with the game dimmed across the whole window. Nothing moves at 16:9 or narrower.Merges with main
RX_VK_SHOT_MENUare drawn with ImGui. In HDR the overlays are converted to HDR10 at the paper white.test_menu_utf8.pychecks that the menu's own text cuts keep whole characters. #1405 (fonts without Latin letters) moved into the game-font reader, which skips a font with no 'A'/'a' and falls back to Libertinus;test_game_font.pycovers it.MKWVK_API_VERSIONis 12 (main took 11 forprompt_face;romfs_dirfollows it).Testing:
tests/run_all.py: all pass excepttests/test_linux_update.py, which fails the same way on main right now (#1525's absolute-symlink refusal vstest_a_part_keeps_toolchain_links). Linux build with DLSS and the Windows DLL cross-build are clean. Headless runs at 1080p, 1440p and 4K Large: every page, A/B into and out of each sub-page, and Game's own / DLSS / DLAA / FSR 3.1 in the Upscaler row. The ultrawide and Large-layout screenshots are being refreshed and will be added to the description.Tested by hand on Linux (KDE Wayland, RTX 4070 Ti), windowed, build
fb70b34, complete dump, DLSS options:Nothing broken found. I only tried the keyboard: the menu didn't open from scripted pad input (HOS_INPUT_LIVE), which is expected since it reads the real devices.