feat(control): an overlay command shows a test player's decisions beside the FPS counter #1358
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1358
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:control-overlay-text"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Linux, RTX 4090, headless (game-free checks; in-game pictures: see Testing) · Needs: Windows (build and run), a real window
Why
The test player (the "pilot", issue #1198) decides what to do next many times a minute, and until now the only way to see why was to read its run files afterwards. Someone watching a run, a recording or the game's own pictures could see the frame times (the FPS counter) but not the decisions. This adds a control-bridge command that puts a short text panel on the game's screen, drawn in the same overlay pass as the FPS counter, so both are visible at once. The pilot uses it to show its goal, the candidates with the chosen one marked, the step and its result, model and cost.
What changed
A client sends
{"cmd":"overlay","text":"...","seconds":N}(overlay_clearor an empty text clears). The bridge checks and cleans the text (at most 2048 bytes and 16 lines of valid UTF-8, control characters dropped), keeps it only while the sending client is connected and for at mostseconds, and hands it to the renderer, which draws it beside the FPS counter at every counter level and alone with the counter Off. Nothing is saved, logged or sent anywhere, and no game state changes.host/control.c,host/control.h: the two commands, owner and expiry on the bridge's thread,unavailablewithout the renderer's hook.gpu/vk/overlay_text.h(new, plain C): the text rules, used by the bridge and run again by the renderer.gpu/vk/vk_api.h,host/rx_backend.c,gpu/vk/vk_replay.cpp:MkwVkRenderer.overlay_text,MKWVK_API_VERSION30 to 31 (main is at 30 since #1383's pose sampling; the number renumbers again at merge if another pull request lands first).gpu/vk/fps_stats.h: the counter's column height, width and corner box as functions (the sums moved out ofdraw_fps; a test checks they agree with the old ones to the pixel);draw_fpspublishes the box it drew.gpu/vk/overlay_panel.h(new): the text store (its own lock, not the menu's) and the placement: in the counter's corner, under it (over it from a bottom corner) only while the panel ends within 70 % of the window's height, else beside it toward the middle, never over it, shrinking to 70 % or cutting lines narrower before giving up (and under it past that reach only when nothing else fits).gpu/vk/vk_menu.cpp:draw_pilot_panelafterdraw_fpsin both record paths (the window's and the headlessRX_VK_SHOT_MENUpictures'); a translucent plate, the menu's font and Menu size scale; a new text size is measured once before the layout is settled and a line is cut to "..." only when it is more than 2 % over its room; a line starting with>is gold; not drawn while the Insert menu is open.scripts/longshot_ctl.py:overlay LINE... [--seconds N]and--clear.docs/control.md(the command, limits, privacy, pictures),docs/development.md(a row for the existingRX_VK_SHOT_MENU=1).shot(no overlay, so a picture sent to a vision model never carries the panel's text), no new switch, nothing when no client sendsoverlay.Measurements
Verdict: nothing to measure: this adds no work to a frame unless a client has set a panel (one atomic read in the overlay pass). No speed claim is made and none was measured.
Flags and settings
None: no new switch, no menu row and no ini key (the panel is drawn by
gpu/vk/vk_menu.cpp's overlay pass but has no setting of its own).RX_VK_SHOT_MENU=1, an existing switch (it has its row ingpu/vk/switches.defalready, so #1309's rule asks for no new row), is now documented indocs/development.md; it is what puts the FPS counter and the panel into a headless run'sRX_VK_SHOTSpictures.Testing
Tested: rebased on 2026-11-17 onto
main @ 67efc4d3e, new headb08a1d56e(the first bullet; Linux 7.2.15 (Arch), i9-13900K, RTX 4090 with NVIDIA 615.78.08, GCC 16.2.1 and Clang 23.1.3; the game-free checks used no window and no GPU; other work was running on the PC, load 20 to 60). The other bullets are the earlier results, each with the head it ran on (38f10348fovermain @ 6d72ca93b, the same five patches); they were not rerun: game-free checks on Linux, a Windows-target syntax check and cross-build, and one windowless run of the game with the pilot (below):main @ 67efc4d3e(2026-11-17: #1309, #1383, #1378, #1379), old head8e1f34ab4, new headb08a1d56e. Changed lines: the same as the old head's except the API number and its doc row.git range-diffpairs all 5 commits: 3=, 2!; the two!commits differ only in lines that name the number or document it, and in the test pins (added and deleted lines compared one by one: 6 and 8 lines in commit 1, 3 and 3 lines in commit 2). One conflict,tests/test_audio_output_devices.py's pinned number (main's== 30from #1383 against this PR's== 30), resolved to== 31with this PR's own message. Renumbered to 31 (main is 30 now): the define and theoverlay_textcomment ingpu/vk/vk_api.h, "renderer API version" indocs/control.md, andtests/test_overlay_panel.py(>= 31, and the comment's pattern: stricter than before, the old>= 30would pass on main alone). The first commit's message says30 -> 31where it said29 -> 30. The doc row:docs/development.md's sentence on the API versions now names version 31, the renderer table'soverlay_text(docs/control.mdalready named the number). No check deleted. #1309's rule (a newRX_,HOS_orLONGSHOT_name read from the environment needs a row ingpu/vk/switches.def): this PR adds no switch;RX_VK_SHOT_MENU, which its docs row names, already has its row.main @ 67efc4d3ewith this PR's changed or new test files copied in: 1 pass, 5 fail (test_audio_output_devices, test_control, test_fps_overlay_order, test_longshot_ctl, test_overlay_panel fail on main's source, as they must); test_switch_table passes on main too (test_switch_tablereads main's own source, where no name is new: this PR adds no switch). The whole suite,tests/run_all.py --jobs 6 --strictin the idle gate with a fresh HOME, each commit in its own clean checkout, one after the other:main @ 67efc4d3e665 passed, 40 failed, 17 skipped in 188 s; this headb08a1d56e666 passed, 40 failed, 17 skipped in 213 s. The failing files are the same 40 on both, compared by name: none fails only on this head.run_all.py --changed 67efc4d3e(the checks that read the 20 changed files; its header counts 202) lists 185 checks by name: all ran in this whole-suite run, and each passed or is one ofmain's own failures. A Linux build of this head,make_game.pybehind the full-build lock and the build gate, no shader preparation, the game's lifted code reused from an earlier build whose lift key equals this head's (checked): exit 0,v3.0.0-rc.7-9-gb08a1d56e, 102 of 518 objects new, 0 failed, 0:01:11. Built, not run.main @ 67efc4d3e): 7 translation units captured, 1 failure, 1 unjudged. Compile for Windows:test_control(control_test.c,host/control.c),test_control_pipe_security,test_fps_overlay_order,test_overlay_panel. The failure istest_audio_output_devices(memmove/tmpfiledeprecated inparser.c: the same fourerror:lines on pristinemain @ 67efc4d3e, run as a control; its fixture lacks_CRT_SECURE_NO_WARNINGS; this PR only moved the pinned number in it). Unjudged:test_longshot_ctl(it starts its fake bridge with a directPopen). Windows execution not run.38f10348f(the same five patches) overmain @ 6d72ca93b. What this rebase changed in lines the game runs: the API number, which host and renderer both take fromvk_api.h(the build above compiled the pair with 31), and nothing else in this PR's lines.tests/test_control.py: new cases against the fake game over the real socket (parsing, the 2048-byte and 16-line caps, UTF-8, dropped controls, expiry once, a later deadline, the owner's disconnect,unavailable). Red first: it did not compile without the header, then failed at the first overlay case withunknown_cmd; green after.tests/test_overlay_panel.py: the text store under a concurrent writer and reader, the counter's column against the old sums, and 51,840 layouts (15 window shapes from 640x360 to 5120x1440 including 4:3, 5:4 and 21:9, three Menu sizes, four counter sizes, six counter levels, four corners): every placed panel is inside the window and clear of the counter; at 4:3 and wider a place is always found.tests/test_longshot_ctl.py(the client, 52 tests).main @ 6d72ca93b, head38f10348f): 7 captured translation units, 1 failure, 1 unjudged driver. Compile for Windows:tests/test_control.py(control_test.c,host/control.c),tests/test_control_pipe_security.py,tests/test_fps_overlay_order.py,tests/test_overlay_panel.py(it defines_CRT_SECURE_NO_WARNINGS). The one failure,tests/test_audio_output_devices.py(memmove/tmpfiledeprecated inparser.c: the same fourerror:lines on pristinemain @ 6d72ca93b; this PR only moved the pinned API number in it); unjudged:tests/test_longshot_ctl.py(it starts the fake bridge with a directPopen). Windows execution was not run.38f10348f, clean checkout): exit 0, 421 compile and link commands (103 new, 318 from matching cache entries), 0 failed, 0 warnings (under the build's own-Wno-everything: not a warning-clean claim). The renderer's 210 units compiled andlongshot_vk.dlllinked (14,367,744 B); the host group's 210 units compiled, including the changedhost/control.candhost/rx_backend.c, andgpu/vk/vk_menu.cppandgpu/vk/vk_replay.cppcompiled new. The game executable was not linked (no Linux lift registered for this head to link it from) and nothing was run on Windows or Wine.tests/run_all.py(--jobs 6 --strict, fresh HOME, idle gate, both runs at the same time on a loaded PC):main @ 6d72ca93b662 passed, 41 failed, 17 skipped in 319 s; this branch@ 38f10348f663 passed, 41 failed, 17 skipped in 310 s. The failing files are the same 41 on both, compared by name (none fails only here):test_bug_report_defaults,test_cpu_report,test_crash_record,test_crt_options,test_damage_debug_menu,test_display_startup_failure,test_dlss_feature_size,test_failed_compile_keeps_objects,test_framegen_preset,test_fsr_uniform_ownership,test_guest_imported,test_hud_layout_row,test_import_reach,test_invalidate_vcore,test_linux_preflight,test_linux_pressure,test_linux_update,test_mirror_budget,test_missing_nup,test_mods_size_table,test_nonuniform_image_arrays,test_parallel_init_order,test_pending_guest_import,test_pipeline_play_cache,test_present_thread,test_profiling_mode,test_readback_timeout_source,test_replay_write_order,test_rumble_menu,test_run_all_changed,test_slow_frames,test_staging_burst,test_switch_table,test_target_image_pool,test_tested_configurations,test_upscaler_round_trip,test_upscaler_rows,test_vcore_escape,test_vk_occlusion_queries,test_vk_swapchain_compat,test_walk_key(they fail on this PC'smainas it stands, for exampletest_import_reachon astd::vector::containscompile error with the system'sclang++; not looked into further).nx2/make_game.pybuild, the rebased head38f10348f):v3.0.0-rc.6-9-g38f10348f (source af14c9ad),longshot546,801,056 B sha25630c67c67939278304b9e345d1ed07d48057d82a7eb05a7e9ddbcb2d846b23026,longshot_vk.so15,573,464 B sha2564a3a176377ca14606928e58e7b62b614940bae8e89fe0925ecb805ae7a965702; 169 s, 96 objects new and 422 from the object cache of the earlier build (main's lift changed, so it was lifted again). The in-game pictures below come from this build.In the game, on the rebased build (Linux 7.2.15, RTX 4090, driver 615.78.08, 2026-11-16, headless; the game's own first log line names the build:
longshot v3.0.0-rc.6-9-g38f10348f (source af14c9ad), built from38f10348f). This branch had been played before at an earlier head on an oldermain(c4aca39d3onac7787515);mainhas moved 37 commits since, so the game was built and played again after the rebase, and this is that run. One run of the pilot (hud3-talk) in Kokiri Forest, saveexplore17-kokiri-child: it walks to a Kokiri girl and talks to her, then goes to the signboard and reads it (rules decider, so no model call). Shader preparation first (45 s, exit 0,@@done shadersfrom the game itself), then the run: the pilot's run took 84.8 s and ended solved, the game ran 83.2 s,RX_VK_SHOT_MENU=1, a picture every 45 presents, 1280x720, 210 pictures. The game's log hascontrol: overlay panel set (11 lines, 235 bytes; it goes with its client or its seconds)once andcontrol: overlay panel cleared (asked)at the end. The wrapper's summary line:exit=143(the pilot's own stop at the end of its run)fatal=0 crash=0 devlost=0 nocore=0 kernel=0 nvrm=0 journal=complete, andcoredumpctllists no core dump since the run started; itsHOME-WRITESnote names~/.cache/longshot-builder, an existing empty folder from 13 November (the note tests existence only). Other work was running on the PC (load 12.7 at the start, 7.3 at the end), so the frame rates in the pictures say nothing about speed.While the pilot played, a second small program changed the FPS counter's level and the Menu size over the run's own control socket, nine seconds per phase, in this order: Graph Medium, Breakdown Medium, Off Medium, Off Large, Graph Large (Breakdown Large, FPS and Frame time were next; the pilot solved its segment and the game was stopped before them). It recorded the renderer present of each change, so every picture is tied to its phase. 53 of the 210 pictures lie inside those phases (Graph Medium 11, Breakdown Medium 12, Off Medium 12, Off Large 12, Graph Large 6); the other 157 are the start-up (black, the opening field in moonlight, the title, the load menu, black again), with neither counter nor panel. All 210 were looked at: the 53 in sets of four at 800x450, the 157 in sets of twenty, and at full size the first picture after the size change and one picture of each of the other phases. What was seen:
vk_007335, the first decision, the panel is under the counter, its lower edge at about y 335 of 720, with the hearts (top left) and the map (lower right) drawn and clear of it.vk_008685, the first picture after the change (7 presents after it), has every line whole; its top line,GOAL talk-to-the-kokiri [open] FOR kokiri-013(45 characters, the longest top line of this run), ends inside the plate. In the first run the same change with the same counter state (vk_005985) had a top line cut to "...". The other eleven Off-Large pictures have the signboard goal's lines whole (GOAL read-the-sign [open] FOR signboard-010, then[met]with aRUN solvedrow, up to 15 lines with+2 more). No picture of the 53 has a panel line cut to "...". The top lines of the two runs differ (the first run's was the signboard goal's, this run's the Kokiri talk goal's), so this is the same change but not the same text.vk_009270the panel is beside the counter (15 lines,RUN solved,+2 more), readable at the larger size, its right edge just left of the counter's, the hearts and the map clear. The first picture of the phase,vk_009225(11 presents after the counter went from Off to Graph), has the counter still filling in (no frame-time row yet) and the panel under it, clear of the counter; the next picture has the counter whole and the panel beside it. A panel goes under the counter only while it ends within 70 % of the window's height, and with the counter's second row in, 15 lines at Large end past that, which I take to be why it moved (not measured): one picture of the six.1. Graph at Medium (a walking scene): the panel under the counter, hearts and map clear. 2. Breakdown at Medium: beside the tall counter. 3. Counter Off at Medium: the panel alone, the dialogue's choices clear. 4. Counter Off, the first picture after the change to Large: every line whole. 5. Graph at Large: beside the counter, 15 lines, readable.
1 and 3 are from the first run (
06096442a), 2 and 4 from this one (38f10348f). 1 and 2: Breakdown at Medium, under the counter down to the corner, and now beside it. 3 and 4: the first picture after the same size change (Medium to Large, counter Off): then a top line cut to "..." although it fitted, now every line whole (the top lines differ: the first run's was the signboard goal's, this run's the Kokiri talk goal's, which is longer).Not tested: a real window (this PC's runs are windowless: the window path calls the same
draw_pilot_panel, unseen); Windows (syntax and cross-build only, nothing run); the pilot's model rows (model, latency, cost) in the game: the run used the rules decider; Breakdown at Large, FPS and Frame time with the panel on this build (the run ended before those phases; the two earlier runs, at06096442aandc4aca39d3on the oldermain, saw Breakdown at Large and the FPS levels, but not on this base); the panel with the Insert menu open (hidden by design, not looked at).Must be tested:
longshot_ctl.py overlay "hello" --seconds 30in a window with the FPS counter on Graph, expecting the panel under the counter, every line whole and no overlap with the counter, the hearts or the map (by hand: look at the corner).longshot_ctl.py overlay "hello" --seconds 30with the FPS counter in each corner, expecting the panel beside or under the counter, never over it, and inside the window (by hand: look at each corner).Risks
What could break:
draw_fps's height sum moved intofps_stats.h(checked to the pixel by a test, so the counter should look as before); a renderer built before this change with a newer host (or the other way round) is refused by the version check, as for every API bump. A client that keeps sending text can keep a panel on the screen over the game: the Insert menu hides it while open, and it goes with the client.Turning it off: nothing is drawn unless a client sends
overlay; revert the PR's commits to remove it.Compatibility:
MKWVK_API_VERSION30 to 31 (a stale renderer library is refused); nothing changes for saves,longshot_options.inior the caches. The number renumbers again at merge if another pull request lands first:mainis at 30 (#1383's pose sampling) and this change takes 31; the other open pull requests that append to the same tables are PR 1426 (takes 32), PR 68 (33) and PR 1411 (34), so whichever merges after another renumbers: a one-line change ingpu/vk/vk_api.hplus the two tests that pin the number,tests/test_audio_output_devices.py(_Static_assert(MKWVK_API_VERSION == 31, ...); this PR moved it from 30 to 31) andtests/test_overlay_panel.py(it asks for 31 or more and for the(version 31, longshotcomment).Checklist
Everyone
type(scope): description) of the status line's type.main(67efc4d3e, 2026-11-17). The first Testing bullet is onb08a1d56eover thatmain: the PR's tests, the whole suite, the Windows-target syntax check and a Linux build. The older tests, builds and the game run are on38f10348fovermain @ 6d72ca93b(the same five patches); the game was not run again.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.MKWVK_API_VERSIONbumped (gpu/vk/vk_api.h'sMkwVkHostorMkwVkRendererchanged).mainand passing here.tests/that reads changed source updated in this PR, not weakened.New on this forge; not opened on GitHub.
WIP: feat(control): an overlay command shows a test player's decisions beside the FPS counterto feat(control): an overlay command shows a test player's decisions beside the FPS counterA client can put a short text panel beside the FPS counter, and it leaves with that client or with its seconds. I read head
8e1f34ab. It applies onto current main. The window runs it still names are open, so this is not an approval.The bridge keeps at most 2048 bytes and 16 lines of UTF-8, drops control characters, and clears the panel when that client disconnects or when
secondspasses. The renderer draws it after the counter, in both the window and the headlessRX_VK_SHOT_MENUpictures, and not while the Insert menu or the start-up screen is up. The bridge's ownshotstill records the picture without this panel. The panel sits under the counter while it ends within 70% of the window, and beside it otherwise, and a line is cut to "..." only when it is more than 2% over its room.MkwVkRenderergainedoverlay_text, and the interface is 30. Main is still 29. If #1034 lands first, this bump is the next number after that one.The headless Linux pictures show the panel clear of the counter, the hearts and the map. Must be tested still asks for a Windows window (the panel under Graph, every line whole) and a Linux window at 4K and at 1280x720, the counter in each corner.
Rebased
8e1f34ab4onto main67efc4d3e; new headb08a1d56e.Patch: the same changed lines as
8e1f34ab4except the API number and its doc row.git range-diffpairs all 5 commits: 3=, 2!; the two!commits differ only in lines that name the number or document it, and in the test pins (added and deleted lines compared one by one: 6 and 8 lines in commit 1, 3 and 3 lines in commit 2). One conflict,tests/test_audio_output_devices.py's pinned number (main's== 30from #1383 against this PR's== 30), resolved to== 31with this PR's own message. Renumbered to 31 (main is 30 now): the define and theoverlay_textcomment ingpu/vk/vk_api.h, "renderer API version" indocs/control.md, andtests/test_overlay_panel.py(>= 31, and the comment's pattern: stricter than before, the old>= 30would pass on main alone). The first commit's message says30 -> 31where it said29 -> 30. The doc row:docs/development.md's sentence on the API versions now names version 31, the renderer table'soverlay_text(docs/control.mdalready named the number). No check deleted. #1309's rule (a newRX_,HOS_orLONGSHOT_name read from the environment needs a row ingpu/vk/switches.def): this PR adds no switch;RX_VK_SHOT_MENU, which its docs row names, already has its row.Carried: the earlier results are of
38f10348fovermain @ 6d72ca93b, the same five patches (git range-diffof38f10348fagainst8e1f34ab4: 5=): the game-free tests, the Windows-target syntax check, the Windows cross-build of the renderer and host, the whole suite (663 passed, 41 failed, 17 skipped) and the in-game run (the pilot'shud3-talkin Kokiri Forest, 210 pictures,v3.0.0-rc.6-9-g38f10348f, Linux, RTX 4090, 2026-11-16, headless). Not rerun; the description names each head.Rerun: the PR's own tests and the ones that read what it changed (test_control, test_overlay_panel, test_longshot_ctl, test_fps_overlay_order, test_audio_output_devices, test_switch_table) on the new head: 6 of 6 pass. The same set run over
main @ 67efc4d3ewith this PR's changed or new test files copied in: 1 pass, 5 fail (test_audio_output_devices, test_control, test_fps_overlay_order, test_longshot_ctl, test_overlay_panel fail on main's source, as they must); test_switch_table passes on main too (test_switch_tablereads main's own source, where no name is new: this PR adds no switch). The whole suite,tests/run_all.py --jobs 6 --strictin the idle gate with a fresh HOME, each commit in its own clean checkout, one after the other:main @ 67efc4d3e665 passed, 40 failed, 17 skipped in 188 s; this headb08a1d56e666 passed, 40 failed, 17 skipped in 213 s. The failing files are the same 40 on both, compared by name: none fails only on this head.run_all.py --changed 67efc4d3e(the checks that read the 20 changed files; its header counts 202) lists 185 checks by name: all ran in this whole-suite run, and each passed or is one ofmain's own failures. A Linux build of this head,make_game.pybehind the full-build lock and the build gate, no shader preparation, the game's lifted code reused from an earlier build whose lift key equals this head's (checked): exit 0,v3.0.0-rc.7-9-gb08a1d56e, 102 of 518 objects new, 0 failed, 0:01:11. Built, not run.Windows: syntax check of the changed C/C++ fixtures (llvm-mingw 20261104, idle gate, base
main @ 67efc4d3e): 7 translation units captured, 1 failure, 1 unjudged. Compile for Windows:test_control(control_test.c,host/control.c),test_control_pipe_security,test_fps_overlay_order,test_overlay_panel. The failure istest_audio_output_devices(memmove/tmpfiledeprecated inparser.c: the same fourerror:lines on pristinemain @ 67efc4d3e, run as a control; its fixture lacks_CRT_SECURE_NO_WARNINGS; this PR only moved the pinned number in it). Unjudged:test_longshot_ctl(it starts its fake bridge with a directPopen). Windows execution not run.Game: not rerun. The earlier run is of
38f10348f(the same five patches) overmain @ 6d72ca93b. What this rebase changed in lines the game runs: the API number, which host and renderer both take fromvk_api.h(the build above compiled the pair with 31), and nothing else in this PR's lines.8e1f34ab45b08a1d56ecView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.