fix(queue): on Windows the job queue waits for a player and sees builds outside it, as on Linux #1523
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1523
Loading…
Reference in a new issue
No description provided.
Delete branch "dukenheimer/Longshot:fix/job-queue-windows-player"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11, RTX 5090 (Python 3.14); Linux (WSL Ubuntu 24.04, Python 3.12) · Needs: none
TL;DR
On Windows, the PC's job queue never noticed someone playing. A timed measurement or a "wait until idle" job could start
while the game was running and spoil both, and
statusalways said nobody was playing. The queue now spots arunning game on Windows the way it already does on Linux, and also sees builds started outside it. New tests fail on
main on Windows and Linux and pass here, including real processes on Windows.
Breakdown
/proc. A game not started by a queued job holds timed and--when-idlejobs, andstatusnames it.longshot.exeprocesses on Windows are told apart byLONGSHOT_JOB; full suites on both OSes show no new failure.Why
AGENTS.md and the measure skill send every game run, measurement and profile on a PC through
scripts/job_queue.py,so that nothing runs over a person playing or over another measurement. On Windows that protection was missing. At
main
310e7c9d:user_playing()returns False wheneverWINDOWS(the check before itsgame_procscall);statussetsgames = []andheavy = []on Windows, so it always printsplaying: nobodyand never a build outside the queue;game_procs,heavy_procs,argv_of,env_ofandtreeread/proconly. On Windows they find nothing, so a timed run never waits for a build outside the queue either.So a
timing/profilejob or a--when-idleticket on Windows starts while someone plays, and its numbers aremeasured over the game. The PC's release-check runner found this.
Prior work: none found. I searched
job_queue,user_playing,game_procs,heavy_procs, "player protection", "playing: nobody","you are playing" and the team chat. #1315 (NitnekLeBorgne) changes only the queue's history. #1085 and #1100 built the
queue and its Windows job objects, and neither covers this.
How to reproduce
Seen on: Windows 11, RTX 5090, main
310e7c9dSettings and switches: defaults
Steps: Start the game (
longshot.exe), then runpython -I scripts/job_queue.py status, or queuepython -I scripts/job_queue.py run fps --mode timing -- <anything>. Game-free:python -I -B tests/test_job_queue.pyfrom this PR's test commit.Expected:
playing: longshot.exe (pid N), and the timed job waits: "fps waits: you are playing"Actual:
playing: nobody, and the timed job starts at onceWhat changed
The queue's process reads now work on Windows, so the player check,
statusand the stray-build check work there asthey do on Linux.
scripts/job_queue.py:_win_snapshot()lists every process with its parent and program (Toolhelp)._win_params()reads a process's command line and environment from its own PEB (NtQueryInformationProcess,ReadProcessMemory), as/proc/<pid>/cmdlineandenviron. It only reads a 64-bit process the user may read; anything else is unreadable, as on Linux.all_pids, the newexes/exe_of/comm_of,argv_of,env_of(Windows'=C:entries are no variables) andtree(by parent, a child born after its parent) use these on Windows.game_procsandheavy_procsfirst match the program's name, so only games' and pythons' memory is read. They also accept backslashed script paths.user_playing()andstatusno longer skip Windows.docs/development.md("On Windows too") and the script's usage say how a player is seen.tests/test_job_queue.py(first commit) adds two classes:WindowsPlayersruns the Windows code over a made-up process table on any OS. It covers a player, a job's game, shader preparation, an unreadable game, builds outside the queue with backslashes, a job's tree and a reused pid.WindowsPlayersReal(Windows only) uses a copy ofping.exenamedlongshot.exe, with and withoutLONGSHOT_JOB, and a python runningnx2/build.py.Measurements
Verdict: fixed. On main the new tests fail on Windows (7) and Linux (5, the real-process ones being Windows-only); here they all pass. Linux's own detection is unchanged.
Hardware: Windows 11, RTX 5090, Python 3.14; WSL Ubuntu 24.04, Python 3.12
Scene: none: game-free checks
Method:
python -I -B tests/test_job_queue.py -von main (with this PR's test) and on this PR, on Windows and Linux. The fix was then broken on purpose and the tests run again. On Linux, a copy ofsleepnamedlongshotwas run with and withoutLONGSHOT_JOBthroughgame_procs/user_playing, on main and here.How to reproduce: the commands above. Rebased since onto main
4b68c42b(its four commits touch onlydocs/development.mdelsewhere among these files);test_job_queue.pygives the same result there. Rebased again onto main1055077d(headf8b14525; main changed neitherscripts/job_queue.pynortests/test_job_queue.py): this PR310e7c9d(with the test: 93f9a3b8)4b68c42b;f8b14525on1055077d)probe, playing: Trueprobe, playing: Truestatuson this PC, a build running beside it under pc-lock(a build outside the queue: pid 181368 ...python.exe -I nx2\make_game...)Flags and settings
None: no switch is added or changed.
LONGSHOT_QUEUE_GAMESandLONGSHOT_QUEUE_HEAVYnow apply on Windows too, withtheir defaults (
longshot.exeis already in the games list).Testing
Tested:
tests/test_job_queue.pyon my own PC (Windows 11), on main and here, and in WSL Ubuntu 24.04, on main and here (table above). The real-process tests ran on Windows: alongshot.exewithoutLONGSHOT_JOBis a player, one withLONGSHOT_JOB=probeis that job's, its arguments are read, and a python runningnx2/build.pyis a build outside the queue unless it's in the caller's tree.python -I scripts/job_queue.py statuson this PC:playing: nobody(no game running) in 0.39 s, and it named the release tester'smake_game.pyrunning beside it as a build outside the queue.python -I -B tests/run_all.py, this PR @ 64d0afbc and main @4b68c42bin a clean checkout, the same runs. Windows 11 (12 jobs): this PR 665 passed, 50 failed; main 664 passed, 51 failed.test_longshot_ctlfailed only on main, a timeout it sometimes has under load, not this PR's. Linux, WSL Ubuntu 24.04 (8 jobs): 594 passed, 50 failed on both. None fails here and not on main.test_job_queuefails on Windows on both, for itstest_ram_file_systemalone (its new cases pass here), and passes on Linux on both. The 50 that fail on both, on this PC: Windowstest_auto_handheld,test_bug_report_defaults,test_build_long_paths,test_compute_warm_apart,test_cpu_report,test_crash_dialog,test_crt_options,test_damage_debug_menu,test_dev_packs,test_discover,test_display_startup_failure,test_dlss_feature_size,test_draw_stream_visibility,test_failed_compile_keeps_objects,test_fsr_image_retirement,test_fsr_uniform_ownership,test_guest_imported,test_hud_layout_row,test_import_reach,test_invalidate_quick,test_invalidate_vcore,test_job_queue,test_libc_hooks,test_lift_lookups,test_linux_preflight,test_linux_pressure,test_mirror_budget,test_missing_nup,test_pads_after_menu,test_parallel_init_order,test_pending_guest_import,test_pipeline_play_cache,test_present_thread,test_profiling_mode,test_psychov35_fsr3_fg,test_readback_timeout_source,test_recorder_sync,test_replay_write_order,test_run_all_changed,test_sdiv_overflow,test_slow_frames,test_staging_burst,test_switch_table,test_target_image_pool,test_tested_configurations,test_upscaler_rows,test_vcore_escape,test_vk_occlusion_queries,test_walk,test_walk_key; Linuxbyte_compile_tracked,test_bake_ds,test_bug_report,test_bug_report_defaults,test_check_third_party,test_control,test_cpu_report,test_crash_groups,test_crt_options,test_crt_shaders,test_damage_debug_menu,test_dev_packs,test_display_startup_failure,test_dlss_feature_size,test_failed_compile_keeps_objects,test_finish_output,test_guest_imported,test_hardware_report,test_hud_layout_row,test_invalidate_quick,test_invalidate_vcore,test_libc_hooks,test_link_skip,test_linux_downgrade,test_linux_pressure,test_linux_update,test_log_open_failure,test_mirror_budget,test_missing_nup,test_mods_size_table,test_oot_state_tap,test_pending_guest_import,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_pipeline_tour,test_present_thread,test_profiling_mode,test_psychov35_fsr3_fg,test_readback_timeout_source,test_release,test_run_all_changed,test_slow_frames,test_switch_table,test_target_image_pool,test_tools_lock,test_upscaler_rows,test_vcore_escape,test_vk_pgo,test_walk_key.Not tested: a real game started by the launcher while a timed job is queued: the tests use real Windows processes named
longshot.exe, and the queue's waiting onuser_playing()is unchanged. Nobody was playing on this PC during these runs.Must be tested:
Risks
What could break: on Windows, the queue now holds timed and
--when-idlejobs while alongshot.exestarted outside the queue runs, which is the point. A game started by a script outside the queue looks like a player until it ends. Run it through the queue (it setsLONGSHOT_JOB), or setLONGSHOT_QUEUE_GAMES. Reading a process's memory needs only read access to the user's own processes and changes nothing in them; an elevated or another user's game isn't readable and counts as nobody's, as on Linux.Turning it off: revert the commit
Compatibility: Nothing changes for a player. On Windows,
job_queue.py status --jsonnow fillsgamesandbuilds_outside.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.716cccd3f064d0afbc6eRebased onto main
4b68c42b(head 64d0afbc); test_job_queue.py gives the same result there. The full run_all.py on Windows and Linux, on this PR and on main, follows after the hat agent's build-slot turn; then I mark it ready.Full
python -I -B tests/run_all.py, this PR @ 64d0afbc against main @4b68c42b(a clean checkout), the same runs:test_longshot_ctl, a timeout it sometimes has under load, not this PR's doing.test_job_queue: on Windows it fails on both, for itstest_ram_file_systemalone; its new cases pass here. On Linux it passes on both.The description lists the 50 checks that fail on both. Marking it ready.
WIP: fix(queue): on Windows the job queue waits for a player and sees builds outside it, as on Linuxto fix(queue): on Windows the job queue waits for a player and sees builds outside it, as on Linux64d0afbc6ef8b145252eFrom the code and the evidence, this looks ready.
I read the Windows process reads against their structures.
PROCESSENTRY32W,PROCESS_BASIC_INFORMATION(six pointers, the PEB at [1]),PEB.ProcessParametersat 0x20, andRTL_USER_PROCESS_PARAMETERS(CommandLineat 0x70/0x78,Environmentat 0x80,EnvironmentSizeat 0x3F0) are all right for x64. The read refuses WOW64 targets and a 32-bit Python. Nothing new is installed (ctypes only) and no new command runs:CommandLineToArgvWonly parses, and the command line it parses is never executed. The PR asks only forPROCESS_QUERY_LIMITED_INFORMATION | PROCESS_VM_READ. That fails for another user's process or an elevated one, so such a game reads asjob: None, which never counts as a player, the same as on Linux. Only processes whose program is inLONGSHOT_QUEUE_GAMES, and pythons, have their memory read. The Builder (LongshotBuilder.exe) and the launcher are not in the games list, so they are never mistaken for a player. A build'slongshot.exe --prepare-shadersstill counts asprepare. On Linux,game_procs,heavy_procsandtreeread the same/procfiles as before. The only changes there are sorted pid order, a case-insensitivepythonmatch and backslash-tolerant script paths, and none of these changes a result in practice.Suggestion (not blocking):
hookshotandlichiiare never seen on Windows. Inscripts/job_queue.py:592, the Windows pre-filter compares the program's full file name with the list. The default list islongshot,longshot.exe,hookshot,lichii, so on Windows onlylongshot.execan match:hookshot.exeorlichii.exerunning there doesn't hold a timed run, while on Linux they do. The same goes for aLONGSHOT_QUEUE_GAMES=longshotsomeone sets. Also accepting the name without.exewould fix both, for examplen = exe.lower(); if WINDOWS and n not in names and n.removesuffix(".exe") not in names: continue. Thehitlookup below it would need the same.Measurements: this is a tooling fix with no game scene, so the evidence is the tests. The new
WindowsPlayerscases run the Windows path over a made-up process table on any OS and would fail without the fix: on main,game_procsandtreeread/proc, anduser_playingreturns early on Windows. The author reports 7 of 7 failing on main and passing here on Windows 11, and 5 of 5 on Linux. They also broke the fix on purpose and showed the right cases fail.WindowsPlayersRealexercises the real Toolhelp and PEB reads on Windows (a copiedping.exenamedlongshot.exe, with and withoutLONGSHOT_JOB). That covers what this changes. A real game started from the launcher while a timed job waits wasn't run, which the PR says. It would be a nice confirmation but isn't needed to merge.@ -418,2 +591,2 @@except OSError:exe = ""for pid, exe in sorted(exes().items()):if WINDOWS and exe.lower() not in names: # (Windows: the program's name; reading every process's memory costs)On Windows this compares the full file name, so the default list's
hookshotandlichii(and aLONGSHOT_QUEUE_GAMES=longshotsomeone sets) never matchhookshot.exeorlichii.exe. Also acceptingexe.lower().removesuffix(".exe")would keep the Windows side the same as Linux.@ -452,3 +622,2 @@out = []for pid in all_pids():if pid in skip:for pid, exe in sorted(exes().items()):Nit: on Linux,
exes()now readlinks/proc/<pid>/exefor every process on each call (every 2 s in_keep_apartwhile a timed run measures), andheavy_procsnever uses the result there.all_pids()off Windows would save that, e.g.for pid, exe in (sorted(exes().items()) if WINDOWS else ((p, "") for p in all_pids())):.