WIP: fix(dev): dev.py ab writes its results up again: the control bridge's socket is no setting of the run #2186
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2186
Loadingβ¦
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:fix/ab-run-only-control"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
dev.py abmeasures a series and then refuses to write it up, every time:no write-up: switches HOS_CONTROL: A2 has 'longshot-control-5d72β¦', A1 'longshot-control-1b71β¦'. The numbers are left in the log, with no results table. Each game run starts the control bridge's supervisor, which setsHOS_CONTROLto a socket name of its own, random each run. The comparison of what the runs had set took that name for a switch the runs differ in.HOS_CONTROLnow joinsRUN_ONLY, beside the build's name and the queue's job switches, which are set for each run and never compared.HOS_CONTROL_MEMandHOS_CONTROL_PEER_SIDchange what a run can do, so they are still compared.Read first:
RUN_ONLYinscripts/devtools/ab.py, and the case intests/test_dev_ab.py.Testing
Tested: not run yet. Found from two
dev.py ab --runs 2series for #2185 on Windows 11, both refused that way.tests/test_dev_ab.py's per-run case now sets a differentHOS_CONTROLon each run and expects no difference; a differentHOS_CONTROL_MEMis still one. Not run yet.Still to test:
dev.py ab --runs 2on any change, expecting the write-up at the end instead ofno write-up(by hand)Measurements
Nothing to measure: the write-up's comparison, not a run.
Risks
A run's control socket can no longer make two runs differ, and nothing else could. Reverting is enough.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). Not run yet.main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.π€ Generated with Claude Code
Possibly related, for context:
The root cause and the red/green are right, but the fix drops the whole
HOS_CONTROLkey, so it also hides the bridge's on/off: one run with the control bridge and one without now compare equal. That is the one way this can still produce a false pass.Root cause / test.
log_factsreadsHOS_CONTROL=<socket>off thevk: switches setline;what_rancompared every switch not inRUN_ONLY; the supervisor gives each run its own random socket, sodifferencesrefused every write-up. The two-line fix is in the right function. I ran the test: on the headtests/test_dev_ab.pyis 33 tests OK; ona2dc5385with only the head'stests/test_dev_ab.pyit is 33 run, 1 failure, exactlytest_run_only_switches_not_comparedwithswitches HOS_CONTROL: A2 has 'longshot-control-5d72β¦', A1 'longshot-control-1b71β¦'. That is the bug's reason, not an incidental failure.Strongest finding: the presence is masked too.
RUN_ONLY = β¦|HOS_CONTROL$removes the key from both runs, andwhat_ranbuildsswitchesonly from keys the run's line had, so set-versus-unset disappears as well. I checked with the PR's ownwhat_ran:switches.defsaysHOS_CONTROL's unset is "off: no control bridge", set "Starts the control bridge on the given socket path": the presence is a run setting (a supervisor, socket and thread), and a run where the bridge was off or failed to start would now pass the comparison even though it did not run the same thing. The fix should keep the key and normalize only the value (e.g.HOS_CONTROLβ a constant such as"set"), so two run-specific socket names compare equal but bridge-on against bridge-off still differs β or keepRUN_ONLYand add an explicit presence check. The test only sets two different values on both runs; add one run withHOS_CONTROLand one without and expect a difference. The Risks line "nothing else could" is not true for the presence.Scope. The regex is anchored, so
HOS_CONTROL_MEMandHOS_CONTROL_PEER_SIDstay compared (I verified both; the test only checksMEMβ addPEER_SID, or a loop over both).RUN_ONLYis used only inwhat_ran; no other reader changes. The value is the supervisor's random socket, and dropping the value is right; the key is the part to keep.Existing A/B folders/readers. The format is unchanged:
plan.json,results.json,measurements.md, and theresults.jsonshape (runs,ran,problems,verdict) are the same; the newran.switchessimply has noHOS_CONTROL. Old refused folders have noresults.json/measurements.md(the tool returned 3, the numbers are in the log), so they still need a re-run; old successful folders still parse, and theswitches_line/log_linesstill show the actual socket. There is no repo reader that re-runsdifferenceson a storedresults.json(onlyab.py/the test/docs), so nothing in-tree re-flags an oldHOS_CONTROL; if a user has such a reader,RUN_ONLYis applied inwhat_ranat collection, not indifferences, so it would still see the old key β apply the same normalization on load if old folders should be re-analyzable.Does it mask the real result? Not for
version,renderer,window,options, or any other switch;HOS_CONTROL_MEM/PEER_SIDstay; thedifferences/verdict/exit-code logic is untouched. The only masked thing is theHOS_CONTROLpresence, above.What remains. The PR's own Still to test β a real
dev.py ab --runs 2write-up β has not run (the checklist says so); the fullmise run testhas not run; and the presence case is untested. I ran only the game-free test; the fix is logic-only and the red/green is genuine.No approval, no claim.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.