WIP: fix(dev): dev.py ab writes its results up again: the control bridge's socket is no setting of the run #2186

Draft
dyst wants to merge 1 commit from dyst/Longshot:fix/ab-run-only-control into main
Member

Summary

dev.py ab measures a series and then refuses to write it up, every time: no write-up: switches HOS_CONTROL: A2 has 'longshot-control-5d72…', A1 'longshot-control-1b71…'. The numbers are left in the log, with no results table. Each game run starts the control bridge's supervisor, which sets HOS_CONTROL to a socket name of its own, random each run. The comparison of what the runs had set took that name for a switch the runs differ in.

HOS_CONTROL now joins RUN_ONLY, beside the build's name and the queue's job switches, which are set for each run and never compared. HOS_CONTROL_MEM and HOS_CONTROL_PEER_SID change what a run can do, so they are still compared.

Read first: RUN_ONLY in scripts/devtools/ab.py, and the case in tests/test_dev_ab.py.

Testing

Tested: not run yet. Found from two dev.py ab --runs 2 series for #2185 on Windows 11, both refused that way.

  • Checks: tests/test_dev_ab.py's per-run case now sets a different HOS_CONTROL on each run and expects no difference; a different HOS_CONTROL_MEM is still one. Not run yet.

Still to test:

  • Windows, any GPU: dev.py ab --runs 2 on any change, expecting the write-up at the end instead of no write-up (by hand)

Measurements

Nothing to measure: the write-up's comparison, not a run.

Risks

A run's control socket can no longer make two runs differ, and nothing else could. Reverting is enough.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. Not run yet.
  • mise run test passes (any failure that also fails on main is named). Not run yet.
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

πŸ€– Generated with Claude Code

## Summary `dev.py ab` measures a series and then refuses to write it up, every time: `no write-up: switches HOS_CONTROL: A2 has 'longshot-control-5d72…', A1 'longshot-control-1b71…'`. The numbers are left in the log, with no results table. Each game run starts the control bridge's supervisor, which sets `HOS_CONTROL` to a socket name of its own, random each run. The comparison of what the runs had set took that name for a switch the runs differ in. `HOS_CONTROL` now joins `RUN_ONLY`, beside the build's name and the queue's job switches, which are set for each run and never compared. `HOS_CONTROL_MEM` and `HOS_CONTROL_PEER_SID` change what a run can do, so they are still compared. Read first: `RUN_ONLY` in `scripts/devtools/ab.py`, and the case in `tests/test_dev_ab.py`. ## Testing **Tested:** not run yet. Found from two `dev.py ab --runs 2` series for #2185 on Windows 11, both refused that way. - **Checks:** `tests/test_dev_ab.py`'s per-run case now sets a different `HOS_CONTROL` on each run and expects no difference; a different `HOS_CONTROL_MEM` is still one. Not run yet. **Still to test:** - Windows, any GPU: `dev.py ab --runs 2` on any change, expecting the write-up at the end instead of `no write-up` (by hand) ## Measurements Nothing to measure: the write-up's comparison, not a run. ## Risks A run's control socket can no longer make two runs differ, and nothing else could. Reverting is enough. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. Not run yet. - [ ] `mise run test` passes (any failure that also fails on `main` is named). Not run yet. - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. πŸ€– Generated with [Claude Code](https://claude.com/claude-code)
fix(dev): dev.py ab writes its results up again: the control bridge's socket is no setting of the run
Some checks failed
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Website build and checks (pull_request) Failing after 17s
CI / Conventional Commit title (pull_request) Successful in 29s
PR description / Description follows the template (pull_request_target) Successful in 32s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 44s
CI / Workflow lint (actionlint) (pull_request) Successful in 47s
Third-party parts / As their notes say (pull_request) Successful in 54s
49911f2bb9
Each game run of dev.py ab starts the control bridge's supervisor, which sets HOS_CONTROL to a socket name of its own
(longshot-control-<random>). The comparison of what each run had set took that name for a switch the runs differ in,
so every series ended "no write-up: switches HOS_CONTROL: A2 has ..., A1 ...", the numbers left in the log. HOS_CONTROL
joins RUN_ONLY, the names the queue and the build set for each run; HOS_CONTROL_MEM and HOS_CONTROL_PEER_SID, which change
what a run can do, are still compared (tests/test_dev_ab.py).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #1726 feat(host): HOS_NATIVE_ZSTD in the in-run A/B β€” Open in-run A/B work touches neighboring dev A/B behavior and tests
  • PR #1802 refactor(dev): docs, tools and process revamp: one dev.py, dev.py ab, one dev folder, a short PR template, faster dev builds β€” Introduced dev.py ab; background for the affected write-up code
Possibly related, for context: - PR #1726 feat(host): HOS_NATIVE_ZSTD in the in-run A/B β€” Open in-run A/B work touches neighboring dev A/B behavior and tests - PR #1802 refactor(dev): docs, tools and process revamp: one dev.py, dev.py ab, one dev folder, a short PR template, faster dev builds β€” Introduced dev.py ab; background for the affected write-up code
dazemc left a comment

The root cause and the red/green are right, but the fix drops the whole HOS_CONTROL key, so it also hides the bridge's on/off: one run with the control bridge and one without now compare equal. That is the one way this can still produce a false pass.

Root cause / test. log_facts reads HOS_CONTROL=<socket> off the vk: switches set line; what_ran compared every switch not in RUN_ONLY; the supervisor gives each run its own random socket, so differences refused every write-up. The two-line fix is in the right function. I ran the test: on the head tests/test_dev_ab.py is 33 tests OK; on a2dc5385 with only the head's tests/test_dev_ab.py it is 33 run, 1 failure, exactly test_run_only_switches_not_compared with switches HOS_CONTROL: A2 has 'longshot-control-5d72…', A1 'longshot-control-1b71…'. That is the bug's reason, not an incidental failure.

Strongest finding: the presence is masked too. RUN_ONLY = …|HOS_CONTROL$ removes the key from both runs, and what_ran builds switches only from keys the run's line had, so set-versus-unset disappears as well. I checked with the PR's own what_ran:

HOS_CONTROL set vs unset -> differences []
HOS_CONTROL_PEER_SID differs -> 1
HOS_CONTROL_MEM differs -> 1

switches.def says HOS_CONTROL's unset is "off: no control bridge", set "Starts the control bridge on the given socket path": the presence is a run setting (a supervisor, socket and thread), and a run where the bridge was off or failed to start would now pass the comparison even though it did not run the same thing. The fix should keep the key and normalize only the value (e.g. HOS_CONTROL β†’ a constant such as "set"), so two run-specific socket names compare equal but bridge-on against bridge-off still differs β€” or keep RUN_ONLY and add an explicit presence check. The test only sets two different values on both runs; add one run with HOS_CONTROL and one without and expect a difference. The Risks line "nothing else could" is not true for the presence.

Scope. The regex is anchored, so HOS_CONTROL_MEM and HOS_CONTROL_PEER_SID stay compared (I verified both; the test only checks MEM β€” add PEER_SID, or a loop over both). RUN_ONLY is used only in what_ran; no other reader changes. The value is the supervisor's random socket, and dropping the value is right; the key is the part to keep.

Existing A/B folders/readers. The format is unchanged: plan.json, results.json, measurements.md, and the results.json shape (runs, ran, problems, verdict) are the same; the new ran.switches simply has no HOS_CONTROL. Old refused folders have no results.json/measurements.md (the tool returned 3, the numbers are in the log), so they still need a re-run; old successful folders still parse, and the switches_line/log_lines still show the actual socket. There is no repo reader that re-runs differences on a stored results.json (only ab.py/the test/docs), so nothing in-tree re-flags an old HOS_CONTROL; if a user has such a reader, RUN_ONLY is applied in what_ran at collection, not in differences, so it would still see the old key β€” apply the same normalization on load if old folders should be re-analyzable.

Does it mask the real result? Not for version, renderer, window, options, or any other switch; HOS_CONTROL_MEM/PEER_SID stay; the differences/verdict/exit-code logic is untouched. The only masked thing is the HOS_CONTROL presence, above.

What remains. The PR's own Still to test β€” a real dev.py ab --runs 2 write-up β€” has not run (the checklist says so); the full mise run test has not run; and the presence case is untested. I ran only the game-free test; the fix is logic-only and the red/green is genuine.

No approval, no claim.

**The root cause and the red/green are right, but the fix drops the whole `HOS_CONTROL` key, so it also hides the bridge's on/off: one run with the control bridge and one without now compare equal. That is the one way this can still produce a false pass.** **Root cause / test.** `log_facts` reads `HOS_CONTROL=<socket>` off the `vk: switches set` line; `what_ran` compared every switch not in `RUN_ONLY`; the supervisor gives each run its own random socket, so `differences` refused every write-up. The two-line fix is in the right function. I ran the test: on the head `tests/test_dev_ab.py` is **33 tests OK**; on `a2dc5385` with only the head's `tests/test_dev_ab.py` it is **33 run, 1 failure**, exactly `test_run_only_switches_not_compared` with `switches HOS_CONTROL: A2 has 'longshot-control-5d72…', A1 'longshot-control-1b71…'`. That is the bug's reason, not an incidental failure. **Strongest finding: the presence is masked too.** `RUN_ONLY = …|HOS_CONTROL$` removes the key from both runs, and `what_ran` builds `switches` only from keys the run's line had, so set-versus-unset disappears as well. I checked with the PR's own `what_ran`: ``` HOS_CONTROL set vs unset -> differences [] HOS_CONTROL_PEER_SID differs -> 1 HOS_CONTROL_MEM differs -> 1 ``` `switches.def` says `HOS_CONTROL`'s unset is "off: no control bridge", set "Starts the control bridge on the given socket path": the presence is a run setting (a supervisor, socket and thread), and a run where the bridge was off or failed to start would now pass the comparison even though it did not run the same thing. The fix should keep the key and normalize only the value (e.g. `HOS_CONTROL` β†’ a constant such as `"set"`), so two run-specific socket names compare equal but bridge-on against bridge-off still differs β€” or keep `RUN_ONLY` and add an explicit presence check. The test only sets two different values on both runs; add one run with `HOS_CONTROL` and one without and expect a difference. The Risks line "nothing else could" is not true for the presence. **Scope.** The regex is anchored, so `HOS_CONTROL_MEM` and `HOS_CONTROL_PEER_SID` stay compared (I verified both; the test only checks `MEM` β€” add `PEER_SID`, or a loop over both). `RUN_ONLY` is used only in `what_ran`; no other reader changes. The value is the supervisor's random socket, and dropping the value is right; the key is the part to keep. **Existing A/B folders/readers.** The format is unchanged: `plan.json`, `results.json`, `measurements.md`, and the `results.json` shape (`runs`, `ran`, `problems`, `verdict`) are the same; the new `ran.switches` simply has no `HOS_CONTROL`. Old **refused** folders have no `results.json`/`measurements.md` (the tool returned 3, the numbers are in the log), so they still need a re-run; old **successful** folders still parse, and the `switches_line`/`log_lines` still show the actual socket. There is no repo reader that re-runs `differences` on a stored `results.json` (only `ab.py`/the test/docs), so nothing in-tree re-flags an old `HOS_CONTROL`; if a user has such a reader, `RUN_ONLY` is applied in `what_ran` at collection, not in `differences`, so it would still see the old key β€” apply the same normalization on load if old folders should be re-analyzable. **Does it mask the real result?** Not for `version`, `renderer`, `window`, `options`, or any other switch; `HOS_CONTROL_MEM`/`PEER_SID` stay; the `differences`/`verdict`/exit-code logic is untouched. The only masked thing is the `HOS_CONTROL` presence, above. **What remains.** The PR's own Still to test β€” a real `dev.py ab --runs 2` write-up β€” has not run (the checklist says so); the full `mise run test` has not run; and the presence case is untested. I ran only the game-free test; the fix is logic-only and the red/green is genuine. No approval, no claim.
Some checks failed
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Website build and checks (pull_request) Failing after 17s
CI / Conventional Commit title (pull_request) Successful in 29s
PR description / Description follows the template (pull_request_target) Successful in 32s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 44s
CI / Workflow lint (actionlint) (pull_request) Successful in 47s
Third-party parts / As their notes say (pull_request) Successful in 54s
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/ab-run-only-control:dyst-fix/ab-run-only-control'
git switch 'dyst-fix/ab-run-only-control'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2186
No description provided.