fix(docs): keep new test runs out of the frozen table #96

Merged
HumanBean merged 1 commit from mooki/Longshot:fix/tested-rows into main 2026-11-14 01:49:22 +00:00
Member

Why

Independent test results should not make unrelated pull requests fight over one table insertion point. This keeps the existing results readable and gives only new runs independent files. See #45.

What changed

  • docs/tested-configurations.md keeps the existing table in place, frozen and in its original order. There are no numbered legacy files, pinned row count, digest or duplicate whitelist to refresh.
  • docs/tested/README.md documents one three-line file per new run: YYYY-MM-DD-topic-tester-run.md. The tester and a distinct run identifier are required; rows must stand alone.
  • scripts/tested_configurations.py validates new files against each other and the current frozen table, then prints the combined view or writes it as UTF-8. Historical repeats and relative references stay untouched.
  • The website's existing repository-document loader builds /tested-configurations/ from those same local files. It watches new, edited and deleted fragments, resolves their links relative to their own directory, and uses the site's existing public-prose rules without changing source rows, code or link destinations. No package or network collection step is added.
  • README and requirements links lead to the combined catalogue. The forge still shows the frozen table and links directly to the new files. Contributor instructions and the PR template name the same destination.
  • The Limits section has one source. The unused --check option, stale-view check and permanent Git-merge test are gone. The description checker itself is unchanged.

Flags and settings

Flag What it does Default now Recommendation Helps on Doesn't help on Untested on
scripts/tested_configurations.py --output FILE New: writes the assembled Markdown view as UTF-8 Without it, prints to stdout; pipes and redirection use UTF-8 Off by default: use it only when a saved view is useful Reading results without the website Game behaviour Native Windows (should help on: Windows)

Documented in the fragment guide and docs/development.md. No game setting changes. Do not commit an assembled view.

Measurements

Branch state: on main 514591a30d707fe9f3ad31eae148c3cc1e750bff; head cfd54df719a46b595697bf417113b1332294db2b; whole game-free suite on this head: 518 passed, 1 failed, 15 skipped (main 514591a30d707fe9f3ad31eae148c3cc1e750bff: 517 passed, 1 failed, 15 skipped). The 1 failing test fails on main the same way: tests/test_menu_i18n.py. The commits are unchanged by rebases apart from the results table.

main: 1ef93e796b
this PR: move/fix/tested-rows-r5 @ 75c5610ac499cc24055bc5a38581f5552a8e8b93
Hardware: Linux x86-64; no GPU work. Node v26.13.0 and npm 11.20.0 for the website.
Scene: none; scripts, documentation and the static website.
Method: game-free checks, as AGENTS.md specifies for changes the game does not run; row-byte comparison, not a speed claim.
How to reproduce: python -I -B tests/test_tested_configurations.py; python -I -B tests/test_check_pr_description.py; python -I -B tests/test_forge_pr.py. In website/, install the unchanged lock with npm ci --ignore-scripts --no-audit --no-fund, then run npm run build, node scripts/check-site.ts and node --test --test-concurrency=1 tests/*.test.ts.

main @ 1ef93e79 this PR @ 75c5610a
Existing rows in their original order 91 91, row bytes unchanged
Extra historical-row files to maintain 0 0
New source-layout regression, using each commit's source table 1 expected failure: assembly marker absent 1 passed within the fragment checks
Named Python checks exercised Not rerun on this base 25 fragment cases, 22 description cases, 25 Forge-command cases passed
Website build and checks Not rerun on this base Offline build passed; 15 pages and 837 local links/assets checked; all 66 Node cases passed

Later-main carry: a temporary fixture combined this guide and marker with all of fbd420241a's table: 109 table lines, including its header and separator, or 107 result rows. All row bytes and order were retained; Python and TypeScript produced identical Markdown without refreshing anything. A copied newly landed row was then rejected as a duplicate of the frozen table. test_later_main_rows_are_read_without_a_refresh_step keeps that behaviour covered. This proves the carry behaviour, not that this branch has already been rebased.

The website lock is unchanged: SHA-256 b051dfed8bee21bc6eb7e583b729f31d45c0ab8717df4d35eeb40cbb7bd88bf0. Its 385 existing packages installed with lifecycle scripts disabled; no package script was enabled.

Earlier conflict evidence, not a replay of this revision: at main 77f605a25 and the original proposal 06d7c456, converting the 18 pinned incoming heads' inserted rows to separate files reduced table conflicts from 13 to 0; all 153 same-base result-only branch pairs changed from 152 conflicts to 0. The issue's original 14-of-16 observation remains a different snapshot. The old numbered-history migration and its counts are not this revision's implementation.

Forgejo 17.0.1's bare-repository conflict check ignored the committed union attribute in the recorded command reproduction. Its sparse merge preparation also conflicted for merge, squash and rebase; the shared non-mergeable status is refused before those styles run. These are historical local command experiments, not a new live server test or an inspection of its private configuration.

Testing

Tested: on Linux, the named Python checks above, including malformed headers and rows, duplicates, dates, filename parts, escaped pipes, ordering, single-source Limits, UTF-8 file output and a forced-ASCII redirected stream. The ASCII stream is a stand-in, not native Windows. The original 91 rows remain byte-identical and in order. The source-layout regression fails as expected on the pinned main table's missing assembly marker, and passes here; this is one negative-control case, not a run of main's whole suite.

The offline website build and existing page checker passed; all 66 Node cases passed serially. Network access was denied: the existing progress and contributor collectors showed their documented unavailable state, not invented empty results. The built catalogue contains all 91 rows; both readers validate the eight-column source format. Chromium inspection at 1600×1000 and an emulated 390×844 viewport found no page-wide overflow; the wide table scrolls within its own container. The installed Unified processor is configured through Astro's supported markdown.processor API; no deprecated plugin setting is retained.

Not tested: the final-head whole repository suite, native Windows execution, live collector data, a new live Forgejo merge, a physical phone browser, game builds, game runs or frame rates. The narrow viewport is browser emulation, not mobile-hardware testing. No game behaviour changes.

Needs other testers:

  • Windows: run the fragment tests, redirect the default command to a file and use --output with a non-ASCII filename; expect UTF-8 arrows and unchanged row text, without setting PYTHONUTF8 (T-AG65)

Screenshots

Nothing changes on screen in the game. Main @ 1ef93e79 has no combined catalogue page. The website's new page with this PR @ 75c5610a, built offline:

This PR @ 75c5610a: the top of the website's Tested configurations page: where new run files go, the empty New runs list and the start of the frozen table.

This PR @ 75c5610a: the frozen table scrolled sideways to its Where and Result columns.

This PR @ 75c5610a: the same page at a phone's width (390 pixels).

Risk and rollback

There are more small files for new runs. Existing open branches that append to the table need their new rows moved to unique files or kept in their PR description until that is done. Old rows are not moved or renumbered. A later main's well-formed rows become part of the frozen table without refreshing a count, digest or generated page; this does not compare the table to Git at runtime or enforce the freeze against an arbitrary manual edit.

The two readers implement the same small format in Python and TypeScript, with tests for each; they must stay in sync if that format changes. Duplicate checks cannot establish that a measurement is true. Website prose follows its existing naming policy; the exact source evidence remains on the forge.

Reverting the commit restores the old workflow. If new fragments have landed, first carry their rows back into the table so evidence is not lost. Do not add a committed combined view: that would restore the shared insertion point.

Reviewer notes

Start with the frozen source page, scripts/tested_configurations.py, and the existing website loader's new catalogue path. This follows the review's simpler-fragments direction, rather than copying all historical rows into files.

Mechanism Trade-off
Keep the table and move rows at merge Contributors can keep results in descriptions, but a maintainer must collect and insert them; no independent result-file history
Union attribute Small in a full checkout, but the recorded bare/sparse Forgejo commands ignored it; the working-tree experiment also silently retained contradictory rows and headers
Dated headings Still one shared file; the earlier same-base experiment had 137 conflicting pairs out of 153
One file per new run, frozen history, website assembly (chosen) Avoids the shared insertion point without renumbering history; adds a small local-file reader to the website
Copy every old row to a numbered archive Unnecessary file churn and stale counts/digests whenever main gains rows; removed in this revision
Retire the catalogue No table to maintain, but readers must find results in individual PRs; not the direction requested in review

The test request T-AG65 replaces T-RKUJ, whose steps used the removed --check. No server configuration change or live test merge is part of this branch.

Issues

See #45

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main.
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
  • Tested in the game, or what wasn't is named under Testing.
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. For these scripts and docs, AGENTS.md specifies game-free checks rather than hardware-timing runs; the checks and pinned row comparison are above.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Flags noted, and the switch reference updated (docs/development.md, the README).
  • Screenshots for a change that shows on screen.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.

New on this forge; not opened on GitHub.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra [email protected]

## Why Independent test results should not make unrelated pull requests fight over one table insertion point. This keeps the existing results readable and gives only new runs independent files. See #45. ## What changed - `docs/tested-configurations.md` keeps the existing table in place, frozen and in its original order. There are no numbered legacy files, pinned row count, digest or duplicate whitelist to refresh. - `docs/tested/README.md` documents one three-line file per new run: `YYYY-MM-DD-topic-tester-run.md`. The tester and a distinct run identifier are required; rows must stand alone. - `scripts/tested_configurations.py` validates new files against each other and the current frozen table, then prints the combined view or writes it as UTF-8. Historical repeats and relative references stay untouched. - The website's existing repository-document loader builds `/tested-configurations/` from those same local files. It watches new, edited and deleted fragments, resolves their links relative to their own directory, and uses the site's existing public-prose rules without changing source rows, code or link destinations. No package or network collection step is added. - README and requirements links lead to the combined catalogue. The forge still shows the frozen table and links directly to the new files. Contributor instructions and the PR template name the same destination. - The Limits section has one source. The unused `--check` option, stale-view check and permanent Git-merge test are gone. The description checker itself is unchanged. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `scripts/tested_configurations.py --output FILE` | New: writes the assembled Markdown view as UTF-8 | Without it, prints to stdout; pipes and redirection use UTF-8 | Off by default: use it only when a saved view is useful | Reading results without the website | Game behaviour | Native Windows (should help on: Windows) | Documented in the fragment guide and `docs/development.md`. No game setting changes. Do not commit an assembled view. ## Measurements <!-- branch-state --> Branch state: on main `514591a30d707fe9f3ad31eae148c3cc1e750bff`; head `cfd54df719a46b595697bf417113b1332294db2b`; whole game-free suite on this head: `518 passed, 1 failed, 15 skipped` (main `514591a30d707fe9f3ad31eae148c3cc1e750bff`: `517 passed, 1 failed, 15 skipped`). The 1 failing test fails on main the same way: `tests/test_menu_i18n.py`. The commits are unchanged by rebases apart from the results table. <!-- /branch-state --> main: 1ef93e796bb0ec5cb95303e0096b30099b9887d8 this PR: move/fix/tested-rows-r5 @ 75c5610ac499cc24055bc5a38581f5552a8e8b93 Hardware: Linux x86-64; no GPU work. Node v26.13.0 and npm 11.20.0 for the website. Scene: none; scripts, documentation and the static website. Method: game-free checks, as AGENTS.md specifies for changes the game does not run; row-byte comparison, not a speed claim. How to reproduce: `python -I -B tests/test_tested_configurations.py`; `python -I -B tests/test_check_pr_description.py`; `python -I -B tests/test_forge_pr.py`. In `website/`, install the unchanged lock with `npm ci --ignore-scripts --no-audit --no-fund`, then run `npm run build`, `node scripts/check-site.ts` and `node --test --test-concurrency=1 tests/*.test.ts`. | | main @ 1ef93e79 | this PR @ 75c5610a | |---|---|---| | Existing rows in their original order | 91 | 91, row bytes unchanged | | Extra historical-row files to maintain | 0 | 0 | | New source-layout regression, using each commit's source table | 1 expected failure: assembly marker absent | 1 passed within the fragment checks | | Named Python checks exercised | Not rerun on this base | 25 fragment cases, 22 description cases, 25 Forge-command cases passed | | Website build and checks | Not rerun on this base | Offline build passed; 15 pages and 837 local links/assets checked; all 66 Node cases passed | **Later-main carry:** a temporary fixture combined this guide and marker with all of `fbd420241a`'s table: 109 table lines, including its header and separator, or 107 result rows. All row bytes and order were retained; Python and TypeScript produced identical Markdown without refreshing anything. A copied newly landed row was then rejected as a duplicate of the frozen table. `test_later_main_rows_are_read_without_a_refresh_step` keeps that behaviour covered. This proves the carry behaviour, not that this branch has already been rebased. The website lock is unchanged: SHA-256 `b051dfed8bee21bc6eb7e583b729f31d45c0ab8717df4d35eeb40cbb7bd88bf0`. Its 385 existing packages installed with lifecycle scripts disabled; no package script was enabled. **Earlier conflict evidence, not a replay of this revision:** at main `77f605a25` and the original proposal `06d7c456`, converting the 18 pinned incoming heads' inserted rows to separate files reduced table conflicts from 13 to 0; all 153 same-base result-only branch pairs changed from 152 conflicts to 0. The issue's original 14-of-16 observation remains a different snapshot. The old numbered-history migration and its counts are not this revision's implementation. Forgejo 17.0.1's [bare-repository conflict check](https://codeberg.org/forgejo/forgejo/src/tag/v17.0.1/services/pull/patch.go) ignored the committed union attribute in the recorded command reproduction. Its [sparse merge preparation](https://codeberg.org/forgejo/forgejo/src/tag/v17.0.1/services/pull/merge_prepare.go) also conflicted for merge, squash and rebase; the shared non-mergeable status is refused [before those styles run](https://codeberg.org/forgejo/forgejo/src/tag/v17.0.1/services/pull/check.go). These are historical local command experiments, not a new live server test or an inspection of its private configuration. ## Testing **Tested:** on Linux, the named Python checks above, including malformed headers and rows, duplicates, dates, filename parts, escaped pipes, ordering, single-source Limits, UTF-8 file output and a forced-ASCII redirected stream. The ASCII stream is a stand-in, not native Windows. The original 91 rows remain byte-identical and in order. The source-layout regression fails as expected on the pinned main table's missing assembly marker, and passes here; this is one negative-control case, not a run of main's whole suite. The offline website build and existing page checker passed; all 66 Node cases passed serially. Network access was denied: the existing progress and contributor collectors showed their documented unavailable state, not invented empty results. The built catalogue contains all 91 rows; both readers validate the eight-column source format. Chromium inspection at 1600×1000 and an emulated 390×844 viewport found no page-wide overflow; the wide table scrolls within its own container. The installed Unified processor is configured through Astro's supported `markdown.processor` API; no deprecated plugin setting is retained. **Not tested:** the final-head whole repository suite, native Windows execution, live collector data, a new live Forgejo merge, a physical phone browser, game builds, game runs or frame rates. The narrow viewport is browser emulation, not mobile-hardware testing. No game behaviour changes. **Needs other testers:** - Windows: run the fragment tests, redirect the default command to a file and use `--output` with a non-ASCII filename; expect UTF-8 arrows and unchanged row text, without setting `PYTHONUTF8` (T-AG65) ## Screenshots Nothing changes on screen in the game. Main @ `1ef93e79` has no combined catalogue page. The website's new page with this PR @ `75c5610a`, built offline: ![This PR @ 75c5610a: the top of the website's Tested configurations page: where new run files go, the empty New runs list and the start of the frozen table.](https://git.magicbean.dev/attachments/7e62911d-81c7-4d69-b4c9-2eb3902d22eb) ![This PR @ 75c5610a: the frozen table scrolled sideways to its Where and Result columns.](https://git.magicbean.dev/attachments/820ae97d-7aea-4546-a16e-5d7cc1a3ae64) ![This PR @ 75c5610a: the same page at a phone's width (390 pixels).](https://git.magicbean.dev/attachments/16305750-47ef-4376-9eb1-21218245c333) ## Risk and rollback There are more small files for new runs. Existing open branches that append to the table need their new rows moved to unique files or kept in their PR description until that is done. Old rows are not moved or renumbered. A later main's well-formed rows become part of the frozen table without refreshing a count, digest or generated page; this does not compare the table to Git at runtime or enforce the freeze against an arbitrary manual edit. The two readers implement the same small format in Python and TypeScript, with tests for each; they must stay in sync if that format changes. Duplicate checks cannot establish that a measurement is true. Website prose follows its existing naming policy; the exact source evidence remains on the forge. Reverting the commit restores the old workflow. If new fragments have landed, first carry their rows back into the table so evidence is not lost. Do not add a committed combined view: that would restore the shared insertion point. ## Reviewer notes Start with the frozen source page, `scripts/tested_configurations.py`, and the existing website loader's new catalogue path. This follows the review's simpler-fragments direction, rather than copying all historical rows into files. | Mechanism | Trade-off | |---|---| | Keep the table and move rows at merge | Contributors can keep results in descriptions, but a maintainer must collect and insert them; no independent result-file history | | Union attribute | Small in a full checkout, but the recorded bare/sparse Forgejo commands ignored it; the working-tree experiment also silently retained contradictory rows and headers | | Dated headings | Still one shared file; the earlier same-base experiment had 137 conflicting pairs out of 153 | | One file per new run, frozen history, website assembly (chosen) | Avoids the shared insertion point without renumbering history; adds a small local-file reader to the website | | Copy every old row to a numbered archive | Unnecessary file churn and stale counts/digests whenever main gains rows; removed in this revision | | Retire the catalogue | No table to maintain, but readers must find results in individual PRs; not the direction requested in review | The test request T-AG65 replaces T-RKUJ, whose steps used the removed `--check`. No server configuration change or live test merge is part of this branch. ## Issues See #45 ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. For these scripts and docs, AGENTS.md specifies game-free checks rather than hardware-timing runs; the checks and pinned row comparison are above. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] Screenshots for a change that shows on screen. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. --- New on this forge; not opened on GitHub. Assisted-by: GPT-6 Astra Co-Authored-By: GPT-6 Astra <[email protected]>
fix(docs): give tested runs independent result files
Some checks failed
Linux AppImage / appimage (pull_request) Blocked by required conditions
Windows builder / Longshot Builder zip (pull_request) Blocked by required conditions
Windows builder / Windows Vulkan back end (llvm-mingw) (pull_request) Blocked by required conditions
Windows builder / dav1d for Windows (llvm-mingw) (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / Python source checks (windows-2022, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (windows-2022, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
06d7c45637
Keep one validated Markdown fragment per run and assemble the reading view without committing it. Preserve every existing row and its order; retain known historical duplicates explicitly. See #45.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
HumanBean left a comment

The direction is right, and the evidence in the description and on issue 45 is thorough. The branch isn't ready to merge yet, though: it's based on an old main, and the page that readers reach loses its data.

Must fix

  • Stale base: 8 of current main's 90 rows are missing. The PR's base is 77f605a2. Main (839e85ff) now has 90 rows. All 82 migrated rows match main byte for byte, but the 8 rows added since then are not in docs/tested/. They sit at the top of the table and 6 of them say as above. Because the archive is numbered newest-first (0001 is the top row), carrying them over as legacy renames all 82 files and changes the digest, the range(1, 83) check and the 82 in the test and the docs. Carrying them over as new fragments means rewriting their as above text. The Rebased on current main box is ticked but isn't true (Forgejo reports mergeable: false).
  • Readers lose the table. website/src/content/docs/requirements.mdx ("Compare tested configurations"), README.md ("estimates from the tested configurations") and CONTRIBUTING.md line 70 ("lists the runs behind it") all send people to docs/tested-configurations.md. After this PR that page has no runs: just two folder links and a Python command. Players arriving from the website can't run that command.

Should fix

  • scripts/tested_configurations.py writes the view to stdout with the locale encoding. The archive contains →, so on Windows a redirected or piped run (> file, | more) fails with an encoding error. -I also ignores PYTHONUTF8. Reconfigure stdout to UTF-8 the way tests/run_all.py and scripts/team_chat.py already do.
  • The "Limits of the original RTX 4070 Ti runs" text is now in two places: the doc page and a string in the script. The two will drift apart.
  • --check has no user: the view is never committed, so nothing is ever stale. Dropping it, together with its flag-table row and its test, makes the change smaller.
  • The Merging test checks Git's own behaviour (distinct files merge, the same line conflicts), not anything in this repo. It's fine as evidence in the description, but it isn't needed as a permanent check.
  • Minor: NAME accepts 2026-11-13-x.md, while the docs ask for topic-tester-run. The filename is the only guard against two people picking the same name. A name that clashes gives an add/add conflict, which is visible, so this is acceptable. Still, the docs should say the tester's name is required, or the regex should require at least three parts.
  • Measured: AGENTS.md says a change the game doesn't run "is measured by its game-free checks", so this box can be ticked with that reason once the branch is rebased and the checks are re-run.

On the decision in issue 45

My recommendation is fragments, but simpler than this PR:

  • Union merge is ruled out. The live experiment on Forgejo 17.0.1 recorded in the issue showed that the forge still marks the PR unmergeable. Union only helps a local rebase.
  • Rows in PR descriptions only would drop the catalogue that the README status table and the website's requirements page are built on, which is a loss for players.
  • Fragments remove the shared insertion point, as the replay shows.

To make fragments cheaper and keep readers served:

  1. Leave the existing rows where they are, as a frozen table on docs/tested-configurations.md. Issue 45 itself proposed this ("The existing table can stay where it is; only new rows need the new path"). That avoids the 82-file archive, the hard-coded digest and duplicate list, and the renumbering on every rebase.
  2. Put only new runs in docs/tested/YYYY-MM-DD-topic-tester-run.md, checked by the script as now.
  3. Assemble the combined table where readers already look, for example in the website build behind the requirements page's link, rather than leaving it to a local command.

If the maintainers want the archive split into files anyway, then generate the migration with a script at rebase time rather than by hand, so that re-basing onto a moving main is mechanical.

The checks look sound and match the project's rules: standard-library only, safe under python -I, deterministic sort by date then name, escaped pipes handled, duplicates refused, and the template, good.md, AGENTS.md, CONTRIBUTING.md and workflow.md all point at the same place. check_pr_description.py reads the template's boxes, so the changed box carries through to it and to forge.py pr create without further edits.

The direction is right, and the evidence in the description and on [issue 45](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/45) is thorough. The branch isn't ready to merge yet, though: it's based on an old main, and the page that readers reach loses its data. **Must fix** - **Stale base: 8 of current main's 90 rows are missing.** The PR's base is `77f605a2`. Main (`839e85ff`) now has 90 rows. All 82 migrated rows match main byte for byte, but the 8 rows added since then are not in `docs/tested/`. They sit at the top of the table and 6 of them say `as above`. Because the archive is numbered newest-first (0001 is the top row), carrying them over as legacy renames all 82 files and changes the digest, the `range(1, 83)` check and the `82` in the test and the docs. Carrying them over as new fragments means rewriting their `as above` text. The *Rebased on current main* box is ticked but isn't true (Forgejo reports `mergeable: false`). - **Readers lose the table.** `website/src/content/docs/requirements.mdx` ("Compare tested configurations"), `README.md` ("estimates from the tested configurations") and `CONTRIBUTING.md` line 70 ("lists the runs behind it") all send people to `docs/tested-configurations.md`. After this PR that page has no runs: just two folder links and a Python command. Players arriving from the website can't run that command. **Should fix** - `scripts/tested_configurations.py` writes the view to stdout with the locale encoding. The archive contains `→`, so on Windows a redirected or piped run (`> file`, `| more`) fails with an encoding error. `-I` also ignores `PYTHONUTF8`. Reconfigure stdout to UTF-8 the way `tests/run_all.py` and `scripts/team_chat.py` already do. - The "Limits of the original RTX 4070 Ti runs" text is now in two places: the doc page and a string in the script. The two will drift apart. - `--check` has no user: the view is never committed, so nothing is ever stale. Dropping it, together with its flag-table row and its test, makes the change smaller. - The `Merging` test checks Git's own behaviour (distinct files merge, the same line conflicts), not anything in this repo. It's fine as evidence in the description, but it isn't needed as a permanent check. - Minor: `NAME` accepts `2026-11-13-x.md`, while the docs ask for `topic-tester-run`. The filename is the only guard against two people picking the same name. A name that clashes gives an add/add conflict, which is visible, so this is acceptable. Still, the docs should say the tester's name is required, or the regex should require at least three parts. - *Measured*: AGENTS.md says a change the game doesn't run "is measured by its game-free checks", so this box can be ticked with that reason once the branch is rebased and the checks are re-run. **On the decision in issue 45** My recommendation is **fragments, but simpler than this PR**: - **Union merge is ruled out.** The live experiment on Forgejo 17.0.1 recorded in the issue showed that the forge still marks the PR unmergeable. Union only helps a local rebase. - **Rows in PR descriptions only** would drop the catalogue that the README status table and the website's requirements page are built on, which is a loss for players. - **Fragments** remove the shared insertion point, as the replay shows. To make fragments cheaper and keep readers served: 1. Leave the existing rows where they are, as a frozen table on `docs/tested-configurations.md`. Issue 45 itself proposed this ("The existing table can stay where it is; only new rows need the new path"). That avoids the 82-file archive, the hard-coded digest and duplicate list, and the renumbering on every rebase. 2. Put only new runs in `docs/tested/YYYY-MM-DD-topic-tester-run.md`, checked by the script as now. 3. Assemble the combined table where readers already look, for example in the website build behind the requirements page's link, rather than leaving it to a local command. If the maintainers want the archive split into files anyway, then generate the migration with a script at rebase time rather than by hand, so that re-basing onto a moving main is mechanical. The checks look sound and match the project's rules: standard-library only, safe under `python -I`, deterministic sort by date then name, escaped pipes handled, duplicates refused, and the template, `good.md`, AGENTS.md, CONTRIBUTING.md and workflow.md all point at the same place. `check_pr_description.py` reads the template's boxes, so the changed box carries through to it and to `forge.py pr create` without further edits.
@ -12,2 +7,3 @@
## Read the runs
## Runs
Browse [new run files](tested/) and the [migrated runs](tested/legacy/), or assemble one Markdown view:
Owner

The website's requirements page ("Compare tested configurations"), the README and CONTRIBUTING.md line 70 ("lists the runs behind it") all link to this page for the data. After this change a reader gets folder links and a Python command instead. The combined table needs to be somewhere readers can open it, for example built into the website or kept on this page for the existing rows.

The website's requirements page ("Compare tested configurations"), the README and CONTRIBUTING.md line 70 ("lists the runs behind it") all link to this page for the data. After this change a reader gets folder links and a Python command instead. The combined table needs to be somewhere readers can open it, for example built into the website or kept on this page for the existing rows.
@ -99,0 +51,4 @@
## Existing rows and branches
The 82 rows from main `77f605a2` are in `tested/legacy/0001.md` through `0082.md`, byte-for-byte in their original
Owner

Main is at 839e85ff with 90 rows. The 8 added since 77f605a2 sit at the top of the table, and 6 of them use as above. They are missing from docs/tested/. Because 0001 is the newest row, carrying them over as legacy renames all 82 files and changes the digest, the range(1, 83) check and the 82 here and in the test. Keeping the old table frozen on this page, with only new runs as fragments, avoids that every time main moves.

Main is at `839e85ff` with 90 rows. The 8 added since `77f605a2` sit at the top of the table, and 6 of them use `as above`. They are missing from `docs/tested/`. Because 0001 is the newest row, carrying them over as legacy renames all 82 files and changes the digest, the `range(1, 83)` check and the `82` here and in the test. Keeping the old table frozen on this page, with only new runs as fragments, avoids that every time main moves.
@ -0,0 +13,4 @@
ROOT = Path(__file__).resolve().parents[1]
HEADER = "| Date | Build | OS / desktop | GPU / driver | CPU / RAM | Settings | Where | Result |"
SEPARATOR = "|---|---|---|---|---|---|---|---|"
NAME = re.compile(r"(\d{4}-\d{2}-\d{2})-[a-z0-9]+(?:-[a-z0-9]+)*\.md")
Owner

This accepts 2026-11-13-x.md, but the docs ask for topic-tester-run. The filename is the only thing that keeps two people from picking the same name, so either require at least three parts here or say in the docs that the tester's name is required.

This accepts `2026-11-13-x.md`, but the docs ask for `topic-tester-run`. The filename is the only thing that keeps two people from picking the same name, so either require at least three parts here or say in the docs that the tester's name is required.
@ -0,0 +90,4 @@
"\n## Migrated runs (original order)\n\n"
"The three pre-existing duplicate pairs are retained; 'as above' refers only to this archive.\n\n" +
HEADER + "\n" + SEPARATOR + "\n" + "".join(row + "\n" for row in legacy) +
"\n## Limits of the original RTX 4070 Ti runs\n\n"
Owner

This text is also on docs/tested-configurations.md, so the two copies will drift. Read it from the page, or leave it out of the generated view.

This text is also on `docs/tested-configurations.md`, so the two copies will drift. Read it from the page, or leave it out of the generated view.
@ -0,0 +100,4 @@
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
target = parser.add_mutually_exclusive_group()
target.add_argument("--output", type=Path, help="write the assembled view as UTF-8 (do not commit it)")
target.add_argument("--check", type=Path, help="fail if this generated view is missing or out of date; write nothing")
Owner

The view is never committed, so --check never has a stale file to catch. I'd drop it, along with its test and its flag-table row.

The view is never committed, so `--check` never has a stale file to catch. I'd drop it, along with its test and its flag-table row.
@ -0,0 +112,4 @@
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(content, encoding="utf-8", newline="\n")
else:
sys.stdout.write(content)
Owner

The archive contains →. On Windows, when stdout is redirected or piped it uses the ANSI code page, so this line raises an encoding error, and -I ignores PYTHONUTF8. Use sys.stdout.reconfigure(encoding="utf-8") first, as tests/run_all.py and scripts/team_chat.py do.

The archive contains `→`. On Windows, when stdout is redirected or piped it uses the ANSI code page, so this line raises an encoding error, and `-I` ignores `PYTHONUTF8`. Use `sys.stdout.reconfigure(encoding="utf-8")` first, as `tests/run_all.py` and `scripts/team_chat.py` do.
@ -0,0 +150,4 @@
@unittest.skipUnless(shutil.which("git"), "Git is needed for the bare merge regression")
class Merging(unittest.TestCase):
Owner

This tests Git's own merge behaviour, not code in this repo. It's good evidence for the description, but I don't think it needs to stay as a permanent check.

This tests Git's own merge behaviour, not code in this repo. It's good evidence for the description, but I don't think it needs to stay as a permanent check.
mooki force-pushed fix/tested-rows from 06d7c45637
Some checks failed
Linux AppImage / appimage (pull_request) Blocked by required conditions
Windows builder / Longshot Builder zip (pull_request) Blocked by required conditions
Windows builder / Windows Vulkan back end (llvm-mingw) (pull_request) Blocked by required conditions
Windows builder / dav1d for Windows (llvm-mingw) (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / Python source checks (windows-2022, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (windows-2022, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to cfd54df719
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
2026-11-13 20:21:48 +00:00
Compare
Author
Member

Thank you for the review and for the answer on issue 45. Head cfd54df71 on main 514591a30; the whole game-free suite on it: 518 passed, 1 failed, 15 skipped.

Reworked this around the simpler split requested in review:

  • History: the existing rows stay in docs/tested-configurations.md, unchanged and in order. Only new runs use separate files. There are no numbered legacy files, pinned row count, digest or duplicate whitelist to refresh when main gains rows.
  • Readers: /tested-configurations/ uses the website's existing loader to assemble the frozen table and new files. The forge still shows the frozen table and a direct link to the fragment directory. README and requirements links point readers to the combined catalogue. No new package or network collection step is added.
  • UTF-8: redirected stdout is explicitly UTF-8, following the repository's unicode_output pattern. The arrow-containing ASCII-pipe test is a stand-in, not a claim of native Windows testing.
  • Limits: both readers take the one Limits section from the source document; neither keeps a second copy.
  • Unused option and Git test: removed --check, its flag entry and stale-view test, and the permanent test of Git's merge behaviour. The earlier Git experiment remains historical evidence only.
  • Filenames: require at least three parts after the date; the guide explicitly requires the tester and a distinct run identifier.
  • Measurements: use the game-free checks as AGENTS.md specifies for scripts and documentation, not a new hardware-timing experiment. The updated description names what ran and keeps the current-main and whole-suite boxes open until the final head has been checked.

The later-main regression also covers adding another frozen row without any refresh step, then refusing a new fragment that copies that newly landed row. See #45.

The test request for this pull request is reissued with this push, because its old steps used the removed --check option. An earlier run of the suite on this head failed nx2/test_texture_phi_dominance.py, tests/test_menu_i18n.py (compilers get killed while this PC is busy); the last run, on the same commit, is the one reported.

Thank you for the review and for the answer on issue 45. Head `cfd54df71` on main `514591a30`; the whole game-free suite on it: 518 passed, 1 failed, 15 skipped. Reworked this around the simpler split requested in review: - **History:** the existing rows stay in `docs/tested-configurations.md`, unchanged and in order. Only new runs use separate files. There are no numbered legacy files, pinned row count, digest or duplicate whitelist to refresh when main gains rows. - **Readers:** `/tested-configurations/` uses the website's existing loader to assemble the frozen table and new files. The forge still shows the frozen table and a direct link to the fragment directory. README and requirements links point readers to the combined catalogue. No new package or network collection step is added. - **UTF-8:** redirected stdout is explicitly UTF-8, following the repository's `unicode_output` pattern. The arrow-containing ASCII-pipe test is a stand-in, not a claim of native Windows testing. - **Limits:** both readers take the one Limits section from the source document; neither keeps a second copy. - **Unused option and Git test:** removed `--check`, its flag entry and stale-view test, and the permanent test of Git's merge behaviour. The earlier Git experiment remains historical evidence only. - **Filenames:** require at least three parts after the date; the guide explicitly requires the tester and a distinct run identifier. - **Measurements:** use the game-free checks as AGENTS.md specifies for scripts and documentation, not a new hardware-timing experiment. The updated description names what ran and keeps the current-main and whole-suite boxes open until the final head has been checked. The later-main regression also covers adding another frozen row without any refresh step, then refusing a new fragment that copies that newly landed row. See #45. The test request for this pull request is reissued with this push, because its old steps used the removed `--check` option. An earlier run of the suite on this head failed `nx2/test_texture_phi_dominance.py`, `tests/test_menu_i18n.py` (compilers get killed while this PC is busy); the last run, on the same commit, is the one reported.
mooki changed title from WIP: fix(docs): give tested runs independent result files to WIP: fix(docs): keep new test runs out of the frozen table 2026-11-13 20:22:18 +00:00
Author
Member

The test request for this pull request is reissued as T-AG65, replacing T-RKUJ: the Windows steps now use the redirected output and --output in place of the removed --check, and add one step that a copied row is refused. The title follows the new design.

The test request for this pull request is reissued as T-AG65, replacing T-RKUJ: the Windows steps now use the redirected output and `--output` in place of the removed `--check`, and add one step that a copied row is refused. The title follows the new design.
mooki changed title from WIP: fix(docs): keep new test runs out of the frozen table to fix(docs): keep new test runs out of the frozen table 2026-11-13 20:22:40 +00:00
HumanBean deleted branch fix/tested-rows 2026-11-14 01:49:23 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Some workflows are waiting to be reviewed.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!96
No description provided.