fix(docs): keep new test runs out of the frozen table #96
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!96
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:fix/tested-rows"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Independent test results should not make unrelated pull requests fight over one table insertion point. This keeps the existing results readable and gives only new runs independent files. See #45.
What changed
docs/tested-configurations.mdkeeps the existing table in place, frozen and in its original order. There are no numbered legacy files, pinned row count, digest or duplicate whitelist to refresh.docs/tested/README.mddocuments one three-line file per new run:YYYY-MM-DD-topic-tester-run.md. The tester and a distinct run identifier are required; rows must stand alone.scripts/tested_configurations.pyvalidates new files against each other and the current frozen table, then prints the combined view or writes it as UTF-8. Historical repeats and relative references stay untouched./tested-configurations/from those same local files. It watches new, edited and deleted fragments, resolves their links relative to their own directory, and uses the site's existing public-prose rules without changing source rows, code or link destinations. No package or network collection step is added.--checkoption, stale-view check and permanent Git-merge test are gone. The description checker itself is unchanged.Flags and settings
scripts/tested_configurations.py --output FILEDocumented in the fragment guide and
docs/development.md. No game setting changes. Do not commit an assembled view.Measurements
Branch state: on main
514591a30d707fe9f3ad31eae148c3cc1e750bff; headcfd54df719a46b595697bf417113b1332294db2b; whole game-free suite on this head:518 passed, 1 failed, 15 skipped(main514591a30d707fe9f3ad31eae148c3cc1e750bff:517 passed, 1 failed, 15 skipped). The 1 failing test fails on main the same way:tests/test_menu_i18n.py. The commits are unchanged by rebases apart from the results table.main:
1ef93e796bthis PR: move/fix/tested-rows-r5 @ 75c5610ac499cc24055bc5a38581f5552a8e8b93
Hardware: Linux x86-64; no GPU work. Node v26.13.0 and npm 11.20.0 for the website.
Scene: none; scripts, documentation and the static website.
Method: game-free checks, as AGENTS.md specifies for changes the game does not run; row-byte comparison, not a speed claim.
How to reproduce:
python -I -B tests/test_tested_configurations.py;python -I -B tests/test_check_pr_description.py;python -I -B tests/test_forge_pr.py. Inwebsite/, install the unchanged lock withnpm ci --ignore-scripts --no-audit --no-fund, then runnpm run build,node scripts/check-site.tsandnode --test --test-concurrency=1 tests/*.test.ts.1ef93e79Later-main carry: a temporary fixture combined this guide and marker with all of
fbd420241a's table: 109 table lines, including its header and separator, or 107 result rows. All row bytes and order were retained; Python and TypeScript produced identical Markdown without refreshing anything. A copied newly landed row was then rejected as a duplicate of the frozen table.test_later_main_rows_are_read_without_a_refresh_stepkeeps that behaviour covered. This proves the carry behaviour, not that this branch has already been rebased.The website lock is unchanged: SHA-256
b051dfed8bee21bc6eb7e583b729f31d45c0ab8717df4d35eeb40cbb7bd88bf0. Its 385 existing packages installed with lifecycle scripts disabled; no package script was enabled.Earlier conflict evidence, not a replay of this revision: at main
77f605a25and the original proposal06d7c456, converting the 18 pinned incoming heads' inserted rows to separate files reduced table conflicts from 13 to 0; all 153 same-base result-only branch pairs changed from 152 conflicts to 0. The issue's original 14-of-16 observation remains a different snapshot. The old numbered-history migration and its counts are not this revision's implementation.Forgejo 17.0.1's bare-repository conflict check ignored the committed union attribute in the recorded command reproduction. Its sparse merge preparation also conflicted for merge, squash and rebase; the shared non-mergeable status is refused before those styles run. These are historical local command experiments, not a new live server test or an inspection of its private configuration.
Testing
Tested: on Linux, the named Python checks above, including malformed headers and rows, duplicates, dates, filename parts, escaped pipes, ordering, single-source Limits, UTF-8 file output and a forced-ASCII redirected stream. The ASCII stream is a stand-in, not native Windows. The original 91 rows remain byte-identical and in order. The source-layout regression fails as expected on the pinned main table's missing assembly marker, and passes here; this is one negative-control case, not a run of main's whole suite.
The offline website build and existing page checker passed; all 66 Node cases passed serially. Network access was denied: the existing progress and contributor collectors showed their documented unavailable state, not invented empty results. The built catalogue contains all 91 rows; both readers validate the eight-column source format. Chromium inspection at 1600×1000 and an emulated 390×844 viewport found no page-wide overflow; the wide table scrolls within its own container. The installed Unified processor is configured through Astro's supported
markdown.processorAPI; no deprecated plugin setting is retained.Not tested: the final-head whole repository suite, native Windows execution, live collector data, a new live Forgejo merge, a physical phone browser, game builds, game runs or frame rates. The narrow viewport is browser emulation, not mobile-hardware testing. No game behaviour changes.
Needs other testers:
--outputwith a non-ASCII filename; expect UTF-8 arrows and unchanged row text, without settingPYTHONUTF8(T-AG65)Screenshots
Nothing changes on screen in the game. Main @
1ef93e79has no combined catalogue page. The website's new page with this PR @75c5610a, built offline:Risk and rollback
There are more small files for new runs. Existing open branches that append to the table need their new rows moved to unique files or kept in their PR description until that is done. Old rows are not moved or renumbered. A later main's well-formed rows become part of the frozen table without refreshing a count, digest or generated page; this does not compare the table to Git at runtime or enforce the freeze against an arbitrary manual edit.
The two readers implement the same small format in Python and TypeScript, with tests for each; they must stay in sync if that format changes. Duplicate checks cannot establish that a measurement is true. Website prose follows its existing naming policy; the exact source evidence remains on the forge.
Reverting the commit restores the old workflow. If new fragments have landed, first carry their rows back into the table so evidence is not lost. Do not add a committed combined view: that would restore the shared insertion point.
Reviewer notes
Start with the frozen source page,
scripts/tested_configurations.py, and the existing website loader's new catalogue path. This follows the review's simpler-fragments direction, rather than copying all historical rows into files.The test request T-AG65 replaces T-RKUJ, whose steps used the removed
--check. No server configuration change or live test merge is part of this branch.Issues
See #45
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.New on this forge; not opened on GitHub.
Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra [email protected]
llun referenced this pull request2026-11-13 03:26:44 +00:00
simonwjackson referenced this pull request2026-11-13 06:13:31 +00:00
simonwjackson referenced this pull request2026-11-13 06:13:40 +00:00
simonwjackson referenced this pull request2026-11-13 06:13:51 +00:00
simonwjackson referenced this pull request2026-11-13 06:20:59 +00:00
simonwjackson referenced this pull request2026-11-13 06:29:05 +00:00
The direction is right, and the evidence in the description and on issue 45 is thorough. The branch isn't ready to merge yet, though: it's based on an old main, and the page that readers reach loses its data.
Must fix
77f605a2. Main (839e85ff) now has 90 rows. All 82 migrated rows match main byte for byte, but the 8 rows added since then are not indocs/tested/. They sit at the top of the table and 6 of them sayas above. Because the archive is numbered newest-first (0001 is the top row), carrying them over as legacy renames all 82 files and changes the digest, therange(1, 83)check and the82in the test and the docs. Carrying them over as new fragments means rewriting theiras abovetext. The Rebased on current main box is ticked but isn't true (Forgejo reportsmergeable: false).website/src/content/docs/requirements.mdx("Compare tested configurations"),README.md("estimates from the tested configurations") andCONTRIBUTING.mdline 70 ("lists the runs behind it") all send people todocs/tested-configurations.md. After this PR that page has no runs: just two folder links and a Python command. Players arriving from the website can't run that command.Should fix
scripts/tested_configurations.pywrites the view to stdout with the locale encoding. The archive contains→, so on Windows a redirected or piped run (> file,| more) fails with an encoding error.-Ialso ignoresPYTHONUTF8. Reconfigure stdout to UTF-8 the waytests/run_all.pyandscripts/team_chat.pyalready do.--checkhas no user: the view is never committed, so nothing is ever stale. Dropping it, together with its flag-table row and its test, makes the change smaller.Mergingtest checks Git's own behaviour (distinct files merge, the same line conflicts), not anything in this repo. It's fine as evidence in the description, but it isn't needed as a permanent check.NAMEaccepts2026-11-13-x.md, while the docs ask fortopic-tester-run. The filename is the only guard against two people picking the same name. A name that clashes gives an add/add conflict, which is visible, so this is acceptable. Still, the docs should say the tester's name is required, or the regex should require at least three parts.On the decision in issue 45
My recommendation is fragments, but simpler than this PR:
To make fragments cheaper and keep readers served:
docs/tested-configurations.md. Issue 45 itself proposed this ("The existing table can stay where it is; only new rows need the new path"). That avoids the 82-file archive, the hard-coded digest and duplicate list, and the renumbering on every rebase.docs/tested/YYYY-MM-DD-topic-tester-run.md, checked by the script as now.If the maintainers want the archive split into files anyway, then generate the migration with a script at rebase time rather than by hand, so that re-basing onto a moving main is mechanical.
The checks look sound and match the project's rules: standard-library only, safe under
python -I, deterministic sort by date then name, escaped pipes handled, duplicates refused, and the template,good.md, AGENTS.md, CONTRIBUTING.md and workflow.md all point at the same place.check_pr_description.pyreads the template's boxes, so the changed box carries through to it and toforge.py pr createwithout further edits.@ -12,2 +7,3 @@## Read the runs## RunsBrowse [new run files](tested/) and the [migrated runs](tested/legacy/), or assemble one Markdown view:The website's requirements page ("Compare tested configurations"), the README and CONTRIBUTING.md line 70 ("lists the runs behind it") all link to this page for the data. After this change a reader gets folder links and a Python command instead. The combined table needs to be somewhere readers can open it, for example built into the website or kept on this page for the existing rows.
@ -99,0 +51,4 @@## Existing rows and branchesThe 82 rows from main `77f605a2` are in `tested/legacy/0001.md` through `0082.md`, byte-for-byte in their originalMain is at
839e85ffwith 90 rows. The 8 added since77f605a2sit at the top of the table, and 6 of them useas above. They are missing fromdocs/tested/. Because 0001 is the newest row, carrying them over as legacy renames all 82 files and changes the digest, therange(1, 83)check and the82here and in the test. Keeping the old table frozen on this page, with only new runs as fragments, avoids that every time main moves.@ -0,0 +13,4 @@ROOT = Path(__file__).resolve().parents[1]HEADER = "| Date | Build | OS / desktop | GPU / driver | CPU / RAM | Settings | Where | Result |"SEPARATOR = "|---|---|---|---|---|---|---|---|"NAME = re.compile(r"(\d{4}-\d{2}-\d{2})-[a-z0-9]+(?:-[a-z0-9]+)*\.md")This accepts
2026-11-13-x.md, but the docs ask fortopic-tester-run. The filename is the only thing that keeps two people from picking the same name, so either require at least three parts here or say in the docs that the tester's name is required.@ -0,0 +90,4 @@"\n## Migrated runs (original order)\n\n""The three pre-existing duplicate pairs are retained; 'as above' refers only to this archive.\n\n" +HEADER + "\n" + SEPARATOR + "\n" + "".join(row + "\n" for row in legacy) +"\n## Limits of the original RTX 4070 Ti runs\n\n"This text is also on
docs/tested-configurations.md, so the two copies will drift. Read it from the page, or leave it out of the generated view.@ -0,0 +100,4 @@parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])target = parser.add_mutually_exclusive_group()target.add_argument("--output", type=Path, help="write the assembled view as UTF-8 (do not commit it)")target.add_argument("--check", type=Path, help="fail if this generated view is missing or out of date; write nothing")The view is never committed, so
--checknever has a stale file to catch. I'd drop it, along with its test and its flag-table row.@ -0,0 +112,4 @@args.output.parent.mkdir(parents=True, exist_ok=True)args.output.write_text(content, encoding="utf-8", newline="\n")else:sys.stdout.write(content)The archive contains
→. On Windows, when stdout is redirected or piped it uses the ANSI code page, so this line raises an encoding error, and-IignoresPYTHONUTF8. Usesys.stdout.reconfigure(encoding="utf-8")first, astests/run_all.pyandscripts/team_chat.pydo.@ -0,0 +150,4 @@@unittest.skipUnless(shutil.which("git"), "Git is needed for the bare merge regression")class Merging(unittest.TestCase):This tests Git's own merge behaviour, not code in this repo. It's good evidence for the description, but I don't think it needs to stay as a permanent check.
06d7c45637cfd54df719Thank you for the review and for the answer on issue 45. Head
cfd54df71on main514591a30; the whole game-free suite on it: 518 passed, 1 failed, 15 skipped.Reworked this around the simpler split requested in review:
docs/tested-configurations.md, unchanged and in order. Only new runs use separate files. There are no numbered legacy files, pinned row count, digest or duplicate whitelist to refresh when main gains rows./tested-configurations/uses the website's existing loader to assemble the frozen table and new files. The forge still shows the frozen table and a direct link to the fragment directory. README and requirements links point readers to the combined catalogue. No new package or network collection step is added.unicode_outputpattern. The arrow-containing ASCII-pipe test is a stand-in, not a claim of native Windows testing.--check, its flag entry and stale-view test, and the permanent test of Git's merge behaviour. The earlier Git experiment remains historical evidence only.The later-main regression also covers adding another frozen row without any refresh step, then refusing a new fragment that copies that newly landed row. See #45.
The test request for this pull request is reissued with this push, because its old steps used the removed
--checkoption. An earlier run of the suite on this head failednx2/test_texture_phi_dominance.py,tests/test_menu_i18n.py(compilers get killed while this PC is busy); the last run, on the same commit, is the one reported.WIP: fix(docs): give tested runs independent result filesto WIP: fix(docs): keep new test runs out of the frozen tableThe test request for this pull request is reissued as T-AG65, replacing T-RKUJ: the Windows steps now use the redirected output and
--outputin place of the removed--check, and add one step that a copied row is refused. The title follows the new design.WIP: fix(docs): keep new test runs out of the frozen tableto fix(docs): keep new test runs out of the frozen table