fix(dev): dev.py update backs up all a build writes, per folder, one update at a time (#1932's review) #1945
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1945
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:fix/dev-update-review"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A follow-up to #1932 (
dev.py update), which was merged before BeanBot's review of it was answered. This fixes thereview's six blocking points; the command's options and the window are unchanged.
bin/) or a half-done build could be left mixed after a rollback, with.longshot_installed.jsondescribing thenewer build. A backup now keeps everything
nx2/make_game.py's finish can write, replace or remove (BUILT,BUILT_DIRSand the FFmpeg libraries, in the folder and inbin/: the game, the renderer and libraries,image.bin,.longshot_installed.json, the shader list, the texts, prompts and DLSS runtime, the settings files,the tool's marks), with a note of which of them were there (
backup.json). Putting it back restores those andremoves what wasn't there (files a build added, a finish's
.newcopies).(
update/backups/<name>-<hash of its path>/, the last three of each), and--rollbacktakes the named folder'snewest and checks that folder's game.
argv[0], resolved against the updater's own folder, with the game folder. It nowreads each
longshotprocess's program path (/proc/<pid>/exe; Windows:QueryFullProcessImageNameW) and matchesthe folder and its
bin/; a path it can't read still counts as running.rollback now holds
update/update.lock(common.Lock) from the recovery to its end; a second one stops at once andsays so; a
--checkbeside a running update lists and recovers nothing.it. A stopped update is now put back only once its build job (the queue's live
update-...ticket) has ended too;until then the next command says the build is still running and how to cancel it.
build.environment()andtools.lock. A tree other than this checkout now gets its environment from its own devtools(
build.environmentandwindows_prebuilt, run in a process of its own:tree_environment); a tree too old to havethem is refused.
BeanBot's second review (of
06988e9b), fixed ina7b0311eande4930e66:start time, recorded before the child can write anything), its queue job and the queue folder it is in. Recovery puts
the backup back only once all three are gone and the game in the backup's own folder is not running; until then it
keeps the record and says why. A check of folder B no longer restores folder A while A's game runs.
--no-write-fetch-head), not the shared FETCH_HEAD that aconcurrent check could rewrite between an update's fetch and its read.
file is per process.
.partial-folder and renamed when complete, so a half-made one (a fulldisk, a stop) is never listed or chosen; ordered by their creation time.
game.txt): recovered after upgrading, and listed for--rollbackof their folder.av*/sw*.dll.newis removed by a restore.a moment is tried again for up to 20 s; the folder's marks and installed-file note go back last, so a restore that
still stops leaves them describing the build that is there; a failed removal or copy stops it with the error.
--rollbackkeepslongshot_options.iniandlongshot_env.txtas they are now; a failed update'srecovery still restores them.
longshot_vk.soon Linux).romfs/,.longshot_exefs/)or the caches (
cache/); the next build or the game's first start makes those again.Read first:
scripts/devtools/update.py(tracked,back_up,put_back,_swap,running_in,exe_path,recover,update_lock,tree_environment,cmd_update/update_locked).Testing
Tested: Windows 11 Pro 26200, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D, Python 3.13.
tests/test_dev_update.py, 40 cases, all pass (the ten from the second review fail on06988e9b, the held-file one ona7b0311e). New: a launcher-shaped folder (bin/game and renderer, theinstalled-file note, texts) put back whole after a build that replaced some of it, added an FFmpeg library, prompts,
image.binand a.newcopy, then failed; backups and--rollbackper folder (another folder's game running stopsits rollback); one update at a time (an update and a rollback stop while the lock is held, a check lists and leaves the
stopped update's record, which the next command puts back); recovery waiting for a live
update-...queue ticket andignoring other jobs; the running check by program path (the folder, its
bin/, another folder, an unreadable path);exe_pathof this process; a tree's own environment (a stand-in devtools in another tree, a tree without devtoolsrefused, this checkout's in process). Against
main'supdate.pythe nine new or changed cases fail; here all pass.Checks:
test_dev_update,test_doc_anchors,byte_compile_trackedpass.A real update with these fixes:
dev.py update --game <a folder built from main @ 87c0d7b4> --pick 1915 --with 1929(the folder's commit predates
update.py, so the build took that tree's own environment: fix 6). #1915 wascherry-picked onto
87c0d7b4and open #1929 merged on top; the backup kept 14 paths (the game, the renderer, the SDLlibraries,
image.bin,.longshot_installed.json, the shader list, the pipeline states,language.txt, the options,the mark,
texts/,prompts/,ngx/), 1.6 GB, with itsbackup.json; the build (with DLSS, from the tree's ownenvironment) and its shader step took 4.3 min; the folder records the pick and the open pull request. The game then
ran 60 s with no window, with no
CRASHor[error]line in its log.The second review's fixes, for real (the same folder, main @
f1f2dc5bthen2b3de5b2):byte, no record left;
--rollbackright after an update first stopped onlongshot.exe, which Windows held for a moment just afterthe build, after it had put back the folder's marks, so the folder said "old commit" while it held the new game.
Fixed in
e4930e66(the retry, the marks last). Run again the same way: the update (0.8 min, nothing changed tocompile), then the rollback at once: the game, the renderer and the mark exactly as before, no staged copy left;
the
longshot_env.txtedit made after the update kept.Still to test:
the builder ends (by hand: someone with Linux)
Measurements
Nothing to measure: a fix to a development tool, the game unchanged (a backup's size is in Tested and Risks).
Risks
Developers only: nothing reaches players. Backups take more room in the development folder's
update/backups/(1.6 GBeach here, three per folder). A folder backed up by #1932's version has no
backup.json; its old backups are nolonger read (delete
update/backups/'s older folders). A revert is enough.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). (run_all.py --changed, the checks thatread the changed files: 5 pass,
test_oot_state_tapskipped for want of gcc here.)main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Recommendation: Request changes. Per-folder backups and executable-path checks address real updater bugs. Recovery can still overwrite a live build, and a failed backup can hide the last usable rollback.
The updater keeps more of each installation, selects backups by game folder, and builds older commits with their own tools. This improves the developer workflow, but it adds substantial disk I/O and still leaves gaps in what a rollback restores.
Does it help: Yes for the reported Windows cases. The author reports nine new or changed cases failing with main's updater and 31 passing on the PR, plus one 4.3-minute real update and a 60-second headless smoke run. That run copied 14 paths totaling 1.6 GB, or about 4.8 GB at three backups of that size per folder. Linux cancellation and recovery remain unproven, and the added backup and restore time has no base comparison.
Other systems: The source change affects the developer CLI and its GUI subprocesses. The builder still runs shader preparation on the local GPU, and rollback can leave its cache outputs behind. Windows with NVIDIA has one reported smoke run. Linux, including Steam Deck and AMD configurations that use this workflow, lacks end-to-end evidence. Launcher-built folders and the installed-file note are covered by a mocked Windows failure case. No game runtime, renderer, save-format, audio, or input code changed.
Switches: No CLI options or production environment switches added, changed, or removed. The test sets the existing LONGSHOT_QUEUE_DIR to a temporary directory.
Reviewed:
06988e9b06against80bb1ccc18: Read AGENTS.md, CONTRIBUTING.md, the Forgejo PR template, docs/dev/standards.md, the commit and diff, both changed Python files, and the changed documentation sections. Traced the relevant builder, queue, lock, process enumeration, and launcher layout code. Read the testing summary and checklist. The rest of the builder and documentation was read selectively. No review-checklist file or sibling Tooling checkout was present. There were no comments, earlier reviews, or separate run logs.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Waiting on the author (a run, a measurement or a decision; no edit does it)
Suggestions (the author may take or leave)
Testing: Static review only. The author reports 31 passing Windows unit cases, five passing changed-file checks with one GCC-dependent skip, and one Windows update and smoke run. Linux end-to-end results, a full suite result, and head CI were absent from the supplied evidence.
Overlaps and merge order: This follows merged #1932. The author used #1915 and #1929 as update inputs. No merge dependency is apparent in the supplied material.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -68,0 +72,4 @@_BESIDE = ("longshot", "longshot_vk.so", "libSDL2-2.0.so.0", "renhdrum.so") if not common.WINDOWS else \("longshot.exe", "longshot_vk.dll", "SDL2.dll", "SDL3.dll", "renhdrum.dll")BUILT = _BESIDE + tuple(os.path.join("bin", n) for n in _BESIDE) + ("image.bin", ".longshot_installed.json", "exefs_patches.txt", "longshot_shaders.bin",The builder also writes romfs and .longshot_exefs in unpacked-input mode and can move or replace pipeline caches. Cover those writes or isolate them and narrow the documented guarantee.
@ -309,3 +369,3 @@n += 1d = os.path.join(part("backups"), "%s-%s-%d" % (stamp, tag, n))d = os.path.join(root, "%s-%s-%d" % (stamp, tag, n))os.makedirs(d)A failed snapshot is visible to rollback before backup.json exists. Publish completed snapshots atomically and prune only completed snapshots.
@ -331,1 +409,4 @@shutil.copytree(src, dst, symlinks=True)else:os.makedirs(os.path.dirname(dst), exist_ok=True)shutil.copy2(src, dst)A full disk can truncate the live image during restore, and ignored removal errors can leave new files behind. Stage replacements and keep failed restores recoverable.
Explicit rollback reverts options and environment edits made since the update. Preserve those user edits even when failed-update recovery needs to restore settings.
@ -301,0 +342,4 @@names = os.listdir(os.path.join(game, d) if d else game)except OSError:continuefound += [os.path.join(d, n) if d else n for n in sorted(names) if n.startswith(("av", "sw")) and n.endswith(".dll")]A first-time avcodec-61.dll.new has no base DLL for tracked() to discover. Include staged FFmpeg names and test a stop before the swap.
@ -355,3 +444,4 @@jobs[0].get("name")))if os.path.isdir(p.get("backup", "")):say("the update to %s did not finish: the game folder is put back as it was" % p.get("commit", "?")[:8])put_back(p["backup"])Checking folder B can restore folder A while A is running. Check the backup's own destination before any restore.
@ -331,0 +391,4 @@"""The folder as the backup found it: what it had copied back, what it didn't have removed (a build's new files, ahalf-done finish's too). Returns the folder"""with open(os.path.join(d, "backup.json"), encoding="utf-8") as f:b = json.load(f)Pending records from #1932 still reference game.txt backups. Handle that format so an interrupted update can recover after upgrading the checkout.
@ -345,0 +422,4 @@def build_jobs():"""The job queue's live update builds (names update-...): a build an updater left running when it was killed"""s = jobqueue.settings()return [t for t in jobqueue.read_tickets(jobqueue.queue_dir(s), clean=False)A delegated child can still be starting before it publishes a ticket, and nested queue builds may never publish one. Track the build lifetime from handoff through the last writer.
@ -352,12 +437,24 @@ def recover():start = procs.proc_start(p.get("pid", 0))if start is not None and start == p.get("start"):die("an update is running (process %s): wait for it, or stop it first" % p.get("pid"))jobs = build_jobs()A global update-* match can block on an unrelated folder or miss a build that uses another queue directory. Save and check the pending build's exact identity.
@ -469,4 +587,3 @@if a.build_here: # (run_build's step, in the tree being built)return build_here(ROOT, os.path.abspath(a.game), a.commit)recover()s = settings(a)settings(a) writes the remembered destination and shared temp file before the lock. Serialize those writes and keep lock-bypassing checks read-only.
@ -482,0 +602,4 @@except BlockingIOError:if not (a.check or a.list_json):die("another dev.py update is running on this PC: wait for it, or stop it first")lock = None # (a check while an update runs: it lists, and leaves the recovery to the update)This concurrent check still overwrites the shared FETCH_HEAD. Isolate each fetch result so an update cannot pick up the check's branch.
@ -405,0 +430,4 @@self.assertEqual(self.read("bin/longshot_vk.dll"), "old renderer")self.assertEqual(self.read(".longshot_installed.json"), "old note")self.assertEqual(self.read("texts/en.txt"), "old text")for gone in ("bin/avcodec-61.dll", "prompts", "image.bin", "bin/longshot_vk.dll.new", update.MARK):Linux tracks longshot_vk.so, so this staged DLL assertion fails there. Use the platform renderer name and cover DLL staging in a Windows-specific case.
06988e9b06e4930e6683Thanks for the second review (of
06988e9b). Each blocking point, ate4930e66(rebased onto main @2b3de5b2):--build-herechild is started with Popen and its pid and start time go into the pending record before it can write anything; recovery waits for it as well as for the updater. A build run inside an existing queue job runs in the updater or that child, so the two pid checks cover it.update-<commit>) and the queue folder it was queued in; recovery checks only that ticket, in that folder.--no-write-fetch-head,+refs/heads/<branch>:refs/dev-update/fetch-<pid>-<ns>), reads it and deletes it. A test interleaves another branch's fetch between the two and reads the right commit (it fails on06988e9b).settings()is read only;remember()runs only with the update lock held, through a per-process temp file. A check beside a running update writes nothing..partial-…and renamed when complete;backups()lists only folders withbackup.jsonand orders them by their creation time; the next backup removes stale partials.game.txtare still put back (for their own folder) and listed for--rollback..dll.new: tracked, so a restore removes it (in the launcher-folder test).romfs/,.longshot_exefs/orcache/, which the next build or the game's first start makes again.longshot.exe, which Windows held for a moment, after it had already put back the folder's marks. Each swap now retries for up to 20 s, the marks and the installed-file note go back last, and a restore that still stops says which file held it (a recovery keeps its record; a rollback can be run again). Re-run the same way on this PC, it completes.--rollbackleaveslongshot_options.iniandlongshot_env.txtas they are now; a failed update's recovery still restores them.Waiting on the author:
test_dev_update.py;run_all.py --changed: 5 pass,test_oot_state_tapskipped for want of gcc here.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.