WIP: feat(saves): the game backs its saves up and restores them into an empty folder #1428

Draft
TrashPanda wants to merge 3 commits from TrashPanda/Longshot:feat/save-backups into main
Contributor

Type: feat · Risk: medium · Tested on: Linux, game-free only · Needs: Windows and Linux, a start and a save, in the game

Why

A game started without the launcher keeps its saves in the game folder (user\save), and the launcher's builds before the shared folder did too: uninstalling to update, building into a new folder or deleting the old one took them along. A deleted or broken save in the shared folder is gone as well. The launcher backs saves up only before it copies over them.

Ported from Hookshot 0e264748 (NotRexed): git.notrexed.dev/Hookshot/Hookshot@0e264748af

The tracker was searched for "save backup", "lost saves" and HOS_SAVE_BACKUP: no pull request or issue does this. Nearby open work is other save tools: #1397 (the save script's Windows default), #1401 and #1403 (editing and sharing saves).

How to reproduce

Seen on: Hookshot's players, who lost saves updating. Not run on Longshot.
Settings and switches: defaults.
Steps: start the game directly (no launcher) from a game folder, save, delete the game folder, build into a new one, start.
Expected: the saves are there.
Actual: on main the new folder starts with no saves.

What changed

host/save_backup.c (new) copies the save folder into the folder the launcher and scripts/longshot_saves.py already back up to (docs/saves.md): %LOCALAPPDATA%\Longshot\oot-save-backups on Windows, $XDG_DATA_HOME/oot-save-backups on Linux. A backup is game-<YYYYmmdd-HHMMSS> holding a copy of the save folder, the launcher's own layout, so it goes back the same way. One is made at each start and after a save mount's nn::fs::Commit (at most every 20 s), only when something is newer than the last; the 20 newest game- folders stay, and the launcher's and the script's are never touched. A start whose save folder holds no file takes the newest backup in before the game reads its saves, and logs it; a folder with saves is never touched.

Unlike Hookshot, only the player's own saves are backed up and restored: the user folder the game picks itself, or on Windows the launcher's %APPDATA%\Longshot. A run given another --user (a test's, the farm's, a second profile) and a scripted run (HOS_HEADLESS, HOS_CONTROL) make and restore none, so a Test Farm run "from a new save" is never filled with a volunteer's saves, and a test's saves never become the newest backup. HOS_SAVE_BACKUPS=<folder> names another place and lifts that rule (a portable launcher could pass its own folder); HOS_SAVE_BACKUP=0 and HOS_SAVE_RESTORE=0 turn them off.

  • host: host/save_backup.c, host/save_backup.h (new); host/hos_main.c calls hos_save_backup_start right after the user folder is made, knowing whether --user named it; host/nn_fs.c's Commit backs up after a save mount's commit.
  • test: tests/test_save_backup.py (new, from Hookshot's) builds save_backup.c as it is (on Linux over host/posix/win32_posix.c's file functions) and runs it in a temporary folder.
  • docs: docs/saves.md (the game's own backups), docs/development.md (the three switches).

Measurements

Verdict: the game-free check fails before the feature and passes here. Nothing in the game has been measured.

Hardware: Linux, no game run.
Scene: none.
Method: python -I -B tests/test_save_backup.py on the test commit and on this head.
How to reproduce: check out ad9f34fb (the test alone, on main 544229a2) and run the command; then this head.

Before: main @ 544229a2 plus the test @ ad9f34fb After: this PR @ ed8cbd38
tests/test_save_backup.py fails: nn::fs::Commit does not back the saves up after a save mount's commit PASS

python -I -B tests/run_all.py on this head (Linux, 16 at once): 602 passed, 39 failed, 81 skipped (no Vulkan or SPIR-V headers here). The failures are the ones main 544229a2 has on this PC, except test_appimage_package_retry and test_compile_stops_on_exception, which failed under the full run's load and pass alone on this head.

Flags and settings

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
HOS_SAVE_BACKUP The game backs its saves up absent on unless =0 A game-<time> copy at start and after a save Untested in the game On
HOS_SAVE_RESTORE An empty save folder takes the newest backup absent on unless =0 Saves back after a reinstall Untested in the game On
HOS_SAVE_BACKUPS Another backups folder absent unset Also backs up a run with its own --user Untested Unset; a portable launcher could set it

Testing

Tested: Linux, game-free. The new check (backups made only when something is newer, the 20 newest kept and the launcher's left, an empty folder restored and a full one untouched, the switches, and no backups for another --user, HOS_HEADLESS or HOS_CONTROL); the checks that read nn_fs.c and the save and UTF-8 path checks.

Not tested: the game, Windows, a host build (a64_suyu.h is not on this PC). Opened with --force because Joey asked; this PC is not running the game.

Must be tested:

  • Windows, started from the launcher and directly: start, save, check a game-<time> folder in %LOCALAPPDATA%\Longshot\oot-save-backups and the log line saves: backed up N files of .... Then empty the save folder and start: saves: none in ...: the newest backup (...) copied in, and the save loads.
  • Linux: the same with ~/.local/share/oot-save-backups.
  • A run with its own --user: no backup line.

Risks

What could break: a player who empties the save folder on purpose to start over gets the newest backup back at the next start (HOS_SAVE_RESTORE=0, or delete the backups too). Each backup copies the whole save folder (a few MB) at most every 20 s while the game saves. On Windows the portable launcher passes its own --user, so its saves get no backups unless it sets HOS_SAVE_BACKUPS.

Turning it off: HOS_SAVE_BACKUP=0, HOS_SAVE_RESTORE=0.

Compatibility: nothing in the save files or options. The backups folder is shared with the launcher; the game only writes and prunes its own game- folders. MKWVK_API_VERSION is unchanged.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why. Not ticked yet: Testing's Tested names only game-free checks, and the change touches host/ with no in-game run yet.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks. Not ticked yet: Risks names a save copy every 20 s and Measurements says nothing in the game is measured.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). The failures also fail on main here; two failed only under load.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A game-free test added, failing on main and passing here.
  • Docs updated.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built here; save_backup.c compiles in the check on Linux over the shim's file functions, and uses the A file functions, UTF-8 under the game's manifest.
**Type:** feat · **Risk:** medium · **Tested on:** Linux, game-free only · **Needs:** Windows and Linux, a start and a save, in the game ## Why A game started without the launcher keeps its saves in the game folder (`user\save`), and the launcher's builds before the shared folder did too: uninstalling to update, building into a new folder or deleting the old one took them along. A deleted or broken save in the shared folder is gone as well. The launcher backs saves up only before it copies over them. Ported from Hookshot `0e264748` (NotRexed): https://git.notrexed.dev/Hookshot/Hookshot/commit/0e264748af0677fde54d790cd7cf00405dd0bc86 The tracker was searched for "save backup", "lost saves" and `HOS_SAVE_BACKUP`: no pull request or issue does this. Nearby open work is other save tools: #1397 (the save script's Windows default), #1401 and #1403 (editing and sharing saves). ## How to reproduce **Seen on:** Hookshot's players, who lost saves updating. Not run on Longshot. **Settings and switches:** defaults. **Steps:** start the game directly (no launcher) from a game folder, save, delete the game folder, build into a new one, start. **Expected:** the saves are there. **Actual:** on main the new folder starts with no saves. ## What changed `host/save_backup.c` (new) copies the save folder into the folder the launcher and `scripts/longshot_saves.py` already back up to (`docs/saves.md`): `%LOCALAPPDATA%\Longshot\oot-save-backups` on Windows, `$XDG_DATA_HOME/oot-save-backups` on Linux. A backup is `game-<YYYYmmdd-HHMMSS>` holding a copy of the save folder, the launcher's own layout, so it goes back the same way. One is made at each start and after a save mount's `nn::fs::Commit` (at most every 20 s), only when something is newer than the last; the 20 newest `game-` folders stay, and the launcher's and the script's are never touched. A start whose save folder holds no file takes the newest backup in before the game reads its saves, and logs it; a folder with saves is never touched. Unlike Hookshot, only the player's own saves are backed up and restored: the user folder the game picks itself, or on Windows the launcher's `%APPDATA%\Longshot`. A run given another `--user` (a test's, the farm's, a second profile) and a scripted run (`HOS_HEADLESS`, `HOS_CONTROL`) make and restore none, so a Test Farm run "from a new save" is never filled with a volunteer's saves, and a test's saves never become the newest backup. `HOS_SAVE_BACKUPS=<folder>` names another place and lifts that rule (a portable launcher could pass its own folder); `HOS_SAVE_BACKUP=0` and `HOS_SAVE_RESTORE=0` turn them off. - host: `host/save_backup.c`, `host/save_backup.h` (new); `host/hos_main.c` calls `hos_save_backup_start` right after the user folder is made, knowing whether `--user` named it; `host/nn_fs.c`'s `Commit` backs up after a save mount's commit. - test: `tests/test_save_backup.py` (new, from Hookshot's) builds `save_backup.c` as it is (on Linux over `host/posix/win32_posix.c`'s file functions) and runs it in a temporary folder. - docs: `docs/saves.md` (the game's own backups), `docs/development.md` (the three switches). ## Measurements **Verdict:** the game-free check fails before the feature and passes here. Nothing in the game has been measured. Hardware: Linux, no game run. Scene: none. Method: `python -I -B tests/test_save_backup.py` on the test commit and on this head. How to reproduce: check out `ad9f34fb` (the test alone, on main `544229a2`) and run the command; then this head. | | Before: main @ 544229a2 plus the test @ ad9f34fb | After: this PR @ ed8cbd38 | |---|---|---| | `tests/test_save_backup.py` | fails: nn::fs::Commit does not back the saves up after a save mount's commit | PASS | `python -I -B tests/run_all.py` on this head (Linux, 16 at once): 602 passed, 39 failed, 81 skipped (no Vulkan or SPIR-V headers here). The failures are the ones `main` `544229a2` has on this PC, except `test_appimage_package_retry` and `test_compile_stops_on_exception`, which failed under the full run's load and pass alone on this head. ### Flags and settings | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `HOS_SAVE_BACKUP` | The game backs its saves up | absent | on unless `=0` | A `game-<time>` copy at start and after a save | Untested in the game | On | | `HOS_SAVE_RESTORE` | An empty save folder takes the newest backup | absent | on unless `=0` | Saves back after a reinstall | Untested in the game | On | | `HOS_SAVE_BACKUPS` | Another backups folder | absent | unset | Also backs up a run with its own `--user` | Untested | Unset; a portable launcher could set it | ## Testing **Tested:** Linux, game-free. The new check (backups made only when something is newer, the 20 newest kept and the launcher's left, an empty folder restored and a full one untouched, the switches, and no backups for another `--user`, `HOS_HEADLESS` or `HOS_CONTROL`); the checks that read `nn_fs.c` and the save and UTF-8 path checks. **Not tested:** the game, Windows, a host build (`a64_suyu.h` is not on this PC). Opened with `--force` because Joey asked; this PC is not running the game. **Must be tested:** - Windows, started from the launcher and directly: start, save, check a `game-<time>` folder in `%LOCALAPPDATA%\Longshot\oot-save-backups` and the log line `saves: backed up N files of ...`. Then empty the save folder and start: `saves: none in ...: the newest backup (...) copied in`, and the save loads. - Linux: the same with `~/.local/share/oot-save-backups`. - A run with its own `--user`: no backup line. ## Risks **What could break:** a player who empties the save folder on purpose to start over gets the newest backup back at the next start (`HOS_SAVE_RESTORE=0`, or delete the backups too). Each backup copies the whole save folder (a few MB) at most every 20 s while the game saves. On Windows the portable launcher passes its own `--user`, so its saves get no backups unless it sets `HOS_SAVE_BACKUPS`. **Turning it off:** `HOS_SAVE_BACKUP=0`, `HOS_SAVE_RESTORE=0`. **Compatibility:** nothing in the save files or options. The backups folder is shared with the launcher; the game only writes and prunes its own `game-` folders. `MKWVK_API_VERSION` is unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [ ] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. Not ticked yet: Testing's Tested names only game-free checks, and the change touches host/ with no in-game run yet. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [ ] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. Not ticked yet: Risks names a save copy every 20 s and Measurements says nothing in the game is measured. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). The failures also fail on `main` here; two failed only under load. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A game-free test added, failing on `main` and passing here. - [x] Docs updated. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built here; `save_backup.c` compiles in the check on Linux over the shim's file functions, and uses the A file functions, UTF-8 under the game's manifest.
A game folder's saves go with it when it is uninstalled to update, built
into a new folder or deleted, and a deleted save in the shared folder is
gone too. This check fails until the game backs its own saves up into the
launcher's backups folder, restores the newest into an empty save folder,
and does neither for another --user or a scripted run.

Ported-from: 0e264748af
Co-authored-by: NotRexed <[email protected]>
Assisted-by: Claude Opus 5.5
A game folder's saves went with it when it was uninstalled to update, built
into a new folder or deleted. host/save_backup.c copies the save folder into
the launcher's backups folder (docs/saves.md: %LOCALAPPDATA%\Longshot\
oot-save-backups, $XDG_DATA_HOME/oot-save-backups) as game-<time>, at each
start and after a save mount's commit (at most every 20 s), only when
something is newer than the last; the 20 newest game- folders stay, and the
launcher's and the script's are never touched. A start whose save folder
holds no file takes the newest one in; a folder with saves is never touched.

Only the player's own saves: no --user, or on Windows the launcher's
%APPDATA%\Longshot. Another --user and a scripted run (HOS_HEADLESS,
HOS_CONTROL) make and restore none, so a farm run's new save is never
filled with a player's. HOS_SAVE_BACKUPS=<folder> moves them and lifts the
user-folder rule; HOS_SAVE_BACKUP=0 and HOS_SAVE_RESTORE=0 turn them off.

Ported-from: 0e264748af
Co-authored-by: NotRexed <[email protected]>
Assisted-by: Claude Opus 5.5
docs(saves): the game's own backups, and their switches
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 9s
ed8cbd38e4
Co-authored-by: NotRexed <[email protected]>
Assisted-by: Claude Opus 5.5
Author
Contributor

@Beanstalk Windows and Linux, any GPU, started the way a player starts it (the launcher, or the game folder with no --user; not a scripted run, since HOS_HEADLESS and HOS_CONTROL turn the backups off). Three steps: (1) start and save in the game; expected a game-<time> folder in the backups folder (%LOCALAPPDATA%\Longshot\oot-save-backups on Windows, ~/.local/share/oot-save-backups on Linux) and the log line saves: backed up N files of .... (2) Close, empty the save folder, start again; expected saves: none in ...: the newest backup (...) copied in, and the save from step 1 loads. (3) A run with its own --user folder: no saves: backed up line. Not run in the game yet; Joey asked for this pull request so the farm can take that run.

@Beanstalk Windows and Linux, any GPU, started the way a player starts it (the launcher, or the game folder with no --user; not a scripted run, since HOS_HEADLESS and HOS_CONTROL turn the backups off). Three steps: (1) start and save in the game; expected a `game-<time>` folder in the backups folder (`%LOCALAPPDATA%\Longshot\oot-save-backups` on Windows, `~/.local/share/oot-save-backups` on Linux) and the log line `saves: backed up N files of ...`. (2) Close, empty the save folder, start again; expected `saves: none in ...: the newest backup (...) copied in`, and the save from step 1 loads. (3) A run with its own --user folder: no `saves: backed up` line. Not run in the game yet; Joey asked for this pull request so the farm can take that run.
Member

@TrashPanda only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@TrashPanda only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Owner

Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).

  • Tested on my own PC: not ticked: Testing's Tested names only game-free checks, and the change touches host/ with no in-game run yet. Run it yourself on your PC (in the game, for a change the game runs) and say where and what you saw in Testing's Tested; or, when the fix's own case can't be run by hand, add a **Not testable by hand:** line saying why and what stands in for it. Then tick it.
  • Nothing gets worse that I know of: not ticked: Risks names a save copy every 20 s and Measurements says nothing in the game is measured. Measure that cost and state it in Risks, then tick it.
Brought up to the [ready-for-review standard](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/CONTRIBUTING.md#ready-for-review-out-of-wip) (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know). - *Tested on my own PC*: not ticked: Testing's Tested names only game-free checks, and the change touches host/ with no in-game run yet. Run it yourself on your PC (in the game, for a change the game runs) and say where and what you saw in Testing's *Tested*; or, when the fix's own case can't be run by hand, add a `**Not testable by hand:**` line saying why and what stands in for it. Then tick it. - *Nothing gets worse that I know of*: not ticked: Risks names a save copy every 20 s and Measurements says nothing in the game is measured. Measure that cost and state it in Risks, then tick it.
Member

On Linux this PR backs saves up to $XDG_DATA_HOME/oot-save-backups, while the launcher keeps its backups in $XDG_DATA_HOME/longshot/save-backup (since LongshotLauncher#80), so the launcher won't list the game's backups there. #1473 moves the save script's Linux default to the launcher's folder (issue: MagicBeanDev/LongshotLauncher#188). Could this PR use longshot/save-backup on Linux too, so every backup is in one place? Windows is already the same everywhere (%LOCALAPPDATA%\Longshot\oot-save-backups).

On Linux this PR backs saves up to `$XDG_DATA_HOME/oot-save-backups`, while the launcher keeps its backups in `$XDG_DATA_HOME/longshot/save-backup` (since LongshotLauncher#80), so the launcher won't list the game's backups there. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1473 moves the save script's Linux default to the launcher's folder (issue: https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/issues/188). Could this PR use `longshot/save-backup` on Linux too, so every backup is in one place? Windows is already the same everywhere (`%LOCALAPPDATA%\Longshot\oot-save-backups`).
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 9s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/save-backups:TrashPanda-feat/save-backups'
git switch 'TrashPanda-feat/save-backups'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1428
No description provided.