WIP: fix(saves): journaled save commits: a crash mid-save keeps the last save #2453

Draft

hckrman101 wants to merge 6 commits from hckrman101/fix/journaled-saves into main

hckrman101 commented

2026-11-11 19:41:28 +00:00

(Migrated from github.com)

A crash, a kill or a power cut in the middle of a save can leave a save slot half written. Hookshot's nn::fs::Commit
did nothing, so the game's writes went straight onto the save files.

What changes

The save mount's root is now a working copy, save-tx/<user id>/work. A Commit only renames folders, so it needs no
free space:

Every file and folder of the working copy is flushed to disk.
The working copy is renamed ready-<n>. This is the commit point.
save/<user id> is renamed old, and ready-<n> is renamed save/<user id>.
old goes to save-backup/<user id>/<UTC time>. The newest 3 are kept.

Then the game's next working copy is made. That is the only step that needs space. If the disk is too full for it,
the rest of the run writes the save in place, as Hookshot did before, and the log says so. The next start uses the
journal again.

At the next start, a commit that was cut short is finished if it passed the commit point and rolled back if not. A
Commit that changed nothing writes nothing.

When things go wrong:

Changes the game never committed (a crash, or a close before saving) are not loaded, as on the console. They
are never committed or deleted either: they are renamed to save-tx/<user id>/uncommitted-<time>, and the newest 3
are kept. A working copy that cannot be renamed is not mounted again; that run writes in place.
A Commit that fails. This happens when a file is still open for writing, or another program holds a folder. The
save stays as it was and the working copy stays mounted. The next Commit or the last Unmount tries again. The game is
told the save worked either way; whether it would show its own "save failed" dialog was not tried.
A Commit whose working copy is gone fails; it never commits an empty save.
Two Hookshots on one data folder. Each save's journal has a session lock (save-tx/<user id>/lock, flock or
LockFileEx). A second Hookshot gets no journal: it writes in place, as before, and logs why.
A stale waiting commit. If the save cannot be replaced at step 3 (Windows: a file held open by another program),
the commit waits in save-tx and is the save the game loads. It is put in place only over the save it was made over,
checked by a fingerprint of names, sizes and times. If something replaced the save since, such as
HOS_SAVE_JOURNAL=0 or a copied-in save, the newer save stays. The waiting commit goes to
save-backup/<user id>/unapplied-<time>, or, if it cannot be moved, stays in save-tx without being loaded.
Links and other drives. The journal is off for a save folder that is a link (a cloud folder), or one on another
drive than save-tx and save-backup. The save is written in place. On Windows only symbolic links and junctions
count as links; OneDrive placeholders are copied like any file.
A failed rollback. If the old save cannot be moved back, the game stops with a message rather than start on a new,
empty save. A run without the journal stops too.

Other details:

Save layout. save/<user id> keeps the emulators' layout, so existing saves load as they are. docs/saves.md
describes save-tx and save-backup.
Journal off. With the journal off for a run, a commit waiting in save-tx is the save that gets written.
MountSaveDataReadOnly mounts the newest save.
Copy script. scripts/oot_saves.py refuses to copy while a commit waits in save-tx.
Control bridge. Its saves command lists the working copy, which is what the game has now.
Windows. MoveFileEx with a flush of both folders, FlushFileBuffers, and \\?\ for long paths. Save files are
opened with FILE_SHARE_DELETE, so the game's own read handles never block the renames.
Switches. HOS_SAVE_JOURNAL=0 writes the save in place as before. HOS_SAVE_CRASH_AT=<n> ends the process at a
commit step, for testing.

Credit

host/save_journal.c is adapted from valentderah's
totk-pc-recomp runtime/fs_host.c, which is
under the MIT licence (Copyright (c) 2026 Valiantsin Dzerakh). That file is itself adapted from MacNativeGames
games/totk/runtime/fs_host.c (MIT). Both are named at the top of the file, with the permission notice, and in the
readme's licence list and docs/design.md's reuse table. Hookshot's changes:

one journal per user id;
no hard links (scripts/oot_saves.py and players write save files in place);
the commit runs on the calling thread;
the commit is renames only;
the file calls are written for both Windows and POSIX.

Verified

Game-free test. tests/test_save_journal.py builds host/save_journal.c into a small driver at -O2 -Werror.
It ends the process at each of the commit's steps. After each one, the next start has the old save or the new one,
whole, and the new one exactly from the commit point on. It also checks:
- a save from before the journal loads unchanged;
- uncommitted changes are set aside (newest 3);
- backups rotate;
- a held-up commit waits and lands;
- two processes on one save;
- a Commit without a working copy;
- a stale waiting commit;
- links;
- a failed Commit (kept aside, not committed);
- a failed rollback, and an empty save made over one.
Faults (Linux). A second driver makes write() fail with ENOSPC and makes rename() fail, or end the process,
at a chosen call. It covers:
- a full disk at Commit: the commit is made, and the log says no next working copy could be made;
- a kill before each of the commit's 7 renames;
- a working copy written after a failed Commit, then a crash;
- a save written in place after a failed Commit;
- a torn working copy that cannot be set aside;
- a stale waiting commit that cannot be moved aside;
- a committed empty save cut short before its backup;
- a journal-off run finding a waiting commit.
Each ends with the old save or the new one, whole, never mixed, torn or silently rolled back.
Copy script test. tests/test_oot_saves.py.
Full test suite. python -I -B tests/run_all.py passes, apart from 3 tests this PC cannot run (no zstandard or
tkinter).
Compiles. host/nn_fs.c and host/save_journal.c compile without warnings for Linux (gcc, clang) and Windows
(llvm-mingw).
In the game. On Linux (RTX 3090), headless, on a copy of a test save, with this branch merged with main:
- Two pause-menu Saves made 4 commits ("made: renamed in"). Each took 127 to 240 ms, including the copy of the
  next working copy.
- save/<user id> held the new slot afterwards, and the replaced saves were in save-backup (the newest 3).
- At the restart, the log said "dropped the last run's working copy (it held the save)". The game reached play on
  the saved slot, and its Commit at load wrote nothing.

Not tested

A kill during an in-game save. Covered by the game-free tests only.
A real full disk. Only the injected faults were tested, so the in-place switch after a commit has not run in the
game.
Windows. The rename, flush and reparse-tag code was compiled for Windows but not run. Not tried: a save folder held
open by another program, long paths, or a OneDrive folder. The Windows CI runs the game-free test.
Commit time on a slow disk. Only measured on this PC's SSD (127 to 240 ms). The commit runs on the game's saving
thread, and making the next working copy is a copy of the save (about 1 MB).

Nearby: #2317 moves saves to a per-user data folder. This PR keeps save-tx and save-backup beside save under
hos_user_dir, so it should follow that move unchanged.

--- A crash, a kill or a power cut in the middle of a save can leave a save slot half written. Hookshot's `nn::fs::Commit` did nothing, so the game's writes went straight onto the save files. ### What changes The save mount's root is now a working copy, `save-tx/<user id>/work`. A Commit only renames folders, so it needs no free space: 1. Every file and folder of the working copy is flushed to disk. 2. The working copy is renamed `ready-<n>`. This is the commit point. 3. `save/<user id>` is renamed `old`, and `ready-<n>` is renamed `save/<user id>`. 4. `old` goes to `save-backup/<user id>/<UTC time>`. The newest 3 are kept. Then the game's next working copy is made. That is the only step that needs space. If the disk is too full for it, the rest of the run writes the save in place, as Hookshot did before, and the log says so. The next start uses the journal again. At the next start, a commit that was cut short is finished if it passed the commit point and rolled back if not. A Commit that changed nothing writes nothing. When things go wrong: - **Changes the game never committed** (a crash, or a close before saving) are not loaded, as on the console. They are never committed or deleted either: they are renamed to `save-tx/<user id>/uncommitted-<time>`, and the newest 3 are kept. A working copy that cannot be renamed is not mounted again; that run writes in place. - **A Commit that fails.** This happens when a file is still open for writing, or another program holds a folder. The save stays as it was and the working copy stays mounted. The next Commit or the last Unmount tries again. The game is told the save worked either way; whether it would show its own "save failed" dialog was not tried. - **A Commit whose working copy is gone** fails; it never commits an empty save. - **Two Hookshots on one data folder.** Each save's journal has a session lock (`save-tx/<user id>/lock`, flock or LockFileEx). A second Hookshot gets no journal: it writes in place, as before, and logs why. - **A stale waiting commit.** If the save cannot be replaced at step 3 (Windows: a file held open by another program), the commit waits in `save-tx` and is the save the game loads. It is put in place only over the save it was made over, checked by a fingerprint of names, sizes and times. If something replaced the save since, such as `HOS_SAVE_JOURNAL=0` or a copied-in save, the newer save stays. The waiting commit goes to `save-backup/<user id>/unapplied-<time>`, or, if it cannot be moved, stays in `save-tx` without being loaded. - **Links and other drives.** The journal is off for a save folder that is a link (a cloud folder), or one on another drive than `save-tx` and `save-backup`. The save is written in place. On Windows only symbolic links and junctions count as links; OneDrive placeholders are copied like any file. - **A failed rollback.** If the old save cannot be moved back, the game stops with a message rather than start on a new, empty save. A run without the journal stops too. Other details: - **Save layout.** `save/<user id>` keeps the emulators' layout, so existing saves load as they are. `docs/saves.md` describes `save-tx` and `save-backup`. - **Journal off.** With the journal off for a run, a commit waiting in `save-tx` is the save that gets written. `MountSaveDataReadOnly` mounts the newest save. - **Copy script.** `scripts/oot_saves.py` refuses to copy while a commit waits in `save-tx`. - **Control bridge.** Its `saves` command lists the working copy, which is what the game has now. - **Windows.** MoveFileEx with a flush of both folders, FlushFileBuffers, and `\\?\` for long paths. Save files are opened with FILE_SHARE_DELETE, so the game's own read handles never block the renames. - **Switches.** `HOS_SAVE_JOURNAL=0` writes the save in place as before. `HOS_SAVE_CRASH_AT=<n>` ends the process at a commit step, for testing. ### Credit `host/save_journal.c` is adapted from valentderah's [totk-pc-recomp `runtime/fs_host.c`](https://github.com/valentderah/totk-pc-recomp/blob/main/runtime/fs_host.c), which is under the MIT licence (Copyright (c) 2026 Valiantsin Dzerakh). That file is itself adapted from MacNativeGames `games/totk/runtime/fs_host.c` (MIT). Both are named at the top of the file, with the permission notice, and in the readme's licence list and `docs/design.md`'s reuse table. Hookshot's changes: - one journal per user id; - no hard links (`scripts/oot_saves.py` and players write save files in place); - the commit runs on the calling thread; - the commit is renames only; - the file calls are written for both Windows and POSIX. ### Verified - **Game-free test.** `tests/test_save_journal.py` builds `host/save_journal.c` into a small driver at `-O2 -Werror`. It ends the process at each of the commit's steps. After each one, the next start has the old save or the new one, whole, and the new one exactly from the commit point on. It also checks: - a save from before the journal loads unchanged; - uncommitted changes are set aside (newest 3); - backups rotate; - a held-up commit waits and lands; - two processes on one save; - a Commit without a working copy; - a stale waiting commit; - links; - a failed Commit (kept aside, not committed); - a failed rollback, and an empty save made over one. - **Faults (Linux).** A second driver makes `write()` fail with ENOSPC and makes `rename()` fail, or end the process, at a chosen call. It covers: - a full disk at Commit: the commit is made, and the log says no next working copy could be made; - a kill before each of the commit's 7 renames; - a working copy written after a failed Commit, then a crash; - a save written in place after a failed Commit; - a torn working copy that cannot be set aside; - a stale waiting commit that cannot be moved aside; - a committed empty save cut short before its backup; - a journal-off run finding a waiting commit. Each ends with the old save or the new one, whole, never mixed, torn or silently rolled back. - **Copy script test.** `tests/test_oot_saves.py`. - **Full test suite.** `python -I -B tests/run_all.py` passes, apart from 3 tests this PC cannot run (no zstandard or tkinter). - **Compiles.** `host/nn_fs.c` and `host/save_journal.c` compile without warnings for Linux (gcc, clang) and Windows (llvm-mingw). - **In the game.** On Linux (RTX 3090), headless, on a copy of a test save, with this branch merged with main: - Two pause-menu Saves made 4 commits ("made: renamed in"). Each took 127 to 240 ms, including the copy of the next working copy. - `save/<user id>` held the new slot afterwards, and the replaced saves were in `save-backup` (the newest 3). - At the restart, the log said "dropped the last run's working copy (it held the save)". The game reached play on the saved slot, and its Commit at load wrote nothing. ### Not tested - **A kill during an in-game save.** Covered by the game-free tests only. - **A real full disk.** Only the injected faults were tested, so the in-place switch after a commit has not run in the game. - **Windows.** The rename, flush and reparse-tag code was compiled for Windows but not run. Not tried: a save folder held open by another program, long paths, or a OneDrive folder. The Windows CI runs the game-free test. - **Commit time on a slow disk.** Only measured on this PC's SSD (127 to 240 ms). The commit runs on the game's saving thread, and making the next working copy is a copy of the save (about 1 MB). Nearby: #2317 moves saves to a per-user data folder. This PR keeps `save-tx` and `save-backup` beside `save` under `hos_user_dir`, so it should follow that move unchanged.

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 21:26:53 +00:00

Joey-Magnus (Migrated from github.com) left a comment

The journal is on by default (HOS_SAVE_JOURNAL=0 turns it off), but no in-game save with changes, reload, or kill during a save has been run yet. Uncommitted changes are now dropped, where before Commit did nothing, so any save path that never calls Commit would silently lose progress. The game is also told a save worked when it failed, and the review fixes haven't been built. The crash-recovery design is solid in the game-free test. One real save/reload/kill run before marking ready would settle it.

The journal is on by default (`HOS_SAVE_JOURNAL=0` turns it off), but no in-game save with changes, reload, or kill during a save has been run yet. Uncommitted changes are now dropped, where before `Commit` did nothing, so any save path that never calls `Commit` would silently lose progress. The game is also told a save worked when it failed, and the review fixes haven't been built. The crash-recovery design is solid in the game-free test. One real save/reload/kill run before marking ready would settle it.