WIP: fix(linux): crash and unclean-end records are published on exFAT and FAT drives #1060

Draft
zeldaootr wants to merge 3 commits from zeldaootr/Longshot:crash-record-no-link into main
Contributor

Type: fix · Risk: low · Tested on: Linux x86-64 (game-free checks); Steam Deck OLED (a normal start, ext4) · Needs: steamdeck or Linux with the game on an exFAT or FAT card

Why

With the game on an exFAT or FAT drive on Linux (a Steam Deck's SD card formatted on a PC, a USB stick), no crash record and no unclean-end record is ever kept, so the crash window and a bug report have nothing to show. The records are published with a hard link, which never replaces another record, and those drives have no hard links. Each crash or unclean end then leaves its files behind in bug-reports/running/, and a later run that the system gives the same process id keeps no record of its own at all.

The tracker was searched (exFAT, hard link, crash record, renameat2, unclean): no issue or pull request is about this.

How to reproduce

Seen on: game-free, Linux (Ubuntu 24.04, x86-64, gcc 13.3 and clang 23), with link() failing as exFAT's and vfat's do (EPERM); in the game, any Linux PC or Steam Deck with the game folder on an exFAT or FAT drive.
Settings and switches: defaults.
Steps: run python -I -B tests/test_crash_record_no_link.py on main with this PR's test commit; in the game, with the game folder on an exFAT card, start it and kill -SEGV it, then look in bug-reports/.
Expected: one bug-reports/pending/crash-<time>.txt, and nothing left in bug-reports/running/.
Actual: nothing in pending/; running/<pid>.record and running/<pid>.running stay there, at the next start too.

What changed

Where link() is refused for want of hard links (EPERM, EOPNOTSUPP), the record is moved into pending/ without replacing anything there: by the kernel's no-replace rename (renameat2 with RENAME_NOREPLACE, which exFAT and vfat support), else onto a name first taken with O_CREAT|O_EXCL. Both are plain system calls, as the crash path needs (it runs in the signal handler), and a name another record has moves on to the next name as before.

  • publish: publish in host/hos_crash_record.c tries move_noreplace (new) after a link() refused with EPERM, EOPNOTSUPP or ENOTSUP. move_noreplace calls renameat2 through syscall() (no dependency on a glibc that wraps it), and falls back to the O_EXCL reservation when the kernel or the drive refuses the flag (EINVAL, ENOSYS, EOPNOTSUPP). A failed move removes its reservation. The crash path (hos_crash_record_write, in the handler) and the start's recovery (recover_one, an unclean end or a crash staged but not published) both go through publish. Windows is unchanged (MoveFileExW, which never replaces a file).
  • docs: the crash record section of docs/development.md says the never-overwrite rule holds on drives without hard links too.
  • test: tests/test_crash_record_no_link.py (new).

Measurements

Verdict: fixed in the game-free test: on main no record is published when link() is refused as on exFAT; with this PR a crash, an unclean end and a taken name each publish exactly once, never over another record, and running/ is left empty, with and without renameat2. In the game only a normal start on a Steam Deck's internal ext4 drive, which takes the old link() path: clean. The no-hard-link path has not run in the game (Must be tested).

Hardware: Ubuntu 24.04 x86-64, gcc 13.3.0 and clang 23.1.5 (no GPU involved)
Scene: none (game-free)
Method: tests/test_crash_record_no_link.py with each compiler, at the test commit and at the fix, plus three mutations of the fix; one run each (the test is deterministic)
How to reproduce: CXX=g++ python -I -B tests/run_all.py test_crash_record_no_link, then CXX=clang++

Before: main @ bbe421f5 + the test (30323df4) After: this PR @ dbfa1d10
a crash's record, link() refused (EPERM) not published, running/ keeps it published once, running/ empty
an unclean end, published by the next start (not reached) published once, running/ empty
a name another record has (not reached) next name, the other untouched
the same without renameat2 (ENOSYS, EINVAL) (not reached) as above
before (30323df4, g++ and clang++):
FAIL: renameat2, crash: exit 1 (FAIL: the crash record was not published)
after (dbfa1d10, g++ and clang++):
PASS: renameat2: crash, unclean end and a taken name, all published once with no hard links
PASS: no renameat2 (ENOSYS): crash, unclean end and a taken name, all published once with no hard links
PASS: no renameat2 (EINVAL): crash, unclean end and a taken name, all published once with no hard links
mutations:
  the fallback not called:          FAIL: renameat2, crash: exit 1 (FAIL: the crash record was not published)
  the reservation without O_EXCL:   FAIL: no renameat2 (ENOSYS): another record was overwritten
  renameat2 without RENAME_NOREPLACE: FAIL: renameat2: another record was overwritten

Flags and settings

None.

Testing

Tested:

  • In the game on a Steam Deck OLED (SteamOS 3.9.5, kernel 7.2.7-valve2, Mesa 26.2.0-devel RADV), headless, started as from Steam: a normal start to the title, 6 minutes there, then quit. Exit 0; bug-reports/pending/ and bug-reports/running/ empty after the quit; no device-loss, VK_ERROR, FATAL or CRASH: line; the log the same as the development base's start but for timing-dependent lines. The game folder was on the Deck's internal ext4 drive, where link() works, so this shows the change leaves the usual path alone and nothing more: the Deck has no exFAT or FAT drive mounted, so the new path could not be shown in the game. The build was a local test pack on an older development base (main 9c9d69ed) carrying this change's host/ diff as it is here. Recorded in docs/tested/2026-11-16-crash-record-no-link-zeldaootr-deck-run01.md.
  • Game-free on Linux (Ubuntu 24.04, x86-64): the new test compiles the whole of host/hos_crash_record.c with link() failing as on exFAT and runs it in separate processes, as the game would; it fails at the test commit and passes at the fix with g++ 13.3 and clang++ 23.1, on main bbe421f5, and each of the three mutations above makes it fail. tests/test_crash_record.py (#75's checks, link() working) passes unchanged. CXX=g++ and CXX=clang++ python -I -B tests/run_all.py --changed forge/main: 17 passed, 2 failed, 0 skipped each; the 2 that fail (test_walk_key, test_vk_occlusion_queries) fail the same way on main. The whole run, CXX=g++ python -I -B tests/run_all.py: 569 passed, 13 failed, 82 skipped; main @ bbe421f5 the same way: 567 passed, 14 failed, 82 skipped. Every check that fails here fails on main too: test_bug_report_defaults, test_crt_options, test_guest_imported, test_link_skip, test_linux_preflight, test_linux_update, test_make_game_cache, test_missing_nup, test_mods_size_table, test_pending_guest_import, test_run_all_changed, test_vk_occlusion_queries, test_walk_key (main also failed test_pipeline_cache_save, a timing check). The skips are checks whose tools (the Vulkan headers and the like) are not on this PC. host/hos_crash_record.c compiled whole with -fsyntax-only -Wall -Wextra for Linux (gcc and clang, with the build's -D_GNU_SOURCE) and for Windows (llvm-mingw 20261104): no warning, as on main.

Not tested: in the game on an exFAT or FAT drive; a real exFAT or vfat mount (the test stands in for link()'s refusal and uses the test folder's own renameat2); a kernel older than 3.15 (no renameat2: the O_EXCL way, which the test runs too).

Must be tested:

  • steamdeck, or any Linux PC, with the game folder on an exFAT (or FAT) card or USB stick (by hand: a person with such a card; look at the folders):
    1. Start the game to the title, then kill -SEGV <its pid>. Expect, within a few seconds, one bug-reports/pending/crash-*.txt and nothing in bug-reports/running/. The process itself may take a while longer to end: the system writes its core dump after the record (minutes on a Steam Deck); that is not this change.
    2. Start it again, kill -9 it at the title, then start it once more and quit. Expect one bug-reports/pending/unclean-*.txt and running/ empty.
    3. On main the same steps leave pending/ empty and <pid>.record / <pid>.running in running/.

Risks

What could break: only the publication of a record when link() is refused; a drive with hard links takes the same link() path as before. On a drive without them, between the O_EXCL reservation and the rename (only without renameat2), the record's name holds an empty file for a moment.

Turning it off: revert the commit.

Compatibility: Nothing changes for a player but that records are kept on exFAT and FAT drives. Files a run left in bug-reports/running/ before this change are published by the next start (an unclean end, or the crash's own record).

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md).
  • A game-free test added for the fix, failing on main and passing here.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • Docs updated.
**Type:** fix · **Risk:** low · **Tested on:** Linux x86-64 (game-free checks); Steam Deck OLED (a normal start, ext4) · **Needs:** steamdeck or Linux with the game on an exFAT or FAT card ## Why With the game on an exFAT or FAT drive on Linux (a Steam Deck's SD card formatted on a PC, a USB stick), no crash record and no unclean-end record is ever kept, so the crash window and a bug report have nothing to show. The records are published with a hard link, which never replaces another record, and those drives have no hard links. Each crash or unclean end then leaves its files behind in `bug-reports/running/`, and a later run that the system gives the same process id keeps no record of its own at all. The tracker was searched (`exFAT`, `hard link`, `crash record`, `renameat2`, `unclean`): no issue or pull request is about this. ## How to reproduce **Seen on:** game-free, Linux (Ubuntu 24.04, x86-64, gcc 13.3 and clang 23), with `link()` failing as exFAT's and vfat's do (`EPERM`); in the game, any Linux PC or Steam Deck with the game folder on an exFAT or FAT drive. **Settings and switches:** defaults. **Steps:** run `python -I -B tests/test_crash_record_no_link.py` on main with this PR's test commit; in the game, with the game folder on an exFAT card, start it and `kill -SEGV` it, then look in `bug-reports/`. **Expected:** one `bug-reports/pending/crash-<time>.txt`, and nothing left in `bug-reports/running/`. **Actual:** nothing in `pending/`; `running/<pid>.record` and `running/<pid>.running` stay there, at the next start too. ## What changed Where `link()` is refused for want of hard links (`EPERM`, `EOPNOTSUPP`), the record is moved into `pending/` without replacing anything there: by the kernel's no-replace rename (`renameat2` with `RENAME_NOREPLACE`, which exFAT and vfat support), else onto a name first taken with `O_CREAT|O_EXCL`. Both are plain system calls, as the crash path needs (it runs in the signal handler), and a name another record has moves on to the next name as before. - publish: `publish` in `host/hos_crash_record.c` tries `move_noreplace` (new) after a `link()` refused with `EPERM`, `EOPNOTSUPP` or `ENOTSUP`. `move_noreplace` calls `renameat2` through `syscall()` (no dependency on a glibc that wraps it), and falls back to the `O_EXCL` reservation when the kernel or the drive refuses the flag (`EINVAL`, `ENOSYS`, `EOPNOTSUPP`). A failed move removes its reservation. The crash path (`hos_crash_record_write`, in the handler) and the start's recovery (`recover_one`, an unclean end or a crash staged but not published) both go through `publish`. Windows is unchanged (`MoveFileExW`, which never replaces a file). - docs: the crash record section of `docs/development.md` says the never-overwrite rule holds on drives without hard links too. - test: `tests/test_crash_record_no_link.py` (new). ## Measurements **Verdict:** fixed in the game-free test: on main no record is published when `link()` is refused as on exFAT; with this PR a crash, an unclean end and a taken name each publish exactly once, never over another record, and `running/` is left empty, with and without `renameat2`. In the game only a normal start on a Steam Deck's internal ext4 drive, which takes the old `link()` path: clean. The no-hard-link path has not run in the game (Must be tested). Hardware: Ubuntu 24.04 x86-64, gcc 13.3.0 and clang 23.1.5 (no GPU involved) Scene: none (game-free) Method: `tests/test_crash_record_no_link.py` with each compiler, at the test commit and at the fix, plus three mutations of the fix; one run each (the test is deterministic) How to reproduce: `CXX=g++ python -I -B tests/run_all.py test_crash_record_no_link`, then `CXX=clang++` | | Before: main @ bbe421f5 + the test (30323df4) | After: this PR @ dbfa1d10 | |---|---|---| | a crash's record, `link()` refused (EPERM) | not published, `running/` keeps it | published once, `running/` empty | | an unclean end, published by the next start | (not reached) | published once, `running/` empty | | a name another record has | (not reached) | next name, the other untouched | | the same without `renameat2` (ENOSYS, EINVAL) | (not reached) | as above | ``` before (30323df4, g++ and clang++): FAIL: renameat2, crash: exit 1 (FAIL: the crash record was not published) after (dbfa1d10, g++ and clang++): PASS: renameat2: crash, unclean end and a taken name, all published once with no hard links PASS: no renameat2 (ENOSYS): crash, unclean end and a taken name, all published once with no hard links PASS: no renameat2 (EINVAL): crash, unclean end and a taken name, all published once with no hard links mutations: the fallback not called: FAIL: renameat2, crash: exit 1 (FAIL: the crash record was not published) the reservation without O_EXCL: FAIL: no renameat2 (ENOSYS): another record was overwritten renameat2 without RENAME_NOREPLACE: FAIL: renameat2: another record was overwritten ``` ### Flags and settings None. ## Testing **Tested:** - In the game on a Steam Deck OLED (SteamOS 3.9.5, kernel 7.2.7-valve2, Mesa 26.2.0-devel RADV), headless, started as from Steam: a normal start to the title, 6 minutes there, then quit. Exit 0; `bug-reports/pending/` and `bug-reports/running/` empty after the quit; no device-loss, `VK_ERROR`, `FATAL` or `CRASH:` line; the log the same as the development base's start but for timing-dependent lines. The game folder was on the Deck's internal ext4 drive, where `link()` works, so this shows the change leaves the usual path alone and nothing more: **the Deck has no exFAT or FAT drive mounted, so the new path could not be shown in the game.** The build was a local test pack on an older development base (main 9c9d69ed) carrying this change's `host/` diff as it is here. Recorded in `docs/tested/2026-11-16-crash-record-no-link-zeldaootr-deck-run01.md`. - Game-free on Linux (Ubuntu 24.04, x86-64): the new test compiles the whole of `host/hos_crash_record.c` with `link()` failing as on exFAT and runs it in separate processes, as the game would; it fails at the test commit and passes at the fix with g++ 13.3 and clang++ 23.1, on main bbe421f5, and each of the three mutations above makes it fail. `tests/test_crash_record.py` (#75's checks, `link()` working) passes unchanged. `CXX=g++` and `CXX=clang++ python -I -B tests/run_all.py --changed forge/main`: 17 passed, 2 failed, 0 skipped each; the 2 that fail (`test_walk_key`, `test_vk_occlusion_queries`) fail the same way on main. The whole run, `CXX=g++ python -I -B tests/run_all.py`: 569 passed, 13 failed, 82 skipped; main @ bbe421f5 the same way: 567 passed, 14 failed, 82 skipped. Every check that fails here fails on main too: `test_bug_report_defaults`, `test_crt_options`, `test_guest_imported`, `test_link_skip`, `test_linux_preflight`, `test_linux_update`, `test_make_game_cache`, `test_missing_nup`, `test_mods_size_table`, `test_pending_guest_import`, `test_run_all_changed`, `test_vk_occlusion_queries`, `test_walk_key` (main also failed `test_pipeline_cache_save`, a timing check). The skips are checks whose tools (the Vulkan headers and the like) are not on this PC. `host/hos_crash_record.c` compiled whole with `-fsyntax-only -Wall -Wextra` for Linux (gcc and clang, with the build's `-D_GNU_SOURCE`) and for Windows (llvm-mingw 20261104): no warning, as on main. **Not tested:** in the game on an exFAT or FAT drive; a real exFAT or vfat mount (the test stands in for `link()`'s refusal and uses the test folder's own `renameat2`); a kernel older than 3.15 (no `renameat2`: the `O_EXCL` way, which the test runs too). **Must be tested:** - steamdeck, or any Linux PC, with the game folder on an exFAT (or FAT) card or USB stick (by hand: a person with such a card; look at the folders): 1. Start the game to the title, then `kill -SEGV <its pid>`. Expect, within a few seconds, one `bug-reports/pending/crash-*.txt` and nothing in `bug-reports/running/`. The process itself may take a while longer to end: the system writes its core dump after the record (minutes on a Steam Deck); that is not this change. 2. Start it again, `kill -9` it at the title, then start it once more and quit. Expect one `bug-reports/pending/unclean-*.txt` and `running/` empty. 3. On main the same steps leave `pending/` empty and `<pid>.record` / `<pid>.running` in `running/`. ## Risks **What could break:** only the publication of a record when `link()` is refused; a drive with hard links takes the same `link()` path as before. On a drive without them, between the `O_EXCL` reservation and the rename (only without `renameat2`), the record's name holds an empty file for a moment. **Turning it off:** revert the commit. **Compatibility:** Nothing changes for a player but that records are kept on exFAT and FAT drives. Files a run left in `bug-reports/running/` before this change are published by the next start (an unclean end, or the crash's own record). ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] Docs updated.
hos_crash_record.c publishes a staged record with link() and gives up on anything but EEXIST. exFAT and vfat (a Steam Deck's SD card) refuse link() with EPERM, so no crash or unclean-end record was ever published and running/ filled up. Compiles the writer with link() failing as there, with and without renameat2, and checks a crash, an unclean end and a taken name each publish exactly once and leave running/ empty. Fails on main.

Assisted-by: Claude
A record is published by link() and unlink(), so that it never replaces another one. exFAT and vfat have no hard links (link() says EPERM): with the game on such a drive, often a Steam Deck's SD card, no crash or unclean-end record was published, the staged <pid>.record and <pid>.running files stayed in running/, and a run given the same pid again had no record. Where link() is refused for want of hard links, the record is moved with the kernel's no-replace rename (renameat2), else onto a name first taken with O_EXCL; both are plain system calls, as the crash path needs, and neither replaces a record there.

Assisted-by: Claude
docs(tested): a normal start with the change on a Steam Deck
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
1e9608176d
Assisted-by: Claude
Author
Contributor

Tested in game: Steam Deck OLED (SteamOS 3.9.5, Mesa 26.2.0-devel RADV), headless, started as from Steam: a normal start to the title, 6 minutes there, quit exit 0, bug-reports/pending and running empty, no FATAL/CRASH/device-lost line. The game folder was on the internal ext4 drive (hard links work), so the new exFAT/FAT path itself has not run in the game: the Deck has no such drive mounted. (A local test pack on an older development base with this change's host diff.)

Tested in game: Steam Deck OLED (SteamOS 3.9.5, Mesa 26.2.0-devel RADV), headless, started as from Steam: a normal start to the title, 6 minutes there, quit exit 0, bug-reports/pending and running empty, no FATAL/CRASH/device-lost line. The game folder was on the internal ext4 drive (hard links work), so the new exFAT/FAT path itself has not run in the game: the Deck has no such drive mounted. (A local test pack on an older development base with this change's host diff.)
Owner

Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).

  • Tested on my own PC: ticked: Testing's Tested and the author's Tested in game comment describe a normal start on the author's Steam Deck OLED, though the exFAT path was not run.
  • Nothing gets worse that I know of: ticked: Measurements gives a verdict, and Risks names only a brief empty file on drives without hard links, with no speed cost.
Brought up to the [ready-for-review standard](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/CONTRIBUTING.md#ready-for-review-out-of-wip) (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know). - *Tested on my own PC*: ticked: Testing's Tested and the author's Tested in game comment describe a normal start on the author's Steam Deck OLED, though the exFAT path was not run. - *Nothing gets worse that I know of*: ticked: Measurements gives a verdict, and Risks names only a brief empty file on drives without hard links, with no speed cost.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'crash-record-no-link:zeldaootr-crash-record-no-link'
git switch 'zeldaootr-crash-record-no-link'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1060
No description provided.