fix(build): the object store's clean-up skips a folder it cannot read #487

Open

jarynclouatre wants to merge 1 commit from jarynclouatre/Longshot:fix/object-store-prune-unreadable into main

pull from: jarynclouatre/Longshot:fix/object-store-prune-unreadable

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 0 Commits 1 Files changed 2 +55 -3

jarynclouatre commented 2026-11-14 16:28:26 +00:00

Contributor

• Copy link
• Copy Markdown

Why

With LONGSHOT_OBJ_STORE set, a folder in the store that cannot be listed (its permissions changed, or removed while the clean-up runs) makes a finished build fail: the game has compiled and installed, then make_game.py exits 1 with a PermissionError (or FileNotFoundError) before shader preparation and "done".

prune in nx2/object_store.py (from #371) calls os.listdir on the store and on each folder outside any try, and make_game.py's main calls prune without a guard. share() already treats an unreadable store as a miss.

What changed

• build (nx2/object_store.py, prune): an OSError from listing the store returns (0, 0), one from listing a folder skips that folder; the other folders are cleaned as before. Compile and link errors still fail the build: they don't go through prune.
• tests (tests/test_object_store.py): a folder that vanishes between the two listings (through a stand-in os.listdir) beside a readable one with an old lone object: prune removes the readable one's object and does not raise. Off Windows and not as root, the same with a mode 000 folder, and a mode 000 store returns (0, 0). A store whose listing fails returns (0, 0), and a store root that is a file returns (0, 0).

Flags and settings

None.

Measurements

main: 9c9d69ed (2026-11-14)
this PR: fix/object-store-prune-unreadable @ 9eb3f892
Hardware: none used (game-free checks on a Linux x86-64 PC, Python 3.14)
Scene: none, nothing built
Method: python3 -I -B tests/run_all.py --jobs 4, two runs each (main twice, then the branches of this batch in turn, twice). The new cases also ran against main's nx2/object_store.py.
How to reproduce: the commands above in a checkout of each commit

	main @ 9c9d69ed	this PR @ 9eb3f892
tests/run_all.py (run 1, run 2)	485 passed, 2 failed, 83 skipped (both)	485 passed, 2 failed, 83 skipped (both)
failing checks	test_bake_portability.py, test_walk_key.py	the same two
tests/test_object_store.py	FileNotFoundError: [Errno 2] No such file or directory: '.../store/0000000000000003'	pass

test_bake_portability.py fails here for a missing Python module (zstandard). test_walk_key.py's fixture doesn't link on main (undefined reference to hos_oot_autoload_input).

Testing

Tested: the game-free suite on a Linux PC, as in Measurements.

Not tested: a full build with an unreadable store folder.

Needs other testers: None.

Screenshots

Nothing on screen.

Risk and rollback

A folder the clean-up cannot read is not cleaned; its objects stay until it can be read again. Reverting the commit puts the unguarded listing back.

Issues

None filed. Found reading prune after #371.

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened.

## Why With `LONGSHOT_OBJ_STORE` set, a folder in the store that cannot be listed (its permissions changed, or removed while the clean-up runs) makes a finished build fail: the game has compiled and installed, then `make_game.py` exits 1 with a `PermissionError` (or `FileNotFoundError`) before shader preparation and "done". `prune` in `nx2/object_store.py` (from #371) calls `os.listdir` on the store and on each folder outside any `try`, and `make_game.py`'s `main` calls `prune` without a guard. `share()` already treats an unreadable store as a miss. ## What changed - build (`nx2/object_store.py`, `prune`): an `OSError` from listing the store returns `(0, 0)`, one from listing a folder skips that folder; the other folders are cleaned as before. Compile and link errors still fail the build: they don't go through `prune`. - tests (`tests/test_object_store.py`): a folder that vanishes between the two listings (through a stand-in `os.listdir`) beside a readable one with an old lone object: `prune` removes the readable one's object and does not raise. Off Windows and not as root, the same with a mode 000 folder, and a mode 000 store returns `(0, 0)`. A store whose listing fails returns `(0, 0)`, and a store root that is a file returns `(0, 0)`. ## Flags and settings None. ## Measurements main: 9c9d69ed (2026-11-14) this PR: fix/object-store-prune-unreadable @ 9eb3f892 Hardware: none used (game-free checks on a Linux x86-64 PC, Python 3.14) Scene: none, nothing built Method: `python3 -I -B tests/run_all.py --jobs 4`, two runs each (main twice, then the branches of this batch in turn, twice). The new cases also ran against main's `nx2/object_store.py`. How to reproduce: the commands above in a checkout of each commit | | main @ 9c9d69ed | this PR @ 9eb3f892 | |---|---|---| | `tests/run_all.py` (run 1, run 2) | 485 passed, 2 failed, 83 skipped (both) | 485 passed, 2 failed, 83 skipped (both) | | failing checks | `test_bake_portability.py`, `test_walk_key.py` | the same two | | `tests/test_object_store.py` | `FileNotFoundError: [Errno 2] No such file or directory: '.../store/0000000000000003'` | pass | `test_bake_portability.py` fails here for a missing Python module (zstandard). `test_walk_key.py`'s fixture doesn't link on main (`undefined reference to hos_oot_autoload_input`). ## Testing **Tested:** the game-free suite on a Linux PC, as in Measurements. **Not tested:** a full build with an unreadable store folder. **Needs other testers:** None. ## Screenshots Nothing on screen. ## Risk and rollback A folder the clean-up cannot read is not cleaned; its objects stay until it can be read again. Reverting the commit puts the unguarded listing back. ## Issues None filed. Found reading `prune` after #371. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.

jarynclouatre added 1 commit 2026-11-14 16:28:27 +00:00

fix(build): the object store's clean-up skips a folder it cannot read ... 9eb3f89237

prune listed the store and each of its folders outside any error handling,
and make_game calls it after the game is compiled and installed. A folder
that could not be listed (its permissions changed, or removed while the
clean-up ran) raised, and make_game stopped with exit 1 before preparing the
shaders.

A store that cannot be listed now has nothing to clean, and a folder that
cannot be listed is skipped; the others are cleaned as before.

jarynclouatre changed title from WIP: fix(build): the object store's clean-up skips a folder it cannot read to fix(build): the object store's clean-up skips a folder it cannot read 2026-11-14 16:31:24 +00:00

Some checks are pending

Hide checks

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

Some workflows are waiting to be reviewed.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'fix/object-store-prune-unreadable:jarynclouatre-fix/object-store-prune-unreadable'

git switch 'jarynclouatre-fix/object-store-prune-unreadable'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!487

No description provided.