fix(ci): recover partially enrolled AppImage toolchain PPAs #2275

Merged

Arkex-ai merged 2 commits from fix/appimage-ppa-recovery into main

2026-11-11 16:41:48 +00:00

Arkex-ai commented

2026-11-11 13:09:45 +00:00

(Migrated from github.com)

Summary

A transient PPA signing-key failure can leave AppImage retries unable to repair the source. Launchpad metadata stalls can also consume the job's full hour before retries finish. This keeps enrollment repair ahead of the signed-index refresh and gives each enrollment attempt a deadline.

Fixes #2266.

Technical changes

Retry base prerequisites and PPA/toolchain setup as independent phases, each with three attempts, a 20 second delay and the final failing status preserved.
Enroll/repair with --no-update, then explicitly refresh signed indexes before installing dependent packages. Package/repository/authentication policy remains unchanged.
Limit each enrollment to 180 seconds, with a 5 second forced-termination grace period. Normal failures, timeout status 124 and forced-termination status 137 remain failures; subsequent attempts revisit partial enrollment first.
Execute the real workflow shell against stateful authored commands. No source/key deletion or insecure apt option.

Validation

Tested 027c1f6c on main 49f75dc8:

Nine methods / 19 cases pass: healthy ordering, transient/persistent failures at all six stages, third-attempt key recovery, nonstandard exit status, recovery after timeout and persistent timeout/forced-kill exhaustion. The three stall cases use real GNU timeout with authored children and an accelerated test clock.
Original source passes the healthy control but fails partial-key recovery. Six executed mutations are rejected, including removal of the enrollment deadline and forced-kill grace.
Full Linux: 351 passed / 0 failed / 23 skipped. Full Windows: 341 passed / 3 failed / 30 skipped. Both match the immutable exact base, including the same Windows NUP file-link privilege, compute-guard header and recorder destructor negative-control failures. jq was supplied on Linux; the POSIX bootstrap test runs there.
Pinned actionlint 1.7.12 passes; official release archive SHA256 verified. Ubuntu Jammy software-properties 0.99.22.9 source confirms enrollment calls add_source then add_key even for an already-present source.

No real package/network operation occurs in local tests. Hosted current-head checks still need to complete; a persistent Launchpad outage is not repaired by a retry/deadline. No gameplay/rendering change or runtime claim.

Claimed before implementation in dev-work. The separate container-build PR #1936 retains the same package/PPA contract. Earlier-head AppImage cancellation and this deadline follow-up are documented in the conversation; exact current-head CI links follow.

Current-head workflow	Result
Game-free source regressions	Needs attention
Linux AppImage	Needs attention
PR title	Passed
Windows builder	Passed

## Summary A transient PPA signing-key failure can leave AppImage retries unable to repair the source. Launchpad metadata stalls can also consume the job's full hour before retries finish. This keeps enrollment repair ahead of the signed-index refresh and gives each enrollment attempt a deadline. Fixes #2266. ## Technical changes - Retry base prerequisites and PPA/toolchain setup as independent phases, each with three attempts, a 20 second delay and the final failing status preserved. - Enroll/repair with `--no-update`, then explicitly refresh signed indexes before installing dependent packages. Package/repository/authentication policy remains unchanged. - Limit each enrollment to 180 seconds, with a 5 second forced-termination grace period. Normal failures, timeout status `124` and forced-termination status `137` remain failures; subsequent attempts revisit partial enrollment first. - Execute the real workflow shell against stateful authored commands. No source/key deletion or insecure apt option. ## Validation Tested `027c1f6c` on main `49f75dc8`: - **Nine methods / 19 cases pass**: healthy ordering, transient/persistent failures at all six stages, third-attempt key recovery, nonstandard exit status, recovery after timeout and persistent timeout/forced-kill exhaustion. The three stall cases use real GNU timeout with authored children and an accelerated test clock. - Original source passes the healthy control but fails partial-key recovery. Six executed mutations are rejected, including removal of the enrollment deadline and forced-kill grace. - Full Linux: **351 passed / 0 failed / 23 skipped**. Full Windows: **341 passed / 3 failed / 30 skipped**. Both match the immutable exact base, including the same Windows NUP file-link privilege, compute-guard header and recorder destructor negative-control failures. `jq` was supplied on Linux; the POSIX bootstrap test runs there. - Pinned actionlint 1.7.12 passes; official release archive SHA256 verified. Ubuntu Jammy software-properties 0.99.22.9 source confirms enrollment calls `add_source` then `add_key` even for an already-present source. No real package/network operation occurs in local tests. Hosted current-head checks still need to complete; a persistent Launchpad outage is not repaired by a retry/deadline. No gameplay/rendering change or runtime claim. Claimed before implementation in [dev-work](https://discord.com/channels/1568810484818378804/1569019786766188665/1570058346193625282). The separate container-build PR #1936 retains the same package/PPA contract. Earlier-head AppImage cancellation and this deadline follow-up are documented in the conversation; exact current-head CI links follow.  | Current-head workflow | Result | |---|---| | [Game-free source regressions](https://github.com/NotRexed/Hookshot/actions/runs/43040125028) | Needs attention | | [Linux AppImage](https://github.com/NotRexed/Hookshot/actions/runs/43040124951) | Needs attention | | [PR title](https://github.com/NotRexed/Hookshot/actions/runs/43040121770) | Passed | | [Windows builder](https://github.com/NotRexed/Hookshot/actions/runs/43040124904) | Passed |

Arkex-ai commented

2026-11-11 14:36:33 +00:00

(Migrated from github.com)

Enrollment deadline follow-up at 027c1f6c: the earlier AppImage reached GitHub's one-hour limit while still in System packages (run). Each PPA enrollment now gets 180 seconds plus 5 seconds before forced termination, so all three retries can be reached without weakening signed-index checks.

The new timeout/forced-termination controls pass, alongside the existing stateful recovery cases and six causal mutations. Full Linux (351 passed, 0 failed, 23 skipped) and Windows (341 passed, 3 failed, 30 skipped) match exact main 49f75dc8. Fresh hosted checks are starting; persistent Launchpad unavailability still fails the job.

Enrollment deadline follow-up at `027c1f6c`: the earlier AppImage reached GitHub's one-hour limit while still in System packages ([run](https://github.com/NotRexed/Hookshot/actions/runs/43026639935)). Each PPA enrollment now gets 180 seconds plus 5 seconds before forced termination, so all three retries can be reached without weakening signed-index checks. The new timeout/forced-termination controls pass, alongside the existing stateful recovery cases and six causal mutations. Full Linux (351 passed, 0 failed, 23 skipped) and Windows (341 passed, 3 failed, 30 skipped) match exact main `49f75dc8`. Fresh hosted checks are starting; persistent Launchpad unavailability still fails the job.

hckrman101 (Migrated from github.com) approved these changes

2026-11-11 16:28:12 +00:00

hckrman101 (Migrated from github.com) left a comment

Looks good. Splitting the bootstrap into two retried phases is the right fix for #2266: a source left without its key is now repaired by the next add-apt-repository --no-update before anything runs apt-get update against it, instead of the base refresh failing with NO_PUBKEY on every retry. Keeping the explicit apt-get update right after enrollment means the signed-index refresh is unchanged, and the final failing status is still preserved. The 180s + 5s kill-after deadline per enrollment is a sensible way to keep three attempts inside the job's hour. I checked the workflow diff against main: linux-appimage.yml has not changed there since #2132/#2124, so there is no conflict, and the PR is mergeable.

CI: the red jobs (AppImage, Linux Python parts 1/2, SPIR-V fixtures) all fail on Temporary failure resolving 'azure.archive.ubuntu.com' / archive.ubuntu.com in the runners' own apt steps, not in anything this PR touches. Windows tests, the Windows Vulkan build, actionlint and the title check pass. Please re-run the failed jobs once before merging, so the AppImage job gets to actually exercise the new path.

Nit: the timeout only covers enrollment; apt-get update right after it can still hang on Launchpad. That is fine for this PR (the description says so), just worth knowing if the hour limit is hit again.

Looks good. Splitting the bootstrap into two retried phases is the right fix for #2266: a source left without its key is now repaired by the next `add-apt-repository --no-update` before anything runs `apt-get update` against it, instead of the base refresh failing with NO_PUBKEY on every retry. Keeping the explicit `apt-get update` right after enrollment means the signed-index refresh is unchanged, and the final failing status is still preserved. The 180s + 5s kill-after deadline per enrollment is a sensible way to keep three attempts inside the job's hour. I checked the workflow diff against main: `linux-appimage.yml` has not changed there since #2132/#2124, so there is no conflict, and the PR is mergeable. CI: the red jobs (AppImage, Linux Python parts 1/2, SPIR-V fixtures) all fail on `Temporary failure resolving 'azure.archive.ubuntu.com'` / `archive.ubuntu.com` in the runners' own apt steps, not in anything this PR touches. Windows tests, the Windows Vulkan build, actionlint and the title check pass. Please re-run the failed jobs once before merging, so the AppImage job gets to actually exercise the new path. Nit: the timeout only covers enrollment; `apt-get update` right after it can still hang on Launchpad. That is fine for this PR (the description says so), just worth knowing if the hour limit is hit again.

Arkex-ai commented

2026-11-11 16:41:30 +00:00

(Migrated from github.com)

Thanks for the review. Agreed on the remaining apt-update timeout limit. Could a maintainer rerun the failed jobs for AppImage and source/SPIR-V checks before merging? Those attempts stopped on runner DNS failures before exercising the changed path. I can't restart these upstream runs with my repository permissions.

Thanks for the review. Agreed on the remaining apt-update timeout limit. Could a maintainer rerun the failed jobs for [AppImage](https://github.com/NotRexed/Hookshot/actions/runs/43040124951) and [source/SPIR-V checks](https://github.com/NotRexed/Hookshot/actions/runs/43040125028) before merging? Those attempts stopped on runner DNS failures before exercising the changed path. I can't restart these upstream runs with my repository permissions.