ci: CircleCI's setup-main stays pending on every main commit since Nov 15 17:57 UTC, so no checks run and no gate posts #1412

Open
opened 2026-11-17 13:32:08 +00:00 by TrashPanda · 0 comments
Contributor

@drcrow: since 17:57 UTC on Nov 15, CircleCI's setup job has stayed pending on every main commit. It never starts, so no checks run and mb/gate posts nothing. I have no CircleCI access, so this is what Forgejo's commit statuses show. The cause is a guess for someone who can open the CircleCI org to confirm.

What the statuses show

  • It worked at cutover. On #417's merge (deb9f781, Nov 15 17:25 UTC), ci/circleci: setup-main went pending, then success a minute later, and the continued workflow posted lint, third-party, test-linux, guide-screenshots, website, spirv-fixtures, windows-builder and windows-vulkan. Seven more main commits did the same, the last one b7a014b8 (17:52:55 UTC).
  • From the next commit on, it never finishes. On 1576a274 (17:57:35 UTC) and on every main commit since, through 544229a2 (Nov 17 03:50 UTC), setup-main is posted pending by circleci-bot with a workflow link, and nothing follows. I checked the first 40 commits after #417 one by one, then sampled every fourth up to today: 55 pending and none finished after b7a014b8.
  • Tag pushes do the same: ci/circleci: setup-noop-tag and setup-release stay pending on the nightly-tagged commits.
  • Nothing in the repository changed at the break. b7a014b8..1576a274 touches no file under .circleci/, and neither does any later commit. The config is the one that worked eight times in a row.

Example on today's head 544229a2: ci/circleci: setup-main pending since 2026-11-17T03:50:23Z, https://app.circleci.com/workflow/dec97c4c-183f-403b-9016-6393817f08d0/job/b9dd9a65-b3b2-4c8c-8fe8-759afcdd9d38

Likely cause (not confirmed)

mb/setup (Orb src/jobs/setup.yml) runs on CircleCI's cloud Docker, cimg/base small, not a self-hosted runner, so no runner of ours can be offline. A cloud job queued forever usually means the organisation can't start jobs: monthly credits used up, a spend limit, or a concurrency cap. It stopped about 30 minutes after cutover. In that time eight full main pipelines ran (force-full: true: large Docker jobs, an ARM job and the Windows machine jobs), so used-up credits is my first guess.

What would confirm or fix it

  • CircleCI org → Plan / Usage: credits left, any spend limit or block. Open one of the stuck workflows above: does it say "queued" or "not running: plan"?
  • If it is credits: on main, only run the full set when nightly needs it (or on a schedule), and leave the per-commit run to the path areas. Pull requests already work that way.
  • If it is something else, please say what: Forgejo's own checks still run, so main isn't unguarded, but the CircleCI gate never reports, and #1145's description of CI is ahead of what runs.

Found during a red-main sweep. The game-free failures on main are a separate problem, being fixed in #1323, #1363, #1409 and #1410.

@drcrow: since 17:57 UTC on Nov 15, CircleCI's setup job has stayed **pending** on every `main` commit. It never starts, so no checks run and `mb/gate` posts nothing. I have no CircleCI access, so this is what Forgejo's commit statuses show. The cause is a guess for someone who can open the CircleCI org to confirm. ## What the statuses show - **It worked at cutover.** On #417's merge (`deb9f781`, Nov 15 17:25 UTC), `ci/circleci: setup-main` went pending, then success a minute later, and the continued workflow posted lint, third-party, test-linux, guide-screenshots, website, spirv-fixtures, windows-builder and windows-vulkan. Seven more `main` commits did the same, the last one `b7a014b8` (17:52:55 UTC). - **From the next commit on, it never finishes.** On `1576a274` (17:57:35 UTC) and on every `main` commit since, through `544229a2` (Nov 17 03:50 UTC), `setup-main` is posted `pending` by circleci-bot with a workflow link, and nothing follows. I checked the first 40 commits after #417 one by one, then sampled every fourth up to today: 55 pending and none finished after `b7a014b8`. - **Tag pushes do the same:** `ci/circleci: setup-noop-tag` and `setup-release` stay pending on the nightly-tagged commits. - **Nothing in the repository changed at the break.** `b7a014b8..1576a274` touches no file under `.circleci/`, and neither does any later commit. The config is the one that worked eight times in a row. Example on today's head `544229a2`: `ci/circleci: setup-main` pending since 2026-11-17T03:50:23Z, https://app.circleci.com/workflow/dec97c4c-183f-403b-9016-6393817f08d0/job/b9dd9a65-b3b2-4c8c-8fe8-759afcdd9d38 ## Likely cause (not confirmed) `mb/setup` (Orb `src/jobs/setup.yml`) runs on CircleCI's cloud Docker, `cimg/base` `small`, not a self-hosted runner, so no runner of ours can be offline. A cloud job queued forever usually means the organisation can't start jobs: monthly credits used up, a spend limit, or a concurrency cap. It stopped about 30 minutes after cutover. In that time eight full `main` pipelines ran (`force-full: true`: large Docker jobs, an ARM job and the Windows machine jobs), so used-up credits is my first guess. ## What would confirm or fix it - CircleCI org → Plan / Usage: credits left, any spend limit or block. Open one of the stuck workflows above: does it say "queued" or "not running: plan"? - If it is credits: on `main`, only run the full set when `nightly` needs it (or on a schedule), and leave the per-commit run to the path areas. Pull requests already work that way. - If it is something else, please say what: Forgejo's own checks still run, so `main` isn't unguarded, but the CircleCI gate never reports, and #1145's description of CI is ahead of what runs. Found during a red-main sweep. The game-free failures on `main` are a separate problem, being fixed in #1323, #1363, #1409 and #1410.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1412
No description provided.