Skip to content

Dedicated CI Runner — Plan

This is a planning doc, not implementation. It captures the proposed shape of moving PR test execution off any individual engineer's laptop onto a dedicated Azure VM, plus the storage isolation refactor that needs to land first. Comments and pushback welcome before any of the referenced implementation PRs open.

The plan covers three implementation PRs that will land in sequence:

  1. Phase 0B (PR 1) — small repo refactor that extracts the shared storage <add key> entries from Web.config / App.config into per-project appSettings.local.config sidecars. Affects every dev's daily flow; details below.
  2. Phase 1 (PR 2) — adds .github/workflows/pr-ci.yml. CI runs on every PR against a dedicated Azure Windows VM registered as a self-hosted GitHub Actions runner.
  3. Phase 3 (PR 3) — adds .github/workflows/auto-review.yml for Claude + Codex automated PR reviews triggered by rakletai reviewer request or @rakletai mention.

Problem statement

The browse smoke pipeline that landed in #13715 / #13720 / #13734 needs a running Raklet site (IIS + SQL + wildcard DNS + grunt-built assets) to drive a browser against. Today the only machine with that environment is whichever engineer happens to be running scripts/dev/pr-ready-check.ps1 — which means that engineer's IIS site is busy serving tests instead of being usable for their own dev work. Adding more automation (Claude/Codex PR review, additional smoke suites) compounds the bottleneck.

A second, related bottleneck: Application/Web.config:22-120 and seven other projects share five Azure storage accounts (rakletlocalfiles, rakletlocalwebjobs, emailblasterlocalstorage, emailblasterlocalqueues, rakletv0storage) whose keys are committed in source. Every dev's local app and every future CI run hits the same queues and blob containers, so test data and queue messages from one machine bleed into another's runs.

Constraints

  • Old-style ASP.NET Framework solution (~50 csproj). No SDK-style projects, no .NET 8, no straightforward containerization.
  • Wildcard DNS + IIS host headers are required for tenant routing. GitHub-hosted runners cannot stand this up per-run; see the comment in .github/workflows/browse-ui-tests.yml:4-9.
  • TLDs are environment signals: .org = local/dev, .net = test, .com = prod. CI uses .org via a sealed DNS view on the VM.
  • Application/Infrastructure/SubdomainRoute.cs:23-47 parses the host up to the first dot as the org permalink. Nested subdomains (e.g. pr-13735.ci.raklet.org) break routing, so the CI VM uses the canonical *.raklet.org hostnames in its sealed view.
  • The GitHub Actions plan has a 3,000 hosted-runner minutes/month quota, currently exhausted. Self-hosted runners do not consume from that quota, so the new workflows add zero billable minutes.

Premises

  1. The bottleneck is "any one engineer's laptop is the test host" — moving the test host anywhere else solves it.
  2. GitHub-hosted runners can't host the Raklet stack.
  3. Tests must use canonical hostnames (admin.raklet.org, etc.) — the routing layer assumes that shape.
  4. One VM = one runner = one slot is acceptable for current throughput (3 devs, ~15 open PRs at peak). Adding parallelism = adding VMs, not stacking runners on one VM.
  5. Per-PR branch previews (the topic of docs/agents/iis-worktree-branch-previews.md) are real engineering. They require SubdomainRoute changes and are explicitly deferred.

Architecture

+--------------------------------------------------------+
|  Azure VM (Standard D4s_v5, always-on, ~$180/mo)       |
|  raklet-ci-resource-group / West Europe                |
|                                                        |
|  +--------------------------------------------------+  |
|  |  GH Actions self-hosted runner agent             |  |
|  |  labels: self-hosted, windows, raklet-ci, slot-1 |  |
|  +--------------------------------------------------+  |
|         |                                              |
|         | (jobs pushed from github.com)                |
|         v                                              |
|  +--------------------------------------------------+  |
|  |  Acrylic DNS (sealed): *.raklet.org -> 127.0.0.1 |  |
|  |  IIS site: *.raklet.org bindings + self-signed   |  |
|  |  SQL Server Developer (gold .bak restored)       |  |
|  |  Source checkout @ C:\repos\rakletv3 (worktree)  |  |
|  |  appSettings.local.config files (slot-1 keys)    |  |
|  +--------------------------------------------------+  |
|         |                                              |
|         | (reads/writes via Azure SDK)                 |
|         v                                              |
+--------------------------------------------------------+
          |
          v
+--------------------------------------------------------+
|  raklet-ci-resource-group (West Europe)                |
|  +------------------------+ +-----+ +------------+     |
|  | rakletci01storage      | | 02  | | 03         |     |
|  | (active, slot-1)       | |idle | |idle        |     |
|  +------------------------+ +-----+ +------------+     |
+--------------------------------------------------------+

When parallelism is genuinely needed: provision VM 2, label it slot-2, bake slot-2 keys into its sidecar configs. Slot-2 storage already exists.

Key decisions

  • Pre-created pool of 3 Azure storage accounts (rakletci01storage, rakletci02storage, rakletci03storage), one per future VM. Slots 2 and 3 sit idle on day one (pennies/month). Real Azure accounts, not emulator, so zero compatibility risk with the existing SDK code path.
  • CI mechanism: GitHub Actions self-hosted runner, not a polling cron. Push-based triggers, native PR Checks UI, built-in concurrency groups with cancel-in-progress: true so rapid push-rebases preempt in-flight runs.
  • Slot identity is a property of the VM, not a runtime workflow parameter. The VM's gitignored appSettings.local.config files are written once at provisioning time with that slot's keys. Workflow doesn't pick a slot; it inherits.
  • Always-on VM (~$180/mo) rather than auto-shutdown. The team's async-dev pattern produces PRs outside business hours; CI sleeping through them would block the workflow the runner is meant to unblock.
  • Per-run cleanup clears blob and queue contents in place — never deletes the containers themselves. Avoids Azure's tombstone period that can cause 409 Conflict on rapid delete-then-create.
  • No DB reset between runs in v1. Trade-off documented; mitigation is the standard pattern of using unique per-run identifiers in tests (e.g. pr-${PR_NUMBER}-event-${TIMESTAMP}) and not destroying gold-state seed data. If accumulated state causes flake, the fall-back is SQL Server snapshot revert per run (~half day to add).
  • Migration PRs get special handling. See "Migration PR handling" below.

Implementation phases

Phase 0A — Provision the VM (one-time, ~half day)

  • Create resource group raklet-ci-resource-group in West Europe.
  • Provision rakletci01storage, rakletci02storage, rakletci03storage (slot 1 active, 2-3 idle standbys).
  • Provision the VM: Windows Server 2022, Standard D4s_v5, Premium SSD, always-on (no auto-shutdown), in the same resource group.
  • Install: SQL Server Developer, IIS (matching docs/local-iis-setup.md), .NET Framework SDK + dev pack, Node 22, Bun 1.3.10, Chrome, Git, gh CLI, VS Build Tools.
  • Configure Acrylic DNS: *.raklet.org → 127.0.0.1.
  • Install self-signed *.raklet.org cert in LocalMachine\My and the trust store, per docs/local-iis-setup.md §1.
  • Restore the prod-anonymized SQL .bak as the gold reference.
  • Register the VM as a self-hosted GH Actions runner with labels self-hosted, windows, raklet-ci, slot-1.
  • Install claude and codex CLIs under a runner-specific identity.

Exit criteria: the VM serves a working Raklet site at canonical *.raklet.org hostnames using slot-1 storage; the runner shows online in repo Settings → Actions → Runners.

Phase 0B — appSettings refactor PR (lands before Phase 1)

This is the change every dev should review carefully because it touches the daily flow.

For each of 8 projects (Application, Raklet.Backend, Raklet.Login, Raklet.Crm, Raklet.WebJobs, Raklet.WebJobs.Secondary, Raklet.WebJobs.Scheduled5mins, Raklet.WebJobs.ScheduledHourly):

  1. Move the storage <add key> entries out of Web.config / App.config into a new appSettings.local.config file alongside the existing connectionStrings.local.config.
  2. Add a small structural change in the project's main config so those keys are loaded via configSource="appSettings.local.config" (or equivalent — the exact mechanic depends on whether the project can use a top-level <appSettings configSource> or needs a sub-section pattern).
  3. Commit an appSettings.local.config.template sibling so new clones know what shape to use.

Repo changes:

  • Add **/appSettings.local.config to .gitignore.
  • git rm --cached the existing committed copies (they remain on disk for current devs; new clones get them via the .template files).

What this means for devs after the merge:

  • The first time you pull, your local appSettings.local.config files will become "untracked but present." Git won't touch them on checkouts.
  • If you do a fresh clone after this merges, copy each appSettings.local.config.template to appSettings.local.config in the same directory and fill in your credentials (the same dev storage keys that are committed in Web.config today work fine for local dev).

Mandatory regression check before this PR lands: for each of the 8 projects, actually boot the project (or smoke-test storage resolution by constructing an Azure SDK client from ConfigurationManager.AppSettings["StorageConnectionString"]) and verify the client can list blobs in the expected account. "Returns non-null from AppSettings" is not enough; the SDK has to actually resolve and connect.

Phase 1 — CI on every PR (~3 days)

New workflow .github/workflows/pr-ci.yml:

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]

concurrency:
  group: pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  ci:
    runs-on: [self-hosted, windows, raklet-ci]
    steps:
      - uses: actions/checkout@v4
      - run: scripts/ci/apply-ci-storage-config.ps1
      - run: scripts/ci/validate-slot-config.ps1
      - run: scripts/ci/sync-db-to-master.ps1
      - run: scripts/ci/clear-slot-storage.ps1
      - run: scripts/ci/apply-pr-migrations.ps1     # no-op for non-migration PRs
      - run: scripts/dev/build-fast.ps1
      - run: scripts/dev/invoke-backend-grunt.ps1
      - run: scripts/ci/swap-iis-to-checkout.ps1
      - run: scripts/dev/pr-ready-check.ps1
      - run: dotnet test
      - if: always()
        run: scripts/ci/revert-pr-migrations.ps1
      - uses: actions/upload-artifact@v4
        with:
          name: ci-artifacts
          path: |
            screenshots/
            logs/

A small follow-up commit removes the "GitHub-hosted runner has no Raklet app" comment block at the top of .github/workflows/browse-ui-tests.yml since the constraint goes away.

Phase 3 — Auto PR review (~2 days)

New workflow .github/workflows/auto-review.yml triggered by:

  • pull_request: ready_for_review
  • pull_request: review_requested (filtered to rakletai)
  • issue_comment: created (filtered to comments mentioning @rakletai)

The workflow runs Claude and Codex against the diff with locked-down tool access: claude -p --allowedTools="" (no Bash, no Write, no network beyond the model call) and codex --sandbox read-only. The only thing either engine can produce is text, and the only thing the workflow does with that text is post it as a PR comment via the GitHub API. No merge, no approve, no push.

A consolidated comment is posted per review and edited in place on subsequent runs so the PR doesn't accumulate stale auto-review threads.

On first activation, the workflow records HEAD SHAs for all open PRs without reviewing — otherwise the first run would post auto-reviews on ~15 already-ready PRs at once. After that, only new SHAs trigger reviews. Hard cap of 3 reviews per cycle and 20 per day as belt-and- suspenders against runaway behavior.

Concurrency model

One self-hosted runner = one job at a time (GitHub Actions native semantics). With cancel-in-progress: true:

  • Two different PRs → second queues until first finishes.
  • Same PR, new push → previous run cancels, new run takes its slot.
  • CI workflow and auto-review workflow share the runner, so they serialize against each other too.

This is intentional. Tests run against a single shared SQL DB and a single IIS site at canonical hostnames — running two PRs in parallel against shared infrastructure would be a flake-fest. Serialization is the feature.

Adding parallelism = adding a VM. The slot-2 storage account is already provisioned; the only work is VM 2 provisioning + registering its runner agent.

Migration PR handling

CI's DB tracks master's schema continuously. The polling step at the start of every workflow runs dotnet ef database update against the current master state, applying any newly-merged migrations.

For PRs that add a migration (detected via git diff --name-only ... | grep -E 'Migrations/'):

  1. Sync DB to master (above).
  2. Apply the PR's new migration: dotnet ef database update.
  3. Run tests.
  4. Revert the PR's migration (always, even on failure): dotnet ef database update -TargetMigration <master-latest>.

Open verification before this is relied on: the revert step requires EF Down methods to be correct and reversible. Many teams stop maintaining Down methods because day-to-day flow is forward-only. A quick spot-check of three recent Raklet migrations should confirm whether reverts actually work cleanly. If they don't, the fallback is to detect migration changes and post a "validate via migration release process" status check instead of running CI for those PRs, matching the existing migration-branch-first policy.

Hardware

  • VM: Azure Standard D4s_v5, 4 vCPU, 16 GB RAM, Premium SSD.
  • Cost: ~$180/mo always-on.
  • DNS: Acrylic, *.raklet.org → 127.0.0.1 on the VM (sealed view, no leakage to prod DNS or other dev machines).
  • Cert: self-signed *.raklet.org, same approach as docs/local-iis-setup.md §1.
  • Storage: 3 accounts in raklet-ci-resource-group. Idle cost is pennies/month; usage cost is pennies/month.
  • GH Actions minutes: zero from quota (self-hosted runners are free from the 3,000-min budget).

Out of scope

These problems are acknowledged but explicitly deferred:

  1. Local dev storage isolation. Devs continue sharing the rakletlocal* accounts in day-to-day work. The Phase 0B refactor sets up the structural fix (per-machine appSettings.local.config) so this is solvable later without another refactor — but actually provisioning per-dev storage accounts is its own decision.
  2. Per-PR branch previews. Covered in docs/agents/iis-worktree-branch-previews.md. Requires SubdomainRoute + RegexHelper + UrlService changes and a per-PR IIS-site provisioner. Real engineering, separate project.
  3. Containerizing the app. Windows containers on .NET Framework is a multi-month project that doesn't move toward .NET 8 modernization. Strategic decision for another day.
  4. Removing AccountKey=... values from Web.config source control. Real cleanup, separate effort. The new CI accounts created here will NOT be committed; their keys live only in the VM's gitignored sidecars and Azure.
  5. Migrating existing GH-hosted workflows to the self-hosted runner. A real Phase 4 opportunity — tests.yml is the most likely candidate to recoup the metered GH Actions overage — but defer until the new runner has been stable for a couple of weeks.

Open items and watch-outs

  1. EF Down migration reliability. Spot-check three recent migrations before the migration-handling logic in Phase 1 is relied on. If Down methods are broken, fall back to "skip CI for migration PRs."
  2. Non-migration test pollution. Without a per-run DB reset, shared state across runs is possible. Mitigations: tests use per-run unique identifiers, never destroy gold-state seed data. Plan B is to add SQL Server snapshot revert (~half day) if flake appears.
  3. Self-signed cert + outbound HTTPS to webhook sandboxes (Stripe/Iyzico/PayPal). Same risk profile as dev laptops today; the cert in the system trust store should resolve it. If a webhook test fails with RemoteCertificateNameMismatch, the fix is verifying the trust store install on Phase 0.
  4. Storage cleanup wall time. If clearing 5 containers + 3 queues per run grows past ~30s, switch from "delete blobs" to a different pattern.
  5. First-build cost after a checkout that clears bin/obj/. Mitigation is one-line YAML (clean: false on the checkout step) if it bites.

Success criteria

  1. No engineer's dev box runs a scheduled Raklet job after Phase 3 ships.
  2. A PR opened by any engineer gets a CI status check within build + test + queue_wait; queue wait is <10 min under normal load.
  3. Median time from git push → CI status posted is <30 min for a full-suite PR.
  4. Auto-review posts a comment within ~5 min of trigger.
  5. Zero regressions in production member portal behavior caused by this work.
  6. Adding a second runner takes <1 hour end-to-end (VM 2 provisioned
  7. registered + slot 2 keys baked).

Dependencies

  • Azure tenant access (create resource group, VM, storage accounts).
  • GitHub repo admin rights (register self-hosted runner, manage secrets).
  • A prod-anonymized SQL .bak snapshot.
  • pr-ready-check.ps1 working in headless mode (already does, per #13734).
  • claude and codex CLI subscriptions on the runner's identity.
  • Azure CLI installed on the runner.

Phase 0 acceptance checks

Before Phase 0B and Phase 1 implementation start, three small de-risking steps:

  1. Provision the resource group + three storage accounts via az CLI. ~5 minutes.
  2. Verify EF Down migrations work on three recent Raklet migrations. Run Update-Database then -TargetMigration <previous>; confirm clean undo, no orphan tables, no errors.
  3. Smoke-test the appSettings refactor mechanic on one project (Application is fine). Move the storage <add key> entries to a new appSettings.local.config, set configSource, run the app, confirm storage still resolves.

If step 2 fails, the migration-handling design needs to change to "skip CI for migration PRs" before any other work starts. If step 3 fails, the configSource mechanic needs to be debugged before doing the other 7 projects.