Skip to content

Background WebJobs on CI-VM-1 — master-pinned, decoupled from runners

How the four Raklet background WebJobs run on the shared dev/CI box (CI-VM-1), why they are pinned to the canonical master checkout instead of a runner workspace, and the decisions behind that. Companion to Canonical IIS hosts pinned to master (same pattern, for the IIS web hosts) and CI runners & workflow policy.

The four WebJobs

The box runs the Raklet background processors continuously, as host-level Windows services + Scheduled Tasks (registered by scripts/ci/setup-ci-webjobs.ps1):

Job Mechanism Cadence Role
RakletWebJobs nssm service continuous Azure Storage Queue processing (reindex, etc.)
RakletWebJobsSecondary nssm service continuous recalculate-balance + secondary queues
RakletWebJobs5mins Scheduled Task every 5 min 5-minute recurring jobs
RakletWebJobsHourly Scheduled Task every 60 min hourly recurring jobs

They process the box's local *.raklet.org work (the pr-ci browse suites). .raklet.net (the Azure test env) has its own cloud WebJobs — these are not those.

The problem this fixes

Historically the four WebJobs ran out of r1's runner workspace bin (C:\actions-runner\_work\rakletv3\rakletv3\Raklet.WebJobs*\bin\Debug). One root cause, three failure modes:

  1. Non-deterministic binary. The WebJobs were whatever branch r1 last built — usually a feature PR, not master. A PR that broke WebJob code left broken background processing running for the whole box until r1 next built something else.
  2. Build-lock coupling to r1. The processes hold Services.dll / Models.dll open from r1's bin, so every r1 build had to pause → build → resume the WebJobs (the CiWebJobsPause / CiWebJobsResume SYSTEM-task band-aid). During every r1 build, box-wide queue processing stalled. It was also r1-only and required a careful run-only privilege grant to avoid a NETWORK SERVICE → SYSTEM escalation path.
  3. Cross-PR contamination. QueueService uses one AzureWebJobsStorage account with global, non-org-scoped queue names. Every runner's deployment enqueues into the same queues, drained by the same single WebJob set — so a PR-branch binary processed messages enqueued by other runners' tests.

Key insight — one runtime already serves all orgs

Queue messages carry their own org context in the payload, so a single WebJob runtime correctly drains every per-runner org's queues (ci1–ci4, the ENG-293 isolation orgs) at once. WebJobs do not need to be multiplied per runner. The only thing wrong with the single runtime was where it lived and which branch it was — not its cardinality.

Decisions (2026-06-28)

Decision Choice Why
Where the WebJob runtime lives Master-pinned in C:\repos\rakletv3, decoupled from all runner workspaces Mirrors canonical IIS pinning. Kills the non-deterministic binary, the build-lock, and the cross-PR contamination in one move. No runner ever builds into this bin, so MSBuild on any runner can never collide with a running WebJob.
Do PRs need their own WebJob code exercised pre-merge? No — master-pinned is enough Browse suites test UI/API; background processing is rarely what a PR changes, and one master WebJob drains all orgs' queues correctly. Per-PR WebJob code testing waits for Increment B per-PR previews (each preview runs its own WebJob instance from its own bin).

The model

C:\repos\rakletv3   (canonical master checkout — civm1 owns it)
 ├─ Application, Raklet.Admin, Raklet.Api, Raklet.Backend, Raklet.Login, Raklet.Crm
 │     → served by the canonical *.raklet.org IIS hosts  (pin-canonical-hosts.ps1)
 └─ Raklet.WebJobs*  bin\Debug\*.exe
       → run by RakletWebJobs / …Secondary / …5mins / …Hourly  (THIS doc)

C:\actions-runner[-2/-3/-4]\_work\…   (r1–r4 runner workspaces)
       → pr-ci builds + browse suites only.  NOTHING here runs the WebJobs anymore.

Rule: the box's WebJobs always run master from C:\repos\rakletv3, rebuilt and restarted on every master advance by the same hourly task that pins the canonical IIS hosts. They are never run from, or pointed at, a runner workspace.

How it stays current — the hourly pin task owns it

The existing Raklet-PinCanonicalHosts scheduled task (runs as civm1, a local admin with git credentials) now also pins the WebJobs. Its action gained -IncludeWebJobs:

pin-canonical-hosts.ps1 -Update -IfNeeded -IncludeWebJobs -MasterRepo C:\repos\rakletv3

On each tick:

  • No new master commits → exits in under a second (the -IfNeeded fast path). WebJobs keep running the current master untouched.
  • Master advanced → git reset --hard origin/master, rebuild the six web projects + admin SPA, repoint/recycle the IIS hosts, warm + verify (backend first, so DB migrations apply before anything serves the new schema), then stop → rebuild the four WebJob projects → restart them from the same checkout. Web code and WebJob code therefore always run the same commit.
  • A host drifted (no new commits, but a vdir moved) → re-assert pins without rebuilding; WebJobs are just ensured-running.

Because the same civm1 admin task owns both the rebuild and the WebJob processes, stopping them to free their DLLs is a plain Stop-Service — no cross-runner privilege dance, no SYSTEM-task escalation surface. The DB migrations the WebJobs depend on are applied by the backend warm earlier in the same run (see Database migrations and warmup order).

Operations

# One-time (cold box / re-image): register the services + tasks against master,
# then pin once so the bin is built. Run elevated.
pwsh scripts\ci\setup-ci-webjobs.ps1                 # WorkspacePath defaults to C:\repos\rakletv3
pwsh scripts\dev\pin-canonical-hosts.ps1 -IncludeWebJobs

# The hourly engine (registers/refreshes the scheduled task). Run elevated, once.
pwsh scripts\dev\register-pin-canonical-task.ps1     # now passes -IncludeWebJobs

# Force a refresh by hand (e.g. after a hotfix lands on master):
pwsh scripts\dev\pin-canonical-hosts.ps1 -Update -IncludeWebJobs

# Health check — services running + pointed under C:\repos\rakletv3:
Get-Service RakletWebJobs,RakletWebJobsSecondary | Format-Table Name,Status,StartType
# NOTE: the services are nssm-wrapped, so Win32_Service.PathName is just nssm.exe.
# The REAL wrapped target is nssm's Application value in the service registry:
foreach ($s in 'RakletWebJobs','RakletWebJobsSecondary') {
  "{0} -> {1}" -f $s, (Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s\Parameters").Application
}
Get-ScheduledTask RakletWebJobs5mins,RakletWebJobsHourly |
  ForEach-Object { "{0} -> {1}" -f $_.TaskName, $_.Actions[0].Execute }

All WebJob targets (the nssm Application values and the task action Execute paths) should be under C:\repos\rakletv3\. If pin-canonical-hosts.ps1 -IncludeWebJobs warns that a WebJob is not pointed at this checkout, re-run setup-ci-webjobs.ps1 to repoint it.

What changed in pr-ci

The pr-ci build job (r1) no longer pauses/resumes WebJobs — there is nothing on r1 to lock anymore. Removed from pr-ci.yml: the "Pause ci-vm-1 WebJobs services" and "Resume ci-vm-1 WebJobs services" steps. Retired (kept only for history / rollback):

  • scripts/ci/webjobs-build-pause.ps1
  • scripts/ci/setup-webjobs-build-control.ps1 (the CiWebJobsPause/Resume SYSTEM tasks + run-only grant it created are now inert; safe to leave or unregister).

What this does NOT do

  • It does not exercise a PR's own WebJob code before merge (see Increment B below for the piece that has shipped). A PR that changes queue/WebJob behaviour is still validated by its UI/API surface by default; the master WebJob remains the one always processing the shared/CI-org queues. Full per-PR WebJob code testing (a live enqueue → background-drain → assert round trip on every relevant PR) is still open — see "What Increment B still needs" below.
  • It does not change the shared storage/DB model. All runners still share one AzureWebJobsStorage and one SQL DB on the box; isolation is at the org level (ENG-293). The single master WebJob draining shared queues per-org is correct and intended.
  • Watch item: if AzureWebJobsStorage ever points at a storage account shared with another live environment (e.g. the dev VM), two boxes' WebJobs would compete to drain the same queues. Confirm the box's AzureWebJobsStorage is box-local before assuming isolation.

Increment B — per-PR WebJob testing (ENG-315)

Tracks closing the "does not exercise a PR's own WebJob code" gap above. Four sub-problems, per the ticket: (1) queue isolation, (2) an isolated PR runtime, (3) start/kill lifecycle, (4) an opt-in trigger. (1)-(3) shipped in the PR that added this section; (4) shipped as a narrow, best-effort slice. Full live queue-message round-trip testing is still open — see "What Increment B still needs".

What shipped

  • Queue isolation, via a different storage backend, not renamed queues. Services/Backbone/QueueService.cs and every [QueueTrigger("...")] across the four WebJob projects keep their exact existing (global, non-org-scoped) queue names — renaming the ~200 hardcoded trigger attributes to inject a per-runner prefix was considered and rejected as too invasive/high-risk for one PR (it touches the production hot path of every queue consumer). Instead, scripts/ci/run-pr-webjobs.ps1 points an isolated runtime's AzureWebJobsStorage connection string at a throwaway, per-run Azurite emulator instance instead of the real shared account — the alternative the ticket itself calls out. A completely different storage backend can never see, steal, or contaminate the master-pinned runtime's messages, by construction, with zero application code changes.
  • Isolated PR runtime, scoped to the calling runner. The script runs Raklet.WebJobs.exe / Raklet.WebJobs.Secondary.exe (the two queue-driven continuous WebJobs; Scheduled5mins/ScheduledHourly use an internal [NoAutomaticTrigger] timer loop, not queue triggers, and are opt-in via -WebJobProjects) straight from the calling workspace's own build output — never C:\repos\rakletv3 and never the nssm-wrapped RakletWebJobs* services. Any DLL lock this creates is scoped to that runner's own workspace, exactly like every other pr-ci build step; it can never collide with the master-pinned runtime.
  • Lifecycle. run-pr-webjobs.ps1 -Action start / -Action stop. stop is idempotent (safe no-op if nothing is tracked), self-heals a leftover state file from a prior run that didn't stop cleanly, and kills the FULL process tree per tracked pid (taskkill /T) — needed because the Azurite npm shim launches node.exe as a child process, so killing only the tracked pid would orphan the real server. Each project's own connectionStrings.local.config is backed up before the isolated connection string is written in and restored on stop, so a build immediately after a stopped run never inherits a dead Azurite endpoint.
  • Opt-in trigger (path filter). pr-ci.yml build job: a git diff --name-only step against Raklet.WebJobs*/, Services/Backbone/QueueService.cs, Services/Backbone/QueueType.cs, and Models/WebJobModels/ gates a start → 10s liveness check → (job-end, always()) stop sequence. Every one of those steps is continue-on-error: true, so this can never fail the build check — it is additive coverage, not a required gate.

What that liveness check actually proves (and doesn't)

The wired-in check only proves the changed WebJob project builds and stays alive for 10s after boot in isolation — it catches startup-time breakage (bad config binding, a DI failure, an unhandled startup exception). It does not enqueue a message or assert any queue gets drained, because the deployed IIS app (what the browse suites in the test job drive) and this isolated runtime intentionally use different storage backends — that separation is the whole isolation mechanism. Nothing reaches the isolated runtime unless something explicitly calls QueueService.SendMessage against its Azurite connection string.

What Increment B still needs

  • A real queue round-trip assertion. The shape of "enqueue → isolated runtime drains it → assert the side effect" depends on which queue a given PR changes; scripting a generic one was left undone rather than guessed. Likely shape: a small integration/handler test (already the Tier 1 default — see the ENG-315 Linear issue) constructs a CloudQueueClient against the same connection string run-pr-webjobs.ps1 prints out, sends a message, and polls for the expected DB/side-effect change.
  • Making it a required gate, once the round-trip assertion exists and has a track record of not flaking — today's steps are advisory-only (continue-on-error: true).
  • Raklet.WebJobs.Scheduled5mins / Raklet.WebJobs.ScheduledHourly aren't started by the opt-in CI step by default (see "What shipped" above); wire them in if a PR's change needs them exercised too.
  • Deciding whether the deployed IIS app itself should ever point at the isolated storage (so a browser-driven test could produce a message the isolated runtime drains) is a materially bigger change (it touches every IIS-served project's own connection string, not just the WebJob projects) and was deliberately not attempted here.

Rollback

Revert this PR (restores the pr-ci pause/resume steps and the runner-workspace default), then re-point the services back at the r1 workspace with setup-ci-webjobs.ps1 -WorkspacePath C:\actions-runner\_work\rakletv3\rakletv3 and re-run setup-webjobs-build-control.ps1 so the pause/resume grant exists again.