Canonical IIS hosts pinned to master (+ worktree teardown guard)

The dev / CI VM (CI-VM-1) serves one set of canonical *.raklet.org IIS sites. Historically each site's vdir pointed at whichever worktree was last pointed at it (appcmd set vdir), so a canonical host could silently drift onto a disposable agent worktree — and when that worktree was removed, the host 500'd. This page documents the durable fix:

1. Pin every canonical bare host to the master checkout (C:\repos\rakletv3).
2. Guard worktree teardown so removing a worktree can never strand a canonical host on a deleted path again.

See also Local IIS setup (the shared-box coordination lock) and IIS worktree branch previews (the pr<ticket>-* per-branch preview model that PR work should use instead of repointing canonical hosts).

The incident this prevents

crm.raklet.org was pointed at the worktree …\.claude\worktrees\mrr-dashboard-customer-cache-refactor\Raklet.Crm. That worktree was later git worktree removed while IIS still served it, so every request to crm.raklet.org returned:

HTTP Error 500.19 — Cannot read configuration file
Error Code   0x80070003   (ERROR_PATH_NOT_FOUND)
Config File  …\.claude\worktrees\mrr-dashboard-customer-cache-refactor\Raklet.Crm\web.config

The config wasn't malformed — the whole directory was gone. This is exactly the failure mode Local IIS setup rule #3 warns against ("never git worktree remove a worktree IIS still points at"), but that rule was advisory and got violated.

The model

Site (apppool)	Host	Project (under master checkout)
v3	*.raklet.org (e.g. gercek.raklet.org)	Application
admin	admin.raklet.org	Raklet.Admin
api	api.raklet.org	Raklet.Api
backend	backend.raklet.org	Raklet.Backend (admin SPA bundle)
login	login.raklet.org	Raklet.Login
crm	crm.raklet.org	Raklet.Crm

Rule: canonical bare hosts always serve master from C:\repos\rakletv3. They are never repointed at a worktree. Per-PR / per-branch work runs on the prefixed preview hosts (pr<ticket>-admin.raklet.org, …) provisioned by create-branch-preview.ps1, which only ever creates preview-<ticket>-* IIS objects and never touches the canonical sites.

gercek.raklet.org is the v3 site via the *.raklet.org wildcard binding; gercek is the org permalink served, not a separate site. Pinning v3 to the master Application is what makes "gercek is always latest master" true.

pin-canonical-hosts.ps1 — assert / restore canonical = master

Re-runnable. Repoints all six canonical vdirs to C:\repos\rakletv3\<project>, recycles each pool, and warms + verifies each host. Optionally fast-forwards master and rebuilds first. Run elevated (IIS edits need the admin token — see Why elevation).

# Re-assert canonical = master with what's already built (fast; no rebuild):
pwsh scripts\dev\pin-canonical-hosts.ps1 -SkipBuild

# Pull latest master, rebuild the 6 web projects + the admin SPA bundle, then pin:
pwsh scripts\dev\pin-canonical-hosts.ps1 -Update

What -Update does, in order: git fetch + git pull --ff-only on the master checkout → MSBuild each of the six web projects (Configuration=Debug) → grunt build-local for Raklet.Backend (the admin SPA bundle, which is not produced by MSBuild) → repoint + recycle + verify. Without -Update it pins (and rebuilds unless -SkipBuild) against the checkout as-is.

backend is pinned/warmed first, on purpose. Raklet.Backend is the only app that migrates the databases — its Startup runs MigrateDatabaseToLatestVersion (for RakletEmail, RakletSms, RakletDb); every other app calls Database.SetInitializer<RakletDb>(null) and never migrates. Cold-starting backend first applies any pending migrations before v3/login/admin serve requests against the new model, so they don't 500 on missing columns (e.g. org creation). A failed backend warm is therefore fatal and aborts before the dependent hosts are touched.

The script refuses to run if the master checkout is not on the master branch, so it can never pin canonical hosts to a feature branch by mistake.

How master actually stays current (the scheduled task IS the engine)

Pinning alone does not advance the checkout — it keeps the hosts pointed at C:\repos\rakletv3, but that checkout only moves to a newer master when something runs pin-canonical-hosts.ps1 -Update. Nothing runs it on its own. So "gercek is always latest master" is only true once you register the scheduled task below; without it, master goes stale until someone runs -Update by hand.

Register it with the helper (run elevated, once):

pwsh scripts\dev\register-pin-canonical-task.ps1   # hourly (default -IntervalMinutes 60)

It creates a task that runs pin-canonical-hosts.ps1 -Update -IfNeeded hourly. On a tick with no new master commits it exits in under a second; when master advanced it rebuilds + repoints; if a host drifted onto a worktree it re-asserts the pins without rebuilding (so it doubles as the self-heal). Pass -IntervalMinutes to change the cadence.

Run-as account (the one thing you must decide). A non-interactive task with -RunLevel Highest needs a stored password, and the account must be both a local Administrator (for the IIS edits) and hold git credentials for the private repo (for the git pull). On CI-VM-1 that is civm1. SYSTEM / NETWORK SERVICE won't work — they have no git credentials for the private remote, so their pull would fail. The helper prompts for the credential (or pass -Credential).

Safety: if master advanced but the rebuild fails — or backend can't apply a pending DB migration — the script aborts before repointing/recycling, so the hosts keep serving the last-good build instead of a broken commit. A broken master commit therefore stalls the update; it doesn't break the canonical sites.

Event-driven alternative: instead of polling, a small job on the self-hosted runner can run pin-canonical-hosts.ps1 -Update on every push to master. That keeps the box exactly current with zero lag but couples the dev/CI VM's canonical sites to the CI pipeline. The scheduled poll above is simpler and is the default.

remove-worktree-safe.ps1 — the teardown guard

Use this instead of raw git worktree remove. Before removing, it checks every canonical vdir; if any physical path resolves inside the target worktree, it repoints that vdir back to the master checkout and recycles the pool first, then removes the worktree. So a teardown can never leave a canonical host on a deleted path.

# Safe removal (repoints any canonical vdir off the worktree, then removes it):
pwsh scripts\dev\remove-worktree-safe.ps1 -Worktree C:\repos\rakletv3\.claude\worktrees\my-feature

# Worktrees with uncommitted changes are refused unless you pass -Force:
pwsh scripts\dev\remove-worktree-safe.ps1 -Worktree …\my-feature -Force

It runs elevated only when it actually has to touch IIS (a canonical vdir was on the worktree); a worktree no canonical host points at is removed without needing the admin token.

This guards the canonical sites. The preview-<ticket>-* sites have their own teardown in remove-branch-preview.ps1; the two are complementary.

Why elevation is required

appcmd / applicationHost.config edits require the elevated admin token. civm1 is a full local Administrator, but a headless session runs with the filtered (medium-integrity) token under UAC (ConsentPromptBehaviorAdmin=5, no auto-elevate), so IIS writes are denied until the session is elevated. Options:

• Run the agent / terminal "as Administrator" — the session then has the elevated token and these scripts work directly. Simplest for one-off runs.
• Scheduled Task with /RL HIGHEST (as above) — runs elevated without an interactive UAC prompt; the right fit for the self-heal loop.
• az vm run-command invoke runs as NT AUTHORITY\SYSTEM (no UAC) once the VM has an az login / managed identity — see CI runners & workflow policy.

Verifying

After a pin, every canonical host should answer (302s to a same-host login/landing are healthy; only a 500/timeout is a problem):

# All six canonical hosts (backend included - it serves the admin SPA bundle and
# is easy to forget; a stale/broken backend silently breaks the admin app).
foreach ($u in 'https://backend.raklet.org/','https://gercek.raklet.org/',
               'https://admin.raklet.org/','https://api.raklet.org/',
               'https://login.raklet.org/','https://crm.raklet.org/') {
  # curl.exe, not Invoke-WebRequest: PS5 IWR chokes on IIS per-request SSL renegotiation.
  curl.exe -k -s -o NUL -w "$u %{http_code}`n" --max-time 150 $u
}

Confirm the served physical paths with the shared-box status helper:

pwsh scripts\dev\iis-deploy-lock.ps1 status   # Site / Pool / PhysicalPath for all canonical sites

All six PhysicalPath values should be under C:\repos\rakletv3\.