Test build optimization + unified YAML Build/Deploy pipeline¶
Date: 2026-06-15 · Author: investigation triggered by a 30-minute test build (build #6940)
This is the research record behind the new
pipelines/azure-test-build-deploy.yml.
It answers "why does the test build take 30 minutes?", lists the fixes ranked by
leverage, and describes a single multi-stage YAML pipeline that replaces the
split classic build + classic release for the Test environment.
The existing classic pipelines are left untouched — this is additive.
TL;DR¶
- The 30 min build is dominated by one single-process MSBuild step (12.8 min), an uncached NuGet restore (5.2 min), and a slow container artifact upload (≈6.6 min across two tasks).
- The single highest-leverage fix is
maximumCpuCount=true(/m) on the build — the classic build currently compiles 39 projects on one core. - Caching (NuGet + npm) and a faster artifact publish recover most of the rest.
- Realistic outcome of the four fixes together: ~30 min → ~12–15 min, with incremental no-op runs lower.
- A security finding: the classic release stores live TEST SQL passwords and a storage account key as plaintext release variables. The new pipeline moves them to a secret variable group.
1. Where the 30 minutes goes (build #6940, def #2 "Test Build - rakletv3")¶
Agent pool: Azure Pipelines (Microsoft-hosted). Source: GitHub rakletadmin/rakletv3, branch test. Triggers: CI + scheduled.
| Step | Time | % | Notes |
|---|---|---|---|
| Build solution (VSBuild) | 12.8 min | 43% | 39-project .NET Framework solution + Web Deploy packaging |
| Artifact out — AzureBlob File Copy (3.7) + Publish Artifact: drop (2.9) | 6.6 min | 22% | two separate uploads (CDN assets + deploy zips) |
| NuGet restore | 5.2 min | 17% | packages.config, hundreds of packages, verbosity=Detailed |
| npm install (1.4) + grunt build-test (0.8) | 2.2 min | 7% | Raklet.Backend admin-SPA assets |
| checkout + 6× "Check Disk Space" + cleanup + overhead | ~3 min | 10% |
Root causes¶
- MSBuild is single-process. The VSBuild task has
maximumCpuCount = false, so the 39 projects compile serially on one core. This is the biggest lever. - NuGet restore is uncached and runs at
verbosity=Detailed. Microsoft-hosted agents are ephemeral, butCache@2persists to the cloud pipeline cache, so a cache hit removes most of the 5.2 min. - Artifact upload is slow and doubled.
PublishBuildArtifacts(Container,StoreAsTar=false) is the slow legacy task. The two upload steps carry different payloads (CDN staticdist/*to blob vs. the Web Deploy zips), so both are needed — but the deploy-zip publish can use the fasterPublishPipelineArtifact. npm install --forcethen deletenode_modules. Not cached. TheRemove bin/objandRemove node_modulescleanup tasks are there to shrink the artifact, not to manage a cache (on a hosted agent there is no persistence to defeat). Caching the toolchain +npm ciis the win.
Note: an earlier read of these cleanup tasks assumed a self-hosted agent. The agent is Microsoft-hosted ("Azure Pipelines"), so the cleanups are purely artifact-size management — correct to keep, not the bottleneck.
2. Fixes, ranked by leverage¶
| # | Fix | Where | Est. saving |
|---|---|---|---|
| 1 | maximumCpuCount: true (parallel /m build) |
VSBuild task | ~6–7 min |
| 2 | Cache@2 NuGet packages keyed on **/packages.config + drop restore verbosity to Normal |
restore step | ~4 min on hit |
| 3 | PublishPipelineArtifact instead of PublishBuildArtifacts (Container) |
artifact step | ~1.5 min |
| 4 | Cache@2 node_modules keyed on package-lock.json + npm ci |
frontend step | ~1 min on hit |
| 5 | (later) packages.config → PackageReference migration |
all .csproj |
structural; faster transitive restore |
Fixes 1–4 are encoded in the new YAML build stage. #5 is a larger refactor tracked separately.
2b. Build agent: hosted vs self-hosted (benchmarked 2026-06-16)¶
The fixes above are for the Microsoft-hosted agent the build runs on today (worker
Azure Pipelines 1, pool 14, 4 vCPU, cold caches). A bigger lever for a daily-release
cadence is a self-hosted agent with more cores + warm caches + incremental build.
Agent topology (confirmed):
| Agent | Specs | Role today |
|---|---|---|
| Microsoft-hosted "Azure Pipelines" | 4 vCPU, cold | runs the build |
devops VM |
2 vCPU / 8 GB (pool SelfHostedPool2026) |
runs the deploys (release #31) |
ci-vm-1 |
8 vCPU / 32 GB, VS 2026 + warm repo, ~idle | browse-smoke runners; candidate build agent |
Benchmark on ci-vm-1 (idle box, warm NuGet packages, real build args incl. /p:GenerateSerializationAssemblies=Off):
| Build phase | Hosted today (#6940) | ci-vm-1 (8-core, warm) |
|---|---|---|
| NuGet restore | 5.2 min | 1.7 min |
| Build + Web Deploy packaging | 12.8 min (single-core /m:1) |
4.8 min (clean /m:8, all 10 zips) |
| Incremental rebuild (small diff) | n/a (always clean) | ~0.3 min |
The hosted 12.8 min is dominated by maximumCpuCount=false (effectively /m:1). On 8 cores
with warm packages a full clean build+package is ~4.8 min; a small-diff incremental build is
seconds. Estimated full build stage on ci-vm-1: ~9–10 min clean, a few minutes incremental,
vs ~30 min today. Numbers are best-case (idle, warm) — concurrent browse-smoke load raises them.
Caveats: ci-vm-1 also runs 4 browse-smoke GitHub Actions runners (CPU contention risk);
needs an AzDO agent installed + a pool; for the incremental win the YAML build must use
clean: false; single VM = single point of failure (keep hosted as fallback). Do not use
the 2-core devops VM for builds — slower than hosted. Tracked in ENG-117.
Next bottleneck: once the build is fast, deploy (on the 2-core devops VM) + the test
stages become the long pole on release cadence. Tracked in ENG-118.
3. Current deploy-to-test architecture (for context)¶
Deployment today is classic release #31 "Test with Unit", fully documented in test-with-unit-pipeline.md. Summary of what the new pipeline must reproduce:
- Consumes the build's
dropartifact (Web Deploy zips). - Backend deploys first, then Admin / Login / v3 / api / Crm in parallel.
- Each service: deploy package to the
prepslot → start slot → WarmUp → swap prep→production (atomic blue/green) → stop prep (Backend's prep is left running and cleaned up later by the "Stop PREP Webjobs" stage). - Resource group
raklet-test-resource-group, planraklet-test-serviceplan, single ARM service connection (id44aa1609-…).
| App Service | Package | Slot |
|---|---|---|
| raklet-test-backend | Raklet.Backend.zip |
prep → production |
| raklet-test-admin | Raklet.Admin.zip |
prep → production |
| raklet-test-login | Raklet.Login.zip |
prep → production |
| raklet-test-v3 | Application.zip |
prep → production |
| raklet-test-api | Raklet.Api.zip |
prep → production |
| raklet-test-crm | Raklet.Crm.zip |
prep → production |
Plus, in the classic release: Redis deployment-key add/clear, Start/Stop continuous WebJobs, WarmUp task groups, a 10-minute integration-test quiesce window, the VSTest Unit/Integration stage, and the browse-smoke GitHub Actions dispatch.
4. The new pipeline¶
pipelines/azure-test-build-deploy.yml
— a multi-stage YAML pipeline:
- Stage
Build— the four optimizations above, same MSBuild publish args and same packages as the classic build. - Stage
DeployTest—deploymentjobs targeting the AzDOraklet-testenvironment (so approvals/checks can gate it). Backend first; the five front-end services deploy in parallel via a sharedtemplates/deploy-appservice.yml.
Safety guards baked in¶
trigger: none— importing the file cannot deploy to the shared TEST env by accident.- Secrets come only from the
raklet-test-secretsvariable group — none are inline. - The deploy stage binds to a named AzDO environment, which is where branch approvals / pre-deploy checks attach.
Parity gaps (phase 2 — not yet ported)¶
Classic task groups don't translate 1:1 to YAML, so these remain on release #31 until ported and validated:
- WarmUp PREP task groups (HTTP JIT warming)
- Redis deployment-key add/clear
- Stop PREP Webjobs stage + the 10-minute quiesce window
- VSTest Unit/Integration stage
- Browse-smoke GitHub Actions dispatch
Until parity is reached, release #31 stays the source of truth for a full test deploy. Use the new pipeline first for build-time validation and timing.
5. Security finding — plaintext secrets in release variables¶
Release #31's variables hold, in plaintext (not marked secret):
RakletV3andRakletEmailContext— TEST SQL connection strings with passwordsAzureWebJobsStorage— storage account key
These are readable by anyone with release-edit access and are not redacted in logs.
Recommendation: move them into a variable group backed by Azure Key Vault (or at
minimum mark them secret), and rotate the SQL password + storage key since they have
been exposed in the definition for a long time. The new YAML pipeline references the
raklet-test-secrets group precisely so secrets never live in the pipeline body.
6. Go-live checklist¶
- Create variable group
raklet-test-secrets(Library) with the connection strings,AzureWebJobsStorage, and the Redis deployment key — ideally Key-Vault-backed. - Set
azureServiceConnectionto the ARM service-connection name (the one release #31 uses, id44aa1609-fc30-47c3-890f-a70b0dfda1c9). - Register the YAML as a new pipeline (Pipelines → New → GitHub → existing YAML).
- Run manually against
testand compare build timing + deploy result to release #31. - Port the phase-2 parity items, validate each.
- Only then flip
trigger: none→branches: include: [ test ]and retire the classic build/release for Test.
7. Open questions / inputs needed¶
- ARM service-connection name — the REST API returned 403 for the service-endpoint list under the current PAT, so only the connection id is known.
- Deploy parity scope — do we want full release-#31 parity in YAML (webjobs, redis, warmup, unit tests, browse-smoke), or keep those on the classic release and use YAML for build + deploy only?
- Authorization — confirmation before anything is registered/run against the shared TEST environment.